Web crawler blacklist generation method, identification method and device

Abstract

The present disclosure provides a method for generating, a method for identifying, and a device for a web crawler blacklist, which relate to big data and machine learning in the field of artificial intelligence technology. The scheme includes: obtaining network access traffic in a first time period, classifying and processing the network access traffic in the first time period based on the network access traffic in a second time period, and obtaining a classification result of the network access traffic in the first time period. The classification result of the network access traffic in the first time period indicates whether the network access traffic in the first time period is web crawler traffic. The second time period is a time period before the first time period. Based on the classification result of the network access traffic in the first time period and the classification result of the network access traffic in the second time period, a web crawler blacklist is generated, thereby avoiding the disadvantage of low accuracy caused by manually generating the web crawler blacklist and improving the accuracy and reliability of the generated web crawler blacklist.

Description

Technical Field

[0001] The present disclosure relates to big data and machine learning in the field of artificial intelligence technology, and in particular to a method for generating, a method for identifying, and a device for generating a web crawler blacklist. Background Art

[0002] Web crawler traffic refers to traffic that automatically crawls World Wide Web information according to certain rules. This traffic is not real traffic and can therefore also be called cheating traffic. To maintain the security of World Wide Web information, it is usually necessary to identify web crawler traffic, and identifying web crawler traffic usually relies on web crawler blacklists.

[0003] In the prior art, web crawler blacklists are usually generated manually, such as determining criteria for determining web crawler traffic based on the experience of technicians, and generating the web crawler blacklist based on the criteria.

[0004] However, manually generating a web crawler blacklist makes the web crawler blacklist susceptible to human subjective factors, resulting in technical problems such as low accuracy and reliability of the web crawler blacklist. Summary of the Invention

[0005] The present disclosure provides a method for generating a web crawler blacklist, a method for identifying a web crawler blacklist, and a device for improving the accuracy of the generated web crawler blacklist.

[0006] According to a first aspect of the present disclosure, a method for generating a web crawler blacklist is provided, comprising:

[0007] Obtain network access traffic in the first time period;

[0008] Classifying the network access traffic in the first time period according to the network access traffic in the second time period to obtain a classification result of the network access traffic in the first time period, wherein the classification result of the network access traffic in the first time period indicates whether the network access traffic in the first time period is web crawler traffic, and the second time period is a time period before the first time period;

[0009] A web crawler blacklist is generated according to the classification result of the network access traffic in the first time period and the classification result of the network access traffic in the second time period.

[0010] According to a second aspect of the present disclosure, a method for identifying web crawler traffic is provided, comprising:

[0011] Obtain the network access traffic to be identified;

[0012] Identify and process the network access traffic to be identified according to the web crawler blacklist to obtain an identification result;

[0013] The identification result is used to characterize whether the network access traffic to be identified is web crawler traffic, and the web crawler blacklist is determined based on the first aspect.

[0014] According to a third aspect of the present disclosure, a device for generating a web crawler blacklist is provided, comprising:

[0015] A first acquiring unit, configured to acquire network access traffic in a first time period;

[0016] a classification unit, configured to classify the network access traffic in the first time period according to the network access traffic in the second time period to obtain a classification result of the network access traffic in the first time period, wherein the classification result of the network access traffic in the first time period indicates whether the network access traffic in the first time period is web crawler traffic, and the second time period is a time period before the first time period;

[0017] The generating unit is configured to generate a web crawler blacklist according to the classification result of the network access traffic in the first time period and the classification result of the network access traffic in the second time period.

[0018] According to a fourth aspect of the present disclosure, a device for identifying web crawler traffic is provided, comprising:

[0019] A second acquiring unit, configured to acquire network access traffic to be identified;

[0020] An identification unit is used to identify and process the network access traffic to be identified according to the network crawler blacklist to obtain an identification result;

[0021] The identification result is used to characterize whether the network access traffic to be identified is web crawler traffic, and the web crawler blacklist is determined based on the first aspect.

[0022] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0023] at least one processor; and

[0024] a memory communicatively connected to the at least one processor; wherein,

[0025] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method of the first aspect or the second aspect.

[0026] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method according to the first aspect or the second aspect.

[0027] According to the seventh aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect or the second aspect.

[0028] The present disclosure provides a method for generating, a method for identifying, and a device for a web crawler blacklist. The method classifies and processes the network access traffic of a first time period according to the network access traffic of a second time period to obtain a classification result of the network access traffic of the first time period. The method generates a web crawler blacklist based on the classification results corresponding to the second time period and the first time period. This avoids the disadvantage of low accuracy caused by manually generating a web crawler blacklist, and improves the accuracy and reliability of the generated web crawler blacklist.

[0029] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0031] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0032] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0033] Figure 3 This is a schematic diagram of the principle of a method for generating a web crawler blacklist according to an embodiment of the present disclosure;

[0034] Figure 4 is a schematic diagram according to a third embodiment of the present disclosure;

[0035] Figure 5 is a schematic diagram of the principle of a method for generating a web crawler blacklist according to another embodiment of the present disclosure;

[0036] Figure 6 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0037] Figure 7 2 is a schematic diagram of a method for generating a web crawler blacklist according to another embodiment of the present disclosure;

[0038] Figure 8 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0039] Figure 9 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0040] Figure 10 is a schematic diagram according to a seventh embodiment of the present disclosure;

[0041] Figure 11 is a schematic diagram according to an eighth embodiment of the present disclosure;

[0042] Figure 12 is a schematic diagram according to a ninth embodiment of the present disclosure;

[0043] Figure 13 The present invention is a block diagram of an electronic device for implementing the method for generating a web crawler blacklist and the method for identifying web crawler traffic according to the embodiments of the present disclosure. DETAILED DESCRIPTION

[0044] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0045] In order to improve the network environment, it is usually necessary to identify web crawler traffic, and usually a web crawler blacklist is generated, and web crawler traffic is filtered out from various traffic based on the web crawler blacklist.

[0046] Web crawler traffic refers to traffic generated by web crawlers accessing the internet. A web crawler (also known as a web spider, web robot, or web chaser) is a program or script that automatically crawls the World Wide Web according to specific rules. Other less commonly used names include ants, automatic indexers, simulators, or worms.

[0047] By detecting and blocking web crawler traffic, we can reduce the crawling of website content (especially core website content), waste of server resources, and the impact of low-quality content on overall content quality.

[0048] In one example, a web crawler blacklist may be manually determined so that web crawler traffic can be detected and blocked based on the web crawler blacklist.

[0049] For example, a criterion for determining web crawler traffic is determined based on the experience of technical personnel, and a web crawler blacklist is generated based on the criterion.

[0050] However, when this method is used to generate a web crawler blacklist, it is easily affected by the subjective factors of technical personnel, resulting in a technical problem that the accuracy of the generated web crawler blacklist is relatively low.

[0051] In another example, a web crawler blacklist may be determined using a network model so that web crawler traffic can be detected and blocked based on the web crawler blacklist.

[0052] For example, a sample data set is collected, which includes positive sample data and negative sample data, where the positive sample data can be non-web crawler traffic and the negative sample data is web crawler traffic. A network model is trained using the positive sample data and the negative sample data. The network model is used to detect web crawler traffic, thereby determining a web crawler blacklist.

[0053] However, this method requires labeling of each sample data to determine the positive sample data and negative sample data in the sample data set, and the labeling process is still a manual labeling process, so there is a possibility of labeling errors, such as labeling negative sample data as positive sample data, or labeling positive sample data as negative sample data, which causes technical problems such as low training accuracy and reliability, and the reliability of the network model will be relatively reduced over time.

[0054] In order to avoid at least one of the above-mentioned technical problems, the inventors of the present disclosure, through creative work, obtained the inventive concept of the present disclosure: combining the network access traffic in the previous time period, classifying and processing the network access traffic in the later time period, and generating a web crawler blacklist based on the classification results of the network access traffic in the previous time period and the classification results of the network access traffic in the later time period.

[0055] Based on the above-mentioned inventive concept, the present disclosure provides a method for generating, a method for identifying, and a device for a web crawler blacklist, which are applied to big data and machine learning in the field of artificial intelligence technology to improve the accuracy of the generated web crawler blacklist.

[0056] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure, as shown in Figure 1 As shown, the method for generating a web crawler blacklist according to an embodiment of the present disclosure includes:

[0057] S101: Obtain network access traffic in a first time period.

[0058] Exemplarily, the executing entity of this embodiment can be a device for generating a web crawler blacklist (hereinafter referred to as a generating device), and the generating device can be a server (such as a local server, a cloud server, a server cluster, etc.), or a computer, or a terminal device, or a processor, or a chip, etc., which is not limited in this embodiment.

[0059] Among them, the "first" in the first time period is used to distinguish it from the second time period in the following text, and is used to distinguish different time periods, but cannot be understood as a limitation on the first time period, such as a limitation on the duration of the first time period.

[0060] S102: Classify the network access traffic in the first time period according to the network access traffic in the second time period to obtain a classification result of the network access traffic in the first time period.

[0061] The classification result of the network access traffic in the first time period indicates whether the network access traffic in the first time period is web crawler traffic, and the second time period is a time period before the first time period.

[0062] That is to say, the first time period and the second time period are two time periods adjacent in time, and the second time period is the time period before the first time period, that is, any time in the second time period is earlier than any time in the first time period, and any time in the first time period is later than any time in the second time period.

[0063] The classification result characterizes whether the corresponding access traffic is web crawler traffic. Accordingly, the classification result of the network access traffic in the first time period characterizes whether the network access traffic in the first time period is web crawler traffic. In this embodiment, whether the network access traffic in the first time period is web crawler traffic is determined in combination with the network access traffic in the second time period, that is, whether the network access traffic in the later time period is web crawler traffic is determined based on the network access traffic in the previous time period, so as to generate a web crawler blacklist through online learning (online learning) and incremental learning (positive-unlabeled learning, PU), thereby improving the accuracy and recall rate of the web crawler blacklist. Technical effect.

[0064] S103: Generate a web crawler blacklist according to the classification result of the network access traffic in the first time period and the classification result of the network access traffic in the second time period.

[0065] In combination with the above analysis, the classification result of the network access traffic in the second time period can be used to indicate whether the network access traffic in the second time period is web crawler traffic.

[0066] It should be understood that the first time period and the second time period are only for adaptive explanation, and different time periods may exist, and the web crawler blacklist may be generated by combining the classification results corresponding to different time periods.

[0067] That is, the web crawler blacklist may be generated by combining the classification results corresponding to two time periods, or by combining the classification results corresponding to more time periods.

[0068] Combined with the above analysis, it can be seen that the embodiment of the present disclosure provides a method for generating a web crawler blacklist, including: obtaining network access traffic in a first time period, classifying and processing the network access traffic in the first time period according to the network access traffic in the second time period, and obtaining a classification result of the network access traffic in the first time period, wherein the classification result of the network access traffic in the first time period characterizes whether the network access traffic in the first time period is web crawler traffic, and the second time period is a time period before the first time period. According to the classification result of the network access traffic in the first time period and the classification result of the network access traffic in the second time period, a web crawler blacklist is generated. In this embodiment, by classifying and processing the network access traffic in the first time period according to the network access traffic in the second time period, the classification result of the network access traffic in the first time period is obtained, so as to generate a web crawler blacklist according to the classification results corresponding to the second time period and the first time period. The technical feature avoids the disadvantage of low accuracy caused by manually generating a web crawler blacklist, and improves the accuracy and reliability of the generated web crawler blacklist.

[0069] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure, as shown in Figure 2 As shown, the method for generating a web crawler blacklist according to an embodiment of the present disclosure includes:

[0070] S201: Receive network access traffic transmitted in real time by a distributed publish-subscribe messaging system.

[0071] It can be understood that in order to avoid redundant elaboration, the technical features of this embodiment that are the same as those of the above-mentioned embodiments will not be described in detail in this embodiment.

[0072] The distributed publish-subscribe messaging system may be Kafka, which is a high-throughput distributed publish-subscribe messaging system.

[0073] For example, Figure 3 As shown, the distributed publish-subscribe messaging system can obtain network access traffic and transmit the obtained network access traffic to the generating device. This embodiment does not limit the transmission method. For example:

[0074] In one example, the distributed publish-subscribe messaging system may report the acquired network access traffic to the generating device. In another example, the distributed publish-subscribe messaging system may also report the network access traffic to the generating device based on a preset time interval.

[0075] S202: Perform streaming computing on the real-time transmitted network access traffic based on a distributed data stream engine to obtain the network access traffic in a first time period.

[0076] Among them, the distributed data flow engine can be used for streaming computing (flink) processing to split the real-time network access traffic into network access traffic in time periods with a certain duration as the unit.

[0077] Exemplarily, the distributed data flow engine uses streaming computing to process network access traffic and obtain network access traffic within 5 minutes.

[0078] In this embodiment, network access traffic is obtained based on a distributed publish-subscribe message system, and the distributed publish-subscribe message system has a high throughput. Therefore, the network access traffic used to generate the web crawler blacklist has a high timeliness, so the efficiency of generating the web crawler blacklist can be improved; and through loss calculation processing, the web crawler blacklist can be generated by combining time periods, and the disadvantages of resource consumption caused by cumbersome operations can be relatively avoided, thereby improving the technical effect of saving resources and rationally utilizing resources.

[0079] S203: Extracting access feature data of the network access traffic in the first time period.

[0080] The access feature data is used to distinguish different access users.

[0081] The network access traffic in the first time period is generated based on different access users. The access feature data refers to data that can be used to distinguish the access users in the first time period. This embodiment does not limit the content of the access feature data.

[0082] Exemplarily, the access characteristic data may be an Internet Protocol (IP) address or a user identification (UID).

[0083] Combine Figure 3 It can be seen that the network access traffic in the first time period can be stored in a high-performance analytical data management system (clickhouse) with column storage, and feature extraction processing can be performed on the network access traffic in the first time period in clickhouse to obtain access feature data of the network access traffic in the first time period.

[0084] S204: Classify the network access traffic in the first time period according to the network access traffic and access feature data in the second time period to obtain a classification result of the network access traffic in the first time period.

[0085] Combined with the above analysis, this step can be understood as: combining the network access traffic of the second time period and the data used to distinguish the visiting users of the first time period, classifying the network access traffic of the first time period, so that the classification result of the network access traffic of the first time period has a strong distinguishing and targeted technical effect relative to each visiting user.

[0086] S205: Generate a web crawler blacklist according to the classification result of the network access traffic in the first time period and the classification result of the network access traffic in the second time period.

[0087] For example, a web crawler blacklist can be generated by combining the network model. Figure 3 As shown, feature data can be input into the network model to obtain a web crawler blacklist.

[0088] In some embodiments, the network model may include an ensemble classifier, and the ensemble classifier includes a base classifier, so as to generate a web crawler blacklist by combining the ensemble classifier. The ensemble classifier may be generated by an ensemble classifier algorithm (sea) for adaptive data stream concept drift.

[0089] The web crawler blacklist may be generated by combining an ensemble classifier with random sampling, or by replacing a base classifier in the ensemble classifier.

[0090] Now combined Figure 4 This paper demonstrates the use of a combination of integrated classifiers and random sampling to generate a web crawler blacklist. Figure 4 is a schematic diagram according to the third embodiment of the present disclosure, as shown in Figure 4 As shown, the method for generating a web crawler blacklist according to an embodiment of the present disclosure includes:

[0091] S401: Obtain network access traffic in a first time period.

[0092] Similarly, in order to avoid redundant elaboration, the technical features of this embodiment that are the same as those of the above embodiments will not be described in detail in this embodiment.

[0093] S402: Extracting access feature data of network access traffic in a first time period.

[0094] The access feature data is used to distinguish different access users.

[0095] S403: Sampling each access feature data to obtain access feature data assigned to each base classifier.

[0096] For example, Figure 5 As shown, the network model may include a sampling layer and an integrated classifier, and the sampling layer is used to perform sampling processing on each access feature data.

[0097] The ensemble classifier includes N base classifiers, such as Figure 5 The base classifiers 1, 2, and N shown in the figure perform sampling processing on all access feature data based on the sampling layer to allocate access feature data to each base classifier, that is, access feature data 1 input to base classifier 1, access feature data 2 input to base classifier 2, and so on, up to access feature data N input to base classifier N.

[0098] In some embodiments, S403 may include: using a bootstrap sampling algorithm to perform sampling processing on each access feature data to obtain access feature data assigned to each base classifier, wherein the number of sampling times is equal to the number of base classifiers.

[0099] Exemplarily, in combination with the above embodiment, the number of base classifiers is N, and N samplings are performed based on the bootstrap sampling algorithm, so that each of the N base classifiers is allocated with corresponding access feature data.

[0100] In this embodiment, the access feature data is sampled by using a bootstrap sampling algorithm and the same number of sampling times as the number of base classifiers, so that the sampling process can be random and comprehensive, thereby enabling the subsequent base classifiers to be trained based on their respective corresponding access feature data with higher accuracy and reliability.

[0101] S404: For each base classifier, train each base classifier based on the access feature data corresponding to each base classifier to obtain a trained base classifier.

[0102] Among the access feature data, at least one access feature data is marked with a sample tag, and the network access traffic corresponding to the access feature data marked with the sample tag is the network crawler traffic.

[0103] That is, in this embodiment, each access feature data may include access feature data marked with a sample tag, and may also include access feature data not marked with a sample tag.

[0104] Relatively speaking, this embodiment can label part of the access feature data, and may not label all access features, and may only label the access feature data whose network access traffic is web crawler traffic, which is equivalent to only labeling negative sample data, thereby reducing labeling costs and improving efficiency.

[0105] In this embodiment, by training each base classifier in combination with sampling processing, the training of each base classifier can be made independent of each other and do not affect each other, thereby achieving the technical effect of avoiding noise interference and improving the reliability and effectiveness of training.

[0106] Each base classifier is trained based on the network access traffic in the second time period.

[0107] S405: Clearing the sample tags of the access feature data marked with the sample tags.

[0108] S406: Classify the access feature data after the clearing process based on each trained base classifier to obtain classification results corresponding to the network access traffic of each access user in the first time period.

[0109] For example, combined with the above analysis, it can be seen that one or more access feature data in each access feature data is marked with a sample label, and in S404, each base classifier is trained based on the access feature data corresponding to each base classifier, thereby obtaining the trained base classifier corresponding to each base classifier.

[0110] That is to say, in this embodiment, the integrated classifier (determined based on the network access traffic of the second time period) is first trained based on the access feature data marked with sample tags (specifically, each base classifier in the integrated classifier is trained), and then the sample tags are cleared to obtain access feature data without sample tags, and the access data without sample tags is again input into the integrated classifier (specifically, input into the base classifier in the integrated classifier), and the classification results corresponding to the network access traffic of each visiting user in the first time period are output.

[0111] For example, if access feature data 1, access feature data 2 to access feature data M are access feature data assigned to base classifier 1, and access feature data 1 and access feature data 2 are access feature data marked with sample tags, then access feature data 1, access feature data 2 to access feature data M are input into base classifier 1 to obtain trained base classifier 1; the sample tags of access feature data 1 and access feature data 2 are cleared, and the cleared access feature data 1 and 2, as well as other access feature data among the M access feature data, are input into the trained base classifier 1, and the classification results corresponding to the network access traffic corresponding to access feature data 1, access feature data 2 to access feature data M are output.

[0112] In this embodiment, by combining the access feature data marked with sample tags and the access feature data after clearing processing to determine the classification results corresponding to the network access traffic in the first time period, the network access traffic with incorrect markings can be corrected, and the network access traffic corresponding to the access feature data without sample tags can be classified and processed, thereby improving the accuracy and reliability of the classification processing.

[0113] Combined with the above analysis, it can be seen that each base classifier is trained based on the network access traffic in the second time period. Therefore, the trained base classifier is equivalent to being determined based on the network access traffic in the second time period, that is, the training process of the base classifier is an incremental training process, and it is also a process of continuous iterative updating. Therefore, the base classifier can have strong timeliness and reliability, thereby improving the technical effect of the accuracy and reliability of classification processing based on the classifier.

[0114] S407: Generate a web crawler blacklist based on the classification result of the network access traffic in the first time period.

[0115] Exemplarily, the web crawler blacklist may be generated in units of time periods, and the final web crawler blacklist may be determined by combining the web crawler blacklists corresponding to each time period, or the web crawler blacklist may be determined by combining the classification results corresponding to each time period.

[0116] Now combined Figure 6 The method of replacing the base classifier in the ensemble classifier to generate a web crawler blacklist is demonstrated. Figure 6 is a schematic diagram according to a fourth embodiment of the present disclosure, as shown in Figure 6 As shown, the method for generating a web crawler blacklist according to an embodiment of the present disclosure includes:

[0117] S601: Obtain network access traffic in a first time period.

[0118] Similarly, in order to avoid redundant elaboration, the technical features of this embodiment that are the same as those of the above embodiments will not be described in detail in this embodiment.

[0119] S602: Extract access feature data of network access traffic in a first time period.

[0120] The access feature data are used to distinguish different access users, and at least one access feature data among the access feature data is marked with a sample tag.

[0121] S603: Clear the sample mark of the access feature data marked with the sample mark.

[0122] S604: Input the access feature data that has been cleared and the access feature data that is not marked with sample tags into each base classifier to obtain the classification results corresponding to each base classifier.

[0123] For example, Figure 7 As shown, if the integrated classifier includes N base classifiers, namely base classifier 1, base classifier 2, and up to base classifier N, the access feature data that has been cleaned and the access feature data that is not marked with sample tags are input into base classifier 1, and the classification results of the network access traffic corresponding to each access feature data are output. The access feature data that has been cleaned and the access feature data that is not marked with sample tags are input into base classifier 2, and the classification results of the network access traffic corresponding to each access feature data are output, until the access feature data that has been cleaned and the access feature data that is not marked with sample tags are input into base classifier N, and the classification results of the network access traffic corresponding to each access feature data are output.

[0124] S605: Generate a classification result of the network access traffic in the first time period according to the classification results corresponding to each base classifier.

[0125] The base classifier is determined based on the network access traffic in the second time period, and the second time period is a time period before the first time period.

[0126] That is to say, the classification result of the network access traffic in the first time period is determined based on the network access traffic in the second time period and the classification results corresponding to each base classifier. By using different base classifiers to determine the classification results of the network access traffic in the first time period respectively, the accuracy and reliability of the classification results of the generated network access traffic in the first time period can be improved.

[0127] In some embodiments, S605 may include: if the classification result corresponding to any base classifier characterizes that the network access traffic in the first time period is web crawler traffic, then the classification result of the network access traffic in the first time period is determined as the classification result characterizing that the network access traffic in the first time period is web crawler traffic.

[0128] In combination with the above embodiments, if the classification results of one or more base classifiers among the N base classifiers characterize that the network access traffic in the first time period is web crawler traffic, then the classification result of the network access traffic in the first time period is determined as the classification result characterizing that the network access traffic in the first time period is web crawler traffic, so as to avoid missing the network access traffic that is web crawler traffic, thereby improving the technical effect of the security and reliability of network access.

[0129] For example, if among the N base classifiers, the classification result of base classifier 1 indicates that the network access traffic in the first time period is web crawler traffic, then the classification result of the network access traffic in the first time period is determined as the classification result indicating that the network access traffic in the first time period is web crawler traffic.

[0130] It should be noted that the network access traffic in the first time period may be generated by different visiting users. For each base classifier, the classification result of the network access traffic of each visiting user can be determined, that is, whether the network access traffic of each visiting user is web crawler traffic.

[0131] Correspondingly, if the classification result of the base classifier 1 for the network access traffic of a certain visiting user is characterized as web crawler traffic, and if the classification result of the base classifier 2 for the network access traffic of the visiting user is characterized as web crawler traffic, then the network access traffic of the visiting user is determined to be web crawler traffic.

[0132] In some embodiments, the classification result of the network access traffic of a certain access user may be determined by the majority of the classification results of each base classifier, or by a threshold.

[0133] For example, the classification results corresponding to N base classifiers are represented, and if there are K classification results that the network access traffic of a certain visiting user is web crawler traffic, and K ≥ (1 / 2) K, then the network access traffic of the visiting user is determined to be web crawler traffic; otherwise, the network access traffic of the visiting user is determined not to be web crawler traffic.

[0134] S606: Generate a new base classifier according to the network access traffic in the first time period.

[0135] This embodiment does not limit the implementation method of generating a new base classifier.

[0136] In one example, the base network can be trained in an iterative manner to obtain a new base classifier.

[0137] In another example, a new base classifier may be trained based on a guided aggregation algorithm (ProbTagging) and network access traffic in the first time period.

[0138] like Figure 7 As shown, a new base classifier can be trained based on the guided clustering algorithm and the access feature data (which can be the access feature data before the sample labels are cleared or the access feature data after the sample labels are cleared).

[0139] S607: Determine the base classifier with the minimum classification accuracy among the base classifiers according to the classification results corresponding to the base classifiers, and replace the base classifier with the minimum classification accuracy with the new base classifier.

[0140] The new base classifier and other base classifiers in the base classifiers are used to classify and process the network access traffic in the third time period, and the third time period is the next time period of the first time period.

[0141] Combined with the above analysis, it can be seen that each base classifier pair among the N base classifiers will classify and process the network access traffic in the first time period to obtain their own corresponding classification results. According to the classification results of each base classifier, the classification accuracy of each base classifier is determined, and the minimum classification accuracy is determined from each classification accuracy, thereby realizing the replacement of the base classifier, that is, replacing the base classifier with the minimum classification accuracy with the new base classifier.

[0142] For example, in combination with the above embodiments and Figure 7 If the base classifier with the minimum classification accuracy among the N base classifiers is base classifier 2, then the new base classifier will replace base classifier 2.

[0143] Accordingly, the network access traffic in the next time period (ie, the third time period) can be classified and processed by combining the N base classifiers except the base classifier 2 and the new base classifier.

[0144] In this embodiment, by replacing the base classifier with the minimum classification accuracy with a new base classifier, the base classifier with the minimum classification accuracy can be replaced, thereby achieving the technical effect of improving the accuracy and reliability of subsequent classification processing.

[0145] In some embodiments, determining the base classifier with the minimum classification accuracy among the classifiers includes the following steps:

[0146] The first step: determining the total amount of web crawler traffic in the network access traffic of each access user in the first time period according to the classification results corresponding to each base classifier.

[0147] For example, in combination with the above analysis, if the number of access users in the first time period is M, each base classifier will obtain M classification results, that is, the access traffic of one access user corresponds to one classification result.

[0148] For each base classifier, determine the number of access traffic representing the visiting user as web crawler traffic among the M classification results determined by the base classifier, and then summarize the classification results of the N base classifiers to obtain the total number of web crawler traffic in the network access traffic of each of the M visiting users.

[0149] The second step: according to the total number and the classification results corresponding to each base classifier, determine the base classifier with the minimum classification accuracy among the base classifiers.

[0150] In this embodiment, the total number is determined by the classification results corresponding to each base classifier, and then the base classifier with the minimum classification accuracy is determined by combining the classification results corresponding to each base classifier. This can achieve a convenient and quick determination of the base classifier with the minimum classification accuracy, and the accuracy and reliability are relatively high.

[0151] In some embodiments, the second step may include the following sub-steps:

[0152] The first sub-step: for each base classifier, determining the amount of web crawler traffic in the network access traffic of each access user in the first time period determined based on the base classifier.

[0153] The second sub-step: determine the classification accuracy of each base classifier based on the number and total number of web crawler traffic in the network access traffic of each visiting user in the first time period determined by each base classifier, and determine the minimum classification accuracy among the classification accuracy rates.

[0154] For example, based on the above analysis, if the classification results of base classifier 1 show that, out of M classification results, P of the users' network access traffic is web crawler traffic, and the total number is Q, then the classification accuracy of base classifier 1 is (P / Q)*100%. This is repeated until the classification accuracy of base classifier N is obtained, thereby determining the base classifier with the minimum classification accuracy.

[0155] In this embodiment, by combining the amount of web crawler traffic in the network access traffic of each visiting user in the first time period and the total amount, the minimum classification accuracy is determined, which can improve the technical effect of the accuracy and reliability of the determined minimum classification accuracy.

[0156] S608: Generate a web crawler blacklist according to the classification result of the network access traffic in the first time period and the classification result of the network access traffic in the second time period.

[0157] Based on the above analysis, it can be seen that a web crawler blacklist can be generated based on the classification results of the network access traffic in the first time period; the web crawler blacklist can also be generated based on the classification results of the network access traffic in the first time period and the classification results of the network access traffic in the second time period in this embodiment; and the web crawler blacklist can also be generated by combining the classification results of the network access traffic in the first time period, the classification results of the network access traffic in the second time period, and the classification results of the network access traffic in the third time period.

[0158] That is to say, generating a web crawler blacklist can be a process of continuous updating and iteration, so that the generated web crawler blacklist has a technical effect of high timeliness and reliability.

[0159] It is worth noting that if the first time period is the initial time period, that is, the first time period is the first time period after initialization, the base classifier is the basic network, and this embodiment does not limit the structure and type of the basic network.

[0160] In some embodiments, after the web crawler blacklist is generated, the web crawler blacklist can be pushed to a cloud server to accelerate the implementation of the web crawler blacklist.

[0161] Depending on whether the quantities of the base classifiers are proportional and linear, the base classifiers can be divided into offline classifiers and nonlinear classifiers.

[0162] Among them, the linear classifier refers to the proportional and linear relationship between quantities, which can be understood as a function of the base classifier whose first-order derivative is a constant; correspondingly, the nonlinear classifier refers to the non-proportional and non-linear relationship between quantities, which can be understood as a function of the base classifier whose first-order derivative is not a constant.

[0163] Relatively speaking, if the base classifier is a linear classifier, the method described in the third embodiment can be used to generate a web crawler blacklist; if the base classifier is a nonlinear base classifier (such as a tree classifier, etc.), the method described in the fourth embodiment can be used to generate a web crawler blacklist.

[0164] It is worth noting that by combining the types of base classifiers (i.e., nonlinear type and linear type) and adopting different methods to generate web crawler blacklists, the diversity, flexibility, and targeted technical effects of generating web crawler blacklists can be improved.

[0165] Figure 8 is a schematic diagram according to a fifth embodiment of the present disclosure, as shown in Figure 8 As shown, the method for identifying web crawler traffic in an embodiment of the present disclosure includes:

[0166] S801: Obtain network access traffic to be identified.

[0167] The executing entity of the embodiment of the present disclosure may be a device for identifying web crawler traffic (hereinafter referred to as the identification device). The identification device may be the same device as the generating device or a different device, which is not limited in this embodiment.

[0168] S802: Identify and process the network access traffic to be identified according to the web crawler blacklist to obtain an identification result.

[0169] The identification result is used to indicate whether the network access traffic to be identified is web crawler traffic, and the web crawler blacklist is determined based on the method described in any of the above embodiments.

[0170] For example, if the identification result indicates that the network access traffic to be identified is web crawler traffic, the web crawler traffic can be blocked to avoid crawling of website content, waste of server resources, and other disadvantages, thereby improving the technical effect of network access security.

[0171] Figure 9 is a schematic diagram according to a sixth embodiment of the present disclosure, as shown in Figure 9 As shown, the network crawler blacklist generation device 900 of the embodiment of the present disclosure includes:

[0172] The first acquiring unit 901 is configured to acquire network access traffic in a first time period.

[0173] The classification unit 902 is used to classify the network access traffic in the first time period according to the network access traffic in the second time period to obtain a classification result of the network access traffic in the first time period, wherein the classification result of the network access traffic in the first time period indicates whether the network access traffic in the first time period is web crawler traffic, and the second time period is the time period before the first time period.

[0174] The generating unit 903 is configured to generate a web crawler blacklist according to the classification result of the network access traffic in the first time period and the classification result of the network access traffic in the second time period.

[0175] Figure 10 is a schematic diagram according to the seventh embodiment of the present disclosure, as shown in Figure 10 As shown, the network crawler blacklist generation device 1000 of the embodiment of the present disclosure includes:

[0176] The first acquiring unit 1001 is configured to acquire network access traffic in a first time period.

[0177] Combine Figure 10 It can be seen that, in some embodiments, the first acquisition unit 1001 includes:

[0178] The receiving subunit 10011 is configured to receive network access traffic transmitted in real time by the distributed publish-subscribe messaging system.

[0179] The processing sub-unit 10012 is configured to perform stream computing processing on the real-time transmitted network access traffic based on a distributed data stream engine to obtain the network access traffic in a first time period.

[0180] The classification unit 1002 is used to classify the network access traffic in the first time period according to the network access traffic in the second time period to obtain a classification result of the network access traffic in the first time period, wherein the classification result of the network access traffic in the first time period indicates whether the network access traffic in the first time period is web crawler traffic, and the second time period is the time period before the first time period.

[0181] Combine Figure 10 It can be seen that, in some embodiments, the classification unit 1002 includes:

[0182] The extraction subunit 10021 is configured to extract access feature data of the network access traffic in the first time period, wherein the access feature data is used to distinguish different access users.

[0183] The classification subunit 10022 is configured to classify the network access traffic in the first time period according to the network access traffic and access feature data in the second time period to obtain a classification result of the network access traffic in the first time period.

[0184] In some embodiments, the number of access users in the first time period is multiple; the classification subunit 10022 includes:

[0185] The clearing module is used to clear the sample tag if at least one access feature data among the access feature data is marked with a sample tag, wherein the network access traffic corresponding to the access feature data marked with the sample tag is the network crawler traffic.

[0186] The classification module is used to perform classification processing based on the network access traffic of the second time period and the access feature data after clearing processing, and obtain classification results corresponding to the network access traffic of each access user in the first time period.

[0187] In some embodiments, the classification module includes:

[0188] The training submodule is used to train a base classifier according to the network access traffic in the second time period.

[0189] The output submodule is used to input the access feature data into the base classifier and output the classification result of the network access traffic in the first time period.

[0190] In some embodiments, there are multiple base classifiers; the output submodule is used to input access feature data into each base classifier, obtain the classification results corresponding to each base classifier, and generate the classification results of the network access traffic in the first time period based on the classification results corresponding to each base classifier.

[0191] In some embodiments, the output submodule is used to determine the classification result of the network access traffic in the first time period as a classification result representing that the network access traffic in the first time period is web crawler traffic if the classification result corresponding to any base classifier represents that the network access traffic in the first time period is web crawler traffic.

[0192] The generating submodule is used to generate a new base classifier according to the network access traffic in the first time period.

[0193] In some embodiments, the generating submodule is configured to train a new base classifier according to the guided clustering algorithm and the network access traffic in the first time period.

[0194] The determination submodule is used to determine the base classifier with the minimum classification accuracy among the base classifiers according to the classification results corresponding to each base classifier.

[0195] In some embodiments, the number of visiting users in the first time period is multiple; the determination submodule is used to determine the total number of web crawler traffic in the network access traffic of each visiting user in the first time period based on the classification results corresponding to each base classifier, and determine the base classifier with the minimum classification accuracy among each base classifier based on the total number and the classification results corresponding to each base classifier.

[0196] In some embodiments, the determination submodule is used to determine, for each base classifier, the amount of web crawler traffic in the network access traffic of each visiting user in the first time period determined based on the base classifier, and determine the classification accuracy of each base classifier based on the amount and total amount of web crawler traffic in the network access traffic of each visiting user in the first time period determined by each base classifier, and determine the minimum classification accuracy among the classification accuracy rates.

[0197] The replacement submodule is used to replace the base classifier with the minimum classification accuracy with the new base classifier.

[0198] The new base classifier and other base classifiers in the base classifiers are used to classify and process the network access traffic in the third time period, and the third time period is the next time period of the first time period.

[0199] In some embodiments, the classification subunit 10022 further includes:

[0200] The training module is used to train the base classifier according to the access feature data to obtain a trained base classifier.

[0201] Furthermore, the clearing module is used to classify the cleared access feature data based on the trained base classifier to obtain classification results corresponding to the network access traffic of each access user in the first time period.

[0202] Correspondingly, there are multiple base classifiers; the output submodule is used to sample the access feature data to obtain the access feature data assigned to each base classifier, and for each base classifier, train each base classifier based on the access feature data corresponding to each base classifier to obtain the trained base classifier of each base classifier.

[0203] The output submodule is used to sample each access feature data using a bootstrap sampling algorithm to obtain access feature data assigned to each base classifier, wherein the number of sampling times is equal to the number of base classifiers.

[0204] The generating unit 1003 is configured to generate a web crawler blacklist according to the classification result of the network access traffic in the first time period and the classification result of the network access traffic in the second time period.

[0205] Figure 11 is a schematic diagram according to an eighth embodiment of the present disclosure, as shown in Figure 11 As shown, the network crawler traffic identification device 1100 of the embodiment of the present disclosure includes:

[0206] The second acquiring unit 1101 is configured to acquire network access traffic to be identified.

[0207] The identification unit 1102 is configured to identify the network access traffic to be identified according to the web crawler blacklist to obtain an identification result.

[0208] The identification result is used to indicate whether the network access traffic to be identified is web crawler traffic, and the web crawler blacklist is determined based on the web crawler blacklist generation method described in any of the above embodiments.

[0209] Figure 12 is a schematic diagram according to the ninth embodiment of the present disclosure, as shown in Figure 12As shown, the electronic device 1200 in the present disclosure may include: a processor 1201 and a memory 1202 .

[0210] Memory 1202 is used to store programs. Memory 1202 may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc. Memory may also include non-volatile memory, such as flash memory. Memory 1202 is used to store computer programs (such as applications and functional modules that implement the above-mentioned methods), computer instructions, etc. These computer programs and computer instructions may be partitioned and stored in one or more memories 1202. Furthermore, these computer programs, computer instructions, data, etc. may be called by processor 1201.

[0211] The aforementioned computer programs, computer instructions, etc. may be partitioned and stored in one or more memories 1202 . Furthermore, the aforementioned computer programs, computer instructions, etc. may be called by the processor 1201 .

[0212] The processor 1201 is configured to execute the computer program stored in the memory 1202 to implement the various steps in the method involved in the above embodiment.

[0213] For details, please refer to the relevant description in the previous method embodiment.

[0214] The processor 1201 and the memory 1202 may be independent structures or integrated structures. When the processor 1201 and the memory 1202 are independent structures, the memory 1202 and the processor 1201 may be coupled via a bus 1203 .

[0215] The electronic device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0216] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information (such as the network access traffic of visiting users, etc.) involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0217] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0218] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0219] Figure 13 A schematic block diagram of an example electronic device 1300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0220] like Figure 13 As shown, device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. Various programs and data required for the operation of device 1300 can also be stored in RAM 1303. Computing unit 1301, ROM 1302, and RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to bus 1304.

[0221] Various components in device 1300 are connected to I / O interface 1305, including an input unit 1306, such as a keyboard and mouse; an output unit 1307, such as various types of displays and speakers; a storage unit 1308, such as a magnetic disk and optical disk; and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1309 allows device 1300 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0222] The computing unit 1301 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1301 performs the various methods and processes described above, such as a method for generating a web crawler blacklist and a method for identifying web crawler traffic. For example, in some embodiments, the method for generating a web crawler blacklist and the method for identifying web crawler traffic can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded into RAM 1303 and executed by the computing unit 1301, one or more steps of the method for generating a web crawler blacklist and the method for identifying web crawler traffic described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to perform the method for generating a web crawler blacklist and the method for identifying web crawler traffic by any other appropriate means (e.g., by means of firmware).

[0223] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0224] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0225] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0226] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0227] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0228] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0229] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0230] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for generating a web crawler blacklist, comprising: Obtain network access traffic in the first time period; Extracting access feature data of the network access traffic in the first time period, wherein the access feature data is used to distinguish different access users; at least one of the access feature data is marked with a sample tag, and the network access traffic corresponding to the access feature data marked with the sample tag is web crawler traffic; Training a base classifier according to the access feature data to obtain a trained base classifier, wherein the trained base classifiers are multiple; the base classifier is obtained by training based on the network access traffic in the second time period; Clearing the sample tags of the access feature data marked with the sample tags; performing classification processing on the cleared access feature data based on the trained base classifiers to obtain classification results corresponding to the trained base classifiers, and generating a classification result of the network access traffic in the first time period based on the classification results corresponding to the trained base classifiers, wherein the classification result of the network access traffic in the first time period indicates whether the network access traffic in the first time period is web crawler traffic, and the second time period is a time period before the first time period; generating a web crawler blacklist according to the classification results of the network access traffic in the first time period and the classification results of the network access traffic in the second time period; The method further comprises: generating a new base classifier according to the network access traffic in the first time period; According to the classification results corresponding to each trained base classifier, the base classifier with the minimum classification accuracy among the trained base classifiers is determined, and the base classifier with the minimum classification accuracy is replaced by the new base classifier; The new base classifier and other base classifiers in the trained base classifiers are used to classify the network access traffic in the third time period, where the third time period is the next time period of the first time period.

2. The method according to claim 1, further comprising: training a base classifier according to the network access traffic in the second time period; The trained base classifiers are used to classify the access feature data after the clearing process, and the classification results corresponding to the trained base classifiers are obtained, including: The access feature data after the cleaning process is input into each trained base classifier to obtain the classification results corresponding to each trained base classifier.

3. The method according to claim 1, wherein Generating a classification result of the network access traffic in the first time period according to the classification results corresponding to each trained base classifier, including: If the classification result corresponding to any trained base classifier indicates that the network access traffic in the first time period is web crawler traffic, the classification result of the network access traffic in the first time period is determined as the classification result indicating that the network access traffic in the first time period is web crawler traffic.

4. The method according to claim 1, wherein The number of access users in the first time period is multiple; and determining a base classifier with the minimum classification accuracy among the trained base classifiers according to the classification results corresponding to the trained base classifiers, including: Determine the total amount of web crawler traffic in the network access traffic of each access user in the first time period according to the classification results corresponding to each trained base classifier; According to the total number and the classification results corresponding to the trained base classifiers, a base classifier with the minimum classification accuracy among the trained base classifiers is determined.

5. The method according to claim 4, wherein Determining a base classifier with the minimum classification accuracy among the trained base classifiers according to the total number and the classification results corresponding to the trained base classifiers, including: For each trained base classifier, determining the amount of web crawler traffic in the network access traffic of each access user in the first time period determined based on the trained base classifier; Based on the number of web crawler traffic in the network access traffic of each visiting user in the first time period determined by each trained base classifier, and the total number, the classification accuracy of each trained base classifier is determined, and the minimum classification accuracy among the classification accuracy rates is determined.

6. The method according to any one of claims 1 to 5, wherein: Generating a new base classifier according to the network access traffic in the first time period includes: A new base classifier is trained according to the guided clustering algorithm and the network access traffic in the first time period.

7. The method according to claim 1, wherein There are multiple base classifiers; the base classifiers are trained according to the access feature data to obtain trained base classifiers, including: Sampling each access feature data to obtain access feature data assigned to each base classifier; For each base classifier, each base classifier is trained based on the access feature data corresponding to each base classifier to obtain a trained base classifier for each base classifier.

8. The method according to claim 7, wherein: The access feature data are sampled to obtain the access feature data assigned to each base classifier, including: A bootstrap sampling algorithm is used to sample each access feature data to obtain access feature data assigned to each base classifier, wherein the number of sampling times is equal to the number of base classifiers.

9. The method according to any one of claims 1-5, 7-8, wherein Get the network access traffic in the first time period, including: Receive network access traffic transmitted in real time by a distributed publish-subscribe messaging system; Based on the distributed data flow engine, streaming computing is performed on the real-time transmitted network access traffic to obtain the network access traffic in the first time period.

10. A method for identifying web crawler traffic, comprising: Obtain the network access traffic to be identified; Identify and process the network access traffic to be identified according to the web crawler blacklist to obtain an identification result; The identification result is used to characterize whether the network access traffic to be identified is web crawler traffic, and the web crawler blacklist is determined based on the method according to any one of claims 1 to 9.

11. A device for generating a web crawler blacklist, comprising: A first acquiring unit, configured to acquire network access traffic in a first time period; Classification unit, including extraction subunit and classification subunit; The classification subunit includes a cleaning module, a training module and a classification module; The extraction subunit is configured to extract access feature data of the network access traffic of the first time period, wherein the access feature data is used to distinguish different access users; at least one of the access feature data is marked with a sample tag, and the network access traffic corresponding to the access feature data marked with the sample tag is web crawler traffic; The training module is used to train a base classifier according to the access feature data to obtain a plurality of trained base classifiers; the base classifiers are obtained by training based on the network access traffic in the second time period; The clearing module is used to clear the sample marks of the access feature data marked with the sample marks; The classification module is configured to perform classification processing on the cleared access feature data based on the trained base classifiers to obtain classification results corresponding to each trained base classifier, and generate a classification result of the network access traffic in the first time period based on the classification results corresponding to each trained base classifier, wherein the classification result of the network access traffic in the first time period indicates whether the network access traffic in the first time period is web crawler traffic, and the second time period is a time period before the first time period; a generating unit, configured to generate a web crawler blacklist according to the classification result of the network access traffic in the first time period and the classification result of the network access traffic in the second time period; The classification module also includes: A generating submodule, configured to generate a new base classifier according to the network access traffic in the first time period; A determination submodule is used to determine the base classifier with the minimum classification accuracy among the trained base classifiers according to the classification results corresponding to the trained base classifiers; The replacement submodule is used to replace the base classifier with the minimum classification accuracy with the new base classifier; The new base classifier and other base classifiers in the trained base classifiers are used to classify the network access traffic in the third time period, where the third time period is the next time period of the first time period.

12. The device according to claim 11, wherein The classification unit further includes: A training submodule, configured to train a base classifier according to the network access traffic in the second time period; The output submodule is used to input the access feature data after the cleaning process into each trained base classifier to obtain the classification results corresponding to each trained base classifier.

13. The device according to claim 12, wherein The output submodule is used to determine the classification result of the network access traffic in the first time period as a classification result representing that the network access traffic in the first time period is web crawler traffic if the classification result corresponding to any trained base classifier represents that the network access traffic in the first time period is web crawler traffic.

14. The device according to claim 11, wherein The number of visiting users in the first time period is multiple; the determination submodule is used to determine the total number of web crawler traffic in the network access traffic of each visiting user in the first time period based on the classification results corresponding to each trained base classifier, and determine the base classifier with the minimum classification accuracy among the trained base classifiers based on the total number and the classification results corresponding to each trained base classifier.

15. The device according to claim 14, wherein The determination submodule is used to determine, for each trained base classifier, the amount of web crawler traffic in the network access traffic of each visiting user in the first time period determined based on the trained base classifier, and determine the classification accuracy of each trained base classifier based on the amount of web crawler traffic in the network access traffic of each visiting user in the first time period determined by each trained base classifier and the total amount, and determine the minimum classification accuracy among the classification accuracy rates.

16. The device according to any one of claims 11 to 15, wherein: The generating submodule is used to train a new base classifier according to the guided aggregation algorithm and the network access traffic in the first time period.

17. The device according to claim 12, wherein There are multiple base classifiers; the output submodule is used to sample the access feature data to obtain the access feature data assigned to each base classifier, and for each base classifier, train each base classifier based on the access feature data corresponding to each base classifier to obtain the trained base classifier of each base classifier.

18. The device according to claim 17, wherein The output submodule is used to sample each access feature data using a bootstrap sampling algorithm to obtain access feature data assigned to each base classifier, wherein the number of sampling times is equal to the number of base classifiers.

19. The device according to any one of claims 11-15, 17-18, wherein: The first acquiring unit includes: A receiving subunit, configured to receive network access traffic transmitted in real time by a distributed publish-subscribe messaging system; The processing subunit is used to perform streaming computing processing on the real-time transmitted network access traffic based on a distributed data stream engine to obtain the network access traffic in the first time period.

20. A device for identifying web crawler traffic, comprising: A second acquiring unit, configured to acquire network access traffic to be identified; An identification unit is used to identify and process the network access traffic to be identified according to the network crawler blacklist to obtain an identification result; The identification result is used to characterize whether the network access traffic to be identified is web crawler traffic, and the web crawler blacklist is determined based on the method according to any one of claims 1 to 9.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 9; or to enable the at least one processor to perform the method of claim 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9; or, the computer instructions are used to cause the computer to execute the method according to claim 10.

23. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 9; or, when the computer program is executed by a processor, the computer program implements the steps of the method according to claim 10.

Images