Data detection method and device

Abstract

The embodiment of the invention discloses a data detection method and a data detection device. The method comprises the steps of extracting at least one target head field from a first service requestdata packet, performing exception analysis in a preset exception analysis rule library, computing a field exception score corresponding to the first service request data packet according to an analysis result to serve as a target field exception score, and storing an association relation between target terminal information in each target head field and the target field exception score into a storage database; and when an exception query request sent by a service server is received, acquiring a query parameter in the exception query request, and when the query parameter is detected to be matched with the target terminal information in the storage database, using the target field exception score corresponding to the target terminal information as result data to be sent to the service server.By using the method and device provided by the invention, the misjudgment rate and the missing rate of malicious crawlers can be reduced, and network request interruption during the normal service process in the service server is avoided.

Description

technical field [0001] The invention relates to the technical field of the Internet, in particular to a data detection method and device. Background technique [0002] A web crawler is a program that automatically obtains the content of web pages. For a website, a large number of requests from malicious crawlers will consume the performance of the server, and even cause the server to go down. Existing web crawler technologies may include high-frequency script crawlers and collector crawlers. [0003] The current mainstream method for real-time detection of malicious reptiles can specifically be as follows: the business server counts the traffic corresponding to a certain IP (Internet Protocol, Network Protocol) address in a time window in real time, and calculates the traffic according to the traffic and the time interval of the time window. The access frequency corresponding to the IP address, and decide whether to intercept or release the service request containing the I...

AI Technical Summary

Problems solved by technology

[0004] Since the existing methods for real-time detection of malicious crawlers rely on a single dimension (that is, only the dimension of visits), the existing methods for real-time detection of malicious crawlers are prone to high false positive and false positive rates.

In the existing methods for real-time detection of malicious crawlers, it is necessary to open up a temporary storage space in the memory of the business server, and set the threshold and judgment logic at the same time, which will easily lead to a large amount of system resources of the business server being occupied

Moreover, in the existing methods for detecting malicious crawlers in real time, the service server needs to wait for the crawler detection system to perform the crawler detection operation on the business request before continuing the subsequent business process. Therefore, in the process of crawler detection for the business request When an exception occurs, it may cause the interruption of network requests in the normal business process in the business server

Images