URL deduplication method and device, electronic equipment and computer readable storage medium

Abstract

The invention relates to the technical field of network application, and discloses a URL deduplication method and device, electronic equipment and a computer readable storage medium. The URL deduplication method comprises the steps of obtaining a to-be-processed URL, wherein the URL comprises a plurality of fields, and each field is provided with a corresponding field value; if the field value ofthe first preset field in the plurality of fields meets the preset condition, determining a parameter field from the plurality of fields; acquiring a hash value corresponding to the URL based on the determined parameter field; and if the hash value is matched with at least one hash value in pre-stored record information, deleting the URL to carry out duplicate removal. According to the URL deduplication method provided by the invention, different URLs can be prevented from being misjudged as the same URL when the different URLs share the path part and only forward the processing logic in the application program based on the parameter value, and the deduplication accuracy is improved, so that the system can be protected more effectively during network intrusion detection.

Description

technical field [0001] The present application relates to the technical field of network applications, and in particular, the present application relates to a URL deduplication method, device, electronic equipment, and computer-readable storage medium. Background technique [0002] URL (Uniform Resource Locator, Uniform Resource Locator) is a representation method for specifying the location of information on the World Wide Web service program of the Internet, including multiple parts such as protocol, domain name, path, and parameters. [0003] URL detection and filtering is an important part of the network intrusion detection system. At present, URL deduplication usually adopts path rewriting (Rewrite) deduplication, marking and deduplication based on the dynamic parameters of the path part of the URL, when different URLs share the path part , when the processing logic in the application is only forwarded based on parameter values, using path rewriting to deduplicate will ...

AI Technical Summary

Problems solved by technology

[0003] URL detection and filtering is an important part of the network intrusion detection system. At present, URL deduplication usually adopts path rewriting (Rewrite) deduplication, marking and deduplication based on the dynamic parameters of the path part of the URL, when different URLs share the path part , when the processing logic in the application is only forwarded based on the parameter value, using path rewriting to deduplicate will cause different URLs to be misjudged as the same URL, and the accuracy of deduplication is low

Images