URL similarity detection method, device, equipment and storage medium

Abstract

The present application relates to the field of web security technology, and discloses a URL similarity detection method, device, equipment and storage medium, the method comprising: obtaining a source URL and a target URL; segmenting the source URL according to the URL hierarchy and special characters to obtain a source segmentation sequence; segmenting the target URL according to the URL hierarchy and special characters to obtain a target segmentation sequence; using the segmentations in the target segmentation sequence and the segmentations in the source segmentation sequence as the minimum measurement unit to determine the edit distance between the target URL and the source URL; and determining the similarity between the target URL and the source URL according to the edit distance. The present application improves the accuracy of URL similarity detection.

Description

Technical Field

[0001] The present application relates to the field of web security technology, and in particular to a URL similarity detection method, device, equipment and storage medium. Background Art

[0002] URLs, as network address identifiers, usually contain keywords related to page resources or topics or brand keywords of a famous company, etc., which are convenient for people to remember and search. However, attackers often use some confusing words to forge URLs, deceive users, and conduct phishing attacks. Therefore, when using statistical learning methods to detect traffic anomalies, Web application firewalls often evaluate the similarity between the URL to be detected and the historical normal URL.

[0003] The Levenshtein edit distance calculation method can quickly calculate the similarity between two strings, but this method is sensitive to the string length and only measures the edit distance, and cannot accurately measure the similarity between the URL to be detected and the normal URL. Summary of the invention

[0004] The embodiments of the present application provide a URL similarity detection method, apparatus, device and storage medium, which can at least to some extent solve the problem that the Levenshtein edit distance calculation method cannot accurately measure the similarity between URLs, thereby improving the accuracy of URL similarity detection.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0006] According to a first aspect of an embodiment of the present application, a URL similarity detection method is provided, the method comprising:

[0007] Get the source URL and target URL;

[0008] Segmenting the source URL according to the URL's hierarchical structure and special characters to obtain a source segmentation sequence;

[0009] Segmenting the target URL according to the URL's hierarchical structure and special characters to obtain a target segmentation sequence;

[0010] Taking the segmented words in the target segmented word sequence and the segmented words in the source segmented word sequence as the minimum measurement unit, determining the edit distance between the target URL and the source URL;

[0011] The similarity between the target URL and the source URL is determined according to the edit distance.

[0012] In some solutions of the present application, based on the above solution, the method further includes:

[0013] Performing generalization processing on the character string satisfying the preset rule in the source URL to obtain a first generalized variable set;

[0014] Performing generalization processing on the character string in the target URL that meets the preset rule to obtain a second generalized variable set;

[0015] Obtaining a third generalized variable set according to the union of the first generalized variable set and the second generalized variable set;

[0016] Taking the target word segmentation sequence and the word segmentation in the source word segmentation sequence as the minimum measurement unit, determining the edit distance between the target URL and the source URL, including:

[0017] Taking the target word segmentation sequence and the words in the source word segmentation sequence as minimum measurement units, recursively calculating the edit distance between the first M words in the source word segmentation sequence and the first N words in the target word segmentation sequence;

[0018] Among them, if the Mth participle in the source participle sequence and the Nth participle in the target participle sequence both belong to the third generalized variable set, then the edit distance between the first M participles in the source participle sequence and the first N participles in the target participle sequence is less than or equal to the edit distance between the first M-1 participles in the source participle sequence and the first N-1 participles in the target participle sequence, 1≤M, 1≤N.

[0019] In some solutions of the present application, based on the above-mentioned solution URL, the step of determining the edit distance between the target URL and the source URL by taking the target word segmentation sequence and the word segmentation in the source word segmentation sequence as the minimum measurement unit includes:

[0020] The following formula is used to determine the detection matrix of the target URL and the source URL;

[0021]

[0022] Among them, a represents the source word sequence, b represents the source word sequence, lev a，b (i, j) represents the edit distance between the first i words in a and the first j words in b, 0≤i≤m, 0≤j≤n, and L represents the third generalized variable set;

[0023] Determine the lev in the detection matrix a，b (m, n) is the edit distance between the target URL and the source URL.

[0024] In some solutions of the present application, based on the above-mentioned solution URL, determining the similarity between the target URL and the source URL according to the edit distance includes:

[0025] Normalizing the edit distance between the source URL and the target URL to obtain a normalized edit distance;

[0026] The similarity between the target URL and the source URL is determined according to the normalized edit distance.

[0027] In some solutions of the present application, based on the above-mentioned solution URL, the editing distance between the source URL and the target URL is normalized to obtain the normalized editing distance, including:

[0028] Obtaining the number of segmented words in the target segmented word sequence;

[0029] The ratio of the edit distance between the source URL and the target URL to the number of segmented words is used as the normalized edit distance.

[0030] In some solutions of the present application, based on the above-mentioned solution URL, determining the similarity between the target URL and the source URL according to the edit distance includes:

[0031] If the edit distance is greater than or equal to a preset threshold, it is determined that the target URL is not similar to the source URL;

[0032] If the edit distance is less than the preset threshold, it is determined that the target URL is similar to the source URL.

[0033] In some solutions of the present application, based on the above-mentioned scheme URL, the source URL is segmented according to the hierarchy and special characters of the URL to obtain a source segmentation sequence, including:

[0034] The source URL is divided according to the URL hierarchy to obtain multiple sub-units;

[0035] The multiple sub-units are respectively segmented by symbols according to special characters of the URL and a preset regular expression to obtain multiple character substrings;

[0036] The multiple character substrings are segmented respectively by a maximum matching algorithm to obtain the source word segmentation sequence.

[0037] According to a second aspect of an embodiment of the present application, a device for obtaining URL similarity is provided, wherein the device comprises:

[0038] URL acquisition unit, used to acquire source URL and target URL;

[0039] A word segmentation unit, used to segment the source URL according to the URL hierarchy and special characters to obtain a source word segmentation sequence;

[0040] The word segmentation unit is also used to segment the target URL according to the URL hierarchy and special characters to obtain a target word segmentation sequence;

[0041] An edit distance calculation unit, used to determine the edit distance between the target URL and the source URL by taking the segmented words in the target segmented word sequence and the segmented words in the source segmented word sequence as the minimum measurement unit;

[0042] A similarity calculation unit is used to determine the similarity between the target URL and the source URL according to the edit distance.

[0043] According to a third aspect of an embodiment of the present application, there is provided an electronic device, characterized in that it includes:

[0044] one or more processors;

[0045] A storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements the method described in the first aspect above.

[0046] According to a fourth aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored, characterized in that when the computer program is executed, the method described in the first aspect above is implemented.

[0047] The embodiment of the present application performs word segmentation on the source URL and the target URL respectively, and uses the word segmentation as the minimum measurement unit to calculate the edit distance between the source URL and the target URL, which fully considers the structural nature of the URL and improves the accuracy of similarity detection.

[0048] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] In the attached picture:

[0051] Figure 1 A flowchart of a URL similarity detection method provided in an embodiment of the present application.

[0052] Figure 2 A flowchart of a URL segmentation method provided in an embodiment of the present application.

[0053] Figure 3 A flowchart of another URL similarity detection method provided in an embodiment of the present application.

[0054] Figure 4 A flowchart of a URL similarity detection device provided in an embodiment of the present application.

[0055] Figure 5 A schematic diagram of a program product for implementing the above method according to an embodiment of the present application is shown.

[0056] Figure 6 A schematic diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0059] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0060] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0061] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the feature. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0062] URL is a uniform resource locator, which is the address of a standard resource on the Internet. It can be used to access and obtain information resources. URL address is a special string without spaces represented by partial ASCII code. It is different from traditional natural language Chinese and English text data. URL address is a special hierarchical network data with its unique language characteristics: a single URL is not a complete sentence, has a limited length, and usually contains some special strings, such as IP address, date, version number and other fields with special meanings.

[0063] Figure 1 A flowchart of a URL similarity detection method provided in an embodiment of the present application. Figure 1 As shown, the method includes at least the following steps.

[0064] Step 110: Get the source URL and the target URL.

[0065] Step 120: Segment the source URL according to the URL's hierarchical structure and special characters to obtain a source segmentation sequence.

[0066] The URL syntax is extensible, and its standard structure is:

[0067] Protocol type: [ / / server address [: port number]] [ / path] [? query] [# fragment]

[0068] Most URLs consist of three main parts: scheme, domain, and path. The scheme part indicates the transmission protocol used by the URL. Common protocols in the network field are http and https. The server address part usually uses a URL or IP address to indicate the location of the resource on the network. The path part specifies the specific location of the resource file in the server address. It is also a hierarchical structure, with " / " as the separator to separate the entire path.

[0069] According to the URL specification in RFC 1738, only letters and numbers [0-9a-zA-Z], some special symbols "-_.~" [excluding double quotes], and certain reserved words can be used directly in URLs without being encoded.

[0070] Segmenting the source URL according to the URL's hierarchical structure and special characters means splitting the source URL according to the special symbols allowed by the specification in each URL hierarchical structure to obtain a source segmentation sequence.

[0071] For example, the source URL is http: / / 10.1.1.1.abc.xyz.com / foods / apple.html

[0072] The three levels are the scheme level part "http: / / ", the domain level part "67.1.12.3.abc.xyz.com", and

[0073] The path hierarchy part "foods / apple.html".

[0074] Use the special string to split into multiple parts to obtain multiple word segments "http", "10.1.1.", "abc", "xyz", "com", "foods", "apple", and "html".

[0075] Step 130: Segment the target URL according to the URL hierarchy and special characters to obtain a target segmentation sequence.

[0076] Step 130 may use the same method as step 120 to segment the target URL.

[0077] Step 140: Using the words in the target word segmentation sequence and the words in the source word segmentation sequence as minimum measurement units, determine the edit distance between the target URL and the source URL.

[0078] Edit distance, also known as Levenshtein distance, refers to the minimum number of edit operations required to transform two strings from one to another. Permitted edit operations include replacing one character with another, inserting a character, and deleting a character. The smaller the edit distance, the greater the similarity between the two strings.

[0079] Traditional Levenshtein uses a single character in a string as the minimum measurement unit. It directly uses Levenshtein to detect the edit distance between the source URL and the target URL. It can only measure the edit distance between the source URL and the target URL string, without considering that the URL itself is hierarchically structured data. Therefore, the embodiment of the present application uses the segmentation in the target segmentation sequence and the segmentation in the source segmentation sequence as the minimum input unit of the Levenshtein edit distance calculation algorithm to calculate the edit distance between the target URL and the source URL.

[0080] Step 150: Determine the similarity between the target URL and the source URL based on the edit distance.

[0081] The embodiment of the present application performs word segmentation on the source URL and the target URL respectively, and uses the word segmentation as the minimum measurement unit to calculate the edit distance between the source URL and the target URL. Compared with the traditional edit distance algorithm that uses a single character as the minimum editable granularity to calculate the edit distance, the present application fully considers the structural nature of the URL, uses the word segmentation as the minimum editable granularity to calculate the edit distance, and improves the accuracy of similarity detection.

[0082] In some solutions of the present application, based on the above solution, the method further includes:

[0083] Performing generalization processing on the character strings satisfying the preset rules in the source URL to obtain a first generalized variable set;

[0084] Performing generalization processing on the character strings satisfying the preset rules in the target URL to obtain a second generalized variable set;

[0085] A third generalized variable set is obtained according to the union of the first generalized variable set and the second generalized variable set.

[0086] In the specific implementation, it can be refined <ip> 、 <ch> 、 <emial> 、 <time>as well as <cookie>The characteristics of strings such as , design regular expressions according to their respective characteristics, and generalize the source URL and target URL through regular expressions to obtain their respective generalized variable sets. For example, the string of the source URL satisfies <ip> 、 <emial> 、 <time>The first set of generalized variables contains <ip> 、 <emial> 、 <time>; The target URL string satisfies <ip> 、 <emial>The second set of generalized variables contains <ip> 、 <emial>; The third set of generalized variables includes <ip> 、 <emial>.

[0087] Based on the above generalization processing steps, the target word segmentation sequence and the word segmentation sequence in the source word segmentation sequence are used as the minimum measurement unit to determine the edit distance between the target URL and the source URL, including:

[0088] Taking the target word segmentation sequence and the words in the source word segmentation sequence as the minimum measurement unit, recursively calculate the edit distance between the first M words in the source word segmentation sequence and the first N words in the target word segmentation sequence;

[0089] Among them, if the Mth segmentation in the source segmentation sequence and the Nth segmentation in the target segmentation sequence both belong to the third generalized variable set, then the edit distance between the first M segmentations in the source segmentation sequence and the first N segmentations in the target segmentation sequence is less than or equal to the edit distance between the first M-1 segmentations in the source segmentation sequence and the first N-1 segmentations in the target segmentation sequence, 1≤M, 1≤N.

[0090] The Mth word in the source word sequence and the Nth word in the target word sequence both belong to the third generalized variable set, indicating that they have the same properties. Assuming that the first M-1 words in the source word sequence and the first N-1 words in the target word sequence need K editing operations to successfully match, then both parties add the Mth word and the Nth word respectively. If the properties of the two newly matched characters are the same, the number of operations K remains unchanged; if the newly matched characters are different, it means that a replacement operation is required, such as replacing the Mth word in the source word sequence with the Nth word in the target word sequence. At this time, the minimum number of operations for the Mth word in the source word sequence and the Nth word in the target word sequence to successfully match is K+1.

[0091] The embodiment of the present application generalizes the source URL and the target URL, and considers whether the newly added word of each party has the same attribute when recursively calculating the edit distance. If the two have the same attribute, it means that the similarity between the two is high. Therefore, when each party adds a word and then matches, in order to make the two parties match, there is no need to perform editing operations before adding, that is, the edit distance does not need to be added by 1 based on the basis before adding two word parts.

[0092] In some solutions of the present application, based on the above solution, the target word segmentation sequence and the word segmentation sequence in the source word segmentation sequence are used as the minimum measurement unit to determine the edit distance between the target URL and the source URL, including:

[0093] The following formula is used to determine the detection matrix of the target URL and the source URL;

[0094]

[0095] Among them, a represents the source word sequence, b represents the source word sequence, lev a，b (i, j) represents the edit distance between the first i words in a and the first j words in b, 0≤i≤m, 0≤j≤n, L represents the third generalization variable set;

[0096] Determine the lev in the detection matrix a，b (m, n) is the edit distance between the target URL and the source URL.

[0097] If both the source URL and the target URL have empty word segments, the minimum edit distance is 0. If one of the word segments is empty and the other is not empty, the minimum edit distance is the number of currently matched word segments on the non-empty side.

[0098] Assume that it takes K1 edit operations to match a[i-1:] with b[j:]. At this time, a[i] is added to the source word sequence. In order to make a[i:] match b[j:], a[i] needs to be removed. Therefore, the number of edit operations required to match a[i:] with b[j:] is K1+1.

[0099] Similarly, suppose that matching a[i:] with b[j-1:] requires K2 edit operations. At this time, b[j] is added to the target word sequence. In order to match a[i:] with b[j:], b[j] needs to be removed. Therefore, the number of edit operations required to match a[i:] with b[j:] is K2+1.

[0100] Similarly, suppose that a[i-1:] and b[j-1:] need to be modified K3 times to match. In this case, a[i] is added to the source word segmentation sequence and b[j] is added to the target word segmentation sequence. If a[i] and b[j] both belong to the third generalization set, then in order to make a[i:] and b[j:] match, no editing operation is required, and the number of editing operations K3 remains unchanged. If either a[i] or b[j] does not belong to the third generalization set, then in order to make a[i:] and b[j:] match, a replacement operation is required, and the number of editing operations K3+1 is required.

[0101] Figure 2 A flowchart of a URL segmentation method provided in an embodiment of the present application. Figure 2 As shown, the method includes at least the following steps.

[0102] Step 210: Split the source URL according to the URL hierarchy to obtain multiple sub-units.

[0103] As described above, a URL mainly includes three parts: a protocol type (scheme), a server address (domain) and a path (path). Among them, the URL in the server address can be divided into two parts: a free URL (FDN) and a registered URL (RDN). The free URL part can be managed and changed by the URL owner, and the registered URL part needs to be managed and allocated by the URL registrar or URL registration agency. The registered URL part can be further divided from left to right into a secondary subdomain (SLD) and a top-level subdomain (TLD). The top-level subdomain is also called a public suffix, which is managed by the registrar or registration agency. The secondary subdomain is located before the top-level subdomain and is defined by the URL registrant. In the present invention, after hierarchical segmentation, the URL finally obtains five parts, namely: the URL's protocol type (scheme), free URL (FDN), secondary subdomain (SLD), top-level subdomain (TLD) and path (path).

[0104] Step 220: perform symbol segmentation on multiple sub-units according to special characters of the URL and a preset regular expression to obtain multiple character substrings.

[0105] The IP address can be used in the URL to indicate the server address. The path part or FDN part can contain strings in the form of dates, version numbers, or consecutive numbers. Since the IP address, date, and consecutive numbers have specific formats, regular expressions are used to extract strings in specific formats and split the URL.

[0106] Step 230: Segment the multiple character substrings separately using a maximum matching algorithm to obtain a source word segmentation sequence.

[0107] The bidirectional maximum matching algorithm contains two types of matching: forward maximum matching and reverse maximum matching, both of which are based on a pre-prepared dictionary for string matching. The reverse maximum matching algorithm reads a string of unsegmented text from a pointer starting at the end of the string, and checks whether the current string is a word in the dictionary. If it is, insert a space and repeat the process. If not, the pointer moves one to the right, the string length decreases, and the matching process is repeated until a single character remains in the end. If no word is found, a single character is created, which is a non-dictionary word, to represent the final separated word. The forward maximum matching algorithm works similarly, but reads from a pointer starting from the beginning of the string.

[0108] For example, the source URL looks like this:

[0109] / art / 3492 / 3 / 3 / art_35_349341 / doc? / name=3298sfjkasdk&mail=123@sn.com

[0110] Segment it to get "art", "3492", "3", "3", "art", "35", "_", "349341", "_", "doc" "?" "name" "=" "3298sfjkasdk" "&" "mail", "=", "123 @sn.com ”.

[0111] For example, the target URL looks like this:

[0112] / art / 1034 / 11 / 23 / art_35_14891 / doc? / name=klak324345lkl3456jd&mail= y2ln@sd.com

[0113] Segment it to get "art", "1034", "11", "23", "art", "35", "_" "14891", "_" "doc" "?" "name" "=" "klak324345lkl3456jd" "&" "mail", "=", "y2ln @sn.com ”.

[0114] From the above word segmentation results, we can find that the source URL and the target URL have the same number of word segments, and the word segments at the same position of the two satisfy the same preset rules, that is, they belong to the third generalization set. Then, the above edit distance calculation formula can be used to obtain that the edit distance between the two is 0, and the two are similar.

[0115] Figure 3 A flowchart of another URL similarity detection method provided in an embodiment of the present application. Figure 3 As shown, the method includes at least the following steps.

[0116] Step 310: Get the source URL and the target URL.

[0117] Step 320: Word segmentation: the source URL is segmented according to the URL hierarchy and special characters to obtain a source word segmentation sequence; the target URL is segmented according to the URL hierarchy and special characters to obtain a target word segmentation sequence.

[0118] Step 330: Generalization: generalize the character strings in the source URL that meet the preset rules to obtain a first generalized variable set; generalize the character strings in the target URL that meet the preset rules to obtain a second generalized variable set; obtain a third generalized variable set based on the union of the first generalized variable set and the second generalized variable set.

[0119] Step 340: Calculate the edit distance: take the words in the target word segmentation sequence and the words in the source word segmentation sequence as the minimum measurement unit to determine the edit distance between the target URL and the source URL.

[0120] Step 350: Normalize the edit distance: normalize the edit distance between the source URL and the target URL to obtain a normalized edit distance.

[0121] In a specific implementation, the edit distance between the source URL and the target URL is normalized to obtain the normalized edit distance, including:

[0122] Get the number of segmented words in the target segmented word sequence;

[0123] The ratio of the edit distance between the source URL and the target URL to the number of segmented words is taken as the normalized edit distance.

[0124] Step 360: If the normalized edit distance is greater than or equal to a preset threshold, it is determined that the target URL is not similar to the source URL; if the normalized edit distance is less than the preset threshold, it is determined that the target URL is similar to the source URL.

[0125] In an embodiment of the present application, if the editing distance between the target URL and the source URL is less than a preset threshold, it means that the two are very similar, and whether the target URL is a safe URL can be determined by judging the similarity between the target URL and the normal source URL.

[0126] The following introduces an embodiment of a device for executing the above-mentioned URL similarity detection method. For details not omitted in the URL similarity detection device, please refer to the embodiment of the above-mentioned URL similarity detection method.

[0127] Figure 4 A flowchart of a URL similarity detection device provided in an embodiment of the present application. Figure 3 As shown, the device 400 includes at least the following parts.

[0128] URL acquisition unit 410, used to acquire source URL and target URL;

[0129] A word segmentation unit 420 is used to segment the source URL according to the URL's hierarchical structure and special characters to obtain a source word segmentation sequence;

[0130] The word segmentation unit 420 is also used to segment the target URL according to the URL hierarchy and special characters to obtain a target word segmentation sequence;

[0131] The edit distance calculation unit 430 is used to determine the edit distance between the target URL and the source URL by taking the segmented words in the target segmented word sequence and the segmented words in the source segmented word sequence as the minimum measurement unit;

[0132] The similarity calculation unit 440 is used to determine the similarity between the target URL and the source URL according to the edit distance.

[0133] The word segmentation method in the embodiment of the present application fully considers the hierarchical information and special character information of the URL, and divides the URL into multiple structural units.

[0134] refer to Figure 5 As shown, a program product 500 for implementing the above method according to an embodiment of the present application is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0135] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0136] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0137] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0138] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0139] As another aspect, the present application also provides an electronic device capable of implementing the above method. It will be appreciated by those skilled in the art that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software, which may be collectively referred to as a "circuit", "module" or "system" herein.

[0140] Refer to the following Figure 6 The electronic device 600 according to this embodiment of the present application is described. Figure 6 The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0141] like Figure 6 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include but are not limited to: at least one processing unit 610, at least one storage unit 620, and a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610).

[0142] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps of various exemplary implementations of the present application described in the above “Embodiment Method” section of this specification.

[0143] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 621 and / or a cache memory unit 622 , and may further include a read-only memory unit (ROM) 623 .

[0144] The storage unit 620 may also include a program / utility 624 having a set (at least one) of program modules 626, such program modules 626 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0145] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0146] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 660. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 660. As shown, the network adapter 660 communicates with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0147] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0148] In addition, the above-mentioned figures are only schematic illustrations of the processing included in the method according to the exemplary embodiment of the present application, rather than for limiting purposes. It is easy to understand that the processing shown in the above-mentioned figures does not indicate or limit the time sequence of these processing. In addition, it is also easy to understand that these processing can be, for example, synchronously or asynchronously executed in multiple modules. Finally, it should be noted that: it should be understood by those skilled in the art that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-usable program codes.

[0149] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0150] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0151] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.< / emial> < / ip> < / emial> < / ip> < / emial> < / ip> < / time> < / emial> < / ip> < / time> < / emial> < / ip> < / cookie> < / time> < / emial> < / ch> < / ip>

Claims

1. A URL similarity detection method, characterized in that: The method comprises: Get the source URL and target URL; Segmenting the source URL according to the URL's hierarchical structure and special characters to obtain a source segmentation sequence; Segmenting the target URL according to the URL's hierarchical structure and special characters to obtain a target segmentation sequence; Taking the segmented words in the target segmented word sequence and the segmented words in the source segmented word sequence as the minimum measurement unit, determining the edit distance between the target URL and the source URL; Determining the similarity between the target URL and the source URL according to the edit distance; The method further comprises: Performing generalization processing on the character string satisfying the preset rule in the source URL to obtain a first generalized variable set; Performing generalization processing on the character string in the target URL that meets the preset rule to obtain a second generalized variable set; Obtaining a third generalized variable set according to the union of the first generalized variable set and the second generalized variable set; The step of determining the edit distance between the target URL and the source URL by taking the target word segmentation sequence and the word segmentation in the source word segmentation sequence as the minimum measurement unit includes: Taking the target word segmentation sequence and the words in the source word segmentation sequence as minimum measurement units, recursively calculating the edit distance between the first M words in the source word segmentation sequence and the first N words in the target word segmentation sequence; Among them, if the Mth participle in the source participle sequence and the Nth participle in the target participle sequence both belong to the third generalized variable set, then the edit distance between the first M participles in the source participle sequence and the first N participles in the target participle sequence is less than or equal to the edit distance between the first M-1 participles in the source participle sequence and the first N-1 participles in the target participle sequence, 1≤M, 1≤N.

2. The URL similarity detection method according to claim 1, characterized in that: The step of determining the edit distance between the target URL and the source URL by taking the target word segmentation sequence and the word segmentation in the source word segmentation sequence as the minimum measurement unit includes: The following formula is used to determine the detection matrix of the target URL and the source URL; Among them, a represents the source word sequence, b represents the source word sequence, lev a,b (i, j) represents the edit distance between the first i words in a and the first j words in b, 0≤i≤m, 0≤j≤n, L represents the third generalized variable set; Determine the lev in the detection matrix a,b (m, n) is the edit distance between the target URL and the source URL.

3. The URL similarity detection method according to claim 1, characterized in that: Determining the similarity between the target URL and the source URL according to the edit distance includes: Normalizing the edit distance between the source URL and the target URL to obtain a normalized edit distance; The similarity between the target URL and the source URL is determined according to the normalized edit distance.

4. The URL similarity detection method according to claim 3, characterized in that: The step of normalizing the edit distance between the source URL and the target URL to obtain the normalized edit distance includes: Obtaining the number of segmented words in the target segmented word sequence; The ratio of the edit distance between the source URL and the target URL to the number of segmented words is used as the normalized edit distance.

5. The URL similarity detection method according to claim 1, characterized in that: Determining the similarity between the target URL and the source URL according to the edit distance includes: If the edit distance is greater than or equal to a preset threshold, it is determined that the target URL is not similar to the source URL; If the edit distance is less than the preset threshold, it is determined that the target URL is similar to the source URL.

6. The URL similarity detection method according to claim 1, characterized in that: The word segmentation of the source URL according to the URL hierarchy and special characters to obtain a source word segmentation sequence includes: The source URL is divided according to the URL hierarchy to obtain multiple sub-units; The multiple sub-units are respectively segmented by symbols according to special characters of the URL and a preset regular expression to obtain multiple character substrings; The multiple character substrings are segmented respectively by a maximum matching algorithm to obtain the source word segmentation sequence.

7. A URL similarity detection device, characterized in that: The device comprises: URL acquisition unit, used to acquire source URL and target URL; A word segmentation unit, used to segment the source URL according to the URL hierarchy and special characters to obtain a source word segmentation sequence; The word segmentation unit is also used to segment the target URL according to the URL hierarchy and special characters to obtain a target word segmentation sequence; An edit distance calculation unit, used to determine the edit distance between the target URL and the source URL by taking the segmented words in the target segmented word sequence and the segmented words in the source segmented word sequence as the minimum measurement unit; A similarity calculation unit, used to determine the similarity between the target URL and the source URL according to the edit distance; The device is also used to perform generalization processing on the character string that meets the preset rules in the source URL to obtain a first generalization variable set; perform generalization processing on the character string that meets the preset rules in the target URL to obtain a second generalization variable set; and obtain a third generalization variable set according to the union of the first generalization variable set and the second generalization variable set; The step of determining the edit distance between the target URL and the source URL by taking the target word segmentation sequence and the word segmentation in the source word segmentation sequence as the minimum measurement unit includes: Taking the target word segmentation sequence and the words in the source word segmentation sequence as minimum measurement units, recursively calculating the edit distance between the first M words in the source word segmentation sequence and the first N words in the target word segmentation sequence; Among them, if the Mth participle in the source participle sequence and the Nth participle in the target participle sequence both belong to the third generalized variable set, then the edit distance between the first M participles in the source participle sequence and the first N participles in the target participle sequence is less than or equal to the edit distance between the first M-1 participles in the source participle sequence and the first N-1 participles in the target participle sequence, 1≤M, 1≤N.

8. An electronic device, characterized in that: include: one or more processors; A storage device, used to store one or more programs, when the one or more programs are executed by the one or more processors, enables the electronic device to implement the method as described in any one of claims 1-6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed, the method according to any one of claims 1 to 6 is implemented.

Images