A method, device, electronic device and readable storage medium for obtaining URL mode

Abstract

The present invention provides a method, device, electronic device and readable storage medium for obtaining a URL pattern, the method comprising the steps of: segmenting each URL in a URL set respectively, and marking the fields generated after the segmentation with a basic pattern according to the character type, wherein the basic pattern is formed by any character of the same character type as the marked field. Then, based on the fields marked with the basic pattern, URLs with the same number of segments and the same basic pattern at the same segment position in the URL set are obtained to form a URL group. Finally, the fields at the same segment position in the URL group are merged according to the marked basic pattern to obtain the URL pattern of the URL group. The present invention can represent the URL pattern more accurately and avoid over-generalization after the URL merging process.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a URL pattern acquisition method, device, electronic device and readable storage medium. Background Art

[0002] With the continuous development of Internet technology, network data is showing explosive growth, one of the most prominent manifestations is the massive growth in the number of URLs (Uniform Resource Locators). In the fields of Internet data search, data collection, data cleaning, website construction, etc., similar URLs need to be processed. The quality and efficiency of processing will directly affect the use of servers and storage resources, as well as the experience of website users.

[0003] A URL generally consists of five parts:

[0004] <scheme> : / / <netloc> / <path> ？ <query> # <fragement>;

[0005] Among them, scheme: protocol; netloc: domain name; path: path; query: parameter; fragment: anchor.

[0006] At present, the commonly used URL processing method is to use artificial custom rules to replace the same or similar parts of multiple URLs. However, this current representation method can only process specific types of URLs, and the processing results are too generalized, that is, the URL type representation is not accurate enough. Summary of the invention

[0007] In view of this, the purpose of the embodiments of the present invention is to provide a URL pattern acquisition method, device, electronic device and readable storage medium, which can obtain a URL pattern with higher accuracy and solve the problem that the results of URL merging processing in the prior art are too generalized.

[0008] In a first aspect, the present invention provides the following technical solution through an embodiment of the present invention:

[0009] A method for obtaining a URL pattern, comprising:

[0010] Segment each URL in the URL set, and mark each field generated after segmenting the URL with a basic pattern according to the character type, where the basic pattern is formed by any characters of the same character type as the marked field;

[0011] According to the fields marked with the basic pattern, URLs with the same number of segments and the same basic pattern at the same segment position in the URL set are obtained to form a URL group;

[0012] The fields at the same segment position in the URL group are merged according to the marked basic pattern to obtain the URL pattern of the URL group.

[0013] Preferably, the step of merging the fields at the same segment position in the URL group according to the marked basic pattern to obtain the URL pattern of the URL group includes:

[0014] The fields at the same segment position in the URL group are merged according to the marked basic pattern and the number of characters in each field to obtain the URL pattern of the URL group.

[0015] Preferably, the step of merging the fields at the same segment position in the URL group according to the marked basic pattern and the number of characters in each field to obtain the URL pattern of the URL group includes:

[0016] A prefix tree is constructed according to each field obtained after segmenting each URL in the URL group; wherein each branch of the prefix tree is a segmented URL, each node of each branch of the prefix tree is a field obtained by segmentation, nodes of the same depth in the prefix tree constitute a layer of the prefix tree, and each layer in the prefix tree corresponds to a same segmentation position;

[0017] The prefix trees are merged from the root node downwards, and the URL pattern of the URL group is obtained according to the basic pattern and the number of characters of the fields included in each layer of the prefix tree.

[0018] Preferably, the step of merging the fields at the same segment position in the URL group according to the marked basic pattern to obtain the URL pattern of the URL group includes:

[0019] Obtaining a pattern string corresponding to each segment position in the URL group; the pattern string is a string obtained by using a unified expression form after performing a merging operation on each field corresponding to each URL in the URL group at the same segment position;

[0020] The pattern strings corresponding to the respective segment positions in the URL group are combined to obtain the URL pattern of the URL group.

[0021] Preferably, the step of obtaining the pattern string corresponding to each segment position in the URL group includes:

[0022] Merge target fields of the URLs in the URL group that are in the same segment position and have the same field content, where the character type of the target field is letters or numbers;

[0023] After merging, the character content of the target field is used as the pattern string of the segment position.

[0024] Preferably, the step of obtaining the pattern string corresponding to each segment position in the URL group includes:

[0025] Merging target fields having the same characters at the same segment position of each URL in the URL group, wherein the same characters included in the target fields are non-alphabetic and non-numeric types;

[0026] After merging, the pattern string of the segment position is obtained according to the character content and the number of characters of the same characters included in the target field.

[0027] Preferably, the step of obtaining the pattern string corresponding to each segment position in the URL group includes:

[0028] Merge the target fields of each URL in the URL group at the same segmentation position, with the same basic pattern of the mark and different characters;

[0029] After merging, the pattern string of the segment position is obtained according to the basic pattern of the target field tag and the field length of the fields in each URL of the segment position before merging.

[0030] Preferably, the step of obtaining the pattern string corresponding to each segment position in the URL group includes:

[0031] The segment position of the predefined character in the URL group is obtained, and the same character content contained in the field after the segment position of the predefined character is used as the pattern string of the segment position where the field is located.

[0032] Preferably, the number of characters in the field after the segment position of the predefined character is less than or equal to a preset first threshold; and / or

[0033] In any of the path part, the parameter part and the location part of the URL, the number of fields after the predefined character is less than or equal to a preset second threshold; and / or

[0034] In any of the path part, parameter part and location part of the URL, the fields after the predefined characters are marked with the basic pattern.

[0035] Preferably, the step of obtaining URLs having the same number of segments and the same basic pattern at the same segment position according to the fields marked with the basic pattern to form a URL group includes:

[0036] According to the fields marked with the basic pattern, obtaining URLs in the URL set that have the same number of segments and the same basic pattern at the same segment position;

[0037] The URLs with the same key field in the parameter part of the obtained URL are regarded as a URL group.

[0038] In a second aspect, the present invention provides the following technical solution through an embodiment of the present invention:

[0039] A URL pattern acquisition device, comprising:

[0040] A marking module, used to segment each URL in the URL set, and mark each field generated after the URL segmentation with a basic pattern according to the character type, wherein the basic pattern is formed by any characters of the same character type as the marked field;

[0041] A URL group generation module, used to obtain URLs with the same number of segments and the same basic pattern at the same segment position in the URL set according to the fields marked with the basic pattern, to form a URL group;

[0042] The URL pattern generating module is used to merge the fields at the same segment position in the URL group according to the marked basic pattern to obtain the URL pattern of the URL group.

[0043] Preferably, the URL pattern generating module is specifically used to merge the fields at the same segment position in the URL group according to the marked basic pattern and the number of characters in each field to obtain the URL pattern of the URL group.

[0044] Preferably, the URL pattern generation module includes:

[0045] A construction unit, used to construct each field obtained after segmenting each URL in the URL group into a prefix tree; wherein each branch of the prefix tree is a segmented URL, each node of each branch of the prefix tree is a field obtained by segmentation, and each layer in the prefix tree corresponds to a same segmentation position;

[0046] The first URL pattern generating unit is used to merge the prefix trees from the root node downwards, and obtain the URL pattern of the URL group according to the basic pattern and the number of characters of the fields included in each layer of the prefix tree.

[0047] Preferably, the URL pattern generation module includes:

[0048] A pattern string generating unit, used to obtain a pattern string corresponding to each segment position in the URL group; the pattern string is a string obtained by performing a merging operation on each field corresponding to each URL in the URL group at the same segment position in a unified expression form;

[0049] The second URL pattern generating unit is used to combine the pattern strings corresponding to the segment positions in the URL group to obtain the URL pattern of the URL group.

[0050] Preferably, the pattern string generating unit comprises:

[0051] A first merging subunit is used to merge target fields of the URLs in the URL group that are in the same segment position and have the same field content, wherein the character type of the target field is letters or numbers;

[0052] It is also used to, after merging, use the character content of the target field as the pattern string of the segment position.

[0053] Preferably, the pattern string generating unit comprises:

[0054] A second merging subunit is used to merge target fields having the same characters at the same segment position of each URL in the URL group, wherein the same characters included in the target fields are non-alphabetic and non-numeric types;

[0055] It is also used to obtain the pattern string of the segment position according to the character content and the number of characters of the same characters included in the target field after merging.

[0056] Preferably, the pattern string generating unit comprises:

[0057] The third merging subunit is used to merge the target fields of the URLs in the URL group at the same segmentation position, with the same basic pattern of the mark and different characters;

[0058] It is also used to obtain the pattern string of the segment position according to the basic pattern of the target field tag and the field length of the fields in each URL of the segment position before merging after merging.

[0059] Preferably, the pattern string generating unit comprises:

[0060] The fourth merging subunit is used to obtain the segment position of the predefined character in the URL group, and use the same character content contained in the field after the segment position of the predefined character as the pattern string of the segment position where the field is located.

[0061] Preferably, the number of characters in the field after the segment position of the predefined character is less than or equal to a preset first threshold; and / or

[0062] In any of the path part, the parameter part and the location part of the URL, the number of fields after the predefined character is less than or equal to a preset second threshold; and / or

[0063] In any of the path part, parameter part and location part of the URL, the fields after the predefined characters are marked with the basic pattern.

[0064] Preferably, the URL group generation module is specifically used to obtain URLs with the same number of segments and the same basic pattern at the same segment position in the URL set according to the fields after the basic pattern marking; and to treat the URLs with the same key field in the parameter part of the obtained URL as a URL group.

[0065] In a third aspect, the present invention provides the following technical solution through an embodiment of the present invention:

[0066] An electronic device, characterized in that it includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the following operations:

[0067] Segment each URL in the URL set, and mark each field generated after segmentation of the URL with a basic pattern according to the character type, where the basic pattern is formed by any characters of the same character type as the marked field;

[0068] According to the fields marked with the basic pattern, URLs with the same number of segments and the same basic pattern at the same segment position in the URL set are obtained to form a URL group;

[0069] The fields at the same segment position in the URL group are merged according to the marked basic pattern to obtain the URL pattern of the URL group.

[0070] In a fourth aspect, the present invention provides the following technical solution through an embodiment of the present invention:

[0071] A readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0072] Segment each URL in the URL set, and mark each field generated after segmentation of the URL with a basic pattern according to the character type, where the basic pattern is formed by any characters of the same character type as the marked field;

[0073] According to the fields marked with the basic pattern, URLs with the same number of segments and the same basic pattern at the same segment position in the URL set are obtained to form a URL group;

[0074] The fields at the same segment position in the URL group are merged according to the marked basic pattern to obtain the URL pattern of the URL group.

[0075] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0076] A URL pattern acquisition method according to an embodiment of the present invention, compared with the prior art, segments the URL and marks the segmented fields with a basic pattern, wherein the marked basic pattern is an abbreviation composed of characters of the same character type as the field, thereby maximally ensuring the character type characteristics of each field and avoiding distortion; then, URLs with the same number of segments and the same basic pattern at the same segment position are obtained from the segmented URL to form a URL group, thereby obtaining different URLs of the same category. This method can separate multiple different types of URL groups to ensure wider applicability. Finally, the fields at the same segment position in the URL group are merged, and the merging is performed based on the marked basic pattern. The final URL pattern can distinguish different basic patterns or different fields, and the obtained result is more accurate, thereby solving the problem that the URL merging processing result of the prior art is too generalized.

[0077] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0079] Figure 1 A method flow chart of a URL pattern acquisition method provided in the first embodiment of the present invention.

[0080] Figure 2 for Figure 1 Flow chart of the sub-step method of step S30.

[0081] Figure 3 A method flow chart of a URL pattern acquisition method provided in the second embodiment of the present invention.

[0082] Figure 4 A method flow chart of a URL pattern acquisition method provided in the third embodiment of the present invention.

[0083] Figure 5 A functional module diagram of a URL pattern acquisition device provided in the fourth embodiment of the present invention.

[0084] Figure 6 A structural block diagram of a URL pattern acquisition device provided for a specific example of the present invention.

[0085] Figure 7 A schematic diagram of the structure of a server provided for another specific example of the present invention. DETAILED DESCRIPTION

[0086] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0087] The invention provides a URL pattern acquisition method, which is used for processing URL data and obtaining a URL pattern with high accuracy.

[0088] First embodiment

[0089] See also Figure 1 In this embodiment, a method for obtaining a URL pattern is provided. Figure 1 A method flow chart of the method is shown, and the method comprises the following steps:

[0090] Step S10: segment each URL in the URL set respectively, and mark the fields generated after segmenting each URL with a basic pattern according to the character type, wherein the basic pattern is formed by any characters of the same character type as the marked field.

[0091] Step S20: According to the fields marked with the basic pattern, URLs with the same number of segments and the same basic pattern at the same segment position in the URL set are obtained to form a URL group.

[0092] Step S30: Merge the fields at the same segment position in the URL group according to the marked basic pattern to obtain the URL pattern of the URL group.

[0093] In step S10, each URL in the URL set is segmented, that is, the URL is divided into multiple different parts according to the characteristics of the URL. The characteristics of the URL can be: the composition rules of the URL, the character types of the URL, the positions of special characters in the URL, etc. For example, segmentation is performed according to the composition rules of the URL, specifically:

[0094] According to the composition rules of URL, the URL can be divided into 5 parts: <scheme> : / / <netloc> / <path> ？ <query> # <fragement>.

[0095] in:

[0096] scheme: protocol, such as http, https, ftp, etc.

[0097] netloc: domain name, independent identifier of different sites, such as www.sohu.com, www.sina.com.cn, etc.

[0098] path: Path, which identifies the virtual address of a file under the server root path. It may have multiple levels, such as / a / b / c.html, / a / b / d.shtml, / a / b / e.jsp (page formats include: .htm, dhtml, .xhtml, .shtm, .shtml, .ASP, .JSP, .PHP, .ASPX, .jspx, .cgi), etc.

[0099] query: parameter, consisting of multiple groups of (key, value). The key and value in each group can be connected with an equal sign (=), and different groups of (key, value) can be connected with '&', such as key1=value1&key2=value2.

[0100] fragment: Anchor, located at a specific position in the page (the positioning part described later).

[0101] For example: URL

[0102] http: / / www.example.com / an / example.html?lang=zh&loc=bj#nowhere

[0103] The parts of the URL are:

[0104] scheme: http;

[0105] netloc: www.example.com;

[0106] path: / an / example.html;

[0107] query:lang=zh&loc=bj;

[0108] fragment: nowhere.

[0109] Step S10 may include the following operation methods when performing URL segmentation:

[0110] The present embodiment provides a segmentation method that can directly extract the character content of each component from the URL according to the composition rule of the URL, and then segment the extracted character content of each component. The composition rule of the URL can be: the URL includes a scheme part, a netloc part, a path part, a query part, and a fragmentation part.

[0111] The above-mentioned segmentation of the character content of each component extracted may include: the path part may be segmented according to " / " and the character type; the query part may be segmented according to "&". Among them, in the query part, it can be segmented according to "&" to form several "key=value" forms, and the value part after "=" is retained as the field of this part, and the key part before "=" is used as an identification feature ("key=" can also be used as an identification feature together). It should be noted that the key part can be used as an identification feature of URL grouping in step S20 to improve the accuracy of URL group classification; the value part here can be further segmented according to the character type, and the basic mode marking is performed for the segmented fields. For example: the query part is k=v1, "k=" is used as the identification feature, and v1 is segmented into v and 1.

[0112] After the character contents of each extracted component are segmented, in the segments corresponding to the character contents of any component in the scheme part, the netloc part, the path part, the query part, and the fragment part, if there are two or more adjacent segments with the same character type, the adjacent segments with the same character type can be connected into one segment, thereby reducing the number of fields to be subsequently merged.

[0113] For example, if the path part is "a / b / 123.html", the three fields a, b, and 123.html are obtained after segmentation according to " / ". The adjacent fields a and b have the same character type. Then the two fields a and b are connected to finally obtain the two fields ab and 123.html.

[0114] In order to reduce redundant information during specific implementation, the connectors such as ":", " / / ", " / " "?", "#" in the URL can be discarded or filtered after segmentation, that is, when segmenting, only the character content corresponding to each component in the scheme part, netloc part, path part, query part, and fragment part needs to be extracted as the field after segmentation. In this embodiment, the scheme part and the netloc part can also be discarded, so that the segment length of the URL is further simplified while retaining the characteristics of the URL type, and the processing scope of the URL is expanded, so that more complex URLs can be processed.

[0115] This embodiment also provides a second URL segmentation method. Different from the first segmentation method, the character content corresponding to each component in the path part, query part, and fragment part of the URL can be directly extracted, and then the character content corresponding to the three components is segmented according to the character type. For example:

[0116] In URL1:http: / / example.com / test01.html?k=v1#abc, first extract the character contents corresponding to the path part, query part, and fragment part, for example, the path part is test01.html; then segment the character contents corresponding to each component according to the character type, for example, the path part is test01.html, which is segmented into test, 01, ., and html according to the character type; the query part k=v1 is segmented into k, =, v, and 1 according to the character type.

[0117] In step S10, the fields generated after segmentation are marked with a basic mode according to the character type, so that a basic mode can be used to represent multiple fields with different contents but the same character type. The character type can specifically refer to letters, numbers, symbols, etc., wherein letters can be divided into uppercase letters and lowercase letters, and symbols can be divided into standard characters (hereinafter ". " is a standard character) and characters other than standard characters.

[0118] Standard characters include:

[0119] "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~: / ?#[]@!$&'()*+,;=%<>\"{}^|".

[0120] The basic pattern is formed by any characters of the same character type as the marked field. A basic pattern may include one or more characters of the same type and other special symbols. For example, the basic pattern corresponding to the field composed of lowercase letters may be a+z, a*z, a&z, az, az, bz, bz, a / z, [az], [a / z], etc.; the basic pattern corresponding to the field composed of uppercase letters may be AZ, A+Z, A*Z, AZ, BZ, (AZ), A / Z, [AZ], etc.; the basic pattern corresponding to the field composed of numbers may be 0-9, 0+9, 0*9, 0&9, 09, 19, 29, [0-9],

[09] , etc. Other characters except letters and numbers can directly use the characters themselves as their basic patterns because they have certain special meanings or certain recognition, such as ". ", "-", and the marking form can be " / character", "\character", "character", etc.

[0121] It should be noted that, in general, the above step S10 can also be equivalent to: when segmenting the URL, firstly, each character in the path part, query part and fragment part of the URL is marked with a basic pattern, and the character content of other components in the URL except the above three parts and " / " are discarded, and then the adjacent characters with the same basic pattern in the above three parts are merged and segmented to obtain various fields, and each basic pattern corresponds to one field.

[0122] Take the example of the path part "a / b / 123.html" as follows: after discarding " / " from "a / b / 123.html", we get "ab123.html". After marking each character with the basic pattern, it is: [az][az][0-9][0-9][0-9][.][az][az][az][az], and the adjacent basic patterns with the same character type are merged to get [az], [0-9][.][az].

[0123] Step S20: According to the fields marked with the basic pattern, URLs with the same number of segments and the same basic pattern at the same segment position in the URL set are obtained to form a URL group.

[0124] In addition, when there are URLs containing a query part in the URL set, in order to improve the accuracy of classification, the key part included in the query part can be introduced as an identification feature in step S20. That is, according to the field marked with the basic pattern, the URLs with the same number of segments, the same basic pattern at the same segment position, and the same key part at the same segment position in the URL set are obtained to form a URL group.

[0125] After step S20, the URLs in the URL set that have been segmented can be filtered out to form a URL group. For example, URLs with the same segment length and the same field character type at the same segment position can be filtered out, and the filtered URLs can be grouped together. For example:

[0126] URL1: http: / / example.com / 123 / test01.html? k=1#abc

[0127] URL2: http: / / example.com / 456 / test02.html? k=2#abc

[0128] URL3: http: / / example.com / 789 / test03.html? k=3#abc

[0129] Then in the URL set, URL1, URL2, and URL3 can form a group after segmentation, and their segmentation information is shown in Table 1:

[0130] Table 1

[0131] Segment position 1 Segment position 2 Segment position 3 Segment position 4 Segment position 5 Segment position 6 Segment position 7 123 test 01 . html 1 abc 456 test 02 . html 2 abc 789 test 03 . html 3 abc

[0132] Another example:

[0133] URL4: http: / / bbs.qyer.com / thread-1086156-1.html

[0134] URL5: https: / / bbs.niuyou5.com / thread-1728466-1.html

[0135] URL6: http: / / bbs.letv.com / thread-971782-1.html

[0136] Then in the URL set, URL4, URL5, and URL6 can form a group after segmentation, and their segmentation information is shown in Table 2:

[0137] Table 2

[0138] Segment position 1 Segment position 2 Segment position 3 Segment position 4 Segment position 5 Segment position 6 Segment position 7 thread - 1086156 - 1 . html thread - 1728466 - 1 . html thread - 971782 - 1 . html

[0139] Step S30: Merge the fields at the same segment position in the URL group according to the marked basic pattern to obtain the URL pattern of the URL group.

[0140] In step S30, the merging may include using a unified form to represent the fields at the same segment position. Further, the fields at the same segment position in the URL group may be merged according to the marked basic pattern and the number of characters in each field to obtain the URL pattern of the URL group.

[0141] In order to further prevent an overly generalized URL pattern, the number of fields merged at the same segment position may be marked (recorded separately) during merging. The mark is used to indicate how many URLs the URL pattern is merged from.

[0142] When executing the above step S30, due to the differences in the types of fields at various segment positions, when performing a merging operation on various fields at a certain segment position in a URL group, different merging operations can be performed based on the basic mode of the field at the segment position and the number of characters in the field. Therefore, in this embodiment, separate merging operations can be performed on the fields at different segment positions in the same URL group.

[0143] Specifically, Figure 2 As shown, step S30 includes the following sub-steps:

[0144] Step S31: Obtain the pattern string corresponding to each segment position in the URL group; the pattern string is: a string obtained by using a unified expression form after performing a merging operation on each field corresponding to each URL in the URL group at the same segment position.

[0145] Step S32: combining the pattern strings corresponding to the segment positions in the URL group to obtain the URL pattern of the URL group.

[0146] The operation of combining the pattern strings may include: connecting the path part, the query part and the fragment part by using a connector.

[0147] When executing step S31, a specific implementation method may be: merging the target fields of each URL in the URL group at the same segment position and with the same field content; wherein the character type of the target field may be letters (such as segment position 2 in Table 1) or numbers (such as segment position 5 in Table 2); after the merging is completed, the character content included in the target field (the characters constituting the target field) is used as the pattern string of the segment position.

[0148] When executing step S31, another specific implementation method can be: merging multiple target fields having the same characters at the same segment position of each URL in the URL group, and the same characters included in the target field are non-alphabetic and non-numeric types (for example, segment position 4 in Table 1); after the merging operation is completed, the pattern string of the segment position is obtained according to the character content and number of the same characters included in the target field.

[0149] Among them, the expression form of the pattern string of the segment position can be specifically subdivided according to whether the characters included in the target field are special characters used in the regular expression; among them, the special characters used in the regular expression can be such as: ()$*.[], etc., so as to further strengthen the structural characteristics of the merged URL pattern. Specifically, if the characters of the target field are not special characters used in the regular expression, the pattern string can be expressed as: [characters]{number} or (characters){number}, etc.; if the characters of the target field are special characters used in the regular expression, the pattern string can be expressed as [\characters]{number}, or (\characters){number}, etc.; for example, the segment position 4 in Table 1 can be expressed as [\.]{1}; wherein, the characters are the character content of the target field; the number is the number of characters in the target field.

[0150] Among them, symbols such as "[]", "()", etc. are only used to distinguish the pattern strings of each segment position, and do not limit the protection scope of the present invention. Other symbols may be used for distinction, such as "{}", "<>", """, etc. The "\" in [\ character] may also be replaced by other symbols that can be used for distinction (such as " / ", "|", etc.), the same below. In this embodiment, [character]{number} is used, wherein {number} can be omitted when the number is 1, and the number represents the number of characters in a field in the segment position. In this embodiment, [\ character]{number} or [character]{number} is used, wherein {number} can be omitted when the number is 1, for example, segment position 4 in Table 1 can be directly represented as [\.].

[0151] When executing step S31, another specific implementation method can be: merge the target fields (such as segment position 1 and segment position 3 in Table 1) at the same segment position of each URL in the URL group, which have the same basic pattern of marking and different characters; after the merging is completed, obtain the pattern string of the segment position according to the basic pattern of the target field marking and the field length of the fields in each URL at the segment position before the merging.

[0152] In this embodiment, if the field lengths are the same (the field length represents the number of characters, and in this embodiment, the field length is the number of characters), the pattern string of the segment position can be expressed as: "[basic pattern]{length}", and if the character length is 1, it can be expressed as "[basic pattern]"; if the field lengths are different, the pattern string of the segment position can be expressed as: "[basic pattern]+". When the field length is 1, {length} can be omitted, and "+" is used to represent the situation where the field lengths of the merged segment positions are different. Other symbols can also be used to distinguish them, such as -, *, %, !, ~, etc.

[0153] Furthermore, in order to improve the compatibility of the system and reduce the learning cost of implementing the present invention, the fields marked by the basic pattern in the above step S31 and the pattern string formed after merging can be expressed by regular expressions, for example: [AZ], [az], [0-9], [character], {length}, etc. In addition, abbreviations can be replaced in regular expressions, and the abbreviation replacement of regular expressions is a known technical means, which will not be described in detail here.

[0154] Please refer to Table 3. To make the solution in this embodiment clearer, the following example is used to illustrate the following URL set:

[0155] http: / / example.com / 11.html?k=1#abc

[0156] http: / / example.com / 12.html?k=2#abc

[0157] http: / / example.com / 13.html?k=3#abc

[0158] http: / / bbs.qyer.com / thread-1086-1.html

[0159] http: / / bbs.letv.com / thread-982-1.html

[0160] The above URL is segmented and marked in basic mode in step S10, and redundant information is discarded to obtain Table 3:

[0161] Table 3

[0162]

[0163] Step S20 is continued for Table 3 to obtain URL group 1 and URL group 2, as shown in Table 4:

[0164] Table 4

[0165]

[0166] Continuing to take URL group 1 in Table 4 as an example, step S30 is executed for URL group 1 in Table 4. When step S30 is executed, steps S31 and S32 are executed respectively. Table 5 is obtained:

[0167] Table 5

[0168]

[0169] Combining the pattern strings formed by the merged segment positions can produce the URL pattern: [0-9]{2}[\.]html[0-9]abc.

[0170] If we record the number of URLs, we get: [0-9]{2}[\.]html[0-9]abc(3).

[0171] Further, in order to enhance the recognition of the URL pattern structure, a preset connector may be used to connect the path part, the query part, and the fragment part. For example, in this embodiment, the preset connector may include: using " / " as the start connector, using "[\?]" as the connector between the path part and the query part, and using "#" as the connector between the query part and the fragment part, to obtain the URL pattern: / [0-9]{2}[\.]html[\? ][0-9]#abc.

[0172] Furthermore, in order to ensure the recognition of the URL pattern formed after the combination of each pattern string, the identification feature (key part, where the key part is k) retained during segmentation can be added to the specified position, and the URL pattern can be obtained: / [0-9]{2}[\.]html[\? ]k=[0-9]#abc

[0173] Second embodiment

[0174] See also Figure 3 In this embodiment, a method for obtaining a URL pattern is also provided, and the method specifically includes:

[0175] Step S10: segment each URL in the URL set respectively, and mark the fields generated after segmenting each URL with a basic pattern according to the character type, wherein the basic pattern is formed by any characters of the same character type as the marked field.

[0176] Step S20: According to the fields marked with the basic pattern, URLs with the same number of segments and the same basic pattern at the same segment position in the URL set are obtained to form a URL group.

[0177] Step S301: construct a prefix tree according to each field obtained after segmenting each URL in the URL group; wherein each branch of the prefix tree is a segmented URL, each node of each branch of the prefix tree is a field obtained by segmentation, nodes of the same depth in the prefix tree constitute a layer of the prefix tree, and each layer in the prefix tree corresponds to a same segmentation position.

[0178] Step S302: Merge the prefix trees from the root node downwards, and obtain the URL pattern of the URL group according to the basic pattern and the number of characters of the fields included in each layer of the prefix tree.

[0179] The above-mentioned step S10 and step S20 are performed with reference to the first embodiment and will not be described in detail herein.

[0180] Different from the first embodiment, in this embodiment, step S301 needs to be executed to construct a prefix tree, and when constructing the prefix tree, an empty node is used as the root node.

[0181] When executing step S302, the prefix tree is constructed and merged from the root node downward in sequence. If an error occurs during the merging process, the operation can be stopped at the current node to avoid other nodes from continuing to operate and wasting computing resources. It should be noted that the merging operation of each layer of the prefix tree in step S301 in this embodiment can be performed with reference to the merging method of step S30 of the first embodiment.

[0182] The nodes of each layer of the prefix tree are merged (each layer of the prefix tree corresponds to a segment position in the URL group) to obtain the pattern string of the layer. The specific process can be explained by the following example, as shown below (the number in the brackets is the number of nodes):

[0183] URL Group:

[0184] http: / / example.com / 11.html?k=1#abc

[0185] http: / / example.com / 12.html?k=2#abc

[0186] http: / / example.com / 13.html?k=3#abc

[0187] After segmentation, the prefix tree is constructed as follows, with the number of nodes in parentheses:

[0188] / 11(1)[\.]html(1)1(1)abc(1)

[0189] -(3)|—12(1)[\.]html(1)2(1)abc(1)

[0190] \13(1)[\.]html(1)3(1)abc(1)

[0191] Merge the first layer to get the pattern string:

[0192] / [\.]html(1)1(1)abc(1)

[0193] -(3)[0-9]{2}(3)|—[\.]html(1)2(1)abc(1)

[0194] \[\.]html(1)3(1)abc(1)

[0195] Merge the second layer to get the pattern string:

[0196] / html(1)1(1)abc(1)

[0197] -(3)[0-9]{2}(3)[\.]|—html(1)2(1)abc(1)

[0198] \html(1)3(1)abc(1)

[0199] Merge the third layer to get the pattern string:

[0200] / html(1)1(1)abc(1)

[0201] -(3)[0-9]{2}(3)[\.](3)|—html(1)2(1)abc(1)

[0202] \html(1)3(1)abc(1)

[0203] Merge the fourth layer to get the pattern string:

[0204] / 1(1)abc(1)

[0205] -(3)[0-9]{2}(3)[\.](3)html(3)|—2(1)abc(1)

[0206] \3(1)abc(1)

[0207] Merge the fifth layer to get the pattern string:

[0208] / abc(1)

[0209] -(3)[0-9]{2}(3)[\.](3)html(3)[0-9(3)]|—abc(1)

[0210] \abc(1)

[0211] Merge the sixth layer to get the pattern string:

[0212] -(3)[0-9]{3}(3)[\.](3)html(3)[0-9(3)]abc(3)

[0213] Use the connector to combine the pattern strings to get the URL pattern:

[0214] / [0-9]{2}[\.]html[\?]k=[0-9]#abc

[0215] Third embodiment

[0216] Please refer to Figure 4 In this embodiment, a URL pattern acquisition method is provided that can retain the character content in a field at a specified segment position of a URL and use the character content of the field as a pattern string for the segment position where the field is located.

[0217] Different from the first embodiment, in this embodiment, fields with higher recognition can be retained according to the structural characteristics of the URL (the specified segment position can be the character content of the field with higher recognition in the URL, such as the page format suffix in the URL), and the character content of the field is used as the pattern string of the segment position where the field is located to further improve the accuracy of the URL pattern.

[0218] The specific steps include:

[0219] Step S10: segment each URL in the URL set respectively, and mark the fields generated after segmenting each URL with a basic pattern according to the character type, wherein the basic pattern is formed by any characters of the same character type as the marked field.

[0220] Step S20: According to the fields marked with the basic pattern, URLs with the same number of segments and the same basic pattern at the same segment position in the URL set are obtained to form a URL group.

[0221] Step S401: Obtain the pattern string of the designated segment position and the pattern string of the non-designated segment position in the URL group.

[0222] Step S402: Combine the pattern strings corresponding to the segment positions in the URL group to obtain the URL pattern of the URL group.

[0223] In step S401, the pattern string of the non-specified position may be obtained by referring to the method in the first embodiment, which will not be described in detail here.

[0224] Step S402 may be performed with reference to step S30 in the first embodiment, and will not be described in detail.

[0225] In step S401, the method for obtaining the pattern string of the specified position is as follows:

[0226] The segment position of the predefined character in the URL group is obtained, and the same character content contained in the field after the segment position of the predefined character is used as the pattern string of the segment position where the field is located.

[0227] In this way, the character content of the URL with identification function is retained as a pattern string to improve the accuracy of the URL pattern. In order to ensure that the retained character content has a high identification function, a first threshold can be set, and the first threshold is used to limit the number of characters in a field at a certain segment position. The value of the first threshold can be set according to the segment position in the URL where the field to be retained exists, and generally does not exceed 6 characters (that is, the value can be 1, 2, 3, 4, 5, 6). The number of characters in the field after the segment position where the predefined character is located can be limited to be less than or equal to the first threshold; in any of the path part, the query part, and the fragement part, the number of fields after the predefined character is less than or equal to the second threshold, and the second threshold can be 1 or 2. If the above-mentioned first threshold and / or second threshold requirements are met, the character content of the field at the specified position can be retained as the pattern string of the segment position where the specified position is located. Before this, it can be further determined that the fields after the predefined character are all marked with a basic pattern to avoid the predefined character being the end character of the URL. In this way, the designated field of the designated segment position can be located and retained to improve the representation accuracy of the URL pattern with the structural feature of the enhanced URL. Of course, the number of fields and the number of characters can only limit one of the first threshold or the second threshold as required.

[0228] It should be noted that the first threshold and the second threshold can be adjusted according to the situation. For example, when the character content of the field at other segment positions is to be retained as the pattern string of the segment position, the first threshold can be changed and limited accordingly according to the number of characters in the field at the segment position, and the second threshold can be changed and limited according to the number of fields at adjacent positions in the path part, query part, or fragement part of the segment position. For example, the first threshold can be a value greater than 6, such as 7, 8, 9, etc.; the second threshold can be a value greater than 2, such as 3, 4, 5, 6, 7, etc.

[0229] For example, if the page format information in the URL group needs to be retained as a pattern string, the predefined characters can be ". ". The suffix of the page format generally does not exceed 6 characters. When looking for the ". " character, the search range can be located in the path part. In the path part, the number of fields after ". " is less than two, and the fields after ". " are all marked with basic patterns. Then, it can be determined that the field after ". " in the path part is the suffix of the page format, and it is retained, and the character content of the retained field is used as the pattern string of the segment position. The accuracy of the URL pattern is improved. Similarly, if other identification information needs to be retained, the predefined characters can be adjusted to "=", "?", "#", etc., without limitation.

[0230] For example, the URL group:

[0231] URL1: http: / / example.com / 123 / test01.html? k=1#abc

[0232] URL2: http: / / example.com / 456 / test02.html? k=2#abc

[0233] URL3: http: / / example.com / 789 / test03.html? k=3#abc

[0234] The fields after the path part is segmented are:

[0235] test, 01, ., html

[0236] test, 02, ., html

[0237] test, 03, ., html

[0238] The predefined character is ". ". When merging, locate the position of ". ". The field after the character ". " is 1 (i.e. html, less than 2), and the number of characters is 4 (less than 6). When merging, html can be retained as the pattern string of the segment position. After the path part is merged, the result is: test[0-9]{2}[\.]html. This further improves the accuracy of the URL pattern.

[0239] Fourth embodiment

[0240] See also Figure 5 Based on the same inventive concept, this embodiment provides a URL pattern acquisition device, which includes: a marking module 11, a URL group generating module 12 and a URL pattern generating module 13.

[0241] The marking module 11 is used to segment each URL in the URL set, and mark the fields generated after each URL segmentation according to the basic pattern according to the character type, wherein the basic pattern is formed by any characters of the same character type as the marked field.

[0242] The URL group generating module 12 is used to obtain URLs with the same number of segments and the same basic pattern at the same segment position in the URL set according to the fields marked with the basic pattern, so as to form a URL group.

[0243] The URL pattern generating module 13 is used to merge the fields at the same segment position in the URL group according to the marked basic pattern to obtain the URL pattern of the URL group.

[0244] Optionally, the URL pattern generating module 13 is specifically configured to merge fields at the same segment position in the URL group according to the marked basic pattern and the number of characters in each field to obtain the URL pattern of the URL group.

[0245] Optionally, the URL pattern generating module 13 includes:

[0246] A construction unit is used to construct various fields obtained after segmenting each URL in the URL group into a prefix tree; wherein each branch of the prefix tree is a segmented URL, each node of each branch of the prefix tree is a field obtained by segmentation, and each layer in the prefix tree corresponds to a same segmentation position.

[0247] The first URL pattern generating unit is used to merge the prefix trees from the root node downwards, and obtain the URL pattern of the URL group according to the basic pattern and the number of characters of the fields included in each layer of the prefix tree.

[0248] Optionally, the URL pattern generation module 13 includes:

[0249] The pattern string generating unit is used to obtain the pattern string corresponding to each segment position in the URL group; the pattern string is a string obtained by using a unified expression form after performing a merging operation on each field corresponding to each URL in the URL group at the same segment position.

[0250] The second URL pattern generating unit is used to combine the pattern strings corresponding to the segment positions in the URL group to obtain the URL pattern of the URL group.

[0251] Optionally, the pattern string generating unit includes:

[0252] The first merging subunit is used to merge target fields of the URLs in the URL group that are in the same segment position and have the same field content, where the character type of the target field is letters or numbers.

[0253] It is also used to, after merging, use the character content of the target field as the pattern string of the segment position.

[0254] Optionally, the pattern string generating unit includes:

[0255] The second merging subunit is used to merge target fields having the same characters at the same segment position of each URL in the URL group, wherein the same characters included in the target fields are non-alphabetic and non-numeric types.

[0256] It is also used to obtain the pattern string of the segment position according to the character content and the number of characters of the same characters included in the target field after merging.

[0257] Optionally, the pattern string generating unit includes:

[0258] The third merging subunit is used to merge the target fields of the URLs in the URL group at the same segmentation position, with the same basic pattern of the mark and different characters;

[0259] It is also used to obtain the pattern string of the segment position according to the basic pattern of the target field tag and the field length of the fields in each URL of the segment position before merging after merging.

[0260] Optionally, the pattern string generating unit includes:

[0261] The fourth merging subunit is used to obtain the segment position of the predefined character in the URL group, and use the same character content contained in the field after the segment position of the predefined character as the pattern string of the segment position where the field is located.

[0262] Optionally, the number of characters in the field after the segment position of the predefined character is less than or equal to a preset first threshold; and / or in any part of the path part, parameter part and location part of the URL, the number of fields after the predefined character is less than or equal to a preset second threshold; and / or in any part of the path part, parameter part and location part of the URL, the fields after the predefined character are all marked with a basic pattern.

[0263] Optionally, the URL group generation module 12 is specifically used to obtain URLs with the same number of segments and the same basic pattern at the same segment position in the URL set according to the fields after the basic pattern marking; and to treat the URLs with the same key field in the parameter part of the obtained URL as a URL group.

[0264] Fifth embodiment

[0265] This embodiment also provides an electronic device to which the above-mentioned URL pattern acquisition method and device can be applied.

[0266] The electronic device includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing the following operations:

[0267] Each URL in the URL set is segmented respectively, and the fields generated after each URL segmentation are marked with a basic pattern according to the character type, where the basic pattern is formed by any characters of the same character type as the marked field.

[0268] According to the fields marked with the basic pattern, URLs with the same number of segments and the same basic pattern at the same segment position in the URL set are obtained to form a URL group;

[0269] The fields at the same segment position in the URL group are merged according to the marked basic pattern to obtain the URL pattern of the URL group.

[0270] For other operation instructions executable in the electronic device, specific reference may be made to the methods in the first to fifth embodiments, or the executable operations of each module and unit in the device, which will not be described in detail here.

[0271] Figure 6 800 is a structural block diagram of a URL pattern acquisition device 800 shown in a specific example of the present invention. For example, the device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0272] Reference Figure 6 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0273] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0274] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0275] The power component 806 provides power to the various components of the device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0276] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user.

[0277] The touch panel includes one or more touch sensors to sense touch, slide and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0278] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0279] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0280] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0281] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0282] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0283] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0284] A readable storage medium, specifically a non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to execute the URL pattern acquisition method in the above embodiment, the method comprising:

[0285] Each URL in the URL set is segmented separately, and the fields generated after each URL segmentation are marked with a basic pattern according to the character type, where the basic pattern is formed by any characters of the same character type as the marked field; based on the fields marked with the basic pattern, URLs with the same number of segments and the same basic pattern at the same segment position in the URL set are obtained to form a URL group; the fields at the same segment position in the URL group are merged according to the marked basic pattern to obtain the URL pattern of the URL group.

[0286] Figure 7 1 is a schematic diagram of the structure of the server in an embodiment of the present invention. The server 1900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1922 (for example, one or more processors) and memory 1932, and one or more storage media 1930 (for example, one or more mass storage devices) storing application programs 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 can be short-term storage or permanent storage. The program stored in the storage medium 1930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 1922 can be configured to communicate with the storage medium 1930 and execute a series of instruction operations in the storage medium 1930 on the server 1900.

[0287] The server 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input and output interfaces 1958, one or more keyboards 1956, and / or, one or more operating systems 1941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0288] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0289] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the present invention.< / fragement> < / query> < / path> < / netloc> < / scheme> < / fragement> < / query> < / path> < / netloc> < / scheme>

Claims

1. A method for obtaining a URL pattern, characterized in that: include: For each URL in the URL set, according to the composition rules of the URL, the character content of each component is extracted from the URL, and then the character content of each component extracted is segmented; Alternatively, the character content of each component is directly extracted from the URL, and then the character content corresponding to each extracted component is segmented according to the character type; Marking each field generated after the URL segmentation with a basic pattern according to the character type, wherein the basic pattern is formed by any characters of the same character type as the marked field; According to the fields marked with the basic pattern, URLs with the same number of segments and the same basic pattern at the same segment position in the URL set are obtained to form a URL group; The fields at the same segment position in the URL group are merged in different ways according to the marked basic mode and the number of characters of each field to obtain the URL mode of the URL group, wherein the merging is to use a unified form to represent the fields at the same segment position; The step of merging the fields at the same segment position in the URL group according to the marked basic pattern and the number of characters in each field to obtain the URL pattern of the URL group includes: A prefix tree is constructed according to each field obtained after segmenting each URL in the URL group; wherein each branch of the prefix tree is a segmented URL, each node of each branch of the prefix tree is a field obtained by segmentation, nodes of the same depth in the prefix tree constitute a layer of the prefix tree, and each layer in the prefix tree corresponds to a same segmentation position; The prefix trees are merged from the root node downwards, and the URL pattern of the URL group is obtained according to the basic pattern and the number of characters of the fields included in each layer of the prefix tree.

2. The method according to claim 1, characterized in that The step of merging the fields at the same segment position in the URL group according to the marked basic pattern to obtain the URL pattern of the URL group includes: Obtaining a pattern string corresponding to each segment position in the URL group; the pattern string is a string obtained by using a unified expression form after performing a merging operation on each field corresponding to each URL in the URL group at the same segment position; The pattern strings corresponding to the respective segment positions in the URL group are combined to obtain the URL pattern of the URL group.

3. The method according to claim 2, characterized in that The step of obtaining the pattern string corresponding to each segment position in the URL group includes: Merge target fields of the URLs in the URL group that are in the same segment position and have the same field content, where the character type of the target field is letters or numbers; After merging, the character content of the target field is used as the pattern string of the segment position.

4. The method according to claim 2, characterized in that: The step of obtaining the pattern string corresponding to each segment position in the URL group includes: Merging target fields having the same characters at the same segment position of each URL in the URL group, wherein the same characters included in the target fields are non-alphabetic and non-numeric types; After merging, the pattern string of the segment position is obtained according to the character content and the number of characters of the same characters included in the target field.

5. The method according to claim 2, characterized in that: The step of obtaining the pattern string corresponding to each segment position in the URL group includes: Merge the target fields of each URL in the URL group at the same segment position, with the same basic pattern of the mark and different characters; After merging, the pattern string of the segment position is obtained according to the basic pattern of the target field tag and the field length of the fields in each URL of the segment position before merging.

6. The method according to claim 2, characterized in that The step of obtaining the pattern string corresponding to each segment position in the URL group includes: The segment position of the predefined character in the URL group is obtained, and the same character content contained in the field after the segment position of the predefined character is used as the pattern string of the segment position where the field is located.

7. The method according to claim 6, characterized in that The number of characters in the field after the segment position of the predefined character is less than or equal to a preset first threshold; and / or, In any of the path part, the parameter part and the location part of the URL, the number of fields after the predefined character is less than or equal to a preset second threshold; and / or, In any of the path part, parameter part and location part of the URL, the fields after the predefined characters are marked with the basic pattern.

8. The method according to claim 1, characterized in that: The step of obtaining URLs having the same number of segments and the same basic pattern at the same segment position in the URL set according to the fields marked with the basic pattern to form a URL group includes: According to the fields marked with the basic pattern, obtaining URLs in the URL set that have the same number of segments and the same basic pattern at the same segment position; The URLs with the same key field in the parameter part of the obtained URL are regarded as a URL group.

9. A URL pattern acquisition device, characterized in that: include: The marking module is used to extract the character content of each component from each URL in the URL set according to the composition rules of the URL, and then segment the extracted character content of each component; Alternatively, the character content of each component is directly extracted from the URL, and then the character content corresponding to each extracted component is segmented according to the character type; Marking each field generated after the URL segmentation with a basic pattern according to the character type, wherein the basic pattern is formed by any characters of the same character type as the marked field; A URL group generation module, used to obtain URLs with the same number of segments and the same basic pattern at the same segment position in the URL set according to the fields marked with the basic pattern, to form a URL group; A URL pattern generation module, used for merging the fields at the same segment position in the URL group in different ways according to the marked basic pattern and the number of characters of each field to obtain the URL pattern of the URL group, wherein the merging is to use a unified form to represent the fields at the same segment position; The URL pattern generation module includes: A construction unit, used to construct a prefix tree from each field obtained after segmenting each URL in the URL group; wherein each branch of the prefix tree is a segmented URL, each node of each branch of the prefix tree is a field obtained by segmentation, nodes of the same depth in the prefix tree constitute a layer of the prefix tree, and each layer in the prefix tree corresponds to a same segmentation position; The first URL pattern generating unit is used to merge the prefix trees from the root node downwards, and obtain the URL pattern of the URL group according to the basic pattern and the number of characters of the fields included in each layer of the prefix tree.

10. The device according to claim 9, characterized in that URL pattern generation module, including: A pattern string generating unit, used to obtain a pattern string corresponding to each segment position in the URL group; the pattern string is a string obtained by performing a merging operation on each field corresponding to each URL in the URL group at the same segment position in a unified expression form; The second URL pattern generating unit is used to combine the pattern strings corresponding to the segment positions in the URL group to obtain the URL pattern of the URL group.

11. The device according to claim 10, characterized in that The pattern string generating unit comprises: A first merging subunit is used to merge target fields of the URLs in the URL group that are in the same segment position and have the same field content, wherein the character type of the target field is letters or numbers; It is also used to, after merging, use the character content of the target field as the pattern string of the segment position.

12. The device according to claim 10, characterized in that The pattern string generating unit comprises: A second merging subunit is used to merge target fields having the same characters at the same segment position of each URL in the URL group, wherein the same characters included in the target fields are non-alphabetic and non-numeric types; It is also used to obtain the pattern string of the segment position according to the character content and the number of characters of the same characters included in the target field after merging.

13. The device according to claim 10, characterized in that The pattern string generating unit comprises: The third merging subunit is used to merge the target fields of the URLs in the URL group at the same segmentation position, with the same basic pattern of the mark and different characters; It is also used to obtain the pattern string of the segment position according to the basic pattern of the target field tag and the field length of the fields in each URL of the segment position before merging after merging.

14. The device according to claim 10, characterized in that The pattern string generating unit comprises: The fourth merging subunit is used to obtain the segment position of the predefined character in the URL group, and use the same character content contained in the field after the segment position of the predefined character as the pattern string of the segment position where the field is located.

15. The device according to claim 14, characterized in that The number of characters in the field after the segmentation position of the predefined character is less than or equal to a preset first threshold; and / or in any part of the path part, parameter part and location part of the URL, the number of fields after the predefined character is less than or equal to a preset second threshold; and / or In any of the path part, parameter part and location part of the URL, the fields after the predefined characters are marked with the basic pattern.

16. The device according to claim 9, characterized in that The URL group generation module is specifically used to obtain URLs with the same number of segments and the same basic pattern at the same segment position in the URL set according to the fields marked with the basic pattern; and to treat the URLs with the same key field in the parameter part of the obtained URL as a URL group.

17. An electronic device, characterized in that: The system comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the following operations: For each URL in the URL set, according to the composition rules of the URL, the character content of each component is extracted from the URL, and then the character content of each component extracted is segmented; Alternatively, the character content of each component is directly extracted from the URL, and then the character content corresponding to each extracted component is segmented according to the character type; Marking each field generated after the URL segmentation with a basic pattern according to the character type, wherein the basic pattern is formed by any characters of the same character type as the marked field; According to the fields marked with the basic pattern, URLs with the same number of segments and the same basic pattern at the same segment position in the URL set are obtained to form a URL group; The fields at the same segment position in the URL group are merged in different ways according to the marked basic mode and the number of characters in each field to obtain the URL mode of the URL group, including: merging the fields at the same segment position in the URL group according to the marked basic mode and the number of characters in each field to obtain the URL mode of the URL group; the merging is to use a unified form to represent the fields at the same segment position; The step of merging the fields at the same segment position in the URL group according to the marked basic pattern and the number of characters in each field to obtain the URL pattern of the URL group includes: A prefix tree is constructed according to each field obtained after segmenting each URL in the URL group; wherein each branch of the prefix tree is a segmented URL, each node of each branch of the prefix tree is a field obtained by segmentation, nodes of the same depth in the prefix tree constitute a layer of the prefix tree, and each layer in the prefix tree corresponds to a same segmentation position; The prefix trees are merged from the root node downwards, and the URL pattern of the URL group is obtained according to the basic pattern and the number of characters of the fields included in each layer of the prefix tree.

18. The electronic device according to claim 17, characterized in that: The processor executes the one or more programs including instructions for performing the following operations: Obtaining a pattern string corresponding to each segment position in the URL group; the pattern string is a string obtained by using a unified expression form after performing a merging operation on each field corresponding to each URL in the URL group at the same segment position; The pattern strings corresponding to the respective segment positions in the URL group are combined to obtain the URL pattern of the URL group.

19. The electronic device according to claim 18, characterized in that: The processor executes the one or more programs including instructions for performing the following operations: Merge target fields of the URLs in the URL group that are in the same segment position and have the same field content, where the character type of the target field is letters or numbers; After merging, the character content of the target field is used as the pattern string of the segment position.

20. The electronic device according to claim 18, characterized in that The processor executes the one or more programs including instructions for performing the following operations: Merging target fields having the same characters at the same segment position of each URL in the URL group, wherein the same characters included in the target fields are non-alphabetic and non-numeric types; After merging, the pattern string of the segment position is obtained according to the character content and the number of characters of the same characters included in the target field.

21. The electronic device according to claim 18, characterized in that: The processor executes the one or more programs including instructions for performing the following operations: Merge the target fields of each URL in the URL group at the same segment position, with the same basic pattern of the mark and different characters; After merging, the pattern string of the segment position is obtained according to the basic pattern of the target field tag and the field length of the fields in each URL of the segment position before merging.

22. The electronic device according to claim 18, characterized in that: The processor executes the one or more programs including instructions for performing the following operations: The segment position of the predefined character in the URL group is obtained, and the same character content contained in the field after the segment position of the predefined character is used as the pattern string of the segment position where the field is located.

23. The electronic device according to claim 22, characterized in that: The number of characters in the field after the segment position of the predefined character is less than or equal to a preset first threshold; and / or, In any of the path part, the parameter part and the location part of the URL, the number of fields after the predefined character is less than or equal to a preset second threshold; and / or, In any of the path part, parameter part and location part of the URL, the fields after the predefined characters are marked with the basic pattern.

24. The electronic device according to claim 17, characterized in that: The processor executes the one or more programs including instructions for performing the following operations: According to the fields marked with the basic pattern, obtaining URLs in the URL set that have the same number of segments and the same basic pattern at the same segment position; The URLs with the same key field in the parameter part of the obtained URL are regarded as a URL group.

25. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the URL pattern acquisition method as described in any one of claims 1-8 is implemented.

Images