Uniform resource locator duplicate removal method and device, electronic equipment and storage medium

Abstract

The invention provides a uniform resource locator deduplication method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of the Internet, and the method comprises the following steps: constructing a bitmap of a memory database; wherein first mapping values corresponding to a plurality of first hash results of the uniform resource locators are stored in the bitmap; obtaining a to-be-identified uniform resource locator, and determining a plurality of second hash results of the to-be-identified uniform resource locator; querying the bitmap according to the plurality of second hash results to obtain second mapping values corresponding to the plurality of second hash results; and determining the matching degree of the obtained second mapping value and the first mapping value, and judging whether the uniform resource locator to be identified is a repeated uniform resource locator or not according to the matching degree. According to the method and the device, in a cluster high-concurrency environment, the duplicate removal efficiency of a crawler on the URL can be improved, and the occupation of a central processing unit is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and in particular to a uniform resource locator deduplication method, device, electronic device and computer-readable storage medium. Background Art

[0002] The existing web crawler deduplication solution is to directly perform deduplication through Map storage in memory. In a single structure or low concurrency scenario, the deduplication efficiency is quite impressive. However, most of the actual customers are large enterprise users, who generally require cluster deployment. When enterprises perform large-scale Uniform Resource Locator (URL) scanning, the real-time concurrency is quite high. If Map storage is still used to implement URL deduplication, the CPU memory usage will be too high.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The present disclosure provides a uniform resource locator deduplication method, device, electronic device and computer-readable storage medium, which can reduce the CPU memory usage during URL deduplication at least to a certain extent.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to one aspect of the present disclosure, a method for removing duplicate uniform resource locators is provided, and the method for removing duplicate uniform resource locators comprises:

[0007] Constructing a bitmap of the memory database; wherein the bitmap stores first mapping values ​​corresponding to each of a plurality of first hash results of the uniform resource locator;

[0008] Obtaining a uniform resource locator to be identified, and determining a plurality of second hash results of the uniform resource locator to be identified;

[0009] Query the bitmap according to the plurality of second hash results to obtain second mapping values ​​corresponding to the plurality of second hash results;

[0010] The matching degree between the obtained second mapping value and the obtained first mapping value is determined, and whether the to-be-identified uniform resource locator is a repeated uniform resource locator is determined according to the matching degree.

[0011] Optionally, the step of constructing the bitmap of the in-memory database includes: using different preset hash algorithms to hash the acquired uniform resource locators respectively to obtain multiple first hash results of the uniform resource locators; setting a corresponding first mapping value for each of the first hash results of the uniform resource locators; and updating the default numerical value corresponding to the spatial storage position in the bitmap to the first mapping value.

[0012] Optionally, the step of determining the multiple second hash results of the uniform resource locator to be identified includes: using different preset hash algorithms to respectively calculate the hash values ​​of the uniform resource locator to be identified to obtain the multiple second hash results of the uniform resource locator to be identified.

[0013] Optionally, the step of querying the bitmap according to the multiple second hash results to obtain the second mapping values ​​corresponding to each of the multiple second hash results includes: obtaining the first hash results corresponding to each of the multiple first mapping values ​​in the bitmap; when the second hash result is included in the first hash results corresponding to the multiple first mapping values, determining the first mapping value corresponding to the first hash result that is the same as the second hash result as the second mapping value corresponding to the second hash result; when the second hash result is not included in the first hash results corresponding to the multiple first mapping values, determining the default value of the spatial storage position in the bitmap as the second mapping value corresponding to the second hash result.

[0014] Optionally, the step of determining the degree of match between the second mapping value and the first mapping value includes: obtaining a first number of the first mapping value; determining a second number of the second mapping value that is the same as the first mapping value; and determining the degree of match based on a ratio of the second number to the first number.

[0015] Optionally, the step of judging whether the uniform resource locator to be identified is a duplicate uniform resource locator according to the matching degree includes: obtaining a matching degree threshold; when the matching degree is greater than or equal to the matching degree threshold, determining that the uniform resource locator to be identified is a duplicate uniform resource locator; when the matching degree is less than the matching degree threshold, determining that the uniform resource locator to be identified is not a duplicate uniform resource locator.

[0016] Optionally, after the step of determining that the above-mentioned uniform resource locator to be identified is not a duplicate uniform resource locator when the above-mentioned matching degree is less than the above-mentioned matching degree threshold, the above-mentioned uniform resource locator deduplication method also includes: setting corresponding third mapping values ​​for the multiple above-mentioned second hash results of the above-mentioned uniform resource locator to be identified; and updating the above-mentioned bitmap using the above-mentioned third mapping value.

[0017] According to another aspect of the present disclosure, a uniform resource locator deduplication device is provided, and the uniform resource locator deduplication device comprises:

[0018] A bitmap construction module is used to construct a bitmap of the memory database; wherein the bitmap stores first mapping values ​​corresponding to each of a plurality of first hash results of the uniform resource locator;

[0019] A hash processing module, used for obtaining a uniform resource locator to be identified and determining a plurality of second hash results of the uniform resource locator to be identified;

[0020] A numerical query module, used to query the bitmap according to the plurality of second hash results to obtain second mapping values ​​corresponding to the plurality of second hash results;

[0021] The de-duplication identification module is used to determine the matching degree between the obtained second mapping value and the obtained first mapping value, and judge whether the to-be-identified uniform resource locator is a duplicate uniform resource locator according to the matching degree.

[0022] According to another aspect of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the Uniform Resource Locator deduplication method as described in the above embodiment when executing the computer program.

[0023] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the uniform resource locator deduplication method described in the above embodiment is implemented.

[0024] The uniform resource locator deduplication method, device, electronic device and computer-readable storage medium provided by the embodiments of the present disclosure have the following technical effects:

[0025] The present disclosure adopts a bitmap to construct an in-memory database, in which the first mapping values ​​corresponding to multiple first hash results of the uniform resource locator are stored, to obtain the uniform resource locator to be identified, and to determine multiple second hash results of the uniform resource locator to be identified, and to query the bitmap according to the multiple second hash results to obtain the second mapping values ​​corresponding to the multiple second hash results, and to determine the matching degree between the obtained second mapping values ​​and the first mapping values, and to judge whether the uniform resource locator to be identified is a duplicate uniform resource locator according to the matching degree. The technical solution can improve the deduplication efficiency of the crawler for URLs in a high-concurrency cluster environment, reduce the occupancy of the central processing unit, and is conducive to fully integrating multiple services in the cluster and improving the overall performance of the crawler.

[0026] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0028] Figure 1 A schematic diagram of a flow chart of a uniform resource locator deduplication method in an exemplary embodiment of the present disclosure is shown;

[0029] Figure 2 A schematic diagram of an exemplary storage structure of a bitmap is shown;

[0030] Figure 3 An exemplary flow chart of S110 in the uniform resource locator deduplication method provided by an embodiment of the present disclosure is shown;

[0031] Figure 4 An exemplary flow chart of S130 in the method for deduplicating uniform resource locators provided in an embodiment of the present disclosure is shown;

[0032] Figure 5 A schematic diagram of querying a bitmap of Redis is shown;

[0033] Figure 6 A schematic diagram showing the structure of a uniform resource locator deduplication device provided by an embodiment of the present disclosure is shown;

[0034] Figure 7 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0036] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are only examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0037] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0038] The existing web crawler deduplication solution is to directly perform deduplication through Map storage in memory. In a single structure or low concurrency scenario, the deduplication efficiency is quite impressive. However, most of the actual customers are large enterprise users, who generally require cluster deployment. When enterprises perform large-scale Uniform Resource Locator (URL) scanning, the real-time concurrency is quite high. If Map storage is still used to implement URL deduplication, the CPU memory usage will be too high.

[0039] Based on the defects existing in the related technology, the present disclosure provides a uniform resource locator deduplication method, device, electronic device and computer-readable storage medium. A shared memory database is used in the uniform resource locator deduplication method to solve the data consistency problem in the above cluster deployment scenario and the performance problem in high concurrency scenario, and the data is processed in combination with the bitmap data structure in the memory database to solve the problem of large data volume.

[0040] The following is an embodiment of the method for deduplicating uniform resource locators provided by the present disclosure. Figure 1 FIG. 2 shows a flow chart of a method for deduplicating uniform resource locators in an exemplary embodiment of the present disclosure. Figure 1 As shown, a method for deduplicating uniform resource locators provided by an embodiment of the present disclosure includes the following schemes:

[0041] Step S110: construct a bitmap of the in-memory database.

[0042] In an exemplary embodiment, the memory database is also called Redis, which is a shared or public memory database, that is, each service in the cluster can use Redis. The bitmap of the memory database is a bit map, which stores first mapping values ​​corresponding to multiple first hash results of different URLs, that is, each uniform resource locator corresponds to multiple first hash results, and each first hash result corresponds to a first mapping value. Among them, the first hash result includes a first hash value, and the first mapping value is a fixed value, for example, the first mapping value is set to 1.

[0043] like Figure 2 As shown, Figure 2A schematic diagram of an exemplary storage structure of a bitmap is shown. The data storage length of the bitmap is 8, specifically 0-7 bits. The first mapping value corresponding to each bit is 1, and the first mapping value on each bit corresponds to a first hash result, that is, there are a total of 8 first hash results, that is, the same URL corresponds to 8 first hash results.

[0044] Step S120: Obtain a uniform resource locator to be identified, and determine a plurality of second hash results of the uniform resource locator to be identified.

[0045] The URL to be identified can be obtained through the services in the cluster, for example, www.baidu.com can be crawled through Service A, or crawled through a crawler tool. After obtaining the URL to be identified, multiple second hash results of the URL to be identified are calculated, and each second hash result is different.

[0046] Optionally, determining a plurality of second hash results of the to-be-identified uniform resource locator includes the following scheme:

[0047] Different preset hash algorithms are used to respectively calculate the hash values ​​of the above-mentioned uniform resource locator to be identified, so as to obtain multiple second hash results of the above-mentioned uniform resource locator to be identified.

[0048] Since each first mapping value in the bitmap corresponds to a first hash result, that is, multiple first hash results correspond to one URL, then there must be multiple second hash results calculated for the URL to be identified, and the number of second hash results is the same as the number of first hash results. The multiple first hash results of the URL in the bitmap are calculated using different preset hash algorithms, so the algorithm for calculating each second hash result of the URL to be identified also needs to use the preset hash algorithm corresponding to each first hash result. For example, the preset hash algorithm used to calculate the first hash result 1-1 of the URL is MD5 (Message-Digest Algorithm 5), and the preset hash algorithm used to calculate the second hash result 1-1 of the URL to be identified is also MD5. Among them, the multiple second hash results of the URL to be identified will be sorted according to the data structure in the bitmap to correspond to the data structure in the bitmap, so as to facilitate subsequent bitmap queries.

[0049] like Figure 2As shown, the first mapping values ​​corresponding to the 8 first hash results of a URL are [1,1,1,1,1,1,1,1], and the first hash results corresponding to the first mapping values ​​from the 1st to the 8th bit are respectively calculated by hash algorithm 1-hash algorithm 8, that is, the 8 first hash results are a group of hash values, and the group of first hash results corresponds to a group of first mapping values. If the 8 second hash results of the URL to be identified are [h1,h2,h3,h4,h5,h6,h7,h8], that is, these 8 second hash results are also calculated by hash algorithm 1-hash algorithm 8, and the 8 second hash results are a group.

[0050] Step S130: query the bitmap according to the plurality of second hash results to obtain second mapping values ​​corresponding to the plurality of second hash results.

[0051] Since the multiple first mapping values ​​under each URL in the bitmap each correspond to a first hash result, after calculating and obtaining the multiple second hash results of the URL to be identified, the bitmap can be queried through the multiple second hash results of the URL to be identified to obtain which second hash results hit the first hash results corresponding to the multiple first mapping values ​​under each URL in the bitmap. For the second hash result that hits the first hash result, a first fixed value is set for it, for example, the first fixed value is 1, and the second mapping value corresponding to the second hash result is obtained; for the second hash result that does not hit the first hash result, a second fixed value is set for it, for example, the second fixed value is 0, and the second mapping value corresponding to the second hash result is obtained, that is, all the second mapping values ​​corresponding to the multiple second hash results of the URL to be identified may all be 1, or all may be 0, or some may be 1 and the other part may be 0. Among them, since the multiple second hash results of the URL to be identified are calculated as a group of hash values, the group of second hash results corresponds to a group of second mapping values.

[0052] Step S140: determining a matching degree between the obtained second mapping value and the obtained first mapping value, and judging whether the to-be-identified uniform resource locator is a repeated uniform resource locator according to the matching degree.

[0053] After obtaining the second mapping values ​​corresponding to each of the multiple second hash results under the URL to be identified, determine the second mapping values ​​that hit the first mapping value among the multiple second mapping values. The more second mapping values ​​that hit the first mapping value, the higher the matching degree between the obtained multiple second mapping values ​​and the first mapping value. Further, determine whether the uniform resource locator to be identified is a duplicate uniform resource locator based on the matching degree. If the matching degree is higher, it is determined that the URL to be identified is the same as the URL corresponding to a set of first mapping values ​​stored in the bitmap, that is, the URL to be identified is a duplicate URL, which is the URL corresponding to a set of first mapping values ​​stored in the bitmap. If the matching degree is lower, it is determined that the URL to be identified is not the same as the URL corresponding to a set of first mapping values ​​stored in the bitmap, that is, the URL to be identified is not a duplicate URL, and it does not repeat with the URL corresponding to a set of first mapping values ​​stored in the bitmap. In the case where the URL to be identified is not a duplicate URL, the crawler will normally access the web page corresponding to the URL to be identified. If the URL to be identified is a duplicate URL, the URL deduplication module will filter out or discard the URL to be identified, and the crawler will no longer visit the web page corresponding to the URL to be identified, which is beneficial to saving the memory usage of the URL deduplication module in large data volume scenarios and reducing the usage of the central processing unit (CPU).

[0054] According to the above technical solution, this embodiment adopts a bitmap to construct an in-memory database, in which the first mapping values ​​corresponding to multiple first hash results of the uniform resource locator are stored, the uniform resource locator to be identified is obtained, and multiple second hash results of the uniform resource locator to be identified are determined, the bitmap is queried according to the multiple second hash results, the second mapping values ​​corresponding to the multiple second hash results are obtained, the matching degree between the obtained second mapping values ​​and the first mapping values ​​is determined, and whether the uniform resource locator to be identified is a duplicate uniform resource locator is judged according to the matching degree. This technical solution can improve the crawler's deduplication efficiency for URLs in a high-concurrency cluster environment, reduce the occupancy of the central processing unit, and is conducive to fully integrating multiple services in the cluster and improving the overall performance of the crawler.

[0055] like Figure 3 As shown, Figure 3 An exemplary flow chart of S110 in the method for deduplicating uniform resource locators provided in an embodiment of the present disclosure is shown. Optionally, based on the above method embodiment, step S110 includes the following schemes:

[0056] S111: using different preset hash algorithms to perform hash processing on the acquired uniform resource locators respectively, to obtain a plurality of first hash results of the uniform resource locators;

[0057] S112: Setting a corresponding first mapping value for each of the first hash results of the uniform resource locator;

[0058] S113: Update the default value corresponding to the spatial storage position in the bitmap to the first mapping value.

[0059] In an exemplary embodiment, multiple preset hash algorithms are preset, and the hash results calculated by each preset hash algorithm are different. For example, the hash value of data 1 is calculated using the preset hash algorithm 1-1 to obtain hash result 1 of data 1, and the hash value of data 1 is calculated using the preset hash algorithm 1-2 to obtain hash result 2 of data 1. Hash result 1 and hash result 2 are calculated by different algorithms, that is, they are different. For an empty bitmap, after the crawler crawls the URL of the webpage, since there is no data stored in the bitmap, different preset hash algorithms are used to hash the crawled URL according to the customer's data volume to obtain multiple first hash results of the URL, each of which includes a hash value. Among them, the number of first hash results is the same as the number of preset hash algorithms. For example, if there are 8 preset hash algorithms, there are also 8 first hash results of the URL. When URL deduplication is performed subsequently, the URL to be identified obtained also needs to calculate the same number of second hash results as the preset hash algorithm, that is, the URL to be identified corresponds to multiple second hash results, and then when the bitmap query is performed through the multiple second hash results of the URL to be identified, the hash collision problem can be avoided.

[0060] The data storage length corresponding to the spatial storage position in the bitmap is predetermined, and the data storage length is numerically consistent with the number of multiple hash results of the URL. For example, if the data storage length is 8, then the number of multiple hash results of the URL is also 8. Before storing data, the spatial storage position in the bitmap needs to be initialized, that is, the data value corresponding to the spatial storage position is a default value, and the default value is set to 0, for example. After obtaining the multiple first hash results of the above-mentioned URL, a corresponding first mapping value is set for each first hash result of the URL, and then the default value corresponding to the spatial storage position is updated to the first mapping value. Figure 2 As shown, Figure 2 It can be understood that each first hash result under a certain URL in the bitmap sets the corresponding first mapping value. The data storage length of the bitmap is 8, specifically 0-7 bits, and the default value corresponding to each bit in the spatial storage position is 0 ( Figure 2 (not shown), that is, [0,0,0,0,0,0,0,0], after the default value corresponding to each bit is updated, the first mapping value corresponding to each bit is 1, that is, [1,1,1,1,1,1,1,1].

[0061] like Figure 4 As shown, Figure 4 An exemplary flow chart of S130 in the method for deduplicating uniform resource locators provided in an embodiment of the present disclosure is shown. Optionally, based on the above method embodiment, step S130 includes the following schemes:

[0062] S131: Obtain first hash results corresponding to each of the plurality of first mapping values ​​in the bitmap;

[0063] S132: if the second hash result is included in the first hash results corresponding to the plurality of first mapping values, determine the first mapping value corresponding to the first hash result that is the same as the second hash result as the second mapping value corresponding to the second hash result;

[0064] S133: If there is the second hash result that is not included in the first hash results corresponding to the plurality of first mapping values, determine the default value of the spatial storage position in the bitmap as the second mapping value corresponding to the second hash result.

[0065] In an exemplary embodiment, after obtaining multiple second hash results of the URL to be identified, the first hash results corresponding to each of the multiple first mapping values ​​under each URL in the bitmap can be obtained, and then it is determined whether the second hash results of the multiple second hash results of the URL to be identified are included in the first hash results corresponding to the multiple first mapping values ​​under which URLs in the bitmap. If there is a second hash result included in the first hash result corresponding to each of the multiple first mapping values ​​of certain URLs, it means that the second hash result is the same as the first hash result in the first hash result corresponding to each of the multiple first mapping values ​​of certain URLs, then the first mapping value corresponding to the first hash result that is the same as the second hash result is determined as the second mapping value corresponding to the second hash result. For example, if the first mapping value is 1, the second mapping value is the same as the first mapping value, which is also 1.

[0066] If there is a second hash result that is not included in the first hash results corresponding to the first mapping values ​​of some URLs, it means that the second hash result is different from the first hash result in the first hash results corresponding to the first mapping values ​​of some URLs, then the default value of the spatial storage position in the bitmap is determined as the second mapping value corresponding to the second hash result. For example, if the default value is 0, the second mapping value is the same as the default value, which is also 0.

[0067] like Figure 5 As shown, Figure 5A schematic diagram of querying a bitmap of Redis is shown. Assume that the URL to be identified crawled by Service A is URL1, the URL to be identified crawled by Service B is URL2, and the URLs to be identified crawled by Service C are URL3 and URL4 respectively.

[0068] Among them, after querying the bitmap through multiple second hash results of URL1, the second mapping value corresponding to the multiple second hash results of URL1 is [0,0,1,0,0,0,0,0], indicating that according to the calculation order of the second hash results, the third second hash result among the multiple second hash results of URL1 is the same as the second hash result corresponding to the third first mapping value under a certain URL in the bitmap.

[0069] After querying the bitmap through multiple second hash results of URL2, the second mapping values ​​corresponding to the multiple second hash results of URL2 are [0,1,1,0,0,0,0,0], indicating that in the calculation order of the second hash results, the second and third second hash results among the multiple second hash results of URL2 are respectively the same as the second hash results corresponding to the second and third first mapping values ​​under a certain URL in the bitmap.

[0070] After querying the bitmap through multiple second hash results of URL3, the second mapping values ​​corresponding to the multiple second hash results of URL3 are [0,0,0,0,0,0,1,0], indicating that in the calculation order of the second hash results, the seventh second hash result among the multiple second hash results of URL3 is the same as the second hash result corresponding to the seventh first mapping value under a certain URL in the bitmap.

[0071] After querying the bitmap through multiple second hash results of URL4, the second mapping values ​​corresponding to the multiple second hash results of URL4 are [0,0,1,0,0,1,0,0], indicating that in the calculation order of the second hash results, the third and fifth second hash results among the multiple second hash results of URL4 are respectively the same as the second hash results corresponding to the third and fifth first mapping values ​​under a certain URL in the bitmap.

[0072] Optionally, based on the above method embodiment, the matching degree between the second mapping value and the first mapping value obtained by the above determination includes the following schemes:

[0073] Get the first number of the first mapping value above;

[0074] Determine a second number of the second mapped values ​​that is the same as the first mapped value;

[0075] The matching degree is determined according to the ratio of the second number to the first number.

[0076] In an exemplary embodiment, the first number of the multiple first mapping values ​​corresponding to each URL stored in the bitmap is the same, and the first number is the same as the data storage length in the bitmap. For example, the first number is 8. After querying the bitmap through the multiple second hash results of the URL to be identified, the second mapping values ​​corresponding to the multiple second hash results of the URL to be identified are obtained, that is, a group of second mapping values. Then, the second mapping value that is the same as the first mapping value in the group of second mapping values ​​is determined, and the second number of the second mapping value that is the same as the first mapping value is counted, and then the ratio of the second number to the first number is calculated, and the ratio is determined as the above-mentioned matching degree. Among them, the more second numbers, the higher the matching degree, and the greater the possibility that the URL to be identified is a duplicate URL. For example, a group of second mapping values ​​corresponding to the URL to be identified is [1,1,1,0,1,1,0,0], and a group of first mapping values ​​corresponding to a URL in the bitmap is [1,1,1,1,1,1,1,1,1], that is, the second number of the second mapping value that is the same as the first mapping value is 5, that is, the ratio of the second number to the first number is 0.625, and the matching degree is 0.625.

[0077] Optionally, based on the above method embodiment, judging whether the to-be-identified uniform resource locator is a repeated uniform resource locator according to the above matching degree includes the following schemes:

[0078] Get the matching threshold;

[0079] When the matching degree is greater than or equal to the matching degree threshold, determining that the to-be-identified uniform resource locator is a repeated uniform resource locator;

[0080] When the matching degree is less than the matching degree threshold, it is determined that the to-be-identified uniform resource locator is not a repeated uniform resource locator.

[0081] In an exemplary embodiment, the matching threshold is set by the customer according to his actual business needs. The larger the matching threshold is set, the higher the accuracy of URL deduplication. For example, the matching threshold is set to 1. If a group of second mapping values ​​corresponding to the URL to be identified are all 1, the URL to be identified is considered to be a duplicate URL. If one of the group of second mapping values ​​corresponding to the URL to be identified is 0, the URL to be identified is considered not to be a duplicate URL. After obtaining the matching degree of multiple second mapping values ​​under the URL to be identified and multiple first mapping values ​​under a URL in the bitmap, the matching degree is compared with the matching threshold. If the matching degree is greater than or equal to the matching threshold, it is determined that the URL to be identified is a duplicate URL, that is, the URL to be identified is repeated with the URL corresponding to the first mapping value stored in the bitmap, and the URL deduplication module will remove or discard the URL to be identified. If the matching degree is less than the matching threshold, it is determined that the URL to be identified is not a duplicate URL, that is, the URL to be identified is not repeated with the URL corresponding to the first mapping value stored in the bitmap, that is, the URL to be identified is a new URL and has never been crawled before. The URL deduplication module will not remove or discard the URL to be identified.

[0082] Optionally, based on the above method embodiment, after the step of determining that the to-be-identified uniform resource locator is not a duplicate uniform resource locator when the above matching degree is less than the above matching degree threshold, the above uniform resource locator deduplication method further includes the following scheme:

[0083] Setting corresponding third mapping values ​​for the plurality of the second hash results of the uniform resource locators to be identified;

[0084] The bitmap is updated using the third mapping value.

[0085] In an exemplary embodiment, if it is determined that the URL to be identified is not a duplicate URL, it means that the URL to be identified has not been crawled before, then a corresponding third mapping value is set for each of the multiple second hash results of the URL to be identified, and the third mapping value is the same as the first mapping value. Then, the third mapping value corresponding to the multiple second hash results of the URL to be identified is used to update the default value corresponding to the spatial storage position in the bitmap, that is, the default value is replaced with the third mapping value. In this way, the third mapping values ​​corresponding to the multiple second hash results of the URL to be identified will also be stored in the bitmap, thereby realizing the update of the bitmap. When the URL to be identified is crawled again later, the bitmap is queried, and the URL to be identified that is crawled again will be removed or discarded by the URL deduplication module.

[0086] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0087] in, Figure 6 The schematic diagram of the structure of the uniform resource locator deduplication device to which the present disclosure can be applied is shown. Figure 6 The uniform resource locator deduplication device shown in the figure can be implemented as all or part of the electronic device through software, hardware or a combination of both, and can also be integrated into the electronic device or server as an independent module.

[0088] The uniform resource locator deduplication device 600 in the embodiment of the present disclosure includes:

[0089] A bitmap construction module 610 is used to construct a bitmap of the memory database; wherein the bitmap stores first mapping values ​​corresponding to each of a plurality of first hash results of the uniform resource locator;

[0090] The hash processing module 620 is used to obtain the uniform resource locator to be identified and determine a plurality of second hash results of the uniform resource locator to be identified;

[0091] A numerical query module 630, configured to query the bitmap according to the plurality of second hash results to obtain second mapping values ​​corresponding to the plurality of second hash results;

[0092] The de-duplication identification module 640 is used to determine the matching degree between the obtained second mapping value and the obtained first mapping value, and determine whether the to-be-identified uniform resource locator is a duplicate uniform resource locator according to the matching degree.

[0093] In an exemplary embodiment, based on the above-mentioned solution, the bitmap construction module 610 includes:

[0094] A hash calculation unit, used to perform hash processing on the acquired uniform resource locator respectively by using different preset hash algorithms to obtain a plurality of first hash results of the uniform resource locator;

[0095] A first value setting unit, used for setting a corresponding first mapping value for each of the first hash results of the uniform resource locator;

[0096] A value updating unit is used to update the default value corresponding to the spatial storage position in the bitmap to the first mapping value.

[0097] In an exemplary embodiment, based on the aforementioned scheme, the hash processing module 620 is specifically used to determine multiple second hash results of the uniform resource locator to be identified by: using different preset hash algorithms to respectively calculate the hash values ​​of the uniform resource locator to be identified, and obtaining multiple second hash results of the uniform resource locator to be identified.

[0098] In an exemplary embodiment, based on the above solution, the numerical query module 630 includes:

[0099] A value acquisition unit, used to obtain first hash results corresponding to each of the plurality of first mapping values ​​in the bitmap;

[0100] a first value determination unit, configured to determine, when the second hash result is included in the first hash results corresponding to the plurality of first mapping values, a first mapping value corresponding to the first hash result that is the same as the second hash result as the second mapping value corresponding to the second hash result;

[0101] The second value determination unit is used to determine the default value of the spatial storage position in the above-mentioned bitmap as the second mapping value corresponding to the above-mentioned second hash result when the above-mentioned second hash result is not included in the first hash results corresponding to each of the multiple first mapping values.

[0102] In an exemplary embodiment, based on the above solution, the de-duplication identification module 640 determines the matching degree between the obtained second mapping value and the first mapping value, including:

[0103] A first number counting unit, used to obtain the first number of the first mapping value;

[0104] a second number counting unit, used to determine a second number of the second mapping value which is the same as the first mapping value;

[0105] The matching degree calculation unit is used to determine the matching degree according to the ratio of the second number to the first number.

[0106] In an exemplary embodiment, based on the above solution, the de-duplication identification module 640 determines whether the to-be-identified uniform resource locator is a duplicate uniform resource locator according to the above matching degree, including:

[0107] A threshold acquisition unit, used to acquire a matching degree threshold;

[0108] A first judgment unit, configured to determine that the to-be-identified uniform resource locator is a repeated uniform resource locator when the matching degree is greater than or equal to the matching degree threshold;

[0109] The second judgment unit is configured to determine that the to-be-identified uniform resource locator is not a duplicate uniform resource locator when the matching degree is less than the matching degree threshold.

[0110] In an exemplary embodiment, based on the above scheme, the above-mentioned uniform resource locator deduplication device further includes:

[0111] A second value setting unit, used for setting corresponding third mapping values ​​for the plurality of second hash results of the uniform resource locator to be identified;

[0112] The bitmap updating unit is used to update the bitmap by using the third mapping value.

[0113] It should be noted that the Uniform Resource Locator deduplication device provided in the above embodiment only uses the division of the above functional modules as an example when executing the Uniform Resource Locator deduplication method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the Uniform Resource Locator deduplication device and the Uniform Resource Locator deduplication method embodiment provided in the above embodiment belong to the same concept. Therefore, for details not disclosed in the embodiment of the device disclosed in the present invention, please refer to the embodiment of the Uniform Resource Locator deduplication method disclosed in the present invention, which will not be repeated here.

[0114] The serial numbers of the above-mentioned embodiments of the present disclosure are only for description and do not represent the advantages or disadvantages of the embodiments.

[0115] The embodiments of the present disclosure also provide a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the method of any of the above embodiments are implemented. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0116] An embodiment of the present disclosure further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of any of the above-mentioned method embodiments when executing the program.

[0117] Figure 7 The schematic diagram of the electronic device is shown schematically. Figure 7 As shown, the electronic device 700 includes: a processor 701 and a memory 702 .

[0118] In the embodiment of the present disclosure, the processor 701 is the control center of the computer system, which can be the processor of a physical machine or the processor of a virtual machine. The processor 701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 701 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state.

[0119] In the embodiment of the present disclosure, the processor 701 is specifically used to: construct a bitmap of an in-memory database; wherein the bitmap stores first mapping values ​​corresponding to multiple first hash results of a uniform resource locator; obtain a uniform resource locator to be identified, and determine multiple second hash results of the uniform resource locator to be identified; query the bitmap according to the multiple second hash results to obtain second mapping values ​​corresponding to the multiple second hash results; determine the degree of matching between the obtained second mapping values ​​and the first mapping values, and determine whether the uniform resource locator to be identified is a duplicate uniform resource locator based on the degree of matching.

[0120] Furthermore, the processor 701 is also used to: use different preset hash algorithms to perform hash processing on the acquired uniform resource locator respectively to obtain multiple first hash results of the uniform resource locator; set a corresponding first mapping value for each of the first hash results of the uniform resource locator; and update the default numerical value corresponding to the spatial storage position in the bitmap to the first mapping value.

[0121] Furthermore, the processor 701 is further configured to: respectively calculate the hash values ​​of the to-be-identified uniform resource locator using different preset hash algorithms to obtain a plurality of second hash results of the to-be-identified uniform resource locator.

[0122] Furthermore, the processor 701 is also used to: obtain first hash results corresponding to each of the multiple first mapping values ​​in the bitmap; when the second hash result is included in the first hash results corresponding to the multiple first mapping values, determine the first mapping value corresponding to the first hash result that is the same as the second hash result as the second mapping value corresponding to the second hash result; when the second hash result is not included in the first hash results corresponding to the multiple first mapping values, determine the default value of the spatial storage position in the bitmap as the second mapping value corresponding to the second hash result.

[0123] Furthermore, the processor 701 is also used to: obtain a first number of the first mapping value; determine a second number of the second mapping value that is the same as the first mapping value; and determine the degree of matching based on a ratio of the second number to the first number.

[0124] Furthermore, the processor 701 is also used to: obtain a matching degree threshold; when the matching degree is greater than or equal to the matching degree threshold, determine that the to-be-identified uniform resource locator is a duplicate uniform resource locator; when the matching degree is less than the matching degree threshold, determine that the to-be-identified uniform resource locator is not a duplicate uniform resource locator.

[0125] Furthermore, the processor 701 is further configured to: set corresponding third mapping values ​​for the plurality of second hash results of the to-be-identified uniform resource locator; and update the bitmap using the third mapping values.

[0126] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments of the present disclosure, the non-transitory computer-readable storage medium in the memory 702 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 701 to implement the method in the embodiment of the present disclosure.

[0127] In some embodiments, the electronic device 700 further includes: a peripheral device interface 703 and at least one peripheral device. The processor 701, the memory 702 and the peripheral device interface 703 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 703 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a display screen 704, a camera 705 and an audio circuit 706.

[0128] The peripheral device interface 703 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 701 and the memory 702. In some embodiments of the present disclosure, the processor 701, the memory 702, and the peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments of the present disclosure, any one or two of the processor 701, the memory 702, and the peripheral device interface 703 may be implemented on a separate chip or circuit board. The embodiments of the present disclosure are not specifically limited to this.

[0129] The display screen 704 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 704 is a touch display screen, the display screen 704 also has the ability to collect touch signals on the surface or above the surface of the display screen 704. The touch signal can be input to the processor 701 as a control signal for processing. At this time, the display screen 704 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments of the present disclosure, the display screen 704 can be one, and the front panel of the electronic device 700 is set; in other embodiments of the present disclosure, the display screen 704 can be at least two, which are respectively set on different surfaces of the electronic device 700 or are folded; in some other embodiments of the present disclosure, the display screen 704 can be a flexible display screen, which is set on the curved surface or folded surface of the electronic device 700. Even, the display screen 704 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 704 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0130] The camera 705 is used to capture images or videos. Optionally, the camera 705 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the electronic device 700, and the rear camera is arranged on the back of the electronic device 700. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments of the present disclosure, the camera 705 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0131] The audio circuit 706 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 701 for processing. For the purpose of stereo sound collection or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the electronic device 700. The microphone may also be an array microphone or an omnidirectional collection microphone.

[0132] The power supply 707 is used to power various components in the electronic device 700. The power supply 707 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 707 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0133] The structural block diagram of the electronic device 700 shown in the embodiment of the present disclosure does not constitute a limitation on the electronic device 700 , and the electronic device 700 may include more or fewer components than those shown in the figure, or combine certain components, or adopt a different component arrangement.

[0134] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of this specification are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the object features, interactive behavior features and user information involved in this specification are all obtained with full authorization.

[0135] In the description of the present disclosure, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present disclosure can be understood according to specific circumstances. In addition, in the description of the present disclosure, unless otherwise specified, "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previously associated objects are in an "or" relationship.

[0136] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, equivalent changes made according to the claims of the present disclosure are still within the scope covered by the present disclosure.

Claims

1. A method for removing duplicate uniform resource locators, characterized in that: The uniform resource locator deduplication method comprises: Constructing a bitmap of the memory database; wherein the bitmap stores first mapping values ​​corresponding to each of a plurality of first hash results of the uniform resource locator; Obtaining a uniform resource locator to be identified, and determining a plurality of second hash results of the uniform resource locator to be identified; Query the bitmap according to the plurality of second hash results to obtain second mapping values ​​corresponding to the plurality of second hash results respectively; A matching degree between the obtained second mapping value and the first mapping value is determined, and whether the to-be-identified uniform resource locator is a repeated uniform resource locator is determined according to the matching degree.

2. The method for deduplicating uniform resource locators according to claim 1, wherein: The step of constructing a bitmap of the memory database comprises: Using different preset hash algorithms to perform hash processing on the acquired uniform resource locators respectively to obtain multiple first hash results of the uniform resource locators; Setting a corresponding first mapping value for each of the first hash results of the uniform resource locator; The default value corresponding to the spatial storage position in the bitmap is updated to the first mapping value.

3. The method for deduplicating uniform resource locators according to claim 1, wherein: The step of determining a plurality of second hash results of the to-be-identified uniform resource locator comprises: Different preset hash algorithms are used to respectively calculate the hash values ​​of the uniform resource locator to be identified, so as to obtain a plurality of second hash results of the uniform resource locator to be identified.

4. The method for removing duplicate uniform resource locators according to claim 1, wherein: The step of querying the bitmap according to the plurality of second hash results to obtain second mapping values ​​corresponding to the plurality of second hash results respectively comprises: Obtain first hash results corresponding to each of the plurality of first mapping values ​​in the bitmap; If the second hash result is included in the first hash results corresponding to the plurality of first mapping values, respectively, the first mapping value corresponding to the first hash result that is the same as the second hash result is determined as the second mapping value corresponding to the second hash result; If there is a second hash result that is not included in the first hash results corresponding to each of the plurality of first mapping values, a default value of the spatial storage position in the bitmap is determined as the second mapping value corresponding to the second hash result.

5. The method for removing duplicate uniform resource locators according to claim 1, wherein: The step of determining the matching degree between the obtained second mapping value and the first mapping value comprises: Get the first number of the first mapping value; determining a second number of said second mapped values ​​that is identical to said first mapped value; The matching degree is determined according to a ratio of the second number to the first number.

6. The method for deduplicating uniform resource locators according to any one of claims 1 to 5, characterized in that: The step of judging whether the to-be-identified uniform resource locator is a repeated uniform resource locator according to the matching degree comprises: Get the matching threshold; When the matching degree is greater than or equal to the matching degree threshold, determining that the to-be-identified uniform resource locator is a repeated uniform resource locator; When the matching degree is less than the matching degree threshold, it is determined that the to-be-identified uniform resource locator is not a repeated uniform resource locator.

7. The method for removing duplicate uniform resource locators according to claim 6, characterized in that: After the step of determining that the to-be-identified uniform resource locator is not a duplicate uniform resource locator when the matching degree is less than the matching degree threshold, the uniform resource locator deduplication method further includes: Setting corresponding third mapping values ​​for the plurality of second hash results of the uniform resource locator to be identified; The bitmap is updated using the third mapping value.

8. A uniform resource locator deduplication device, characterized in that: The uniform resource locator deduplication device comprises: A bitmap construction module, used to construct a bitmap of the memory database; wherein the bitmap stores first mapping values ​​corresponding to each of a plurality of first hash results of the uniform resource locator; A hash processing module, used for obtaining a uniform resource locator to be identified, and determining a plurality of second hash results of the uniform resource locator to be identified; A numerical query module, used for querying the bitmap according to the plurality of second hash results to obtain second mapping values ​​corresponding to the plurality of second hash results respectively; The de-duplication identification module is used to determine the matching degree between the obtained second mapping value and the first mapping value, and judge whether the to-be-identified uniform resource locator is a duplicate uniform resource locator according to the matching degree.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the uniform resource locator deduplication method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the uniform resource locator deduplication method according to any one of claims 1 to 7 is implemented.

Images