Link deduplication method, device, apparatus and storage medium

Abstract

The application belongs to the field of communication and discloses a link deduplication method, device, equipment and storage medium. The method comprises the following steps: obtaining a to-be-downloaded link, formatting the to-be-downloaded link to obtain a formatted link, matching the formatted link with a link template stored in a preset link mode library to obtain a matching result, and judging whether to perform a deduplication operation on the formatted link according to the matching result. Since the formatted link is matched with the link template stored in the preset link mode library to obtain the matching result, and whether to perform the deduplication operation on the formatted link is judged according to the matching result, compared with the existing mode of directly performing resource downloading after receiving a download link, the above mode can perform the deduplication operation on the download link, reduces repeated downloading of resources, and ensures the quality of the downloaded resources.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a method, apparatus, device, and storage medium for link deduplication. Background Technology

[0002] In daily life, downloading an application usually involves downloading a file through a URI link. With the development of internet technology, these links are no longer fixed addresses; over 90% of them are constantly changing. Downloading resources through such a massive number of URIs often leads to the following problems: when downloading a large number of application links, the files are often extremely large. If each download link is processed, it will result in a massive amount of invalid resources.

[0003] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this invention is to provide a method, apparatus, device, and storage medium for deduplication of links, aiming to solve the technical problem of duplicate downloads and resource redundancy caused by directly downloading resources based on download links in the prior art.

[0005] To achieve the above objectives, the present invention provides a link deduplication method, the method comprising the following steps:

[0006] Obtain the download link, format the download link, and obtain the formatted link;

[0007] The formatted link is matched with the link templates stored in the preset link pattern library to obtain the matching result;

[0008] Determine whether to perform deduplication on the formatted links based on the matching results.

[0009] Optionally, before the step of matching the formatted link with link templates stored in a preset link pattern library to obtain a matching result, the method further includes:

[0010] Retrieve links to be processed from the link history database;

[0011] The link to be processed is subjected to invalid parameter identification to obtain invalid parameter identification results;

[0012] When the invalid parameter identification result contains a parameter to be verified, dynamic parameter identification is performed on the link to be processed based on the invalid parameter identification result to obtain the dynamic parameter identification result.

[0013] A preset link pattern library is generated based on the dynamic parameter identification results and the links to be processed.

[0014] Optionally, the step of generating a preset link pattern library based on the dynamic parameter identification result and the link to be processed includes:

[0015] The valid parameters of the link to be processed are determined based on the dynamic parameter identification results.

[0016] Generate a target link based on the valid parameters and the link to be processed;

[0017] A preset link pattern library is constructed based on the target link.

[0018] Optionally, the step of identifying invalid parameters in the link to be processed and obtaining invalid parameter identification results includes:

[0019] Obtain the link resource file corresponding to the link to be processed;

[0020] For each link to be processed, parameters are deleted one at a time to obtain the link to be verified.

[0021] After each parameter deletion, obtain the corresponding link resource file to be verified.

[0022] The link resource file and the link resource file to be verified are compared to obtain the comparison result;

[0023] Invalid parameter identification results are generated based on the comparison results.

[0024] Optionally, the step of generating invalid parameter identification results based on the comparison results includes:

[0025] If the comparison result is consistent, then the parameter to be deleted is determined to be invalid.

[0026] If the comparison result is inconsistent, then the parameter currently being deleted is determined to be a parameter to be verified.

[0027] Optionally, the step of comparing the linked resource file and the linked resource file to be verified to obtain a comparison result includes:

[0028] The target hash value corresponding to the linked resource file is determined by a preset algorithm;

[0029] The verification hash value corresponding to the link resource file to be verified is determined by the preset algorithm.

[0030] The target hash value and the verification hash value are compared to obtain the comparison result.

[0031] Optionally, the step of performing dynamic parameter identification on the link to be processed based on the invalid parameter identification result to obtain the dynamic parameter identification result includes:

[0032] Based on the invalid parameter identification results, determine the parameters to be verified in the link to be processed;

[0033] Perform format verification on the parameter to be verified and obtain the format verification result;

[0034] The dynamic parameter identification result corresponding to the link to be processed is determined based on the format verification result.

[0035] Optionally, the step of performing format verification on the parameter to be verified and obtaining the format verification result includes:

[0036] Get the preset dynamic parameter format;

[0037] Determine the parameter format corresponding to the parameter to be verified;

[0038] Determine whether the parameter format belongs to the preset dynamic parameter format, and obtain the format verification result.

[0039] Optionally, the step of obtaining the link to be downloaded, formatting the link to be downloaded, and obtaining a formatted link includes:

[0040] Obtain the download link, remove invalid characters from the download link to obtain a valid download link;

[0041] The parameters in the valid download link are adjusted according to the ASCII code table to obtain the adjusted download link.

[0042] The adjusted download link is converted into a preset format to obtain a formatted download link.

[0043] Optionally, after the step of generating a preset link pattern library based on the dynamic parameter identification result and the link to be processed, the method further includes:

[0044] Obtain the link template from the preset link pattern library;

[0045] The link template is subjected to similarity detection to obtain similarity detection results;

[0046] The link templates are classified based on the similarity detection results to obtain classification results;

[0047] The preset link pattern library is optimized based on the classification results.

[0048] Optionally, the step of optimizing the preset link pattern library based on the classification result includes:

[0049] The number of links in each type of pattern is determined based on the classification results;

[0050] Determine whether the number of links is greater than a preset threshold;

[0051] If the value is greater than the specified value, then the preset link pattern library is linked and merged according to the classification result.

[0052] Optionally, the step of determining whether to perform deduplication on the formatted links based on the matching results includes:

[0053] If the matching result is a failure, the formatted link is added to the preset link pattern library.

[0054] Optionally, the step of determining whether to perform deduplication on the formatted links based on the matching results includes:

[0055] If the matching result is successful, then the formatted link is deduplicated.

[0056] Furthermore, to achieve the above objectives, the present invention also provides a link deduplication device, the device comprising:

[0057] The acquisition module is used to acquire the link to be downloaded, format the link to be downloaded, and obtain a formatted link.

[0058] The matching module is used to match the formatted link with the link templates stored in the preset link pattern library to obtain the matching result;

[0059] The deduplication module is used to determine whether to perform deduplication on the formatted links based on the matching results.

[0060] Optionally, the acquisition module is further configured to acquire a link to be processed from a link history library; identify invalid parameters in the link to be processed to obtain an invalid parameter identification result; when the invalid parameter identification result contains a parameter to be verified, perform dynamic parameter identification on the link to be processed based on the invalid parameter identification result to obtain a dynamic parameter identification result; and generate a preset link pattern library based on the dynamic parameter identification result and the link to be processed.

[0061] Optionally, the acquisition module is further configured to determine the valid parameters of the link to be processed based on the dynamic parameter identification result; generate a target link based on the valid parameters and the link to be processed; and construct a preset link pattern library based on the target link.

[0062] Optionally, the acquisition module is further configured to acquire the link resource file corresponding to the link to be processed; delete parameters for each link to be processed by deleting one parameter at a time to obtain the link to be verified; after each parameter deletion, acquire the link resource file to be verified for the corresponding link to be verified; compare the link resource file and the link resource file to be verified to obtain a comparison result; and generate an invalid parameter identification result based on the comparison result.

[0063] Optionally, the acquisition module is further configured to determine that the currently deleted parameter is an invalid parameter if the comparison result is consistent, and to determine that the currently deleted parameter is a parameter to be verified if the comparison result is inconsistent.

[0064] Optionally, the acquisition module is further configured to determine the target hash value corresponding to the linked resource file using a preset algorithm; determine the verification hash value corresponding to the linked resource file to be verified using the preset algorithm; and compare the target hash value and the verification hash value to obtain a comparison result.

[0065] Optionally, the acquisition module is further configured to determine the parameters to be verified in the link to be processed based on the invalid parameter identification result; perform format verification on the parameters to be verified to obtain the format verification result; and determine the dynamic parameter identification result corresponding to the link to be processed based on the format verification result.

[0066] Optionally, the acquisition module is further configured to acquire a preset dynamic parameter format; determine the parameter format corresponding to the parameter to be verified; determine whether the parameter format belongs to the preset dynamic parameter format, and obtain a format verification result.

[0067] Optionally, the acquisition module is further configured to acquire a link to be downloaded, delete invalid characters from the link to be downloaded to obtain a valid link to be downloaded; adjust the order of parameters in the valid link to be downloaded according to the ASCII code table to obtain an adjusted link to be downloaded; and convert the adjusted link to be downloaded into a preset format to obtain a formatted link to be downloaded.

[0068] Optionally, the acquisition module is further configured to acquire link templates in the preset link pattern library; perform similarity detection on the link templates to obtain similarity detection results; classify the link templates according to the similarity detection results to obtain classification results; and optimize the preset link pattern library according to the classification results.

[0069] Optionally, the acquisition module is further configured to determine the number of links in each type of pattern based on the classification result; determine whether the number of links is greater than a preset threshold; if it is greater, then merge the links in the preset link pattern library based on the classification result.

[0070] Optionally, the deduplication module is further configured to add the formatted link to a preset link pattern library if the matching result is a match failure.

[0071] Optionally, the deduplication module is further configured to perform deduplication on the formatted link if the matching result is a successful match.

[0072] Furthermore, to achieve the above objectives, the present invention also proposes a link deduplication device, the device comprising: a memory, a processor, and a link deduplication program stored in the memory and executable on the processor, the link deduplication program being configured to implement the steps of the link deduplication method as described above.

[0073] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a link deduplication program, which, when executed by a processor, implements the steps of the link deduplication method described above.

[0074] This invention obtains a link to be downloaded, formats the link to obtain a formatted link, matches the formatted link with link templates stored in a preset link pattern library to obtain a matching result, and determines whether to perform a deduplication operation on the formatted link based on the matching result. Because this invention matches the formatted link with link templates stored in a preset link pattern library to obtain a matching result, and then determines whether to perform a deduplication operation on the formatted link based on the matching result, compared to existing methods that directly download resources upon receiving a download link, this invention can perform a deduplication operation on the download link, reducing duplicate downloads and ensuring the quality of downloaded resources. Attached Figure Description

[0075] Figure 1 This is a schematic diagram of the structure of the deduplication device for the hardware operating environment involved in the embodiments of the present invention;

[0076] Figure 2 This is a flowchart illustrating the first embodiment of the link deduplication method of the present invention;

[0077] Figure 3 This is a flowchart illustrating the second embodiment of the link deduplication method of the present invention;

[0078] Figure 4 This is a structural block diagram of the first embodiment of the link deduplication device of the present invention.

[0079] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0080] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0081] Reference Figure 1 , Figure 1 This is a schematic diagram of the link deduplication device structure of the hardware operating environment involved in the embodiment of the present invention.

[0082] like Figure 1 As shown, the link deduplication device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0083] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the link deduplication device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0084] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a link deduplication program.

[0085] exist Figure 1 In the link deduplication device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the link deduplication device of the present invention can be set in the link deduplication device, and the link deduplication device calls the link deduplication program stored in the memory 1005 through the processor 1001 and executes the link deduplication method provided in the embodiment of the present invention.

[0086] Based on the aforementioned link deduplication device, this embodiment of the invention provides a link deduplication method, referring to... Figure 2, Figure 2 This is a flowchart illustrating the first embodiment of the link deduplication method of the present invention.

[0087] In this embodiment, the link deduplication method includes the following steps:

[0088] Step S10: Obtain the link to be downloaded, format the link to be downloaded, and obtain the formatted link.

[0089] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a mobile phone, tablet computer, or personal computer, or an electronic device or link deduplication device capable of performing the above functions. The following description uses the link deduplication device as an example to illustrate this embodiment and the subsequent embodiments.

[0090] It should be noted that the link to be downloaded can be the link corresponding to the resource to be downloaded. Formatting the link to be downloaded can involve removing invalid parts of the link. For example, removing anchor text and invalid characters from the link.

[0091] Furthermore, to improve the efficiency of deduplication of the download links, step S10 may include: obtaining the download link; deleting invalid characters from the download link to obtain a valid download link; adjusting the order of parameters in the valid download link according to the ASCII code table to obtain an adjusted download link; and converting the adjusted download link into a preset format to obtain a formatted download link.

[0092] It should be noted that the invalid character removal from the download link can refer to deleting invalid characters such as "?", "," " / ", "、", and "#" from the download link. The order adjustment of parameters in the valid download link according to the ASCII code table can be achieved by sorting the parameters in the link according to the ASCII code table, ensuring that the order of parameters in the link does not affect the parameter recognition result.

[0093] Step S20: Match the formatted link with the link templates stored in the preset link pattern library to obtain the matching result.

[0094] It should be noted that the preset link pattern library can be a data table or file used to store link templates. No two link templates in the preset link pattern library are identical. Matching the formatted link with the link templates stored in the preset link pattern library can be done by determining whether a link template identical to the formatted link exists in the preset link pattern library.

[0095] Step S30: Determine whether to perform deduplication on the formatted links based on the matching results.

[0096] It should be noted that if a link template identical to the formatted link exists in the preset link pattern library, the formatted link is considered redundant and can be removed from the link download list. If no link template identical to the formatted link exists in the preset link pattern library, the formatted link is not redundant and should be downloaded.

[0097] Furthermore, to avoid downloading too many invalid resources due to redundant download links, thus occupying system storage space, step S30 may include: if the matching result is a failure, adding the formatted link to a preset link pattern library; if the matching result is a success, performing a deduplication operation on the formatted link.

[0098] It should be noted that a matching failure means that there is no link template in the preset link pattern library that is the same as the formatted link. This indicates that the formatted link is not a redundant link, and the formatted link is added to the preset link pattern library for download. A matching success means that there is a link template in the preset link pattern library that is the same as the formatted link. This indicates that the formatted link is a redundant link, and a deduplication operation is performed on the formatted link.

[0099] This embodiment obtains a link to be downloaded, formats the link to obtain a formatted link, matches the formatted link with link templates stored in a preset link pattern library to obtain a matching result, and determines whether to perform a deduplication operation on the formatted link based on the matching result. Since this embodiment matches the formatted link with link templates stored in a preset link pattern library to obtain a matching result, and determines whether to perform a deduplication operation on the formatted link based on the matching result, compared to the existing method of directly downloading resources upon receiving a download link, this embodiment can perform a deduplication operation on the download link, reducing duplicate downloads and ensuring the quality of downloaded resources.

[0100] refer to Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the link deduplication method of the present invention.

[0101] Based on the first embodiment described above, in this embodiment, before step S20, the method further includes:

[0102] Step S101: Obtain the links to be processed from the link history database.

[0103] It should be noted that the link history library can be a database or file storing links to be processed. The links to be processed can be links to file resources whose corresponding files need to be downloaded.

[0104] Step S102: Identify invalid parameters in the link to be processed and obtain invalid parameter identification results.

[0105] It should be noted that the invalid parameter identification of the link to be processed can be to identify whether the parameters in the link to be processed are invalid parameters. Specifically, it can be to delete parameters one by one for each link to be processed, and determine whether the link after the parameter deletion can be downloaded normally. If it can be downloaded normally, then the currently deleted parameter is determined to be an invalid parameter.

[0106] Furthermore, to reduce duplicate downloads of resources and ensure the quality of downloaded resources, step S102 may include: obtaining the link resource file corresponding to the link to be processed; deleting parameters from each link to be processed one parameter at a time to obtain a link to be verified; after each parameter deletion, obtaining the link resource file to be verified for the corresponding link to be verified; comparing the link resource file and the link resource file to be verified to obtain a comparison result; and generating an invalid parameter identification result based on the comparison result.

[0107] It should be noted that the link resource file can be the resource file obtained after downloading the link to be processed. The parameter deletion process for each link to be processed, one parameter at a time, can involve deleting each parameter from the link one by one, and then downloading the link after parameter deletion to obtain the link resource file to be verified. For example, if the link to be processed has parameters A, B, and C, parameter A is deleted first. The link to be processed after deleting parameter A is downloaded to obtain the link resource file to be verified. The link resource file and the link resource file to be verified are compared to obtain a comparison result. If the comparison result is consistent, it means that deleting the parameter has no impact on the downloaded resource, and the parameter is invalid. If the comparison result is inconsistent, it means that deleting the parameter affects the download of the resource file, but it may not be a useful parameter; it may be a dynamic parameter that requires further verification. In this case, the parameter is defined as a parameter to be verified. Therefore, the invalid parameter identification result can be whether each parameter is an invalid parameter or a parameter to be verified.

[0108] Furthermore, in order to improve the comparison efficiency between the linked resource file and the linked resource file to be verified, the step of comparing the linked resource file and the linked resource file to be verified to obtain a comparison result includes: determining the target hash value corresponding to the linked resource file through a preset algorithm; determining the verification hash value corresponding to the linked resource file to be verified through the preset algorithm; and comparing the target hash value and the verification hash value to obtain a comparison result.

[0109] It should be noted that the preset algorithm can be the Message-Digest Algorithm (MD5). MD5 is a widely used cryptographic hash function that produces a 128-bit (16-byte) hash value to ensure the integrity and consistency of transmitted information. By performing hash operations on the linked resource file and the linked resource file to be verified according to the MD5 algorithm, the comparison efficiency between the linked resource file and the linked resource file to be verified can be improved.

[0110] Step S103: When the invalid parameter identification result contains a parameter to be verified, perform dynamic parameter identification on the link to be processed based on the invalid parameter identification result to obtain the dynamic parameter identification result.

[0111] It should be noted that when the invalid parameter identification result includes a parameter to be verified, deleting the parameter will have a certain impact on the download result. However, this does not mean that the parameter is a valid parameter. It may be a dynamic parameter. Therefore, when the invalid parameter identification result includes a parameter to be verified, it is necessary to perform dynamic parameter identification on the link to be processed based on the invalid parameter identification result. Specifically, it can be to determine whether the parameter to be verified is a dynamic parameter.

[0112] Furthermore, in order to reduce duplicate downloads of resources and ensure the quality of downloaded resources, step S103 may include: determining the parameters to be verified in the link to be processed based on the invalid parameter identification result; performing format verification on the parameters to be verified to obtain the format verification result; and determining the dynamic parameter identification result corresponding to the link to be processed based on the format verification result.

[0113] It should be noted that common dynamic parameters include time, random strings, and dynamic tokens. Dynamic parameters generally have specific format information. For example, time parameters typically conform to the format "Year Month Day," and dynamic tokens are generally long hexadecimal strings, along with combinations of fixed letters or lengths. Dynamic parameter identification is performed by sequentially judging whether the parameter to be verified meets the preset format. If the parameter to be verified meets any of the above formats, it is determined to be a dynamic parameter, i.e., an invalid parameter. If the parameter to be verified does not meet any of the preset dynamic parameter formats, it is determined to be a valid parameter.

[0114] Furthermore, in order to obtain accurate dynamic parameter verification results, the step of performing format verification on the parameter to be verified and obtaining format verification results includes: obtaining a preset dynamic parameter format; determining the parameter format corresponding to the parameter to be verified; determining whether the parameter format belongs to the preset dynamic parameter format, and obtaining a format verification result.

[0115] It should be noted that the preset dynamic parameter format can be any format that the pre-set dynamic parameters may satisfy.

[0116] Step S104: Generate a preset link pattern library based on the dynamic parameter identification results and the links to be processed.

[0117] It should be noted that the step of generating a preset link pattern library based on the dynamic parameter identification result and the link to be processed can be to generate a target link based on the valid parameters in the dynamic parameter identification result and the protocol part, domain name part, port part, file name part and / or virtual directory part in the link to be processed, and then construct the preset link pattern library based on the target link.

[0118] Furthermore, in order to reduce duplicate downloads of resources and ensure the quality of downloaded resources, step S104 may include: determining the valid parameters of the link to be processed based on the dynamic parameter identification result; generating a target link based on the valid parameters and the link to be processed; and constructing a preset link pattern library based on the target link.

[0119] It should be noted that the effective parameters can be non-dynamic parameters that have been identified and determined through the dynamic parameter identification process.

[0120] Furthermore, to improve the matching efficiency of matching formatted links with link templates stored in the preset link pattern library, after step S104, the method may further include: obtaining link templates from the preset link pattern library; performing similarity detection on the link templates to obtain similarity detection results; classifying the link templates according to the similarity detection results to obtain classification results; and optimizing the preset link pattern library according to the classification results.

[0121] It should be noted that the similarity detection of the link templates can be used to determine whether there are link templates that differ only in a single parameter or path. If so, these link templates with only a single parameter or path difference can be grouped into one category. The preset link pattern library is then optimized by classifying the link templates in the preset link pattern library. The step of optimizing the preset link pattern library based on the classification results includes: determining the number of links in each type of pattern based on the classification results; determining whether the number of links is greater than a preset threshold; if it is greater, then merging the links in the preset link pattern library based on the classification results.

[0122] It should be noted that the preset threshold can be a pre-set threshold for the number of link templates in a certain type of link template. That is, if the number of link templates in a certain category exceeds the preset threshold, the preset link pattern library can be merged according to the classification result. Specifically, it can be that the different parameters or paths in the link templates in the classification result are represented by variables, and then the link templates in the category are merged.

[0123] This embodiment retrieves links to be processed from a link history database; identifies invalid parameters in the links to be processed, obtaining invalid parameter identification results; when the invalid parameter identification results contain parameters to be verified, performs dynamic parameter identification on the links to be processed based on the invalid parameter identification results, obtaining dynamic parameter identification results; and generates a preset link pattern library based on the dynamic parameter identification results and the links to be processed. This embodiment retrieves links to be processed from a link history database; identifies invalid parameters in the links to be processed; when the invalid parameter identification results contain parameters to be verified, performs dynamic parameter identification on the links to be processed, and then generates a preset link pattern library based on the dynamic parameter identification results and the links to be processed. This allows subsequent downloads to be deduplicated based on the preset link pattern library, reducing redundant downloads, ensuring the quality of downloaded resources, and improving user experience.

[0124] Reference Figure 4 , Figure 4 This is a structural block diagram of the first embodiment of the link deduplication device of the present invention.

[0125] like Figure 4 As shown, the link deduplication device proposed in this embodiment of the invention includes:

[0126] The acquisition module 10 is used to acquire the link to be downloaded, format the link to be downloaded, and obtain a formatted link;

[0127] Matching module 20 is used to match the formatted link with the link templates stored in the preset link pattern library to obtain the matching result;

[0128] The deduplication module 30 is used to determine whether to perform deduplication on the formatted links based on the matching results.

[0129] This embodiment obtains a link to be downloaded, formats the link to obtain a formatted link, matches the formatted link with link templates stored in a preset link pattern library to obtain a matching result, and determines whether to perform a deduplication operation on the formatted link based on the matching result. Since this embodiment matches the formatted link with link templates stored in a preset link pattern library to obtain a matching result, and determines whether to perform a deduplication operation on the formatted link based on the matching result, compared to the existing method of directly downloading resources upon receiving a download link, this embodiment can perform a deduplication operation on the download link, reducing duplicate downloads and ensuring the quality of downloaded resources.

[0130] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0131] In addition, for technical details not described in detail in this embodiment, please refer to the link deduplication method provided in any embodiment of the present invention, which will not be repeated here.

[0132] Based on the first embodiment of the link deduplication device of the present invention described above, a second embodiment of the link deduplication device of the present invention is proposed.

[0133] In this embodiment, the acquisition module 10 is further configured to acquire a link to be processed from the link history library; identify invalid parameters in the link to be processed to obtain an invalid parameter identification result; when the invalid parameter identification result contains a parameter to be verified, perform dynamic parameter identification on the link to be processed according to the invalid parameter identification result to obtain a dynamic parameter identification result; and generate a preset link pattern library according to the dynamic parameter identification result and the link to be processed.

[0134] Furthermore, the acquisition module 10 is also used to determine the valid parameters of the link to be processed based on the dynamic parameter identification result; generate a target link based on the valid parameters and the link to be processed; and construct a preset link pattern library based on the target link.

[0135] Furthermore, the acquisition module 10 is also used to acquire the link resource file corresponding to the link to be processed; to delete parameters for each link to be processed by deleting one parameter at a time to obtain the link to be verified; after each parameter deletion, to acquire the link resource file to be verified for the corresponding link to be verified; to compare the link resource file and the link resource file to be verified to obtain a comparison result; and to generate an invalid parameter identification result based on the comparison result.

[0136] Furthermore, the acquisition module 10 is also used to determine that the currently deleted parameter is an invalid parameter if the comparison result is consistent; and to determine that the currently deleted parameter is a parameter to be verified if the comparison result is inconsistent.

[0137] Furthermore, the acquisition module 10 is also used to determine the target hash value corresponding to the linked resource file through a preset algorithm; determine the verification hash value corresponding to the linked resource file to be verified through the preset algorithm; and compare the target hash value and the verification hash value to obtain a comparison result.

[0138] Furthermore, the acquisition module 10 is also used to determine the parameters to be verified in the link to be processed based on the invalid parameter identification result; to perform format verification on the parameters to be verified to obtain the format verification result; and to determine the dynamic parameter identification result corresponding to the link to be processed based on the format verification result.

[0139] Furthermore, the acquisition module 10 is also used to acquire a preset dynamic parameter format; determine the parameter format corresponding to the parameter to be verified; determine whether the parameter format belongs to the preset dynamic parameter format, and obtain a format verification result.

[0140] Furthermore, the acquisition module 10 is also used to acquire a link to be downloaded, delete invalid characters from the link to be downloaded to obtain a valid link to be downloaded; adjust the order of parameters in the valid link to be downloaded according to the ASCII code table to obtain an adjusted link to be downloaded; and convert the adjusted link to be downloaded into a preset format to obtain a formatted link to be downloaded.

[0141] Furthermore, the acquisition module 10 is also used to acquire link templates in the preset link pattern library; perform similarity detection on the link templates to obtain similarity detection results; classify the link templates according to the similarity detection results to obtain classification results; and optimize the preset link pattern library according to the classification results.

[0142] Furthermore, the acquisition module 10 is also used to determine the number of links in each type of pattern according to the classification result; determine whether the number of links is greater than a preset threshold; if it is greater, then merge the links in the preset link pattern library according to the classification result.

[0143] Furthermore, the deduplication module 30 is also used to add the formatted link to a preset link pattern library if the matching result is a matching failure.

[0144] Furthermore, the deduplication module 30 is also used to perform deduplication on the formatted link if the matching result is a successful match.

[0145] Other embodiments or specific implementations of the deduplication device of the present invention can be found in the above-described method embodiments, and will not be repeated here.

[0146] Furthermore, embodiments of the present invention also propose a storage medium storing a link deduplication program, which, when executed by a processor, implements the steps of the link deduplication method described above.

[0147] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0148] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0150] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

[0151] The present invention also discloses A1, a link deduplication method, the link deduplication method comprising the following steps:

[0152] Obtain the download link, format the download link, and obtain the formatted link;

[0153] The formatted link is matched with the link templates stored in the preset link pattern library to obtain the matching result;

[0154] Determine whether to perform deduplication on the formatted links based on the matching results.

[0155] A2. The link deduplication method as described in A1, before the step of matching the formatted link with the link templates stored in the preset link pattern library to obtain the matching result, further includes:

[0156] Retrieve links to be processed from the link history database;

[0157] The link to be processed is subjected to invalid parameter identification to obtain invalid parameter identification results;

[0158] When the invalid parameter identification result contains a parameter to be verified, dynamic parameter identification is performed on the link to be processed based on the invalid parameter identification result to obtain the dynamic parameter identification result.

[0159] A preset link pattern library is generated based on the dynamic parameter identification results and the links to be processed.

[0160] A3. The link deduplication method as described in A2, wherein the step of generating a preset link pattern library based on the dynamic parameter identification result and the links to be processed includes:

[0161] The valid parameters of the link to be processed are determined based on the dynamic parameter identification results.

[0162] Generate a target link based on the valid parameters and the link to be processed;

[0163] A preset link pattern library is constructed based on the target link.

[0164] A4. The link deduplication method as described in A2, wherein the step of identifying invalid parameters in the link to be processed and obtaining the invalid parameter identification result includes:

[0165] Obtain the link resource file corresponding to the link to be processed;

[0166] For each link to be processed, parameters are deleted one at a time to obtain the link to be verified.

[0167] After each parameter deletion, obtain the corresponding link resource file to be verified.

[0168] The link resource file and the link resource file to be verified are compared to obtain the comparison result;

[0169] Invalid parameter identification results are generated based on the comparison results.

[0170] A5. The link deduplication method as described in A4, wherein the step of generating invalid parameter identification results based on the comparison results includes:

[0171] If the comparison result is consistent, then the parameter to be deleted is determined to be invalid.

[0172] If the comparison result is inconsistent, then the parameter currently being deleted is determined to be a parameter to be verified.

[0173] A6. The link deduplication method as described in A4, wherein the step of comparing the link resource file and the link resource file to be verified to obtain the comparison result includes:

[0174] The target hash value corresponding to the linked resource file is determined by a preset algorithm;

[0175] The verification hash value corresponding to the link resource file to be verified is determined by the preset algorithm.

[0176] The target hash value and the verification hash value are compared to obtain the comparison result.

[0177] A7. The link deduplication method as described in A5, wherein the step of performing dynamic parameter identification on the link to be processed based on the invalid parameter identification result to obtain the dynamic parameter identification result includes:

[0178] Based on the invalid parameter identification results, determine the parameters to be verified in the link to be processed;

[0179] Perform format verification on the parameter to be verified and obtain the format verification result;

[0180] The dynamic parameter identification result corresponding to the link to be processed is determined based on the format verification result.

[0181] A8. The link deduplication method as described in A7, wherein the step of performing format verification on the parameter to be verified and obtaining the format verification result includes:

[0182] Get the preset dynamic parameter format;

[0183] Determine the parameter format corresponding to the parameter to be verified;

[0184] Determine whether the parameter format belongs to the preset dynamic parameter format, and obtain the format verification result.

[0185] A9. The link deduplication method as described in A1, wherein the steps of obtaining the link to be downloaded and formatting the link to be downloaded to obtain the formatted link include:

[0186] Obtain the download link, remove invalid characters from the download link to obtain a valid download link;

[0187] The parameters in the valid download link are adjusted according to the ASCII code table to obtain the adjusted download link.

[0188] The adjusted download link is converted into a preset format to obtain a formatted download link.

[0189] A10. The link deduplication method as described in any one of A2-A8, after the step of generating a preset link pattern library based on the dynamic parameter identification result and the links to be processed, further includes:

[0190] Obtain the link template from the preset link pattern library;

[0191] The link template is subjected to similarity detection to obtain similarity detection results;

[0192] The link templates are classified based on the similarity detection results to obtain classification results;

[0193] The preset link pattern library is optimized based on the classification results.

[0194] A11. The link deduplication method as described in A10, wherein the step of optimizing the preset link pattern library based on the classification result includes:

[0195] The number of links in each type of pattern is determined based on the classification results;

[0196] Determine whether the number of links is greater than a preset threshold;

[0197] If the value is greater than the specified value, then the preset link pattern library is linked and merged according to the classification result.

[0198] A12. The link deduplication method as described in any one of A1-A8, wherein the step of determining whether to perform deduplication on the formatted links based on the matching result includes:

[0199] If the matching result is a failure, the formatted link is added to the preset link pattern library.

[0200] A13. The link deduplication method as described in any one of A1-A8, wherein the step of determining whether to perform deduplication on the formatted links based on the matching result includes:

[0201] If the matching result is successful, then the formatted link is deduplicated.

[0202] The present invention also discloses B14, a link deduplication device, the link deduplication device comprising:

[0203] The acquisition module is used to acquire the link to be downloaded, format the link to be downloaded, and obtain a formatted link.

[0204] The matching module is used to match the formatted link with the link templates stored in the preset link pattern library to obtain the matching result;

[0205] The deduplication module is used to determine whether to perform deduplication on the formatted links based on the matching results.

[0206] B15. The link deduplication device as described in B14, wherein the acquisition module is further configured to acquire a link to be processed from a link history library; perform invalid parameter identification on the link to be processed to obtain an invalid parameter identification result; when the invalid parameter identification result contains a parameter to be verified, perform dynamic parameter identification on the link to be processed according to the invalid parameter identification result to obtain a dynamic parameter identification result; and generate a preset link pattern library according to the dynamic parameter identification result and the link to be processed.

[0207] B16. The link deduplication device as described in B15, wherein the acquisition module is further configured to determine the effective parameters of the link to be processed based on the dynamic parameter identification result; generate a target link based on the effective parameters and the link to be processed; and construct a preset link pattern library based on the target link.

[0208] B17. The link deduplication device as described in B15, wherein the acquisition module is further configured to acquire the link resource file corresponding to the link to be processed; delete parameters for each link to be processed by deleting one parameter at a time to obtain the link to be verified; after each parameter deletion, acquire the link resource file to be verified for the corresponding link to be verified; compare the link resource file and the link resource file to be verified to obtain a comparison result; and generate an invalid parameter identification result based on the comparison result.

[0209] B18. The link deduplication device as described in B17, wherein the acquisition module is further configured to determine that the currently deleted parameter is an invalid parameter if the comparison result is consistent; and to determine that the currently deleted parameter is a parameter to be verified if the comparison result is inconsistent.

[0210] The present invention also discloses C19, a link deduplication device, the device comprising: a memory, a processor, and a link deduplication program stored in the memory and executable on the processor, the link deduplication program being configured to implement the steps of the link deduplication method as described in any one of A1 to A13.

[0211] The present invention also discloses D20, a storage medium storing a link deduplication program, wherein when the link deduplication program is executed by a processor, it implements the steps of the link deduplication method as described in any one of A1 to A13.

Claims

1. A method for deduplicating links, characterized in that, The link deduplication method includes the following steps: Obtain the download link, format the download link, and obtain the formatted link; The formatted link is matched with the link templates stored in the preset link pattern library to obtain the matching result; Determine whether to perform deduplication on the formatted links based on the matching results; The step of obtaining the link to be downloaded and formatting the link to obtain a formatted link includes: Remove invalid characters from the download links to obtain valid download links; The parameters in the valid download link are rearranged according to the ASCII code table to obtain a formatted link; Before matching the formatted link with the link templates stored in the preset link pattern library to obtain the matching result, the following steps are included: Perform similarity detection on link templates in the preset link pattern library and obtain similarity detection results; The link templates are classified based on the similarity detection results to obtain classification results; The link templates in the classification results are merged to obtain a preset link pattern library after template merging, and the preset link pattern library after template merging is used as the preset link pattern library.

2. The link deduplication method as described in claim 1, characterized in that, Before the step of matching the formatted link with link templates stored in a preset link pattern library to obtain a matching result, the method further includes: Retrieve the links to be processed from the link history database; The link to be processed is subjected to invalid parameter identification to obtain invalid parameter identification results; When the invalid parameter identification result contains a parameter to be verified, dynamic parameter identification is performed on the link to be processed based on the invalid parameter identification result to obtain the dynamic parameter identification result. A preset link pattern library is generated based on the dynamic parameter identification results and the links to be processed.

3. The link deduplication method as described in claim 2, characterized in that, The step of generating a preset link pattern library based on the dynamic parameter identification result and the link to be processed includes: The valid parameters of the link to be processed are determined based on the dynamic parameter identification results. Generate a target link based on the valid parameters and the link to be processed; A preset link pattern library is constructed based on the target link.

4. The link deduplication method as described in claim 2, characterized in that, The step of identifying invalid parameters in the link to be processed and obtaining the invalid parameter identification result includes: Obtain the link resource file corresponding to the link to be processed; For each link to be processed, parameters are deleted one at a time to obtain the link to be verified. After each parameter deletion, obtain the corresponding link resource file to be verified. The link resource file and the link resource file to be verified are compared to obtain the comparison result; Invalid parameter identification results are generated based on the comparison results.

5. The link deduplication method as described in claim 4, characterized in that, The step of generating invalid parameter identification results based on the comparison results includes: If the comparison result is consistent, then the parameter to be deleted is determined to be invalid. If the comparison result is inconsistent, then the parameter currently being deleted is determined to be a parameter to be verified.

6. The link deduplication method as described in claim 4, characterized in that, The step of comparing the linked resource file and the linked resource file to be verified to obtain a comparison result includes: The target hash value corresponding to the linked resource file is determined by a preset algorithm; The verification hash value corresponding to the link resource file to be verified is determined by the preset algorithm. The target hash value and the verification hash value are compared to obtain the comparison result.

7. The link deduplication method as described in claim 5, characterized in that, The step of performing dynamic parameter identification on the link to be processed based on the invalid parameter identification result to obtain the dynamic parameter identification result includes: Based on the invalid parameter identification results, determine the parameters to be verified in the link to be processed; Perform format verification on the parameter to be verified and obtain the format verification result; The dynamic parameter identification result corresponding to the link to be processed is determined based on the format verification result.

8. A link deduplication device, characterized in that, The link deduplication device includes: The acquisition module is used to acquire the link to be downloaded, format the link to be downloaded, and obtain a formatted link. The matching module is used to match the formatted link with the link templates stored in the preset link pattern library to obtain the matching result; The deduplication module is used to determine whether to perform deduplication on the formatted links based on the matching results. The step of obtaining the link to be downloaded and formatting the link to obtain a formatted link includes: Remove invalid characters from the download links to obtain valid download links; The parameters in the valid download link are rearranged according to the ASCII code table to obtain a formatted link; Before matching the formatted link with the link templates stored in the preset link pattern library to obtain the matching result, the following steps are included: Perform similarity detection on link templates in the preset link pattern library and obtain similarity detection results; The link templates are classified based on the similarity detection results to obtain classification results; The link templates in the classification results are merged to obtain a preset link pattern library after template merging, and the preset link pattern library after template merging is used as the preset link pattern library.

9. A link deduplication device, characterized in that, The device includes: a memory, a processor, and a link deduplication program stored in the memory and executable on the processor, the link deduplication program being configured to implement the steps of the link deduplication method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a link deduplication program, which, when executed by a processor, implements the steps of the link deduplication method as described in any one of claims 1 to 7.

Images