Crawler data deduplication method and related device

Abstract

Embodiments of the present application provide a crawler data filter duplication method and related equipment. The crawler data filter duplication method comprises: determining a crawling type of a target source according to configuration information of the target source; if the crawling type is a file type, performing multiple filter duplications according to download information in a target source file; and if the crawling type is a page type, performing multiple filter duplications according to page information in a target source page. The technical solution of the embodiments of the present application performs multiple filter duplications on target source data of different crawling types by using different specific filter duplication strategies, so that more duplicate data can be filtered out more specifically in the filter duplication process, the time and resource consumption of duplicate crawling are reduced, the filter duplication efficiency of the crawler is improved, the crawler is more efficient in the execution process, the filter duplication of the crawler is more accurate, and the problem of poor filter duplication effect of the filter duplication method is solved.

Description

Technical Field

[0001] This application relates to the fields of computer and communication technology, and more specifically, to a method and related equipment for deduplication of web crawler data. Background Technology

[0002] Before actually performing web crawling, filtering for duplicate pages is crucial. This process can significantly improve crawling efficiency and reduce wasted storage space. However, existing methods for filtering crawled data often rely on simple deduplication based on the target source's URL, which is ineffective. Summary of the Invention

[0003] The embodiments of this application provide a method and related equipment for deduplication of crawler data, which can at least to some extent overcome the problem of poor deduplication effect of existing deduplication methods.

[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0005] According to one aspect of the embodiments of this application, a method for deduplication filtering of crawler data is provided, comprising: determining the crawling type of the target source based on the configuration information of the target source; if the crawling type is a file type, performing multiple deduplication based on the download information in the target source file; if the crawling type is a page type, performing multiple deduplication based on the page information in the target source page.

[0006] In some feasible embodiments of this application, the step of performing multiple deduplication based on the download information in the target source file specifically includes: performing preliminary deduplication using the details page URL of the target source to obtain preliminary deduplication data; and performing further deduplication on the preliminary deduplication data based on the download information in the target source file to obtain final deduplication data.

[0007] In some feasible embodiments of this application, the step of further filtering the preliminary deduplication data based on the download information in the target source file to obtain the final deduplication data specifically includes: filtering the preliminary deduplication data based on the request header information to obtain the requested deduplication data; and filtering the requested deduplication data based on the download information to obtain the final deduplication data.

[0008] In some feasible embodiments of this application, the step of filtering the preliminary deduplication data according to the request header information to obtain the requested deduplication data specifically includes: parsing the request header information to obtain request parameters; and filtering the preliminary deduplication data according to the request parameters to obtain the requested deduplication data.

[0009] In some feasible embodiments of this application, the step of filtering the requested deduplication data according to the download information to obtain the final deduplication data specifically includes: downloading the target source file according to the download information; determining the feature value of the target source file; and performing a repeatability comparison on the requested deduplication data according to the feature value of the target source file to obtain the final deduplication data.

[0010] In some feasible embodiments of this application, the step of performing multiple deduplication based on page information in the target source page specifically includes: performing preliminary deduplication using the details page URL of the target source to obtain preliminary deduplication data; and performing further deduplication on the preliminary deduplication data based on the page information in the target source page to obtain final deduplication data.

[0011] In some feasible embodiments of this application, the page information includes overall page information and page content information. The step of further filtering the preliminary deduplication data based on the page information in the target source page to obtain the final deduplication data specifically includes: filtering the preliminary deduplication data based on the overall page information of the target source page to obtain overall deduplication data; and filtering the overall deduplication data based on the page content information of the target source page to obtain the final deduplication data.

[0012] In some feasible embodiments of this application, the overall page information includes a page version number and an overall page feature value. The step of filtering the preliminary deduplication data based on the overall page information of the target source page to obtain overall deduplication data specifically includes: performing version deduplication on the preliminary deduplication data based on the page version number to obtain version deduplication data; and performing a repeatability comparison on the version deduplication data based on the overall page feature value to obtain overall deduplication data.

[0013] In some feasible embodiments of this application, the page content information includes text time information and page content feature values. The step of filtering the overall deduplication data based on the page content information of the target source page to obtain the final deduplication data specifically includes: determining the update data of the current crawling period based on the text time information; and performing a repeatability comparison on the update data based on the page content feature values ​​to obtain the final deduplication data.

[0014] In some feasible embodiments of this application, the step of using the detail page URL of the target source to perform preliminary deduplication and obtain preliminary deduplication data specifically includes: filtering the detail page URL of the target source for interference items to obtain a unique URL; and comparing the unique URL with the detail page URL in the crawler database to obtain preliminary deduplication data.

[0015] According to one aspect of the embodiments of this application, a web crawler data deduplication device is provided, the web crawler data deduplication device comprising: a type determination module, configured to determine the crawling type of the target source according to the configuration information of the target source; a first deduplication module, configured to perform multiple deduplication based on download information in the target source file if the crawling type is a file type; and a second deduplication module, configured to perform multiple deduplication based on page information in the target source page if the crawling type is a page type.

[0016] In some feasible embodiments of this application, the first deduplication module specifically includes: a preliminary deduplication submodule, used to perform preliminary deduplication using the details page URL of the target source to obtain preliminary deduplication data; and a file deduplication submodule, used to perform further deduplication on the preliminary deduplication data based on the download information in the target source file to obtain final deduplication data.

[0017] In some feasible embodiments of this application, the file deduplication submodule specifically includes: a request deduplication unit, used to deduplication the preliminary deduplication data according to the request header information to obtain request deduplication data; and a download deduplication unit, used to deduplication the request deduplication data according to the download information to obtain final deduplication data.

[0018] In some feasible embodiments of this application, the request deduplication unit is specifically used to perform: parsing the request header information to obtain request parameters; and deduplicating the preliminary deduplication data according to the request parameters to obtain request deduplication data.

[0019] In some feasible embodiments of this application, the download deduplication unit is specifically used to perform: downloading the target source file according to the download information; determining the feature value of the target source file; and performing a repeatability comparison on the requested deduplication data according to the feature value of the target source file to obtain the final deduplication data.

[0020] In some feasible embodiments of this application, the second deduplication module specifically includes: a preliminary deduplication submodule, used to perform preliminary deduplication using the details page URL of the target source to obtain preliminary deduplication data; and a page deduplication submodule, used to perform further deduplication on the preliminary deduplication data based on the page information in the target source page to obtain final deduplication data.

[0021] In some feasible embodiments of this application, the page information includes overall page information and page content information, and the page deduplication submodule specifically includes: an overall deduplication unit, used to deduplication the preliminary deduplication data according to the overall page information of the target source page to obtain overall deduplication data; and a content deduplication unit, used to deduplication the overall deduplication data according to the page content information of the target source page to obtain final deduplication data.

[0022] In some feasible embodiments of this application, the overall page information includes a page version number and an overall page feature value. The overall deduplication unit is specifically used to perform: performing version deduplication on the preliminary deduplication data according to the page version number to obtain version deduplication data; and performing a repeatability comparison on the version deduplication data according to the overall page feature value to obtain overall deduplication data.

[0023] In some feasible embodiments of this application, the page content information includes text time information and page content feature values, and the content deduplication unit is specifically used to perform: determining the update data of the current crawling cycle based on the text time information; and performing a duplication comparison on the update data based on the page content feature values ​​to obtain the final deduplication data.

[0024] In some feasible embodiments of this application, the preliminary deduplication submodule specifically includes: filtering interference items from the detail page URL of the target source to obtain a unique URL; and comparing the unique URL with the detail page URL in the crawler database to obtain preliminary deduplication data.

[0025] According to one aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the crawler data deduplication method as described in the above embodiments.

[0026] According to one aspect of the embodiments of this application, an electronic device is provided, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the web crawler data deduplication method as described in the above embodiments.

[0027] A computer program product includes one or more computer programs, characterized in that, when the one or more computer programs are executed by one or more processors, they implement the steps of the crawler data deduplication method as described in the above embodiments.

[0028] In some embodiments of this application, the technical solutions employ different specific deduplication strategies for target source data of different crawling types, enabling more targeted filtering of duplicate data during the deduplication process. This reduces the time and resource consumption of repeated crawling, improves the deduplication efficiency of the crawler, makes the crawler more efficient during execution, and makes deduplication filtering more accurate, thus solving the problem of poor deduplication effect of existing deduplication methods. Furthermore, the final result obtained after deduplication has less duplicate data compared to existing technologies, reducing unnecessary resource consumption and storage space occupation, alleviating the burden of storage management, and thereby improving overall performance.

[0029] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0031] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown.

[0032] Figure 2 The diagram shows a flowchart of a web crawler data deduplication method provided in an embodiment of this application.

[0033] Figure 3 It shows that according to Figure 2 A flowchart illustrating a specific implementation of step S200 in the crawler data deduplication method shown in the corresponding embodiment.

[0034] Figure 4 It shows that according to Figure 3 A flowchart illustrating a specific implementation of step S220 in the crawler data deduplication method shown in the corresponding embodiment.

[0035] Figure 5 It shows that according to Figure 2 A flowchart illustrating a specific implementation of step S300 in the web crawler data deduplication method shown in the corresponding embodiment.

[0036] Figure 6 It shows that according to Figure 5 The flowchart shown is a specific implementation of step S320 in the crawler data deduplication method.

[0037] Figure 7 A schematic diagram of a web crawler data deduplication device provided in an embodiment of this application is shown.

[0038] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0039] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0040] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0041] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0042] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0043] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown.

[0044] like Figure 1 As shown, the system architecture may include terminal devices (such as...) Figure 1 The device shown includes one or more of a smartphone 101, tablet 102, and portable computer 103 (which could also be a desktop computer, etc.), a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal device and the server 105. The network 104 can include various connection types, such as wired communication links, wireless communication links, etc.

[0045] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers. For example, server 105 could be a server cluster composed of multiple servers.

[0046] Users can use terminal devices to interact with server 105 via network 104 to receive or send messages, etc. Server 105 can be a server that provides various services. For example, a user uses terminal device 103 (or terminal device 101 or 102) to upload configuration information of a target source to server 105. Server 105 can determine the crawling type of the target source based on the configuration information of the target source; if the crawling type is a file, multiple deduplication is performed based on the download information in the target source file; if the crawling type is a page, multiple deduplication is performed based on the page information in the target source page.

[0047] It should be noted that the web crawler data deduplication method provided in this application embodiment is generally executed by server 105, and correspondingly, the web crawler data deduplication device is generally set in server 105. However, in other embodiments of this application, the terminal device may also have similar functions to the server, thereby executing the web crawler data deduplication scheme provided in this application embodiment.

[0048] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0049] Figure 2 A flowchart of a web crawler data deduplication method according to an embodiment of this application is shown. This web crawler data deduplication method can be executed by a server, which can be... Figure 1 The server shown. (Refer to...) Figure 2 As shown, this web crawler data deduplication method includes at least the following:

[0050] S100, determine the crawling type of the target source based on the configuration information of the target source.

[0051] S200, if the capture type is a file type, then multiple deduplication is performed based on the download information in the target source file.

[0052] S300, if the crawling type is page type, then multiple deduplication is performed based on the page information in the target source page.

[0053] In some embodiments of this application, multiple deduplication strategies are applied to target source data of different crawling types to achieve more targeted filtering of duplicate data. This reduces the time and resource consumption of repeated crawling, improves the deduplication efficiency of the crawler, makes the crawler more efficient during execution, and makes deduplication filtering more accurate, thus solving the problem of poor deduplication effect of existing deduplication methods. Furthermore, the final result obtained after deduplication has less duplicate data compared to existing technologies, reducing unnecessary resource consumption and storage space occupation, alleviating the burden of storage management, and thereby improving overall performance.

[0054] In S100, the fetch types include file type and page type.

[0055] For a target source that is a file, the crawler's target is the file. The crawler needs to download the file and store it in its crawler database. For file-type deduplication, the main purpose is to filter out duplicate files.

[0056] For target sources of type page, the crawler's crawling target is the page content. The crawler needs to extract the page content and store it in its crawler database. For file-type deduplication, the main task is to filter out duplicate page content.

[0057] Configuration information may include format information, access information, and request information.

[0058] Target sources that are crawled as pages mainly exist in the form of XML, HTML, PHP, and JSP, while target sources that are crawled as files mainly exist in the form of download links.

[0059] Therefore, based on the configuration information of the target source, the corresponding crawling type can be identified.

[0060] Specifically, in some embodiments, a neural network model can be used to classify the crawling type of the target source. That is, the configuration information of the target source is first feature-engineered to obtain a configuration feature vector, and then the configuration feature vector is input into the crawling type classification model to obtain the crawling type of the target source.

[0061] Specifically, in some embodiments, the training method for the above-mentioned crawling type classification model includes:

[0062] Obtain a configuration feature vector sample set, which contains multiple configuration feature vector samples, each of which is pre-labeled with a corresponding crawling type;

[0063] Input each configuration feature vector sample in the configuration feature vector sample set into the crawling type classification model to obtain the crawling type;

[0064] Based on the output crawl type and the labeled crawl type, the parameters of the crawl type classification model are updated until a predetermined termination condition is met, at which point the training ends and a trained crawl type classification model is obtained.

[0065] Specifically, in some embodiments, updating the parameters of the crawling type classification model based on the output crawling type and the labeled crawling type until a predetermined termination condition is met, thus ending the training and obtaining the trained crawling type classification model, may specifically include:

[0066] If, in the set of configuration feature vector samples, less than a predetermined number of configuration feature vector samples are input into the crawling type classification model and the output crawling type is consistent with the labeled crawling type, then the parameters of the crawling type classification model are updated.

[0067] If more than a predetermined number of configuration feature vector samples in the configuration feature vector sample set are input into the crawling type classification model and the output crawling type is consistent with the labeled crawling type, then the predetermined termination condition is met, the training ends, and the trained crawling type classification model is obtained.

[0068] Specifically, in some other embodiments, updating the parameters of the crawling type classification model based on the output crawling type and the labeled crawling type until a predetermined termination condition is met, thus ending the training and obtaining the trained crawling type classification model, may specifically include:

[0069] Determine the loss function based on the type of crawling output and the type of crawling tags;

[0070] The parameters of the crawling type classification model are updated according to the loss function until the predetermined termination condition is met, at which point the training ends and a trained crawling type classification model is obtained.

[0071] In this embodiment, the predetermined termination condition for the loss function can be that the loss function converges or the loss function is less than a predetermined loss (e.g., 0.001).

[0072] In the S200, for file-based data scraping, it's typically necessary to focus on the uniqueness of files, such as their URL, filename, size, and modification time. By comparing this information, it ensures that only new or modified files are downloaded. This effectively avoids wasted bandwidth and storage resources caused by repeatedly downloading the same file.

[0073] Specifically, in some embodiments, the specific implementation of step S200 can be found in [reference needed]. Figure 3 . Figure 3 It is based on Figure 2 According to the detailed description of step S200 in the web crawler data deduplication method shown in the corresponding embodiment, step S200 in the web crawler data deduplication method may include the following steps:

[0074] S210, perform preliminary deduplication using the URL of the details page of the target source to obtain preliminary deduplication data.

[0075] S220, the preliminary deduplication data is deduplicated again based on the download information in the target source file to obtain the final deduplication data.

[0076] This embodiment successfully solves the problems of data redundancy, file version control, and crawling accuracy through a two-round deduplication mechanism. Specifically, this embodiment performs initial screening using the URL of the target source's detail page, and then performs a second deduplication based on download information. This effectively improves data processing efficiency, reduces network bandwidth consumption, and enhances the freshness and relevance of the crawled data. Simultaneously, this embodiment improves crawling accuracy through detailed comparison of download information and ensures that the crawler database stores only the latest versions of files, improving the timeliness of crawler database file updates.

[0077] In S210, the details page URL is used as the unique identifier of the file. By comparing the already crawled details page URLs, duplicate URL data is quickly filtered out, achieving the purpose of initially filtering out most duplicate requests and quickly reducing the dataset to be processed.

[0078] Specifically, in some embodiments, the specific implementation of step S210 can be found in the following embodiments. This embodiment is based on... Figure 2 According to the detailed description of step S210 in the web crawler data deduplication method shown in the corresponding embodiment, step S210 in the web crawler data deduplication method may include the following steps:

[0079] The URLs of the details pages of the target source are filtered for interference to obtain unique URLs.

[0080] The unique URL is compared with the detail page URLs in the crawler database to obtain preliminary deduplication data.

[0081] In this embodiment, before processing the URL, it is necessary to filter out distractors. This includes filtering parameters, timestamps, etc., to obtain the most unique URL. Before the crawler starts crawling the list page, by comparing the unique URL with the detail page URLs already stored in the database, content that has already been crawled can be quickly identified, thereby reducing unnecessary duplicate crawling and saving time and resources.

[0082] Even after initial deduplication, some unidentified duplicate data may still exist, especially those with different URLs but identical file content. Therefore, in S220, download information is analyzed for more precise comparison, ensuring that the final crawled data consists of new or modified files. This second deduplication strategy effectively improves the accuracy and effectiveness of data crawling, guaranteeing low redundancy in the final data.

[0083] Specifically, in some embodiments, the specific implementation of step S220 can be found in [reference needed]. Figure 4 . Figure 4 It is based on Figure 3According to the detailed description of step S220 in the web crawler data deduplication method shown in the corresponding embodiment, step S220 may include the following steps:

[0084] S222, Based on the request header information, perform deduplication on the preliminary deduplication data to obtain the requested deduplication data.

[0085] S224, Based on the download information, filter the requested deduplication data to obtain the final deduplication data.

[0086] In this embodiment, a dual deduplication strategy using request header information and download information ensures more efficient identification of duplicate content, improves crawling accuracy, and ensures that no valuable data is missed. This solves the problem of accurately identifying and filtering duplicate data during data crawling, especially when the source data is frequently updated. Simultaneously, it reduces duplicate requests, optimizing the use of network resources.

[0087] In S222, the request header information contains important information such as the request source, request type, and content negotiation. By comparing the request headers, it is possible to effectively identify whether different requests are duplicate requests for the same resource. S222 can quickly eliminate duplicate data caused by different request methods, further refining the dataset to be crawled.

[0088] Specifically, in some embodiments, the specific implementation of step S222 can be found in the following embodiments. This embodiment is based on... Figure 4 According to the detailed description of step S222 in the web crawler data deduplication method shown in the corresponding embodiment, step S222 in the web crawler data deduplication method may include the following steps:

[0089] The request header information is parsed to obtain the request parameters.

[0090] Based on the requested parameters, the preliminary deduplication data is deduplicated to obtain the requested deduplication data.

[0091] In this embodiment, the request parameters include content-length (file size), last-modified (last modified time), and content-type (file type). This embodiment obtains the file size, last modified time, and file type parameters by parsing the request header information (i.e., the header request), and then performs comparison and deduplication to achieve faster identification of duplicate data, thereby reducing data processing time and computational costs. It also enables the crawler to handle more complex and diverse deduplication scenarios, improving the accuracy of duplicate data identification.

[0092] In S224, by using download information, it is possible to accurately determine whether two files are completely identical or whether they are newer versions of the file. This effectively avoids duplicate scraping caused by changes in file names or URLs, ensuring that the final scraped data is the truly needed and latest file.

[0093] Specifically, in some embodiments, the specific implementation of step S224 can be found in the following embodiments. This embodiment is based on... Figure 4 According to the detailed description of step S224 in the web crawler data deduplication method shown in the corresponding embodiment, step S224 in the web crawler data deduplication method may include the following steps:

[0094] Download the target source file according to the download information.

[0095] Determine the feature values ​​of the target source file.

[0096] Based on the feature values ​​of the target source file, the duplicate data of the requested deduplication is compared to obtain the final deduplication data.

[0097] In this embodiment, after the data has been filtered for deduplication using URL information and request header information, the file needs to be downloaded to calculate feature values ​​for further deduplication. These feature values ​​include parameters such as the file's MD5 hash, SHA hash, and CRC32 hash, which can serve as unique identifiers for the file. By comparing these unique identifiers, duplicate data can be identified, improving the accuracy of data deduplication. Furthermore, the data requested for deduplication has undergone multiple deduplication processes, significantly reducing its volume compared to the original data, thus decreasing resource consumption and processor load, and improving deduplication efficiency.

[0098] In S300, for page crawling, the focus is primarily on the page content and its changes. For example, it can determine whether to re-crawl by calculating the page's HTML feature values ​​and comparing the plain text of the main content. This process ensures that the crawled data is up-to-date, avoids repeatedly crawling the same pages, and improves the timeliness and relevance of the data.

[0099] Specifically, in some embodiments, the specific implementation of step S300 can be found in [reference needed]. Figure 5 . Figure 5 It is based on Figure 2 According to the detailed description of step S300 in the web crawler data deduplication method shown in the corresponding embodiment, step S300 in the web crawler data deduplication method may include the following steps:

[0100] S310, perform preliminary deduplication using the URL of the details page of the target source to obtain preliminary deduplication data.

[0101] S320, the preliminary deduplication data is deduplicated again based on the page information in the target source page to obtain the final deduplication data.

[0102] In this embodiment, by combining a dual deduplication mechanism using URLs and page information, duplicate data can be identified more accurately, reducing the possibility of false positives and false negatives. Specifically, this embodiment uses the URL of the target source's detail page for initial screening, which can quickly remove a large amount of obviously duplicate data, reducing the amount of data to be processed in the second deduplication step and improving the efficiency of the second deduplication process. During the second deduplication process, since page information is used as the basis for deduplication, different content can be captured more comprehensively, ensuring data diversity and integrity.

[0103] Step S310 is exactly the same as step S210 above. For a detailed description of step S210, please refer to the previous detailed description of step S210. This specification will not repeat it here.

[0104] In S320, after initial deduplication, the crawler uses detailed information from the target source page (such as text content, images, publication time, version number, etc.) to further analyze and filter the initially filtered data to determine if duplicate data exists. By combining page information, this step can analyze the actual similarity of content in greater detail, effectively identifying duplicate data even when URLs differ. This ensures that the final dataset is more accurate and comprehensive, reducing redundancy and invalid information, and optimizing data quality and usability.

[0105] Specifically, in some embodiments, the specific implementation of step S320 can be found in [reference needed]. Figure 6 . Figure 6 It is based on Figure 5 In the detailed description of step S320 of the crawler data deduplication method shown in the corresponding embodiment, the page information in the crawler data deduplication method includes overall page information and page content information, and step S320 may include the following steps:

[0106] S322, Based on the overall page information of the target source page, the preliminary deduplication data is deduplicated to obtain the overall deduplication data.

[0107] S324, Based on the page content information of the target source page, perform deduplication on the overall deduplication data to obtain the final deduplication data.

[0108] In this embodiment, by combining overall page information with page content information for deduplication, duplicate data can be identified more comprehensively, reducing missed and false positives caused by single features. This also effectively reduces the final data volume, significantly improving the quality of the resulting dataset and ensuring data diversity and integrity. Furthermore, this embodiment can effectively handle information within complex webpage structures, making data crawling more flexible and adaptable.

[0109] In S322, the crawler filters the initial duplicate data based on the overall page information of the target source page (such as page title, metadata, navigation structure, etc.), effectively eliminating web pages that are significantly duplicated in content or structure, quickly reducing the amount of data to be processed, and improving the quality of the dataset.

[0110] Specifically, in some embodiments, the specific implementation of step S322 can be found in the following embodiments. This embodiment is based on... Figure 6 The detailed description of step S322 in the crawler data deduplication method shown in the corresponding embodiment is as follows: In the crawler data deduplication method, the overall page information includes the page version number and the overall page feature value. Step S322 may include the following steps:

[0111] Based on the page version number, the preliminary deduplication data is subjected to version deduplication to obtain version deduplication data.

[0112] Based on the overall feature values ​​of the page, the duplication of the version deduplication data is compared to obtain the overall deduplication data.

[0113] In this embodiment, by combining version-based deduplication with overall feature values, irrelevant data can be quickly filtered out, reducing the workload of subsequent processing and improving data processing efficiency. The final overall deduplicated data has both diversity and uniqueness; therefore, this embodiment also improves the quality of the dataset.

[0114] Specifically, this embodiment uses version-based deduplication to effectively distinguish between different versions of web pages, avoiding the repeated crawling of different versions of the same content. This not only reduces data redundancy but also ensures data uptime and timeliness. Furthermore, this embodiment utilizes overall page feature values ​​for duplication comparison, which can more accurately identify content similarities and improve deduplication accuracy.

[0115] It should be noted that the overall page feature value is the hash feature value of the entire page, which can be an MD5 hash value, SHA hash value, CRC32 hash value, or other feature values ​​that can serve as unique identifiers for the file.

[0116] In S324, the crawler further analyzes and filters the overall deduplication data based on page content information (such as text, images, timestamps, etc.) to more accurately identify duplicate content and data, reduce unnecessary duplicate information, and improve the diversity and usability of the data.

[0117] Specifically, in some embodiments, the specific implementation of step S324 can be found in the following embodiments. This embodiment is based on... Figure 6 The detailed description of step S324 in the crawler data deduplication method shown in the corresponding embodiment, wherein the page content information includes text time information and page content feature values, and step S324 may include the following steps:

[0118] Based on the time information in the main text, determine the update data for the current crawling period.

[0119] Based on the page content feature values, the updated data is compared for redundancy to obtain the final deduplication data.

[0120] In this embodiment, the text time information includes time-related information such as publication time, update time, and edit time. By utilizing this time information, the most recently updated webpage content can be quickly identified and captured, ensuring that the acquired data reflects the current state of the webpage and avoiding interference from outdated information. Using page content feature values ​​for duplicate comparison allows for more accurate identification of content similarities and differences, thereby improving the deduplication effect.

[0121] It should be noted that the page content feature value is the hash feature value of the page content, which can be an MD5 hash value, SHA hash value, CRC32 hash value, or other feature values ​​that can serve as unique identifiers for the file.

[0122] Page content feature values ​​can be obtained by converting the main text content from HTML to plain text. Specifically, this can be done by converting HTML to Markdown and then to plain text, thus preserving the content and format of the main text, allowing for the calculation and comparison of feature values ​​from the plain text.

[0123] The following describes an apparatus embodiment of this application, which can be used to execute the web crawler data deduplication method described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the web crawler data deduplication method described above.

[0124] Figure 7 A block diagram of a web crawler data deduplication device according to an embodiment of this application is shown.

[0125] Reference Figure 7As shown, a crawler data deduplication device 700 according to an embodiment of this application includes: a type determination module 710, a first deduplication module 720, and a second deduplication module 730.

[0126] The type determination module 710 is used to determine the crawling type of the target source based on the configuration information of the target source; the first deduplication module 720 is used to perform multiple deduplication based on the download information in the target source file if the crawling type is a file type; and the second deduplication module 730 is used to perform multiple deduplication based on the page information in the target source page if the crawling type is a page type.

[0127] In some feasible embodiments of this application, the first deduplication module specifically includes: a preliminary deduplication submodule, used to perform preliminary deduplication using the details page URL of the target source to obtain preliminary deduplication data; and a file deduplication submodule, used to perform further deduplication on the preliminary deduplication data based on the download information in the target source file to obtain final deduplication data.

[0128] In some feasible embodiments of this application, the file deduplication submodule specifically includes: a request deduplication unit, used to deduplication the preliminary deduplication data according to the request header information to obtain request deduplication data; and a download deduplication unit, used to deduplication the request deduplication data according to the download information to obtain final deduplication data.

[0129] In some feasible embodiments of this application, the request deduplication unit is specifically used to perform: parsing the request header information to obtain request parameters; and deduplicating the preliminary deduplication data according to the request parameters to obtain request deduplication data.

[0130] In some feasible embodiments of this application, the download deduplication unit is specifically used to perform: downloading the target source file according to the download information; determining the feature value of the target source file; and performing a repeatability comparison on the requested deduplication data according to the feature value of the target source file to obtain the final deduplication data.

[0131] In some feasible embodiments of this application, the second deduplication module specifically includes: a preliminary deduplication submodule, used to perform preliminary deduplication using the details page URL of the target source to obtain preliminary deduplication data; and a page deduplication submodule, used to perform further deduplication on the preliminary deduplication data based on the page information in the target source page to obtain final deduplication data.

[0132] In some feasible embodiments of this application, the page information includes overall page information and page content information, and the page deduplication submodule specifically includes: an overall deduplication unit, used to deduplication the preliminary deduplication data according to the overall page information of the target source page to obtain overall deduplication data; and a content deduplication unit, used to deduplication the overall deduplication data according to the page content information of the target source page to obtain final deduplication data.

[0133] In some feasible embodiments of this application, the overall page information includes a page version number and an overall page feature value. The overall deduplication unit is specifically used to perform: performing version deduplication on the preliminary deduplication data according to the page version number to obtain version deduplication data; and performing a repeatability comparison on the version deduplication data according to the overall page feature value to obtain overall deduplication data.

[0134] In some feasible embodiments of this application, the page content information includes text time information and page content feature values, and the content deduplication unit is specifically used to perform: determining the update data of the current crawling cycle based on the text time information; and performing a duplication comparison on the update data based on the page content feature values ​​to obtain the final deduplication data.

[0135] In some feasible embodiments of this application, the preliminary deduplication submodule specifically includes: filtering interference items from the detail page URL of the target source to obtain a unique URL; and comparing the unique URL with the detail page URL in the crawler database to obtain preliminary deduplication data.

[0136] In some embodiments of this application, multiple deduplication strategies are applied to target source data of different crawling types to achieve more targeted filtering of duplicate data. This reduces the time and resource consumption of repeated crawling, improves the deduplication efficiency of the crawler, makes the crawler more efficient during execution, and makes deduplication filtering more accurate, thus solving the problem of poor deduplication effect of existing deduplication methods. Furthermore, the final result obtained after deduplication has less duplicate data compared to existing technologies, reducing unnecessary resource consumption and storage space occupation, alleviating the burden of storage management, and thereby improving overall performance.

[0137] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0138] It should be noted that, Figure 8 The computer system of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0139] like Figure 8 As shown, the computer system includes a Central Processing Unit (CPU) 1801, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 1802 or programs loaded from storage portion 1808 into Random Access Memory (RAM) 1803, such as performing the methods described in the above embodiments. The RAM 1803 also stores various programs and data required for system operation. The CPU 1801, ROM 1802, and RAM 1803 are interconnected via a bus 1804. An Input / Output (I / O) interface 1805 is also connected to the bus 1804.

[0140] The following components are connected to I / O interface 1805: an input section 1806 including a keyboard, mouse, etc.; an output section 1807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1808 including a hard disk, etc.; and a communication section 1809 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1809 performs communication processing via a network such as the Internet. A drive 1810 is also connected to I / O interface 1805 as needed. Removable media 1811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1810 as needed so that computer programs read from them can be installed into storage section 1808 as needed.

[0141] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1809, and / or installed from removable medium 1811. When the computer program is executed by central processing unit (CPU) 1801, it performs various functions defined in the system of this application.

[0142] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0144] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0145] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0146] This specification also provides a computer program product storing at least one instruction, which is loaded and executed by the processor as described above. Figures 1-6 The method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-6 The specific details of the illustrated embodiments will not be elaborated here.

[0147] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0148] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0149] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0150] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for deduplication of crawler data, characterized in that, The method for deduplication of crawler data includes: Based on the configuration information of the target source, determine the crawling type of the target source; If the capture type is a file, then multiple deduplications are performed based on the download information in the target source file; If the crawling type is page type, then multiple deduplications are performed based on the page information in the target source page.

2. The method for deduplication of crawler data as described in claim 1, characterized in that, The multiple deduplication process based on the download information in the target source file specifically includes: Preliminary deduplication is performed using the details page URL of the target source to obtain preliminary deduplication data; The initial deduplication data is further deduplicated based on the download information in the target source file to obtain the final deduplication data.

3. The method for deduplication of crawler data as described in claim 2, characterized in that, The step of further filtering the preliminary deduplication data based on the download information in the target source file to obtain the final deduplication data specifically includes: Based on the request header information, the preliminary deduplication data is deduplicated to obtain the requested deduplication data; Based on the download information, the requested deduplication data is deduplicated to obtain the final deduplication data.

4. The method for deduplication of crawler data as described in claim 3, characterized in that, The step of filtering the preliminary deduplication data according to the request header information to obtain the requested deduplication data specifically includes: Parse the request header information to obtain the request parameters; Based on the requested parameters, the preliminary deduplication data is deduplicated to obtain the requested deduplication data.

5. The method for deduplication of crawler data as described in claim 3, characterized in that, The step of filtering the requested deduplication data based on the download information to obtain the final deduplication data specifically includes: Download the target source file according to the download information; Determine the feature values ​​of the target source file; Based on the feature values ​​of the target source file, the duplicate data of the requested deduplication is compared to obtain the final deduplication data.

6. The method for deduplication of crawler data as described in claim 1, characterized in that, The multiple deduplication based on page information from the target source page specifically includes: Preliminary deduplication is performed using the details page URL of the target source to obtain preliminary deduplication data; The initial deduplication data is further deduplicated based on the page information in the target source page to obtain the final deduplication data.

7. A web crawler data deduplication device, characterized in that, The crawler data deduplication device includes: The type determination module is used to determine the crawling type of the target source based on the configuration information of the target source; The first deduplication filtering module is used to perform multiple deduplication based on the download information in the target source file if the crawling type is a file type. The second deduplication module is used to perform multiple deduplication based on the page information in the target source page if the crawling type is a page type.

8. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the crawler data deduplication method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the crawler data deduplication method as described in any one of claims 1 to 6.

10. A computer program product comprising one or more computer programs, characterized in that, When the one or more computer programs are executed by one or more processors, they implement the steps of the crawler data deduplication method according to any one of claims 1 to 6.

Images