Crawler automatic repair processing method and related device

Abstract

Embodiments of the present application provide a crawler automatic repair processing method and related equipment. The crawler automatic repair processing method comprises: determining a sub-health source according to output statistics in a unit time, the sub-health source being a target source that needs to be repaired; replacing selectors in a selector library one by one to test a detail page of the sub-health source, and obtaining detail page information of the sub-health source; and if the detail page information cannot be obtained, performing automatic repair identification on a URL corresponding to the sub-health source, and obtaining the detail page information of the sub-health source. Embodiments of the present application first statistically analyze the output of each target source, automatically screen out invalid selectors, and timely identify changes in web page structure. At the same time, through automatic repair and selector replacement, the strategy is automatically adjusted to adapt to the new environment, the efficient operation of the crawler is maintained, the continuity and accuracy of the data collection process are ensured, and the demand for manual adjustment of the selector by the operator is reduced.

Description

Technical Field

[0001] This application relates to the fields of computer and communication technology, and more specifically, to an automatic web crawler repair processing method and related equipment. Background Technology

[0002] With the development of computer technology, the importance of internet data has become increasingly prominent. During data collection, web crawlers often face a thorny problem: redesigns of target web pages can cause existing element selectors to become invalid. This not only affects the efficiency of data collection but also increases the workload of operations personnel. Therefore, how to quickly adapt to changes in web page structure and maintain the stability and effectiveness of web crawlers has become an important research direction. Summary of the Invention

[0003] The embodiments of this application provide an automatic web crawler repair processing method and related equipment, which can at least reduce the failure rate of web crawling to a certain extent.

[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0005] According to one aspect of the embodiments of this application, an automatic web crawler repair method is provided, comprising: determining sub-health sources based on output statistics per unit time, wherein the sub-health sources are target sources that need to be repaired; performing detail page tests on the sub-health sources one by one by changing selectors in a selector library to obtain detail page information of the sub-health sources; if detail page information is not obtained, automatically repairing and identifying the URL corresponding to the sub-health source to obtain the detail page information of the sub-health source.

[0006] According to some feasible embodiments of this application, the automatic repair and identification of the URL corresponding to the sub-health source to obtain the details page information of the sub-health source specifically includes: extracting all URLs corresponding to the sub-health source; analyzing each URL to determine the details page URL; and performing details page testing based on the details page URL to obtain the details page information of the sub-health source.

[0007] According to some feasible embodiments of this application, the step of analyzing the URL to determine the details page URL specifically includes: traversing each URL and checking whether each URL contains a specific keyword; if the URL contains a specific keyword, then the URL is determined to be the details page URL.

[0008] According to some feasible embodiments of this application, the step of analyzing the URL to determine the details page URL specifically includes: comparing the feature values ​​of each URL to determine the degree of similarity between the URLs; and determining the details page URL based on the degree of similarity.

[0009] According to some feasible embodiments of this application, the step of performing detail page testing based on the detail page URL to obtain the detail page information of the sub-health source specifically includes: if the detail page information is obtained, generating a new selector; writing the new selector into the selector library and updating the configuration table.

[0010] According to some feasible embodiments of this application, the step of performing detail page testing based on the detail page URL to obtain the detail page information of the sub-health source specifically includes: if the detail page information is not obtained, the detail page URL is redefined and the detail page test is continued; if the detail page information is still not obtained after exceeding a predetermined number of times, the detail page test is terminated and an alarm is triggered.

[0011] According to some feasible embodiments of this application, after testing the sub-health source by changing the selectors one by one in the selector library, the method further includes: if the details page information is obtained, updating the selector data of the current selector to the crawler source configuration table; and updating the frequency of the current selector in the selector library.

[0012] According to some feasible embodiments of this application, the step of determining the source of sub-health based on the output statistics within a unit time period specifically includes: if there is no output for the target source within a unit time period, then updating the sub-health marker corresponding to the target source; and determining the sub-health source based on the sub-health marker.

[0013] According to one aspect of the embodiments of this application, an automatic web crawler repair processing device is provided. The automatic web crawler repair processing device includes: a target determination module, used to determine sub-health sources based on output statistics within a unit time, wherein the sub-health sources are target sources that need to be repaired; a replacement test module, used to perform detail page testing on the sub-health sources by changing selectors one by one in a selector library, and obtain the detail page information of the sub-health sources; and a repair identification module, used to automatically repair and identify the URL corresponding to the sub-health sources if the detail page information cannot be obtained, and obtain the detail page information of the sub-health sources.

[0014] According to some feasible embodiments of this application, the repair identification module specifically includes: a URL extraction submodule, used to extract all URLs corresponding to the sub-health source; a URL analysis submodule, used to analyze each URL to determine the details page URL; and a URL testing submodule, used to perform details page testing based on the details page URL to obtain the details page information of the sub-health source.

[0015] According to some feasible embodiments of this application, the URL analysis submodule specifically includes: a URL traversal unit, used to traverse each URL and check whether each URL contains a specific keyword; and a URL matching unit, used to determine that if the URL contains a specific keyword, the URL is a details page URL.

[0016] According to some feasible embodiments of this application, the URL analysis submodule specifically includes: a feature comparison unit, used to compare the feature values ​​of each URL to determine the degree of similarity between the URLs; and a target determination unit, used to determine the details page URL based on the degree of similarity.

[0017] According to some feasible embodiments of this application, the URL testing submodule specifically includes: a selector generation unit, used to generate a new selector if details page information is obtained; and a selector update unit, used to write the new selector into the selector library and update the configuration table.

[0018] According to some feasible embodiments of this application, the URL testing submodule specifically includes: a retesting unit, used to redetermine the details page URL and continue the details page test if details page information is not obtained; and a stop alarm unit, used to stop the details page test and issue an alarm if details page information is not obtained after a predetermined number of attempts.

[0019] According to some feasible embodiments of this application, the automatic crawler repair processing device further includes: a data update module, used to update the selector data of the current selector to the crawler source configuration table if the details page information is obtained; and a frequency update module, used to update the frequency of the current selector in the selector library.

[0020] According to some feasible embodiments of this application, the target determination module specifically includes: a sub-health marking sub-module, used to update the sub-health mark corresponding to the target source if no output is produced for the target source within a unit time; and a sub-health determination sub-module, used to determine the sub-health source based on the sub-health mark.

[0021] According to one aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the crawler automatic repair processing method as described in the above embodiments.

[0022] According to one aspect of the embodiments of this application, an electronic device is provided, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the crawler automatic repair processing method as described in the above embodiments.

[0023] A computer program product includes one or more computer programs, characterized in that, when the one or more computer programs are executed by one or more processors, they implement the steps of the crawler automatic repair processing method as described in the above embodiments.

[0024] In some embodiments of this application, the output of each target source is first statistically analyzed to automatically screen for failed selectors, enabling timely identification of changes in webpage structure. Simultaneously, through automatic repair and selector replacement, strategies are automatically adjusted to adapt to new environments, maintaining the efficient operation of the crawler, ensuring the continuity and accuracy of the data collection process, reducing the need for manual selector adjustments by operators, and improving work efficiency.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0027] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown.

[0028] Figure 2 The diagram shows a flowchart of an automatic web crawler repair method provided in an embodiment of this application.

[0029] Figure 3 It shows according to Figure 2 A flowchart illustrating a specific implementation of step S300 in the crawler automatic repair processing method shown in the corresponding embodiment.

[0030] Figure 4 It shows according to Figure 3 A flowchart illustrating a specific implementation of step S330 in the crawler automatic repair processing method shown in the corresponding embodiment.

[0031] Figure 5 A schematic diagram of the structure of an automatic crawler repair processing device provided in an embodiment of this application is shown.

[0032] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0033] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0034] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0035] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0036] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0037] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown.

[0038] like Figure 1 As shown, the system architecture may include terminal devices (such as...) Figure 1 The device shown includes one or more of a smartphone 101, tablet 102, and portable computer 103 (which could also be a desktop computer, etc.), a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal device and the server 105. The network 104 can include various connection types, such as wired communication links, wireless communication links, etc.

[0039] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers. For example, server 105 could be a server cluster composed of multiple servers.

[0040] Users can use terminal devices to interact with server 105 via network 104 to receive or send messages, etc. Server 105 can be a server providing various services. For example, a user can use terminal device 103 (or terminal device 101 or 102) to upload output statistics of each target source per unit time to server 105. Server 105 can determine the sub-healthy source based on the output statistics per unit time, and the sub-healthy source is the target source that needs to be repaired; it can then test the details page of the sub-healthy source by changing the selector one by one in the selector library to obtain the details page information of the sub-healthy source; if the details page information cannot be obtained, it can automatically repair and identify the URL corresponding to the sub-healthy source to obtain the details page information of the sub-healthy source.

[0041] It should be noted that the automatic web crawler repair processing method provided in this application embodiment is generally executed by server 105, and correspondingly, the automatic web crawler repair processing device is generally set in server 105. However, in other embodiments of this application, the terminal device may also have similar functions to the server, thereby executing the automatic web crawler repair processing scheme provided in this application embodiment.

[0042] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0043] Figure 2 A flowchart of an automatic web crawler repair processing method according to an embodiment of this application is shown. This automatic web crawler repair processing method can be executed by a server, which may be... Figure 1 The server shown. (Refer to...) Figure 2 As shown, this automatic repair method for web crawlers includes at least the following:

[0044] S100, based on the output statistics per unit time, determine the sub-health source, which is the target source that needs to be repaired.

[0045] S200, the selectors in the selector library are changed one by one to perform a details page test on the sub-health source and obtain the details page information of the sub-health source.

[0046] S300, if the details page information cannot be obtained, the URL corresponding to the sub-health source is automatically repaired and identified to obtain the details page information of the sub-health source.

[0047] In the embodiments of this application, the output of each target source is first statistically analyzed to automatically screen for failed selectors and promptly identify changes in webpage structure. Simultaneously, through automatic repair and selector replacement, the strategy is automatically adjusted to adapt to the new environment, maintaining the efficient operation of the crawler, ensuring the continuity and accuracy of the data collection process, reducing the need for manual selector adjustments by operations personnel, and improving work efficiency.

[0048] In S100, based on the output statistics per unit time, "sub-healthy sources" are identified, which are the target sources for which the selector cannot correctly extract information, i.e., those target sources that need to be repaired. In this step, by analyzing the crawler's running data, the problematic target pages can be quickly located, achieving accurate identification of sub-healthy sources.

[0049] Specifically, in some embodiments, the specific implementation of step S100 can be found in the following embodiments. This embodiment is based on... Figure 2 According to the detailed description of step S100 in the automatic web crawler repair method shown in the corresponding embodiment, step S100 in the automatic web crawler repair method may include the following steps:

[0050] If no output is generated for the target source within a unit of time, the sub-health marker corresponding to the target source is updated.

[0051] The source of sub-health is determined based on the sub-health markers.

[0052] In this embodiment, the failure selector and its corresponding target source are determined by monitoring and statistically analyzing the output of the target source within a unit of time. If the target source does not return valid data within that time period, it is considered to be in a sub-healthy state, and its sub-healthy label is updated. The label may be "sub-healthy" or its sub-healthy value. When the sub-healthy label reaches a predetermined constraint, the target source is determined to be a sub-healthy source. The predetermined constraint may be having a sub-healthy label, having a predetermined number of sub-healthy labels, or the sub-healthy value exceeding a predetermined sub-healthy threshold. The sub-healthy value is positively correlated with the number of times valid data has not been returned.

[0053] This embodiment uses dynamically updated tags to enable the system to quickly identify sources that need adjustment, ensuring that the crawler can respond rapidly to changes in data sources and avoid invalid attempts. Simultaneously, by identifying suboptimal sources, the crawler can adopt corresponding strategies, such as reducing the request frequency to these sources or prioritizing access to other sources, ensuring the overall efficiency and stability of data collection. This also defines the objectives for automatic repair processing.

[0054] In S200, selectors are replaced one by one in the selector library to perform detail page tests on sub-health sources in order to obtain their latest detail page information. By systematically testing different selectors, selectors suitable for the current webpage structure can be found quickly, which significantly improves the success rate of data acquisition and ensures that even if the webpage changes, the required data can still be obtained through automated means.

[0055] In some embodiments, after step S200, the method may further include:

[0056] If the details page information is obtained, the selector data of the current selector will be updated in the crawler source configuration table.

[0057] Update the frequency of the current selector in the selector library.

[0058] In the embodiments of this application, if the details page information is successfully obtained, the selector data of the current selector is updated to the crawler source configuration table and the usage frequency of the current selector is updated, which improves the flexibility and effectiveness of the crawler in data collection and ensures the integrity and availability of the data.

[0059] This embodiment achieves the effect of dynamically adjusting the selection strategy by updating the frequency of selection usage, avoiding the repeated use of invalid selectors and improving the utilization efficiency of crawler resources. This automatic selection repair and update mechanism also enables the crawler to quickly adapt to changes in webpage structure, reducing data acquisition interruptions caused by selector failures.

[0060] In S300, if none of the selectors in the selector library can obtain the details page information, the URL corresponding to the sub-health source needs to be automatically repaired and identified in order to attempt to obtain information, further improving the robustness of data collection. Even if the selector replacement fails, the embodiments of this application can continue to attempt to obtain data through repair and identification, reducing the risk of data loss.

[0061] Specifically, in some embodiments, the specific implementation of step S300 can be found in [reference needed]. Figure 3 . Figure 3 It is based on Figure 2 According to the detailed description of step S300 in the automatic web crawler repair method shown in the corresponding embodiment, step S300 in the automatic web crawler repair method may include the following steps:

[0062] S310, Extract all URLs corresponding to the sub-health source.

[0063] S320, Analyze each of the URLs to determine the details page URL.

[0064] S330, Perform a details page test based on the details page URL to obtain the details page information of the sub-health source.

[0065] In this embodiment, after obtaining all URLs corresponding to the sub-health source, each URL is analyzed to filter out possible detail page URLs. Finally, tests are performed on the selected detail page URLs to obtain detailed page information. The entire process requires no manual intervention, greatly reducing the risk of operational errors and improving the automation level of the crawler. Simultaneously, this embodiment can quickly adapt to changes in webpage structure, obtaining the latest detail page information in a timely manner through analysis and testing. Overall data collection efficiency is significantly improved, ensuring timely information updates.

[0066] In S310, all URLs of the sub-health source are first extracted. Among these URLs, there will be detail page URLs that can be used for detail page testing to obtain detail page information.

[0067] In S320, the extracted URLs are analyzed and filtered to identify which URLs lead to detail pages. Typically, these URLs have a specific format or contain specific keywords.

[0068] Specifically, in some embodiments, the specific implementation of step S320 can be found in the following embodiments. This embodiment is based on... Figure 3 According to the detailed description of step S320 in the automatic web crawler repair method shown in the corresponding embodiment, step S320 may include the following steps:

[0069] Iterate through each URL and check if each URL contains a specific keyword.

[0070] If the URL contains specific keywords, then the URL is determined to be the details page URL.

[0071] In this embodiment, a comprehensive traversal ensures that no potential detail page URLs are missed; keyword matching effectively reduces the possibility of false positives. Only URLs that conform to a specific format and meaning are marked as detail pages, thereby improving the accuracy and reliability of identification. Accurate identification of detail page URLs significantly reduces the error rate in subsequent data collection, ensuring that the collected information is the most valuable.

[0072] Specifically, in some other embodiments, the specific implementation of step S320 can be found in the following embodiments. This embodiment is based on... Figure 3 According to the detailed description of step S320 in the automatic web crawler repair method shown in the corresponding embodiment, step S320 may include the following steps:

[0073] By comparing the feature values ​​of each URL, the degree of similarity between the URLs is determined.

[0074] The URL of the details page is determined based on the degree of similarity.

[0075] In this embodiment, by comparing feature values ​​and similarity, the system can more objectively determine which URLs are detail pages, reducing the risk of misidentification caused by similar URLs. The crawler can be more efficient and accurate in identifying detail page URLs, saving subsequent processing time and resources.

[0076] Specifically, the aforementioned feature values ​​may include information such as the path, parameters, and domain name in the URL. The similarity calculation algorithm may include algorithms such as cosine similarity and Jaccard similarity, and may also include algorithms using neural network models, such as similarity models.

[0077] A similarity calculation algorithm is used to compare extracted feature values ​​to determine the degree of similarity between different URLs. Based on the calculated similarity, a threshold can be set to identify several URLs with relatively similar feature values, and these will be used as detail page URLs. That is, in this embodiment, if URLs with common values ​​and similar prefixes are found on the list page, these URLs are identified as detail page URLs.

[0078] In some embodiments of this application, a feature vector of a URL can be constructed by analyzing information elements such as path, parameters, and domain name in the URL. Then, the feature vectors of all URLs are aggregated into a URL feature matrix. Finally, the URL feature matrix is ​​input into a similarity comparison model to obtain the similarity between the URLs.

[0079] Similarly, in some embodiments, the URLs of the detail page can be obtained by constructing a similarity vector matrix based on the similarity between the URLs and then inputting the similarity vector matrix into a similarity analysis model.

[0080] In addition to the two embodiments described above, in other embodiments of this application, the URL feature vector can be constructed by first analyzing information elements such as path, parameters, and domain name in the URL. Then, the feature vectors of all URLs can be aggregated into a URL feature matrix. Finally, the URL feature matrix is ​​input into a URL filtering model to obtain the details page URL. This URL filtering model includes a similarity comparison sub-model and a similarity analysis sub-model.

[0081] Specifically, the URL feature matrix can be first input into the similarity comparison sub-model to obtain a similarity vector matrix, which contains the similarity data between the aforementioned URLs. Then, this similarity vector matrix is ​​input into the similarity analysis sub-model to obtain the details page URL.

[0082] In this embodiment, the similarity comparison sub-model and the similarity analysis sub-model need to be trained together. In other embodiments, the similarity comparison sub-model and the similarity analysis sub-model can be pre-trained in advance.

[0083] Specifically, the training method for the aforementioned URL filtering model includes:

[0084] Obtain the URL matrix sample set, which contains multiple URL matrix samples, each of which is labeled with a corresponding details page URL tag.

[0085] The URL matrix samples are input one by one into the URL filtering model to obtain the URL results for the details page.

[0086] Based on the output details page URL result and the details page URL tag, the parameters of the URL filtering model are updated until the predetermined termination condition is reached, the training ends, and the trained URL filtering model is obtained.

[0087] In the embodiments of this application, during training, a URL matrix sample set containing multiple URL matrix samples can be obtained first. Each URL matrix sample contains a three-dimensional organ model and is marked with a corresponding details page URL tag. Then, the multiple URL matrix samples are divided into a training set, a validation set, and a test set according to a predetermined ratio. Then, the parameters of the encoder and decoder in the URL filtering model are adjusted and determined according to the URL matrix samples included in the training set, validation set, and test set to obtain the trained URL filtering model.

[0088] It should be noted that the URL matrix samples mentioned above include URL matrix samples containing URLs such as homepage URL, list page URL, and detail page URL, in order to improve the model's ability to recognize various different URLs.

[0089] When training the model, the URL matrix sample set can be divided into a training set, a validation set, and a test set. Then, the model is trained based on the training set, validated based on the validation set, and tested based on the test set to obtain the trained URL filtering model.

[0090] Before training on the training set, the URL matrix samples in the training set can be preprocessed. Preprocessing includes image resizing, normalization, data augmentation, and class encoding.

[0091] The specific methods for data augmentation are to expand, replace, or delete URLs in the URL matrix sample. These operations can better expand the form of the URL matrix sample.

[0092] In some embodiments, GAN networks can also be used to generate URLs that are difficult to identify but not easy to obtain, which helps the algorithm model to accurately identify detail page URLs, reduces false identification and missed identification, and improves the model's accuracy in identifying special detail page URLs.

[0093] After obtaining the enhanced training set, the URL filtering model can be trained based on the enhanced training set, and the parameters and weights in the network can be updated.

[0094] Specifically, the 3D organ model image samples from the training set are input into the URL filtering model to obtain the detail page URL results output by the model. The detail page URL results and detail page URL tags are compared to calculate the loss function. Then, the stochastic gradient descent method is used to minimize the loss function. The parameters and weights in the URL filtering model are updated by backpropagation until the loss function meets a predetermined condition, such as convergence or falling below a predetermined threshold.

[0095] After training, the URL filtering network with updated parameters from the training set can be validated using a validation set. Specifically, the URL filtering network is debugged based on the validation set data. When the loss function meets predetermined conditions, the model parameters for that stage are output. If the loss function does not meet the predetermined conditions, hyperparameters such as the learning rate are automatically adjusted, and the next round of network model training begins.

[0096] When the loss function computed on the validation set meets predetermined conditions, the parameters and weights can be retained, and then the retained parameters are tested based on the test set. Specifically, the input test set data enables the URL filtering network with retained parameters and weights to output detail page URLs and model weights. By comparing the model loss and corresponding weights across multiple rounds, the model weights with the minimum loss are output, thus determining the trained URL filtering model.

[0097] After obtaining the trained URL filtering model, the URL detail page test can be completed based on the URL filtering model.

[0098] Furthermore, data augmentation is not required for the input data in the validation and test sets.

[0099] In other embodiments, URL knowledge graphs can be formed by using each URL as a node and the similarity relationship between URLs as edges. Then, the node with the most edges (or the most adjacent nodes) in the URL knowledge graph is determined as the target node, and the URLs corresponding to the target node and its adjacent nodes are all identified as detail page URLs.

[0100] The adjacent nodes mentioned above may include only directly adjacent nodes, or they may include both directly adjacent nodes and indirectly adjacent nodes.

[0101] Two nodes are directly adjacent if there is an edge between them. Two nodes are indirectly adjacent if there is no edge between them, but both nodes have edges with the same node. For example, if node A and node B have an edge, node B and node C have an edge, but node C and node A have no edge, then node A and node B are directly adjacent, node B and node C are also directly adjacent, and node C and node A are indirectly adjacent.

[0102] Furthermore, the similarity relationship between the URLs mentioned above can refer to whether the URLs have similarity labels. That is, when the similarity between two URLs is higher than a predetermined similarity threshold, they have a similarity label, and an edge is established between the two URLs. The similarity relationship between the URLs mentioned above can also be based on the shared or similar features between the two URLs. The more shared or similar features between the two URLs, the more edges are formed between them.

[0103] In S330, after determining a valid detail page URL, the system will perform access tests on these links to attempt to retrieve information from the page.

[0104] Specifically, in some embodiments, the specific implementation of step S330 can be found in [reference needed]. Figure 4 . Figure 4 It is based on Figure 3 According to the detailed description of step S330 in the automatic web crawler repair method shown in the corresponding embodiment, step S330 may include the following steps:

[0105] S332, if the details page information is obtained, update the selector library and configuration table.

[0106] S334. If the details page information cannot be obtained, the details page test will be terminated and an alarm will be triggered.

[0107] In this embodiment, if the details page information is successfully obtained, the selector library and configuration table are updated according to the process of obtaining the details page information.

[0108] If the details page information is still not obtained, the crawling of the target source can be stopped and manual intervention should be requested. At this time, the target source may contain anti-crawling code.

[0109] In S332, if the details page information is successfully obtained, it means that the URL is still valid, and it also means that the crawler in this embodiment has the possibility of adapting to changes in the webpage structure.

[0110] Specifically, in some embodiments, the specific implementation of step S332 can be found in the following embodiments. This embodiment is based on... Figure 4 According to the detailed description of step S332 in the automatic web crawler repair method shown in the corresponding embodiment, step S332 in the automatic web crawler repair method may include the following steps:

[0111] If the details page information is obtained, a new selector is generated.

[0112] Write the new selector into the selector library and update the configuration table.

[0113] In this embodiment, if the details page information is successfully obtained, it means that the URL is still valid, and it also means that the crawler in this embodiment has the potential to adapt to changes in the webpage structure. At this time, the structure of the page can be automatically analyzed, and a new selector can be generated. This new selector is suitable for crawling the sub-health source, so it will be stored in the selector library, and the corresponding configuration table will be updated for subsequent use. This embodiment can adapt to changes in the webpage structure, improve the flexibility and accuracy of data extraction, and reduce the need for manual intervention, making the selector update process more efficient. At the same time, updating the selector library keeps it up-to-date, making the crawler more reliable and consistent in subsequent data collection. By quickly updating the selector, the crawler can adapt to changes in the target website, reducing data loss or errors caused by selector failure.

[0114] In S334, if the details page information is still not obtained, the target source may have anti-scraping measures, and the application for manual intervention can be suspended.

[0115] Specifically, in some embodiments, the specific implementation of step S334 can be found in the following embodiments. This embodiment is based on... Figure 4 According to the detailed description of step S334 in the automatic web crawler repair method shown in the corresponding embodiment, step S334 may include the following steps:

[0116] If the details page information cannot be obtained, the details page URL will be determined again, and the details page test will continue.

[0117] If the details page information is not obtained after exceeding the predetermined number of attempts, the details page test will be terminated and an alarm will be triggered.

[0118] In this embodiment, if the details page information cannot be obtained, the URL is redefined and different details page URLs are tried continuously to overcome the failure to obtain information due to network or other factors to a certain extent, thereby increasing the success rate of data collection and making the crawler more adaptable to changes, enabling it to quickly find alternative solutions and maintain the continuity of data collection.

[0119] Meanwhile, setting a predetermined limit on the number of attempts and an alarm mechanism can ensure that operators are notified in a timely manner when a fault occurs, avoiding prolonged invalid attempts, thereby preventing the waste of computing resources and improving data acquisition efficiency.

[0120] The following describes an embodiment of the apparatus described in this application, which can be used to execute the automatic web crawler repair processing method described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the automatic web crawler repair processing method described above.

[0121] Figure 5 A block diagram of an automatic crawler repair processing apparatus according to an embodiment of this application is shown.

[0122] Reference Figure 5 As shown, a crawler automatic repair processing device 500 according to an embodiment of this application includes: a target determination module 510, a replacement test module 520, and a repair identification module 530.

[0123] The target determination module 510 is used to determine the sub-health source based on the output statistics within a unit time, and the sub-health source is the target source that needs to be repaired; the replacement test module 520 is used to replace the selector one by one in the selector library to perform detail page testing on the sub-health source and obtain the detail page information of the sub-health source; the repair identification module 530 is used to automatically repair and identify the URL corresponding to the sub-health source if the detail page information cannot be obtained, and obtain the detail page information of the sub-health source.

[0124] According to some feasible embodiments of this application, the repair identification module specifically includes: a URL extraction submodule, used to extract all URLs corresponding to the sub-health source; a URL analysis submodule, used to analyze each URL to determine the details page URL; and a URL testing submodule, used to perform details page testing based on the details page URL to obtain the details page information of the sub-health source.

[0125] According to some feasible embodiments of this application, the URL analysis submodule specifically includes: a URL traversal unit, used to traverse each URL and check whether each URL contains a specific keyword; and a URL matching unit, used to determine that if the URL contains a specific keyword, the URL is a details page URL.

[0126] According to some feasible embodiments of this application, the URL analysis submodule specifically includes: a feature comparison unit, used to compare the feature values ​​of each URL to determine the degree of similarity between the URLs; and a target determination unit, used to determine the details page URL based on the degree of similarity.

[0127] According to some feasible embodiments of this application, the URL testing submodule specifically includes: a selector generation unit, used to generate a new selector if details page information is obtained; and a selector update unit, used to write the new selector into the selector library and update the configuration table.

[0128] According to some feasible embodiments of this application, the URL testing submodule specifically includes: a retesting unit, used to redetermine the details page URL and continue the details page test if details page information is not obtained; and a stop alarm unit, used to stop the details page test and issue an alarm if details page information is not obtained after a predetermined number of attempts.

[0129] According to some feasible embodiments of this application, the automatic crawler repair processing device further includes: a data update module, used to update the selector data of the current selector to the crawler source configuration table if the details page information is obtained; and a frequency update module, used to update the frequency of the current selector in the selector library.

[0130] According to some feasible embodiments of this application, the target determination module specifically includes: a sub-health marking sub-module, used to update the sub-health mark corresponding to the target source if no output is produced for the target source within a unit time; and a sub-health determination sub-module, used to determine the sub-health source based on the sub-health mark.

[0131] In the embodiments of this application, the output of each target source is first statistically analyzed to automatically screen for failed selectors and promptly identify changes in webpage structure. Simultaneously, through automatic repair and selector replacement, the strategy is automatically adjusted to adapt to the new environment, maintaining the efficient operation of the crawler, ensuring the continuity and accuracy of the data collection process, reducing the need for manual selector adjustments by operations personnel, and improving work efficiency.

[0132] Figure 6 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0133] It should be noted that, Figure 6 The computer system of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0134] like Figure 6As shown, the computer system includes a Central Processing Unit (CPU) 1801, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 1802 or programs loaded from storage portion 1808 into Random Access Memory (RAM) 1803, such as performing the methods described in the above embodiments. The RAM 1803 also stores various programs and data required for system operation. The CPU 1801, ROM 1802, and RAM 1803 are interconnected via a bus 1804. An Input / Output (I / O) interface 1805 is also connected to the bus 1804.

[0135] The following components are connected to I / O interface 1805: an input section 1806 including a keyboard, mouse, etc.; an output section 1807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1808 including a hard disk, etc.; and a communication section 1809 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1809 performs communication processing via a network such as the Internet. A drive 1810 is also connected to I / O interface 1805 as needed. Removable media 1811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1810 as needed so that computer programs read from them can be installed into storage section 1808 as needed.

[0136] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1809, and / or installed from removable medium 1811. When the computer program is executed by central processing unit (CPU) 1801, it performs various functions defined in the system of this application.

[0137] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0139] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0140] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0141] This specification also provides a computer program product that stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figures 1-4 The method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-4 The specific details of the illustrated embodiments will not be elaborated here.

[0142] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0143] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0144] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0145] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for automatically repairing web crawlers, characterized in that, include: Based on the output statistics per unit time, the sources of sub-health are identified, and these sources of sub-health are the target sources that need to be repaired. By changing each selector in the selector library, the details page of the sub-health source is tested to obtain the details page information of the sub-health source; If the details page information cannot be obtained, the URL corresponding to the sub-health source will be automatically repaired and identified to obtain the details page information of the sub-health source.

2. The automatic crawler repair method as described in claim 1, characterized in that, The automatic repair and identification of the URL corresponding to the sub-health source, and the acquisition of the details page information of the sub-health source, specifically includes: Extract all URLs corresponding to the sources of sub-health; Analyze each of the URLs to determine the details page URL; Perform a details page test based on the details page URL to obtain the details page information of the sub-health source.

3. The automatic crawler repair method as described in claim 2, characterized in that, The step of analyzing the URL to determine the details page URL specifically includes: Iterate through each URL and check if each URL contains a specific keyword; If the URL contains specific keywords, then the URL is determined to be the details page URL.

4. The automatic crawler repair method as described in claim 2, characterized in that, The step of analyzing the URL to determine the details page URL specifically includes: By comparing the feature values ​​of each URL, the degree of similarity between the URLs is determined; The URL of the details page is determined based on the degree of similarity.

5. The automatic crawler repair method as described in claim 2, characterized in that, The step of performing a details page test based on the details page URL to obtain the details page information of the sub-health source specifically includes: If the details page information is obtained, a new selector is generated; Write the new selector into the selector library and update the configuration table.

6. The automatic crawler repair method as described in claim 2, characterized in that, The step of performing a details page test based on the details page URL to obtain the details page information of the sub-health source specifically includes: If the details page information cannot be obtained, the details page URL will be determined again, and the details page test will continue. If the details page information is not obtained after exceeding the predetermined number of attempts, the details page test will be terminated and an alarm will be triggered.

7. An automatic crawler repair processing device, characterized in that, The automatic crawler repair processing device includes: The target determination module is used to determine the source of sub-health based on the output statistics per unit time, and the sub-health source is the target source that needs to be repaired. The test module is used to perform detail page tests on the sub-health source by changing each selector in the selector library one by one, and to obtain the detail page information of the sub-health source. The repair and identification module is used to automatically repair and identify the URL corresponding to the sub-health source if the details page information cannot be obtained, and obtain the details page information of the sub-health source.

8. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the automatic crawler repair processing method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the crawler automatic repair processing method as described in any one of claims 1 to 6.

10. A computer program product comprising one or more computer programs, characterized in that, When the one or more computer programs are executed by one or more processors, they implement the steps of the crawler automatic repair processing method according to any one of claims 1 to 6.

Images