Crawler crawling time adjustment method, device, computer equipment and storage medium

Abstract

The present application provides a method, device, computer equipment and storage medium for adjusting crawler crawling time, the method comprising: obtaining the page update time of a target website within a preset period, the preset period including at least two periodic target periods, the target period including a first period and a second period; based on the page update time, counting the number of page updates of the target website within each of the first periods, and the number of page updates of the target website within each of the second periods; based on the number of page updates within each of the first periods and the number of page updates within each of the second periods, adjusting the crawler crawling time of the target website to obtain the adjusted crawler crawling time. By adopting this method, the crawler can timely obtain the updated information of the target website within the shortest interval time, thereby improving the timeliness of the crawler's information crawling and reducing the waste of the crawler's bandwidth resources.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a method, device, computer equipment and storage medium for adjusting crawler crawling time. Background Art

[0002] A web crawler (also known as a web chaser) is a program or script that automatically crawls web page information according to certain rules. Setting intervals during the working hours of a web crawler is an effective way to avoid wasting crawler resources and enable the crawler to crawl web page information in a timely manner.

[0003] The traditional crawler time setting method usually sets a fixed time period based on actual business needs, for example, setting a crawl time of 10 am every day or every hour, or adjusting the original crawler interval to a larger or smaller time based on the difference between the actual page update volume of the web pages to be crawled and the preset page update volume. However, whether it is the first method or the second method, it is possible that the web page information of the web pages to be crawled has been updated, but the crawler has not crawled it in time, resulting in a delay in information update.

[0004] Therefore, the existing crawler time setting method has a technical problem that the timeliness of information crawling is not high due to unreasonable crawler interval setting. Summary of the invention

[0005] Based on this, it is necessary to provide a crawler crawling time adjustment method, device, computer equipment and storage medium to address the above technical problems, so as to reasonably adjust the crawling time of the crawler, thereby improving the timeliness of information crawling of the network crawler.

[0006] In a first aspect, the present application provides a method for adjusting crawler crawling time, the method comprising:

[0007] Acquire a page update time of a target website within a preset period, wherein the preset period includes at least two periodically spaced target periods, wherein the target period includes a first period and a second period;

[0008] Based on the page update time, counting the number of page updates of the target website in each of the first time periods, and the number of page updates of the target website in each of the second time periods;

[0009] Based on the number of page updates in each of the first time periods and the number of page updates in each of the second time periods, the crawler crawling time of the target website is adjusted to obtain an adjusted crawler crawling time.

[0010] In some embodiments of the present application, the step of obtaining the page update time of the target website within a preset period includes:

[0011] Obtaining a page update time of the target website within a first target time period, where the first target time period is a target time period in a first preset time period;

[0012] Obtaining a page update time of the target website within a second target period, where the second target period is a target period in a second preset time, and the first preset time and the second preset time are preset times spaced apart by two cycles;

[0013] The page update time within the first target period and the page update time within the second target period are determined as the page update time within the preset period.

[0014] In some embodiments of the present application, the preset periodic time period includes a first target periodic time period and a second target periodic time period at periodic intervals, and the step of counting the number of page updates of the target website in each of the first periodic time period and the number of page updates of the target website in each of the second periodic time period based on the page update time includes:

[0015] Determine, based on the page update times respectively corresponding to the first target period and the second target period, a first time set corresponding to the first target period and a second time set corresponding to the second target period;

[0016] Counting the number of times in the first time set and the second time set to obtain a first page update amount corresponding to the first target time period and a second page update amount corresponding to the second target time period;

[0017] Based on the first time period and the second time period included in each of the target time periods, the first page update amount and the second page update amount are split and processed to obtain the page update amount of the target website in each of the first time periods and the page update amount in each of the second time periods.

[0018] In some embodiments of the present application, the step of splitting the first page update amount and the second page update amount based on the first time period and the second time period included in each of the target time periods to obtain the number of page updates of the target website in each of the first time periods and the number of page updates in each of the second time periods includes:

[0019] Based on a first time period and a second time period included in the first target time period, the first page update amount is split to obtain a third page update amount corresponding to the first time period in the first target time period and a fourth page update amount corresponding to the second time period in the first target time period;

[0020] Based on a first time period and a second time period included in the second target time period, the second page update amount is split to obtain a fifth page update amount corresponding to the first time period in the second target time period and a sixth page update amount corresponding to the second time period in the second target time period;

[0021] The third page update amount and the fifth page update amount are determined as the page update amount of the target website in each of the first time periods, and the fourth page update amount and the sixth page update amount are determined as the page update amount of the target website in each of the second time periods.

[0022] In some embodiments of the present application, the preset periodic period further includes a third target period that is periodically spaced from the first target period and the second target period, and the method further includes:

[0023] Determining a third time set corresponding to the third target time period based on the page update time corresponding to the third target time period;

[0024] Counting the number of times in the third time set to obtain a third page update amount corresponding to the third target time period;

[0025] Based on the first time period and the second time period included in the third target time period, the third page update amount is split to obtain a seventh page update amount corresponding to the first time period in the third target time period and an eighth page update amount corresponding to the second time period in the third target time period;

[0026] The seventh page update amount is determined as the page update amount of the target website in each of the first time periods, and the eighth page update amount is determined as the page update amount of the target website in each of the second time periods.

[0027] In some embodiments of the present application, the step of adjusting the crawler crawling time of the target website based on the number of page updates in each of the first time periods and the number of page updates in each of the second time periods to obtain the adjusted crawler crawling time includes:

[0028] Based on the number of page updates in each of the first time periods, a first average update number is obtained, and based on the number of page updates in each of the second time periods, a second average update number is obtained;

[0029] Obtaining a ratio between the first average update number and the second average update number as a time adjustment ratio;

[0030] If the time adjustment ratio is greater than or equal to a preset time adjustment threshold, the crawler crawling time of the target website is adjusted, and half of the crawler crawling time is determined as the adjusted crawler crawling time.

[0031] In some embodiments of the present application, the step of obtaining a first average update number based on the number of page updates in each of the first time periods, and obtaining a second average update number based on the number of page updates in each of the second time periods, includes:

[0032] Determining a preset attenuation coefficient corresponding to each of the preset periodic time periods;

[0033] Based on the attenuation coefficient, weighted summing is performed on the number of page updates in each of the first time periods to obtain a total number of first page updates corresponding to the first time period;

[0034] Based on the attenuation coefficient, weighted summing is performed on the number of page updates in each of the second time periods to obtain a total number of second page updates corresponding to the second time period;

[0035] The average number of the first page update total amount and the second page update total amount is obtained, and the first average update number and the second average update number are correspondingly obtained.

[0036] In a second aspect, the present application provides a crawler crawling time adjustment device, the device comprising:

[0037] A time acquisition module, used to acquire the page update time of the target website within a preset period, wherein the preset period includes at least two periodically spaced target periods, wherein the target period includes a first period and a second period;

[0038] A time analysis module, for counting the number of page updates of the target website in each of the first time periods and the number of page updates of the target website in each of the second time periods based on the page update time;

[0039] The time adjustment module is used to adjust the crawling time of the target website based on the number of page updates in each of the first time periods and the number of page updates in each of the second time periods to obtain the adjusted crawling time.

[0040] In a third aspect, the present application further provides a server, the server comprising:

[0041] one or more processors;

[0042] Memory; and

[0043] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the crawler crawling time adjustment method.

[0044] In a fourth aspect, the present application also provides a computer-readable storage medium on which a computer program is stored, and the computer program is loaded by a processor to execute the steps in the crawler crawling time adjustment method.

[0045] The above-mentioned crawler crawling time adjustment method, device, computer equipment and storage medium obtain the page update time of the target website within a preset cycle time period to count the number of page updates of the target website in different time periods, so as to obtain the number of page updates of the target website in each first time period and the number of page updates in each second time period, and then use this to analyze the proportion of the number of page updates of the target website between the first half of the time period and the second half of the time period in a target time period, and finally use the proportion to adjust the crawler's crawling time, so that the crawler can timely obtain the updated information of the target website in the shortest interval time, thereby improving the timeliness of the crawler's information crawling and reducing the waste of the crawler's bandwidth resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 It is a flowchart of a method for adjusting crawler crawling time in an embodiment of the present application;

[0048] Figure 2 It is a flowchart of the page update time acquisition step in the embodiment of the present application;

[0049] Figure 3 It is a flowchart of the step of counting the number of page updates in an embodiment of the present application;

[0050] Figure 4 It is a flow chart of the page update amount splitting step in the embodiment of the present application;

[0051] Figure 5 is a flowchart of another page update quantity counting step in an embodiment of the present application;

[0052] Figure 6 It is a flowchart of the crawler crawling time adjustment step in the embodiment of the present application;

[0053] Figure 7 It is a flowchart of the step of obtaining the average update quantity in an embodiment of the present application;

[0054] Figure 8 It is a structural schematic diagram of a crawler crawling time adjustment device in an embodiment of the present application;

[0055] Fig. 9 It is a schematic diagram of the structure of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0057] In the description of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0058] In the description of the present application, the word "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0059] In the embodiments of the present application, it should be noted that since the crawler crawling time adjustment method provided in the present application is executed in a computer device, the processing objects of each computer device exist in the form of data or information. For example, the page update time is actually the page update time information. It can be understood that if the time period, size, quantity, location, etc. are mentioned in the subsequent embodiments, the corresponding data exist for the computer device to process, and the details will not be repeated here.

[0060] In one embodiment, Figure 1 As shown, a crawler crawling time adjustment method is provided. This embodiment uses the method applied to a server as an example for illustration. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be a device that includes both receiving and transmitting hardware, that is, a device with receiving and transmitting hardware that can perform two-way communication on a two-way communication link. Such a device may include: a cellular or other communication device, which has a single-line display or a multi-line display or a cellular or other communication device without a multi-line display. The terminal can specifically be a desktop terminal or a mobile terminal, and the terminal can specifically be one of a mobile phone, a tablet computer, a laptop computer, etc. The server can be an independent server, or a server network or a server cluster composed of servers, which includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets, or a cloud server composed of multiple servers. Among them, the cloud server is composed of a large number of computers or network servers based on cloud computing (Cloud Computing).

[0061] This embodiment mainly uses the method applied to a server as an example to illustrate that the crawler crawling time adjustment method specifically includes steps S101 to S103, which are as follows:

[0062] S101, obtaining a page update time of a target website within a preset period, wherein the preset period includes at least two periodically spaced target periods, and the target period includes a first period and a second period.

[0063] Among them, the target website refers to the website target of the web crawler's current web page update status to be crawled. Since each website has a corresponding page update time when updating its web page, before adjusting the crawler's crawling time for the target website, the page update time of the target website within a preset cycle period can be obtained to analyze how the crawler's crawling time for the updated pages of the target website should be adjusted.

[0064] Among them, the target period can be a time period in seconds, minutes, hours, days, months, etc. The target period selected in the embodiment of the present application is a time period in hours. For example, 10 o'clock to 11 o'clock (10 o'clock-11 o'clock) is a target period. The first period and the second period constitute the target period, so the first period and the second period can take any ratio, and it is only necessary to ensure that the sum of the two is the target period. For example, if the target period is 10 o'clock-11 o'clock, the first period is 10 o'clock-10:20, and the second period is 10:20-11 o'clock; for another example, if the target period is 10 o'clock-11 o'clock, the first period is 10 o'clock-10:30, and the second period is 10:30-11 o'clock. The preset periodic time period is a target time period including at least two periodic intervals. For example, when the target time period includes two periodic intervals, it may be 10:00-11:00 on September 22 and 10:00-11:00 on September 29. September 22 and September 29 are two dates one week apart, so the corresponding target time periods are periodic intervals. For another example, when the target time period includes three periodic intervals, it may be 10:00-11:00 on September 22, 10:00-11:00 on September 29, and 10:00-11:00 on October 6.

[0065] Among them, page update time refers to the time point when the website page is updated. The page update time depends on the time point of the page update operation by the website staff. For example, staff member A updates a page of the target website abc.com at 9 am on January 1, 2020, and its page update time is 2020 / 1 / 1 / 09:00.

[0066] Specifically, the execution entity for obtaining the page update time of the target website can be a server, a terminal, or a third-party platform or system between the terminal and the server, etc., which is not limited in this embodiment. However, it should be noted that since this application proposes that the current crawling time of the crawler can be adjusted in time by analyzing historical data, the data on the page update time should include data for multiple periodic time periods, and the selection of the periodic time period can be set according to actual application requirements, and the page update time data corresponding to each target time period constitutes a set of data. The page update time acquisition steps involved in this embodiment will be described in detail below.

[0067] For example, taking T = 1 hour as the initial interval time of the web crawler, the obtained page update data includes: (2020-10-06, 10-11, 100, [10:01, 10:02, 10:03, 10:03, 10:03, 10:04…]), where "2020-10-06" represents the crawling date of the crawler, "10-11" represents the time period for page updates, "100" represents the number of page updates within the time period, and each time point in "[…]" represents the page update time. At this time, the page update data should at least include: (2020-09-29, 10-11, 90, [10:01, 10:02, 10:03, 10:03, 10:03, 10:04…]). The two copies of the page update data containing historical data will be used as the basis for subsequent analysis to adjust the crawling time of the crawler.

[0068] In one embodiment, Figure 2 As shown, this step includes: S201, obtaining the page update time of the target website within the first target time period, the first target time period is a target time period in the first preset time; S202, obtaining the page update time of the target website within the second target time period, the second target time period is a target time period in the second preset time, the first preset time and the second preset time are preset times separated by two cycles; S203, determining the page update time within the first target time period, and the page update time within the second target time period, as the page update time within the preset cycle time period.

[0069] Among them, the first target period is the target period in the first preset time, and the first preset time can be the current time of the device of the server / terminal, or the historical time of the device, for example, October 6, 2020. The second target period is the target period in the second preset time, and the same as the first target period is that the target periods of the two are the same period, such as an hour period covered by 10 o'clock-11 o'clock, and the difference from the first target period is that the second preset time is a time with a periodic interval with the first preset time, rather than being taken from the first preset time, for example, September 29, 2020. It can be understood that the second preset time can also be the current time of the device of the server / terminal or the historical time of the device, which is not limited in this embodiment and can be set according to actual business needs.

[0070] Specifically, the page update time obtained by the server may include the page update time of the target website in the first target period and the page update time in the second target period. By analyzing the page update time in the two periods, the page update trend of the target website in the past period can be analyzed, and then the crawler can be used to adjust the crawling time of the target website to achieve a balance between timeliness and bandwidth as much as possible.

[0071] For example, the page update time within the first target period (2020-10-06, 10-11) is: [10:01, 10:02, 10:03, 10:03, 10:03, 10:04…], and the page update time within the second target period (2020-09-29, 10-11) is [10:01, 10:02, 10:03, 10:03, 10:03, 10:04…].

[0072] S102: Based on the page update time, count the number of page updates of the target website in each of the first time periods and the number of page updates of the target website in each of the second time periods.

[0073] Specifically, the number of page updates can be obtained by counting the page update time, that is, by counting the page update time of the target website within the preset cycle period, the number of page updates of the target website within each cycle period can be obtained, and the cycle period includes at least two target periods separated by cycles, and each target period can be divided into a first period and a second period. Therefore, the boundary between the first period and the second period can be used as a reference to split the number of page updates corresponding to the entire target period, and the number of page updates of the target website in each first period and each second period can be obtained. Alternatively, after obtaining the page update time of the target website within the preset cycle period, based on the time dividing point between the first period and the second period, the number of page updates of the target website in each first period and the second period are counted respectively. At this time, the total number of page updates in the target period is the sum of the number of page updates in the corresponding first period and the second period. In this embodiment, the steps for obtaining the number of page updates in each period will be described in detail below. However, it should be noted that the first period and the second period included in each target period should be periods with the same start and end time points.

[0074] For example, the target time period of multiple periodic intervals is 10:00-11:00. If the first time period is set to "10:00-10:30" and the second time period is set to "10:30-11:00", then the first time period included in all target time periods is "10:00-10:30", and the first time period included in all target time periods is "10:30-11:00".

[0075] In one embodiment, Figure 3 As shown, the preset periodic time period includes a first target time period and a second target time period of the periodic interval, and this step includes: S301, based on the page update time corresponding to each of the first target time period and the second target time period, determining a first time set corresponding to the first target time period and a second time set corresponding to the second target time period; S302, counting the number of times in the first time set and the second time set, and obtaining a first page update amount corresponding to the first target time period, and a second page update amount corresponding to the second target time period; S303, based on the first time period and the second time period included in each of the target time periods, splitting the first page update amount and the second page update amount to obtain the number of page updates of the target website in each of the first time periods, and the number of page updates in each of the second time periods.

[0076] Among them, based on the description in the above embodiment, when the first target period is (2020-10-06, 10-11), the first time set corresponding to the first target period may be: [10:01, 10:02, 10:03, 10:03, 10:03, 10:04…]; when the second target period is (2020-09-29, 10-11), the second time set corresponding to the second target period may be: [10:01, 10:02, 10:03, 10:03, 10:03, 10:04…]. It can be understood that the specific time points (page update times) included in the above time sets are only examples, and the actual situation depends on the actual page update time of the target website. The page update times corresponding to the target periods obtained by the server at this time exist and are stored in the form of arrays. There may be the same page update time or different page update time. What needs to be analyzed is only the number of elements in each array.

[0077] Specifically, when the preset periodic time period includes a first target period and a second target period of a periodic interval, the server may respectively obtain the page update time of the target website within the first target period, and the page update time of the target website within the second target period, to form a time set corresponding to each target period. By analyzing the amount of time included in each time set, the page update amount corresponding to each target period can be obtained, and the page update amount can be presented by one of the data in the page update data described above. For example, in the page update data: (2020-10-06, 10-11, 100, [10:01, 10:02, 10:03, 10:03, 10:03, 10:04...]), "100" represents the number of page updates within the first target time period (2020-10-06, 10-11).

[0078] More specifically, after obtaining the first page update amount corresponding to the first target time period and the second page update amount corresponding to the second target time period, the first page update amount and the second page update amount can be split according to the current division of the first and second time periods in the first and second target time periods, respectively, to obtain the page update amount of the target website in the two first time periods and the page update amount of the target website in the two second time periods. The page update amount division step involved in this embodiment will be described in detail below.

[0079] In one embodiment, Figure 4 As shown, the step of splitting the first page update amount and the second page update amount based on the first time period and the second time period included in each of the target time periods to obtain the number of page updates of the target website in each of the first time periods and the number of page updates in each of the second time periods includes: S401, splitting the first page update amount based on the first time period and the second time period included in the first target time period to obtain the third page update amount corresponding to the first time period in the first target time period and the fourth page update amount corresponding to the second time period in the first target time period; S402, splitting the second page update amount based on the first time period and the second time period included in the second target time period to obtain the fifth page update amount corresponding to the first time period in the second target time period and the sixth page update amount corresponding to the second time period in the second target time period; S403, determining the third page update amount and the fifth page update amount as the number of page updates of the target website in each of the first time periods, and determining the fourth page update amount and the sixth page update amount as the number of page updates of the target website in each of the second time periods.

[0080] Among them, the third page update amount refers to a partial page update amount in the first page update amount, and the partial page update amount is the page update amount corresponding to the first time period in the first target time period. For example, if the first page update amount is "100", then the third page update amount can be a partial page update amount less than "100", and the specific partial number depends on the first time period in the first target time period.

[0081] Among them, the fourth page update amount refers to a partial page update amount in the first page update amount, and the partial page update amount is the page update amount corresponding to the second time period in the first target time period. For example, if the first page update amount is "100", then the fourth page update amount can be a partial page update amount less than "100", and the specific partial number depends on the second time period in the first target time period.

[0082] Among them, the fifth page update amount refers to the partial page update amount in the second page update amount, and the partial page update amount is the page update amount corresponding to the first time period in the second target time period. For example, if the second page update amount is "90", then the fifth page update amount can be a partial page update amount less than "90", and the specific partial number depends on the first time period in the second target time period.

[0083] Among them, the sixth page update amount refers to the partial page update amount in the second page update amount, and the partial page update amount is the page update amount corresponding to the second time period in the second target time period. For example, the second page update amount is "90", then the sixth page update amount can be a partial page update amount less than "90", and the specific partial number depends on the second time period in the second target time period.

[0084] Specifically, to analyze how to adjust the crawler's crawling time, it is necessary to analyze the number of page updates of the target website in each time period, so as to obtain the page update trend of the target website in the historical time period, and use the page update trend in the historical time period as the basis for adjusting the crawler's crawling time, and adjust the crawler's crawling time to the best, so that the crawler can efficiently and timely obtain the updated website pages of the target website in the shortest interval time. Therefore, after the server obtains the first page update amount corresponding to the first target time period and the second page update amount corresponding to the second target time period, it can split the first page update amount and the second page update amount according to the first time period and the second time period included in the first target time period, and the first time period and the second time period included in the second target time period, respectively, to obtain the third page update amount and the fourth page update amount in the first page update amount, and obtain the fifth page update amount and the sixth page update amount in the second page update amount. At this time, the third page update volume, the fourth page update volume, the fifth page update volume and the sixth page update volume are all page update volumes of the target website within the minimum time period. Analyzing such page update volumes that are smaller than the first / second page update volumes is to analyze whether the current initial interval time of the web crawler can be compressed again, so as to obtain a more accurate interval time and obtain the crawler crawling time that can improve the timeliness of information crawling.

[0085] For example, the page update time in the first target period (2020-10-06, 10-11) is: [10:01, 10:02, 10:03, 10:03, 10:03, 10:04...], and the page update time in the second target period (2020-09-29, 10-11) is [10:01, 10:02, 10:03, 10:03, 10:03, 10:04...]; the first page update amount corresponding to the first target period is "100", and the second page update amount corresponding to the second target period is "90". The first time period of the first target period and the second target period is "10:00-10:30", and the second time period of the first target period and the second target period is "10:30-11:00". Based on the first target time period and the second target time period, the page update time corresponding to the first time period and the second time period is counted, and the update amount of the third page is "80", the update amount of the fourth page is "20", the update amount of the fifth page is "60", and the update amount of the sixth page is "30".

[0086] In one embodiment, Figure 5 As shown, the preset periodic time period also includes a third target time period that is periodically spaced from the first target time period and the second target time period. The method described in the above embodiment also includes: S501, based on the page update time corresponding to the third target time period, determining a third time set corresponding to the third target time period; S502, counting the number of times in the third time set to obtain a third page update amount corresponding to the third target time period; S503, based on the first time period and the second time period included in the third target time period, splitting the third page update amount to obtain a seventh page update amount corresponding to the first time period in the third target time period, and an eighth page update amount corresponding to the second time period in the third target time period; S504, determining the seventh page update amount as the page update amount of the target website in each of the first time periods, and determining the eighth page update amount as the page update amount of the target website in each of the second time periods.

[0087] Among them, the third target time period is the target time period in the third preset time. The third preset time is of the same nature as the first preset time and the second preset time mentioned above, that is, it can be the current time of the server / terminal device, or it can be the historical time of the device. For example, September 22, 2020, the third target time period can be from 10:00 to 11:00 on September 22, 2020, expressed as (2020-09-22, 10-11).

[0088] Specifically, the page update amount of the target website in the third target period can be analyzed by referring to the steps of analyzing the first target period and the second target period described above. Since the above embodiment has emphasized how to adjust the crawler crawling time, it is necessary to analyze the page update trend of the target website in the past historical period, and the historical period can include the current period and the same historical period with a periodic interval with the current period, that is, the target period of at least two periodic periods included in the preset periodic period, and the above embodiment has detailedly explained that when this periodic period includes two periodic interval target periods, the page update amount corresponding to each of the first period and the second period in the two target periods should be analyzed, and the crawler crawling time should be adjusted based on the page update amount as the time adjustment analysis basis, but it should not be limited to the case of only two target periods. Therefore, in this embodiment, it is proposed that the preset periodic period also includes a third target period with a periodic interval with the first target period and the second target period, and the third target period is used to participate in the analysis of the first target period and the second target period, which can improve the recognition accuracy of the page update trend of the target website, thereby making the final crawling time setting more accurate.

[0089] More specifically, the above embodiments have described in detail how to obtain the corresponding time sets based on the page update time of the first / second target time period, and obtain the page update amount corresponding to each target time period based on the time set statistics, and then obtain the page update amount corresponding to the first time period and the second time period in each target time period. The analysis principle of the third target time period involved in this embodiment is the same, so it will not be repeated in this embodiment. However, it should be noted that the first time period and the second time period included in the third target time period are the same time periods as the first time period and the second time period included in the first / second target time period, that is, they have the same start and end time points.

[0090] Furthermore, in actual applications, the server can eventually obtain the number of page updates of the target website in at least three first time periods, such as the third page update amount, the fifth page update amount, and the seventh page update amount, and can also obtain the number of page updates of the target website in at least three second time periods, such as the fourth page update amount, the sixth page update amount, and the eighth page update amount.

[0091] S103: Based on the number of page updates in each of the first time periods and the number of page updates in each of the second time periods, the crawler crawling time of the target website is adjusted to obtain an adjusted crawler crawling time.

[0092] Specifically, after the server analyzes and obtains the number of page updates in each first time period of the target website and the number of page updates in each second time period, the crawler crawling time of the target website can be adjusted based on the proportion of the number of page updates in each time period, that is, the initial interval time of the crawler is adjusted to obtain the best interval time, such as the adjusted crawler crawling time, which helps the crawler to more efficiently obtain the updated pages of the target website. The crawler crawling time adjustment steps involved in this embodiment will be described in detail below.

[0093] In one embodiment, Figure 6 As shown, this step includes: S601, obtaining a first average update number based on the number of page updates in each of the first time periods, and obtaining a second average update number based on the number of page updates in each of the second time periods; S602, obtaining a ratio between the first average update number and the second average update number as a time adjustment ratio; S603, if the time adjustment ratio is greater than or equal to a preset time adjustment threshold, adjusting the crawler time of the target website, and determining half of the crawler time as the adjusted crawler time.

[0094] Among them, the first average update quantity is the average value of the page update quantity corresponding to each first time period. For example, based on the description of the above embodiment, when the first time period includes the first target time period and the first time period within the second target time period, the third page update amount and the fifth page update amount are the page update quantities corresponding to each first time period, and the third page update amount is "80" and the fifth page update amount is "60", then the first average update amount at this time is "70".

[0095] Among them, the second average update amount is the average value of the page update amount corresponding to each second time period. For example, based on the description of the above embodiment, when the second time period includes the first target time period and the second time period within the second target time period, the fourth page update amount and the sixth page update amount are the page update amounts corresponding to each second time period, and the fourth page update amount is "20" and the sixth page update amount is "30", then the second average update amount at this time is "25".

[0096] The preset time adjustment threshold can be set according to actual business needs, for example, 2, 3, or 4. However, it should be noted that the time adjustment threshold should not be 1. If it is set to 1, there will be 64 page updates in the first half hour and 63 page updates in the second half hour. In fact, the crawler crawling time does not need to be adjusted at this time, but the server will adjust the crawler crawling time because the calculation exceeds the threshold.

[0097] Specifically, after the server analyzes and obtains the first average update number corresponding to each first time period, and the second average update number corresponding to each second time period, it can calculate the ratio between the first average update number and the second average update number, and obtain the proportion of the number of page updates of the target website in the first time period and the second time period, which is used as the time adjustment ratio. If the time adjustment ratio is greater than or equal to the preset time adjustment threshold, it means that the number of page updates of the target website in the first time period is greater than the number of page updates in the second time period, and the number of page updates controlled by the initial interval time is no longer balanced. The crawler should crawl the target web page more times in the first time period relative to the second time period. Therefore, the server can adjust the crawler crawling time, that is, adjust the initial interval time for the crawler to crawl web pages to half of its original time period, such as T=1 / 2T, as the adjusted crawler crawling time.

[0098] It can be understood that although this embodiment describes the ratio between the first average update number and the second average update number, which will be compared with the time adjustment threshold, in practice, the ratio between the second average update number and the first average update number can also be compared with the time adjustment threshold, and the principle remains unchanged.

[0099] For example, the first average update number is "70", the second average update number is "25", and the ratio between the first average update number and the second average update number is "2.8". If the time adjustment threshold at this time is "2", the crawler crawling time needs to be adjusted, that is, its initial interval time T=1, the crawler crawling time before adjustment is 10 o'clock and 11 o'clock, and the crawler crawling time after adjustment is 10 o'clock, 10:30, and 11 o'clock.

[0100] In one embodiment, Figure 7 As shown, the steps of obtaining the first average update number based on the number of page updates in each of the first time periods, and obtaining the second average update number based on the number of page updates in each of the second time periods, include: S701, determining a preset attenuation coefficient corresponding to each of the preset periodic time periods; S702, based on the attenuation coefficient, performing weighted summation processing on the number of page updates in each of the first time periods, to obtain the total first page update number corresponding to the first time period; S703, based on the attenuation coefficient, performing weighted summation processing on the number of page updates in each of the second time periods, to obtain the total second page update number corresponding to the second time period; S704, obtaining the average number of the first page update total number and the second page update total number, to obtain the first average update number and the second average update number accordingly.

[0101] The attenuation coefficient is a weighted coefficient for the number of page updates, and its size depends on the distance between the time of the historical data and the current time of the device. For example, the attenuation coefficient of the target period one week before the current time of the device is 0.9, while the attenuation coefficient of the target period two weeks before the current time of the device is 0.8, and so on.

[0102] Specifically, the previous embodiment has detailed how to obtain the first average update number and the second average update number based on the number of page updates in each first time period and the second time period. The solution provided in this embodiment is different from that in that before calculating the average update number corresponding to each time period, the number of page updates in each first time period or second time period needs to be accumulated with its corresponding preset attenuation coefficient. The attenuation coefficient has been explained above as the weighted coefficient acting on the number of page updates, specifically the weighted coefficient acting on the number of page updates in the target time period, that is, there is a weighted coefficient corresponding to different target time periods, and its size depends on the length of time from the current time of the device.

[0103] For example, if the first target period is the current time of the server device, the corresponding preset attenuation coefficient is 1, and the second target period is one week away from the first target period, so the corresponding attenuation coefficient can be 0.9, and so on to obtain the attenuation coefficient corresponding to the third target period is 0.8. At this time, if the number of page updates in each first period includes the third page update amount "80", the fifth page update amount "60" and the seventh page update amount "70", then the total amount of first page updates corresponding to the target website in the first period is: (80+60*0.9+70*0.8)=190; if the number of page updates in each second period includes the fourth page update amount "20", the sixth page update amount "30" and the eighth page update amount "20", then the total amount of second page updates corresponding to the target website in the second period is: (20+30*0.9+20*0.8)=63.

[0104] More specifically, the server calculates the total amount of first page updates of the target website corresponding to the first time period and the total amount of second page updates corresponding to the second time period based on the preset attenuation coefficient corresponding to each preset cycle time period, and then calculates the first average update number and the second average update number according to the number of page updates involved in the total calculation.

[0105] For example, the total number of first page updates is "190", and there are 3 page updates involved in the calculation, then the first average update number is: 190 / 3≈63; the total number of second page updates is "63", and there are 3 page updates involved in the calculation, then the second average update number is: 63 / 3≈21.

[0106] The above crawler crawling time adjustment method obtains the page update time of the target website within a preset cycle time period to count the number of page updates of the target website in different time periods, and can obtain the number of page updates of the target website in each first time period and the number of page updates in each second time period, and then analyzes the proportion of the number of page updates of the target website between the first half of the time period and the second half of the time period in a target time period, and finally uses the proportion to adjust the crawler's crawling time, so that the crawler can timely obtain the updated information of the target website in the shortest interval time, thereby improving the timeliness of the crawler's information crawling and reducing the waste of the crawler's bandwidth resources.

[0107] In order to enable those skilled in the art to fully understand the crawler crawling time adjustment scheme proposed in this application, this application also provides an application scenario, which applies the above-mentioned crawler crawling time adjustment method. Specifically, the application of the crawler crawling time adjustment method in this application scenario will be described as follows:

[0108] Since most existing crawlers crawl the target website during a fixed time period, this method is often not applicable to certain time-sensitive scenarios. For example, in an extreme case, the target website updates 100 pages between 10:00 and 11:00, but 99 of them are updated at 10:01 and the remaining page is updated at 10:59. In this case, the traditional information crawling method may only consider the total number of updated pages within a certain time period, but not the time point of the page update within the time period, and simply use the following formula: X:(N / D)=T':T to adjust the crawling time of the crawler.

[0109] Among them, X represents the normal average update volume of the target website, N represents the total update volume of the target website within time D, D represents the crawling time of the crawler, and T represents the initial interval time of the crawler crawling information. At this time, the normal average update volume of the target website is "100", the actual average update volume within time D (N / D) is "150", and the initial interval time is 60 minutes, then 100:150 = T': 60min, and the adjusted interval time T' = 40min. In other words, the traditional information crawling method will not adjust the crawler's crawling strategy. For the update of the above 100 pages, it cannot handle it because the page update volume does not change within a fixed period of time.

[0110] The method proposed in this application is to adjust the crawler crawling time according to the proportion of page updates in two sub-periods within a fixed time period. Therefore, the crawling strategy will inevitably be adjusted to continuously shorten the interval time for the crawler to crawl information. The target website will not update data at 10:01, and the server will not crawl until 11 o'clock, which greatly improves the timeliness of the crawling.

[0111] For example, the web crawler sets the initial crawling interval for the target website abc.com to 1 hour, and it currently obtains the page update data between 10 and 11 o'clock. After analysis, it is found that the target website updated 63 pages in the first half hour and 21 pages in the second half hour, which exceeds the threshold (assuming the threshold is 2). Therefore, it is necessary to make another hypothetical split for the first half hour and the second half hour. Since the previous target period has been split into 30 minutes, the target period will be split into 15 minutes. It is found that the target website updated 30 pages in the first 15 minutes and 33 pages in the second 15 minutes, which is lower than the threshold, so the first half hour will not be split. In the second half hour, 16 pages were updated in the first 15 minutes and 5 pages were updated in the second 15 minutes, which exceeds the threshold, so the second half hour is determined to be set to 15 minutes. Then look at the number of pages in the first 7.5 minutes and the last 7.5 minutes until it cannot be split. Finally, the adjusted interval time is one climb at 10:30 and one climb at 10:45 based on the original initial interval time.

[0112] In this embodiment, by obtaining the page update time of the target website within a preset cycle time period, the number of page updates of the target website in different time periods is counted, and the number of page updates of the target website in each first time period and the number of page updates in each second time period can be obtained. Then, the proportion of the page updates of the target website between the first half of the time period and the second half of the time period in a target time period is analyzed, and finally the proportion is used to adjust the crawling time of the crawler, so that the crawler can timely obtain the updated information of the target website in the shortest interval time, thereby improving the timeliness of the crawler's information crawling and reducing the waste of the crawler's bandwidth resources.

[0113] In order to better implement the crawler crawling time adjustment method in the embodiment of the present application, based on the crawler crawling time adjustment method, a crawler crawling time adjustment device is also provided in the embodiment of the present application, such as Figure 8 As shown, the crawler crawling time adjustment device 800 includes:

[0114] A time acquisition module 810 is used to acquire the page update time of the target website within a preset period, wherein the preset period includes at least two target periods separated by a period, and the target period includes a first period and a second period;

[0115] A time analysis module 820, for counting the number of page updates of the target website in each of the first time periods and the number of page updates of the target website in each of the second time periods based on the page update time;

[0116] The time adjustment module 830 is used to adjust the crawler crawling time of the target website based on the number of page updates in each of the first time periods and the number of page updates in each of the second time periods to obtain an adjusted crawler crawling time.

[0117] In some embodiments of the present application, the time acquisition module 810 is also used to obtain the page update time of the target website within a first target time period, where the first target time period is a target time period in a first preset time; obtain the page update time of the target website within a second target time period, where the second target time period is a target time period in a second preset time, where the first preset time and the second preset time are preset times separated by two periodic intervals; determine the page update time within the first target time period and the page update time within the second target time period as the page update time within the preset periodic period.

[0118] In some embodiments of the present application, the preset periodic time period includes a first target periodic time period and a second target periodic time period at periodic intervals, and the time analysis module 820 is further used to determine a first time set corresponding to the first target periodic time and a second time set corresponding to the second target periodic time based on the page update times corresponding to the first target periodic time and the second target periodic time respectively; count the number of times in the first time set and the second time set to obtain a first page update amount corresponding to the first target periodic time and a second page update amount corresponding to the second target period; based on the first periodic time and the second periodic time included in each of the target periodic time, split the first page update amount and the second page update amount to obtain the number of page updates of the target website in each of the first periodic time and the number of page updates in each of the second periodic time.

[0119] In some embodiments of the present application, the time analysis module 820 is also used to split the first page update amount based on the first time period and the second time period included in the first target time period, and obtain the third page update amount corresponding to the first time period in the first target time period, and the fourth page update amount corresponding to the second time period in the first target time period; split the second page update amount based on the first time period and the second time period included in the second target time period, and obtain the fifth page update amount corresponding to the first time period in the second target time period, and the sixth page update amount corresponding to the second time period in the second target time period; determine the third page update amount and the fifth page update amount as the number of page updates of the target website in each of the first time periods, and determine the fourth page update amount and the sixth page update amount as the number of page updates of the target website in each of the second time periods.

[0120] In some embodiments of the present application, the time analysis module 820 is also used to determine a third time set corresponding to the third target time period based on the page update time corresponding to the third target time period; count the number of times in the third time set to obtain a third page update amount corresponding to the third target time period; based on a first time period and a second time period included in the third target time period, split the third page update amount to obtain a seventh page update amount corresponding to the first time period in the third target time period, and an eighth page update amount corresponding to the second time period in the third target time period; determine the seventh page update amount as the page update amount of the target website in each of the first time periods, and determine the eighth page update amount as the page update amount of the target website in each of the second time periods.

[0121] In some embodiments of the present application, the time adjustment module 830 is also used to obtain a first average update number based on the number of page updates in each of the first time periods, and to obtain a second average update number based on the number of page updates in each of the second time periods; to obtain a ratio between the first average update number and the second average update number as a time adjustment ratio; if the time adjustment ratio is greater than or equal to a preset time adjustment threshold, the crawler crawling time of the target website is adjusted, and half of the crawler crawling time is determined as the adjusted crawler crawling time.

[0122] In some embodiments of the present application, the time adjustment module 830 is also used to determine a preset attenuation coefficient corresponding to each of the preset cycle time periods; based on the attenuation coefficient, a weighted summation is performed on the number of page updates in each of the first time periods to obtain a total first page update number corresponding to the first time period; based on the attenuation coefficient, a weighted summation is performed on the number of page updates in each of the second time periods to obtain a total second page update number corresponding to the second time period; the average number of the first page update total and the second page update total is obtained, and the first average update number and the second average update number are obtained accordingly.

[0123] In the above embodiment, by obtaining the page update time of the target website within a preset period, the number of page updates of the target website in different time periods is counted, and the number of page updates of the target website in each first time period and the number of page updates in each second time period can be obtained. Then, the proportion of the page updates of the target website between the first half of the time period and the second half of the time period in a target time period is analyzed, and finally the crawling time of the crawler is adjusted according to the proportion, so that the crawler can obtain the updated information of the target website in time within the shortest interval time, thereby improving the timeliness of the crawler's information crawling and reducing the waste of the crawler's bandwidth resources.

[0124] In some embodiments of the present application, the crawler crawling time adjustment device 800 can be implemented in the form of a computer program. The computer program can be implemented in Fig. 9 The computer device shown in the figure is run. The memory of the computer device can store various program modules constituting the crawler crawling time adjustment device 800, for example, Figure 8 The time acquisition module 810, time analysis module 820 and time adjustment module 830 are shown. The computer program composed of various program modules enables the processor to execute the steps of the crawler crawling time adjustment method of each embodiment of the present application described in this specification.

[0125] For example, Fig. 9 The computer device shown can be Figure 8 The time acquisition module 810 in the crawler crawling time adjustment device 800 shown executes step S101. The computer device can execute step S102 through the time analysis module 820. The computer device can execute step S103 through the time adjustment module 830. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external computer device through a network connection. When the computer program is executed by the processor, a crawler crawling time adjustment method is implemented.

[0126] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0127] In some embodiments of the present application, a computer device is provided, comprising one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor in the steps of the above-mentioned crawler crawling time adjustment method. The steps of the crawler crawling time adjustment method here may be the steps of the crawler crawling time adjustment method in each of the above-mentioned embodiments.

[0128] In some embodiments of the present application, a computer-readable storage medium is provided, which stores a computer program, and the computer program is loaded by a processor, so that the processor executes the steps of the above-mentioned crawler crawling time adjustment method. The steps of the crawler crawling time adjustment method here can be the steps of the crawler crawling time adjustment method in each of the above-mentioned embodiments.

[0129] The above is a detailed introduction to a crawler crawling time adjustment method, device, computer equipment and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A crawler crawling time adjustment method, characterized in that: The method comprises: Acquire a page update time of a target website within a preset period, wherein the preset period includes at least two periodically spaced target periods, wherein the target period includes a first period and a second period; Based on the page update time, counting the number of page updates of the target website in each of the first time periods, and the number of page updates of the target website in each of the second time periods; Determining a preset attenuation coefficient corresponding to each of the preset periodic time periods; Based on the attenuation coefficient, weighted summing is performed on the number of page updates in each of the first time periods to obtain a total number of first page updates corresponding to the first time period; Based on the attenuation coefficient, weighted sum processing is performed on the number of page updates in each of the second time periods to obtain a total number of second page updates corresponding to the second time period; Obtaining an average of the total amount of updates to the first page and the total amount of updates to the second page, and correspondingly obtaining a first average update amount and a second average update amount; Obtaining a ratio between the first average update number and the second average update number as a time adjustment ratio; If the time adjustment ratio is greater than or equal to a preset time adjustment threshold, the crawler crawling time of the target website is adjusted, and half of the crawler crawling time is determined as the adjusted crawler crawling time.

2. The crawler crawling time adjustment method according to claim 1, characterized in that: The step of obtaining the page update time of the target website within a preset period includes: Obtaining a page update time of the target website within a first target time period, where the first target time period is a target time period in a first preset time period; Obtaining a page update time of the target website within a second target period, where the second target period is a target period in a second preset time, and the first preset time and the second preset time are preset times spaced apart by two cycles; The page update time within the first target period and the page update time within the second target period are determined as the page update time within the preset period.

3. The crawler crawling time adjustment method according to claim 1, characterized in that: The preset period includes a first target period and a second target period at periodic intervals, and the step of counting the number of page updates of the target website in each of the first period and the number of page updates of the target website in each of the second period based on the page update time includes: Determine, based on the page update times respectively corresponding to the first target period and the second target period, a first time set corresponding to the first target period and a second time set corresponding to the second target period; Counting the number of times in the first time set and the second time set to obtain a first page update amount corresponding to the first target time period and a second page update amount corresponding to the second target time period; Based on the first time period and the second time period included in each of the target time periods, the first page update amount and the second page update amount are split and processed to obtain the page update amount of the target website in each of the first time periods and the page update amount in each of the second time periods.

4. The crawler crawling time adjustment method according to claim 3, characterized in that: The step of splitting the first page update amount and the second page update amount based on the first time period and the second time period included in each target time period to obtain the number of page updates of the target website in each first time period and the number of page updates in each second time period includes: Based on a first time period and a second time period included in the first target time period, the first page update amount is split to obtain a third page update amount corresponding to the first time period in the first target time period and a fourth page update amount corresponding to the second time period in the first target time period; Based on a first time period and a second time period included in the second target time period, the second page update amount is split to obtain a fifth page update amount corresponding to the first time period in the second target time period and a sixth page update amount corresponding to the second time period in the second target time period; The third page update amount and the fifth page update amount are determined as the page update amount of the target website in each of the first time periods, and the fourth page update amount and the sixth page update amount are determined as the page update amount of the target website in each of the second time periods.

5. The crawler crawling time adjustment method according to claim 3 or 4, characterized in that: The preset period also includes a third target period that is periodically spaced from the first target period and the second target period, and the method further includes: Determining a third time set corresponding to the third target time period based on the page update time corresponding to the third target time period; Counting the number of times in the third time set to obtain a third page update amount corresponding to the third target time period; Based on the first time period and the second time period included in the third target time period, the third page update amount is split to obtain a seventh page update amount corresponding to the first time period in the third target time period and an eighth page update amount corresponding to the second time period in the third target time period; The seventh page update amount is determined as the page update amount of the target website in each of the first time periods, and the eighth page update amount is determined as the page update amount of the target website in each of the second time periods.

6. A crawler crawling time adjustment device, characterized in that: The device comprises: A time acquisition module, used to acquire the page update time of the target website within a preset period, wherein the preset period includes at least two periodically spaced target periods, wherein the target period includes a first period and a second period; A time analysis module, for counting the number of page updates of the target website in each of the first time periods and the number of page updates of the target website in each of the second time periods based on the page update time; A time adjustment module, used to determine a preset attenuation coefficient corresponding to each of the preset periodic time periods; Based on the attenuation coefficient, weighted summing is performed on the number of page updates in each of the first time periods to obtain a total number of first page updates corresponding to the first time period; Based on the attenuation coefficient, weighted sum processing is performed on the number of page updates in each of the second time periods to obtain a total number of second page updates corresponding to the second time period; Obtaining an average of the total amount of updates to the first page and the total amount of updates to the second page, and correspondingly obtaining a first average update amount and a second average update amount; Obtaining a ratio between the first average update number and the second average update number as a time adjustment ratio; If the time adjustment ratio is greater than or equal to a preset time adjustment threshold, the crawler crawling time of the target website is adjusted, and half of the crawler crawling time is determined as the adjusted crawler crawling time.

7. A computer device, characterized in that: The computer device comprises: one or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the crawler crawling time adjustment method described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the crawler crawling time adjustment method described in any one of claims 1 to 5.

Images