Data acquisition method and device, storage medium and computer device

Abstract

The application relates to the field of electronic information technology and discloses a data acquisition method and device, a storage medium and computer equipment. The method comprises the following steps: constructing a task list, determining a target site in the task list, acquiring request parameters of each webpage in the target site, then extracting a target webpage in the target site according to the request parameters, calculating the priority of the target webpage, generating a task execution queue based on the priority, sending the task execution queue to at least two host devices, executing a data acquisition task based on the task execution queue, obtaining acquisition data of the target webpage, finally analyzing and cleaning and optimizing the acquisition data to obtain a data acquisition result, and storing the data acquisition result in a database. The above method can acquire data of multiple target webpages through at least two host devices in a high-concurrency mode, and can sort the data acquisition sequence of the multiple target webpages according to the priority, so that the data acquisition mode is flexible and efficient.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electronic information, and in particular to a data collection method and device, a storage medium and computer equipment. BACKGROUND

[0002] With the rapid development of information technology, the World Wide Web has become an important information data carrier. In order to stably and efficiently and accurately obtain data in the massive information of the World Wide Web, various data collection technologies have emerged as the times require. Application of data collection technology can quickly capture data from the World Wide Web and structure it into a database.

[0003] At present, with the increasing types and quantities of business requirements, the amount of data to be collected is also increasing, and the frequency of data to be collected is also increasing. At the same time, the data in the World Wide Web is growing in geometric order, and the data acquisition mechanisms set by various websites are different, resulting in longer task execution time, increased data collection difficulty, lower data collection efficiency, and thus the conventional data collection technology cannot flexibly, quickly and efficiently collect web page data of multiple websites. SUMMARY

[0004] Therefore, the data collection method, device, storage medium and computer equipment provided by the present application mainly aim to solve the technical problem that the conventional data collection technology in the prior art cannot flexibly, quickly and efficiently collect web page data of multiple websites.

[0005] According to a first aspect of the present application, a data collection method is provided, which comprises:

[0006] constructing a task list, determining a target site in the task list, and obtaining a request parameter of each web page in the target site, wherein the target site comprises at least one web page;

[0007] According to the request parameter, a target web page is extracted from the target site, and the priority of the target web page is calculated, and a task execution queue is generated based on the priority of the target web page;

[0008] The task execution queue is sent to at least two host devices, so that the at least two host devices execute data collection tasks one by one based on the task execution queue, and obtain collection data of the target web page;

[0009] The collection data is parsed and cleaned and optimized to obtain a data collection result, and the data collection result is stored in a database.

[0010] According to a second aspect of the present application, a data collection device is provided, which comprises:

[0011] The task acquisition module is configured to construct a task list, determine a target site in the task list, and acquire a request parameter of each web page in the target site, wherein the target site comprises at least one web page.

[0012] The task generation module is configured to extract a target web page in the target site according to the request parameter, calculate a priority of the target web page, and generate a task execution queue based on the priority of the target web page.

[0013] The task execution module is configured to send the task execution queue to at least two host devices, so that the at least two host devices execute data collection tasks one by one based on the task execution queue, and obtain collection data of the target web page.

[0014] The data storage module is configured to parse and clean and optimize the collection data, obtain data collection results, and store the data collection results in a database.

[0015] According to a third aspect of the present application, a storage medium is provided, which stores a computer program, and the program is executed by a processor to implement the above data collection method.

[0016] According to a fourth aspect of the present application, a computer device is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to implement the above data collection method.

[0017] The application provides a data collection method and device, a storage medium and a computer device. The application first constructs a task list, determines a target site in the task list, and obtains request parameters of each webpage in the target site, wherein the target site comprises at least one webpage. Then, the target webpage is extracted in the target site according to the request parameters, and the priority of the target webpage is calculated. The task execution queue is generated based on the priority of the target webpage, and the task execution queue is sent to at least two host devices, so that the at least two host devices execute the data collection task one by one based on the task execution queue, obtain the collection data of the target webpage, finally analyze and clean and optimize the collection data to obtain the data collection result, and store the data collection result in the database. The above method screens the webpages in the target site, determines the target webpage to be collected, and avoids wasting data collection time by directly accessing invalid webpages. The priority of each target webpage is calculated by obtaining the request parameters of the target webpage itself, and the task execution queue generated based on the priority of the target webpage is generated to optimize the data collection sequence of each target webpage. The task execution queue is sent to at least two host devices to execute the data collection task, so that the data of the target webpage can be collected asynchronously, and the efficiency of data collection is improved. Finally, the obtained data collection result is stored in the database, which is convenient for checking and obtaining at any time. The above method collects data of multiple target webpages through at least two host devices with high concurrency, and sorts the multiple target webpages in a targeted manner. The data collection method is flexible and efficient, and the situation that the data collection task is inefficient due to different data acquisition mechanisms of each target site is avoided.

[0018] The above description is only a summary of the technical scheme of the application. In order to more clearly understand the technical means of the application, the application can be implemented according to the content of the specification, and in order to make the above and other purposes, characteristics and advantages of the application more obvious and easy to understand, the following specific embodiments of the application are described. BRIEF DESCRIPTION OF DRAWINGS

[0019] The drawings described herein are used to provide further understanding of the application, and form a part of the application. The schematic embodiments of the application and their descriptions are used to explain the application, and do not constitute an improper limitation on the application. In the drawings:

[0020] Figure 1 A flowchart of a data collection method in an embodiment provided by the application is shown;

[0021] Figure 2 A flowchart of a data collection method in an embodiment provided by the application is shown;

[0022] Figure 3 A principle flowchart of a data collection method in an embodiment provided by the application is shown;

[0023] Figure 4 Fig. 1 shows a structural schematic diagram of a data collection device according to an embodiment of the present application;

[0024] Figure 5 Fig. 2 shows a structural schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] Exemplary embodiments of the present application will be described in detail with reference to the drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0026] An embodiment of the present application provides a data collection method, as shown in the figure, the method comprises the following steps: Figure 1

[0027] 101, constructing a task list, determining a target site in the task list, and obtaining the request parameters of each webpage in the target site, wherein the target site comprises at least one webpage.

[0028] First, the application scenarios applicable to the present method are introduced. With the rapid development of information technology, in order to stably and efficiently and accurately obtain data in the vast amount of information on the World Wide Web, data collection technology is usually applied to quickly capture data from the World Wide Web and structure it into a database. However, with the increasing types and quantities of business needs, the amount of data required for collection is also increasing, and the frequency of data required for collection is also increasing. At the same time, the data in the World Wide Web is growing in geometric order, and the data acquisition mechanisms set by various websites are different, resulting in longer task execution time, increased difficulty of data collection, lower efficiency of data collection, and thus the conventional data collection technology cannot flexibly and quickly collect data from multiple websites.

[0029] Specifically, a site refers to the accurate access address of a website in the Internet, and a website is a collection of web pages for displaying specific content related web pages made by using HTML standard universal markup language and other tools. By obtaining and accessing the site of the website, all web pages in the website can be accessed, and the web pages contained in the website can be one or many. The multiple web pages in the website are usually linked by different web addresses, that is, the link of one web page can open another linked web page.

[0030] ​Further, the application provides a data collection method. First, a task list is constructed, and a target site is determined in the task list. The target site can be directly accessed to access the webpage of the required data collection. There is usually one or more webpages in the target site, and the webpage of the required data collection is in the target site. Therefore, after the target site is determined, the request parameters of each webpage in the target site are obtained, so as to facilitate the screening of all webpages in the target site by using the request parameters to find the webpage of the required data collection.

[0031] 102. According to the request parameter, the target webpage is extracted from the target site, and the priority of the target webpage is calculated. The task execution queue is generated based on the priority of the target webpage.

[0032] Specifically, the priority is a parameter for a computer time-sharing operating system to determine the priority level of each job program, process or task to accept system resources when processing multiple job programs, processes or tasks. The priority is usually a convention, and in general, the system executes the task with a higher priority first, and then executes the task with a lower priority in order of priority. Each job program or task needs to be assigned a priority according to certain standards or requirements, such as the nature or length of the program, or the source of the job, and other specified conditions to determine the priority. Then, the computer determines the order of processing each program according to the priority of each job program, and even in the execution or processing of multiple job programs, processes or tasks, the process with a higher priority can interrupt the process with a lower priority to meet the needs of the user.

[0033] In the embodiment of the application, since the target site usually contains multiple webpages, only specific webpages are target webpages for data collection. Therefore, all webpages in the target site need to be screened in advance to extract the target webpages for data collection, and some invalid webpages or inaccessible webpages in other target sites do not need to be accessed to avoid wasting data collection time and affecting collection efficiency. The screening is performed according to the request parameters of the webpages. After calculating the priority of all target webpages, the task execution queue is generated according to the priority of the target webpages according to the requirement of data collection. In the task execution queue, the target webpages for data collection are arranged in order of priority. The data collection task is executed according to the task execution queue, so that the data of each target webpage can be collected according to the priority order, and the order of data acquisition of each target webpage is differentiated to effectively improve the data acquisition efficiency.

[0034] 103. The task execution queue is sent to at least two host devices, so that the at least two host devices execute the data collection task one by one based on the task execution queue to obtain the collected data of the target webpage.

[0035] Specifically, the data collection task is executed by using a data collection technology to collect data of the target webpage. The data collection is a program or script that automatically captures data information on the World Wide Web according to a preset rule. The task execution queue is sent to at least two host devices, so that the at least two host devices simultaneously run data collection to cooperatively execute the data collection task, and the overall architecture of distributed data collection is achieved.

[0036] In the embodiments of the present application, the host device for generating the task execution queue is a master of distributed data collection, which is mainly used to generate and maintain the task execution queue according to requirements. The master of distributed data collection sends the task execution queue to at least two host devices, and the host devices receiving the task execution queue are slaves of distributed data collection, which are mainly responsible for data collection, data processing and data storage.

[0037] 104. The collected data is parsed and cleaned to obtain data collection results, and the data collection results are stored in a database.

[0038] In the embodiments of the present application, the at least two host devices receiving the task execution queue execute the data collection task, that is, the slaves of distributed data collection are used for data collection, data processing and data storage. The at least two host devices execute the data collection task to obtain collected data, and further process the collected data, specifically including parsing and cleaning the collected data, and storing the obtained data collection results in a database, so as to facilitate real-time viewing and acquisition.

[0039] The application provides a data collection method and device, a storage medium and a computer device. The application first constructs a task list, determines a target site in the task list, and obtains request parameters of each webpage in the target site, wherein the target site comprises at least one webpage. Then, the target webpage is extracted from the target site according to the request parameters, the priority of the target webpage is calculated, the task execution queue is generated based on the priority of the target webpage, the task execution queue is sent to at least two host devices, so that the at least two host devices execute the data collection task one by one based on the task execution queue, the collection data of the target webpage is obtained, the collection data is finally parsed, cleaned and optimized to obtain the data collection result, and the data collection result is stored in the database. The above method screens the webpages in the target site, determines the target webpage to be collected, and avoids wasting data collection time by directly accessing invalid webpages. The priority of each target webpage is calculated by obtaining the request parameters of the target webpage itself, and the task execution queue generated based on the priority of the target webpage is generated to optimize the data collection sequence of each target webpage. The task execution queue is sent to at least two host devices to execute the data collection task, so that the data of the target webpage can be collected asynchronously, and the efficiency of data collection is improved. Finally, the obtained data collection result is stored in the database, so that it can be viewed and obtained at any time. The above method collects the data of multiple target webpages through at least two host devices in high concurrency, and sorts the multiple target webpages, so that the data collection method is flexible and efficient, and the situation that the data collection task is inefficient due to different data acquisition mechanisms of each target site is avoided.

[0040] The application also provides a data collection method, as shown in Figure 2 The method comprises the following steps:

[0041] 201. Construct a task list, determine a target site in the task list, and obtain request parameters of webpages in the target site.

[0042] Specifically, the steps of determining the target site and obtaining the request parameters of each webpage in the target site are described in step 101, and will not be repeated here.

[0043] 202. Extract the target webpage from the target site according to the request parameters, calculate the priority of the target webpage, and generate the task execution queue based on the priority of the target webpage.

[0044] Specifically, the request parameters include a status code, a response time and a result quantity, the status code of each webpage in the target site is acquired, and the status code of each webpage is compared with a preset status code, when the status code of any webpage is the same as the preset status code, the webpage is marked as a target webpage, then the response time and the result quantity of the target webpage are acquired, and the priority of the target webpage is calculated based on the response time and the result quantity of the target webpage, finally, the target webpages are sorted based on the priority of the target webpages according to a sorting rule of the values of the priority from high to low, and a task execution queue is obtained.

[0045] In the embodiment of the application, the status code, the response time and the result quantity of the webpages in the target site are acquired, mainly by acquiring the access logs of the webpages, and then the request parameters are obtained by counting the access logs. The preset status code is 200, that is, the status code of the webpage is 200, which means that the webpage response is normal and can be normally accessed. By screening all the webpages in the target site, the webpages that can normally respond are selected, the webpages that can be normally accessed are marked as target webpages, and an initial task queue is constructed, the target webpages are added to the initial task queue, and the data of the target webpages is prepared to be collected. The webpages with a status code other than 200 represent abnormal webpage response, and the data collection of the abnormal webpage is abandoned, and the status code of other webpages in the target site is compared in turn, until the comparison of the status codes of all the webpages in the target site is completed. The initial task queue contains all the target webpages that need to be collected, and the abnormal webpages will not appear in the initial task queue. By screening the webpages in the target site, the target webpages that can normally respond are obtained, and the attempt to access the abnormal webpages in the target site is avoided, which can waste the data collection time. Setting the status code condition comparison can effectively improve the efficiency of data collection. After the priority of all the target webpages is calculated, the target webpages are sorted according to the priority from high to low, a task execution queue is obtained based on the initial task queue, and the data collection task is executed according to the sorting of the target webpages in the task execution queue to asynchronously crawl the data of all the target webpages, which effectively improves the efficiency of data collection.

[0046] Further, the priority of the target webpage is calculated based on the response time and the result quantity of the target webpage by a preset priority calculation formula, and the priority calculation formula is:

[0047]

[0048] Wherein, P is the priority of the target webpage, t is the response time of the target webpage, and num is the result quantity of the target webpage.

[0049] In this embodiment, after filtering web pages from the target site, an initial queue containing the target web pages is generated. Further calculation of the priority order of the target web pages is needed to determine the task execution queue. Specifically, the priority of a target web page is mainly determined by its response time *t* and the number of returned results *num*. Target web pages with shorter response times and fewer results are given priority for data collection tasks. Considering that the response time difference between most web pages is on the order of milliseconds, a constant needs to be added to the denominator of the priority calculation formula to increase the difference in priority calculation between target web pages, avoid similar values ​​of P affecting the sorting effect, and obtain a more accurate task execution queue.

[0050] 203. Send the task execution queue to at least two host devices and store the task execution queue in the system path of at least two host devices.

[0051] Specifically, when the data acquisition task is disconnected, the data acquisition task is stopped, and the interruption node of the data acquisition task is saved to the system path. When the data acquisition task is reconnected, the interruption node is extracted from the system path, and the data acquisition task continues to be executed from the interruption node.

[0052] In this embodiment, at least two host devices act as slave devices in a distributed data acquisition process to execute data acquisition tasks. When the task execution queue is distributed to the at least two host devices, the task execution queue is pre-stored in the system paths of the at least two host devices. If a host device experiences an unexpected power outage, causing the data acquisition process to be interrupted and the data acquisition task to stop, the interruption point of the data acquisition task can be directly saved in the system path. Once the host device resumes normal operation, the interruption point of the data acquisition task can be directly retrieved from the system path to continue executing the data acquisition task. This avoids interruptions to the data acquisition task due to unexpected situations, preventing the need to restart the data acquisition task. It allows for recovery from the interruption point and effectively improves the stability of the data acquisition task execution.

[0053] 204. At least two host devices execute data acquisition tasks one by one based on the task execution queue and deduplicate access request fingerprints.

[0054] Specifically, first, an access request deduplication set is constructed in at least two host devices, wherein the access request deduplication set includes existing access request fingerprints, and then, based on the task execution queue, a new access request fingerprint corresponding to a target webpage in the task execution queue is obtained, the new access request fingerprint is compared with the existing access request fingerprints, when the new access request fingerprint is consistent with any existing access request fingerprint, the new access request fingerprint is removed, and the target webpage corresponding to the new access request fingerprint in the task execution queue is determined, and the data collection task for the target webpage is stopped, and when the new access request fingerprint is inconsistent with any existing access request fingerprint, the new access request fingerprint is added to the access request deduplication set as an existing access request fingerprint.

[0055] In the embodiment of the present application, when performing a data collection task, an http request, that is, an access request fingerprint, is generated, and in the task execution queue, there can be multiple same access request fingerprints in a target webpage, therefore, an access request deduplication set is set to record new access request fingerprints corresponding to the target webpage, that is, http requests generated in data collection, and whether the new access request fingerprint is duplicated with an existing access request fingerprint in the access request deduplication set is determined, if the fingerprints are duplicated, the new access request fingerprint needs to be removed, and the target webpage corresponding to the new access request fingerprint will not be accessed for data collection, and if the fingerprints are not duplicated, the new access request fingerprint is added to the access request deduplication set as an existing access request fingerprint, until the comparison of all access request fingerprints corresponding to the target webpages in the task execution queue is completed, and it is ensured that all access request fingerprints in the access request deduplication set are unique, which effectively avoids collecting the repeated webpages in the task execution queue, and further improves the efficiency of data collection.

[0056] Specifically, when performing a data collection task on a target webpage, a waiting time RN is set between two data collection tasks, the waiting time RN is a randomly generated waiting time, and the waiting time RN is greater than the response time t as a safety buffer time, which avoids collecting data at a very high rate to hit the target site limit strategy, causing the target site load to be too large, affecting the normal business order and data collection efficiency. By setting the waiting time RN, it can effectively avoid using the same frequency to crawl the target site to hit the limit strategy of the target site, and affect the progress of the data collection task.

[0057] 205、According to the execution state of the data collection task, different data collection strategies are executed to obtain the collection data of the target webpage.

[0058] Specifically, when the at least two host devices detect that the current data collection task is paused, the task execution state of the current data collection task is obtained, if the task execution state is a blocking state, the data collection task of the current target webpage is paused, and based on the task execution queue, the data collection tasks of the remaining target webpages except the current target webpage are executed in turn until the current data collection task is responded, and the data collection task of the current target webpage is continued to be executed, if the task execution state is a non-blocking state, the current data collection task is waited to start, and the current data collection task is continued to be executed.

[0059] In the embodiment of the present application, since multiple data collection tasks are executed in the same target site when the data collection task is executed, after the target site completes the data collection task of the target webpage with the highest priority, the data collection task is continued to be performed, and when the get method in the python_asyncio package is executed in the process of performing the data collection task, the data collection task of the current target webpage is suspended, and thus the state of the current data collection task needs to be judged, if the task execution state of the current data collection task is a blocking state, the current data collection task cannot be continued to be executed in a short time, and thus the data collection tasks of other target webpages are continued to be executed according to the priority order in the task execution list, if the task execution state of the current data collection task is a non-blocking state, the current data collection task is immediately woken up after being suspended, and after the current data collection task is woken up, the current data collection task is continued to be executed, and the execution order is not affected. The above method can track and judge the state of the data collection task in real time, and determine the execution order of each target webpage according to the different states of the data collection task, which has strong flexibility.

[0060] 206, the collected data is parsed and cleaned and optimized to obtain a data collection result, and the data collection result is stored in a database.

[0061] Specifically, the collected data is parsed to obtain a page tag of the target webpage, the collected data is cleaned and optimized based on the page tag to obtain a data collection result, and the data collection result includes a field type of the target webpage, and finally the data collection result is stored in the database in a structured manner according to the field type.

[0062] In the embodiment of the present application, the page label of the target webpage is obtained by analyzing the collected data of the target webpage, and the data is further cleaned and optimized based on the page label, specifically including data cleaning, filtering and optimization, to obtain the collection result, wherein the data cleaning, filtering and optimization are common data processing means, specifically including cleaning the information, unifying the format, filtering and removing noise data from the text content, deeply filtering and optimizing the text content, and finally structuring the collection result according to the field type of the target webpage in the collection result and storing it in the database to realize the storage of the target webpage data, facilitating the viewing and extraction of the data.

[0063] The application provides a data collection method and device, a storage medium and a computer device, and the specific principle is as shown in the Figure 3 The application first constructs a task list, determines the target site in the task list, and obtains the request parameters of the webpages in the target site, then extracts the target webpage in the target site according to the request parameters, calculates the priority of the target webpage, generates a task execution queue based on the priority of the target webpage, sends the task execution queue to at least two host devices, and stores the task execution queue in the system path of the at least two host devices, the at least two host devices execute the data collection tasks one by one based on the task execution queue and remove the access request fingerprints, then execute different data collection strategies according to the execution state of the data collection tasks, obtain the collection data of the target webpage, finally analyze and clean and optimize the collection data to obtain the data collection result, and store the data collection result in the database. The above method obtains the target webpage by obtaining the request parameters of each webpage in the target site and screening the webpages based on the request parameters, calculates the priority of the target webpage, executes the data collection tasks in the at least two host devices according to the obtained task execution queue, removes the access request fingerprints in the process of executing the data collection tasks, executes different data collection strategies according to the execution state of the data collection tasks, and finally stores the data collection result in the database. The above method effectively improves the efficiency of data collection and improves the flexibility of data collection.

[0064] Further, as a specific implementation of the Figure 1 The embodiment of the present application provides a data collection device, as shown in the Figure 4 The device includes a task acquisition module 401, a task generation module 402, a task execution module 403 and a data storage module 404.

[0065] The task acquisition module 401 can be used to construct a task list, determine the target site in the task list, and obtain the request parameters of each webpage in the target site, wherein the target site includes at least one webpage;

[0066] The task generation module 402 can be configured to extract target webpages from the target website according to the request parameters, and calculate the priorities of the target webpages, and generate a task execution queue based on the priorities of the target webpages.

[0067] The task execution module 403 can be configured to send the task execution queue to at least two host devices, so that the at least two host devices execute data collection tasks one by one based on the task execution queue, and obtain collection data of the target webpages.

[0068] The data storage module 404 can be configured to parse and clean the collection data, obtain data collection results, and store the data collection results in a database.

[0069] In a specific application scenario, the task generation module 402 can be configured to obtain a status code of each webpage in the target website, compare the status code of each webpage with a preset status code, mark the webpage as a target webpage when the status code of any webpage is the same as the preset status code, obtain a response time and a result number of the target webpage, calculate the priority of the target webpage based on the response time and the result number of the target webpage, sort the target webpages according to a sorting rule from high to low based on the priorities of the target webpages, and obtain the task execution queue.

[0070] In a specific application scenario, the task generation module 402 can also be configured to calculate the priority of the target webpage based on the response time and the result number of the target webpage by using a preset priority calculation formula, and the priority calculation formula is as follows:

[0071]

[0072] Wherein, P is the priority of the target webpage, t is the response time of the target webpage, and num is the result number of the target webpage.

[0073] In a specific application scenario, the task execution module 403 can be configured to construct an access request de-duplication set in the at least two host devices, wherein the access request de-duplication set includes existing access request fingerprints, obtain a new access request fingerprint corresponding to a target webpage in the task execution queue based on the task execution queue, compare the new access request fingerprint with the existing access request fingerprints, remove the new access request fingerprint when the new access request fingerprint is consistent with any existing access request fingerprint, determine the target webpage corresponding to the new access request fingerprint in the task execution queue, and stop executing a data collection task on the target webpage, and add the new access request fingerprint to the access request de-duplication set as an existing access request fingerprint when the new access request fingerprint is inconsistent with any existing access request fingerprint.

[0074] In a specific application scenario, the task execution module 403 can also be configured to acquire a task execution state of the current data collection task when the at least two host devices detect that the current data collection task is paused, and if the task execution state is a blocking state, pause the execution of the data collection task on the current target webpage, and execute the data collection tasks of the remaining target webpages except the current target webpage in sequence based on the task execution queue until the current data collection task is responded to, and continue to execute the data collection task of the current target webpage, or if the task execution state is a non-blocking state, wait for the current data collection task to start, and continue to execute the current data collection task.

[0075] In a specific application scenario, the task execution module 403 can also be configured to send the task execution queue to the at least two host devices, and store the task execution queue in a system path of the at least two host devices, stop the data collection task when the data collection task is disconnected, save a disconnection node of the data collection task in the system path, and extract the disconnection node from the system path when the data collection task is reconnected, and continue to execute the data collection task from the disconnection node.

[0076] In a specific application scenario, the data storage module 404 can be specifically configured to parse the collected data to obtain a page label of the target webpage, and then clean and optimize the collected data based on the page label to obtain a data collection result, wherein the data collection result includes a field type of the target webpage, and finally store the data collection result in a structured manner in a database according to the field type.

[0077] It should be noted that other corresponding descriptions of the functions of the data collection device provided in the embodiment can be referred to the corresponding descriptions in Figure 1 and Figure 2 , which will not be described here in detail.

[0078] Based on the above method as shown in Figure 1 , accordingly, the embodiment also provides a storage medium having a computer program stored thereon, which is executed by a processor to implement the above data collection method.

[0079] Based on such understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the data collection method of each embodiment scenario.

[0080] Based on the above method as shown in Figure 1 and Figure 2 , and Figure 4The shown data acquisition device embodiment, in order to achieve the above-mentioned purposes, such as Figure 5 The embodiment shown also provides a data acquisition entity device, which comprises a communication bus, a processor, a memory and a communication interface, and can further comprise an input / output interface and a display device, wherein the communication between the various functional units can be completed through the bus. The memory stores a computer program, and the processor is used to execute the program stored in the memory to execute the data acquisition method in the above-mentioned embodiment.

[0081] Optionally, the entity device can further comprise a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module and the like. The user interface can comprise a display screen (Display), an input unit such as a keyboard (Keyboard) and the like. The optional user interface can further comprise a USB interface, a card reader interface and the like. The network interface can optionally comprise a standard wired interface, a wireless interface (such as a WI-FI interface) and the like.

[0082] Those skilled in the art can understand that the structure of the data acquisition entity device provided by the embodiment does not constitute a limitation on the entity device, and can include more or fewer components, or combine certain components, or different component arrangements.

[0083] The storage medium can further comprise an operating system and a network communication module. The operating system is a program for managing the hardware and the to-be-identified software resources of the entity device, supporting the running of the information processing program and other to-be-identified software and / or programs. The network communication module is used to realize the communication between the components in the storage medium and the communication between the other hardware and software in the information processing entity device.

[0084] Those skilled in the art can clearly understand from the description of the above embodiments that the application can be implemented by means of software and necessary general hardware platforms, or by hardware. By applying the technical solutions of the application, a task list is first constructed, a target site is determined in the task list, and request parameters of each web page in the target site are obtained, wherein the target site includes at least one web page, then a target web page is extracted in the target site according to the request parameters, and the priority of the target web page is calculated, a task execution queue is generated based on the priority of the target web page, and the task execution queue is sent to at least two host devices, so that the at least two host devices execute data collection tasks one by one based on the task execution queue, obtain collection data of the target web page, finally, the collection data is parsed and cleaned and optimized to obtain data collection results, and the data collection results are stored in a database. The above method filters the web pages in the target site to determine the target web pages to be collected, avoiding wasting data collection time by directly accessing invalid web pages. The priority of each target web page is calculated by obtaining the request parameters of the target web page itself, and a task execution queue generated based on the priority of the target web page is generated to optimize the data collection order of each target web page. The task execution queue is sent to at least two host devices to execute data collection tasks, which can asynchronously collect data of the target web page and improve the efficiency of data collection. Finally, the obtained data collection results are stored in the database, which is convenient for checking and obtaining at any time. The above method collects data of multiple target web pages by at least two host devices in high concurrency, and sorts the multiple target web pages, so that the data collection method is flexible and efficient, and the situation of inefficient execution of data collection tasks caused by different data acquisition mechanisms of each target site is avoided.

[0085] Those skilled in the art can understand that the drawings are only schematic of a preferred implementation scenario, and the modules or flows in the drawings are not necessarily essential for implementing the application. Those skilled in the art can understand that the modules in the device in the implementation scenario can be distributed in the device in the implementation scenario according to the description of the implementation scenario, or can be changed and located in one or more devices different from the implementation scenario. The modules of the above implementation scenario can be combined into one module, or can be further split into multiple sub-modules.

[0086] The above application numbers are only for description, and do not represent the advantages and disadvantages of the implementation scenario. The above disclosure is only a few specific implementation scenarios of the application, but the application is not limited thereto, and any changes that can be thought of by those skilled in the art should fall within the protection scope of the application.

Claims

1. A data acquisition method, characterized by, The method comprises: constructing a task list, determining a target site in the task list, and obtaining a request parameter of each web page in the target site, wherein the target site comprises at least one web page, and the request parameter comprises a status code, a response time and a result quantity; extracting a target web page from the target site according to the request parameter, calculating a priority of the target web page, and generating a task execution queue based on the priority of the target web page; sending the task execution queue to at least two host devices, so that the at least two host devices execute data collection tasks one by one based on the task execution queue, and obtain collection data of the target web page; parsing and cleaning and optimizing the collection data to obtain data collection results, and storing the data collection results in a database; the method comprises: obtaining the status code of each web page in the target site, and comparing the status code of each web page with a preset status code; when the status code of any web page is the same as the preset status code, the web page is marked as a target web page; obtaining the response time and the result quantity of the target web page, and calculating the priority of the target web page based on the response time and the result quantity of the target web page; based on the priority of the target web page, the target web page is sorted according to the sorting rule of the value of the priority from high to low, and a task execution queue is obtained; the method comprises: based on the response time and the result quantity of the target web page, the priority of the target web page is calculated by a preset priority calculation formula, and the priority calculation formula is: wherein P is the priority of the target web page, t is the response time of the target web page, and num is the result quantity of the target web page.

2. The method of claim 1, wherein, the method comprises: constructing an access request deduplication set in the at least two host devices, wherein the access request deduplication set comprises existing access request fingerprints; based on the task execution queue, obtaining a new access request fingerprint corresponding to the target web page in the task execution queue, comparing the new access request fingerprint with the existing access request fingerprints; when the new access request fingerprint is consistent with any of the existing access request fingerprints, the new access request fingerprint is removed, the target web page corresponding to the new access request fingerprint is determined in the task execution queue, and the data collection task of the target web page is stopped; when the new access request fingerprint is inconsistent with any of the existing access request fingerprints, the new access request fingerprint is added to the access request deduplication set as an existing access request fingerprint.

3. The method of claim 2, wherein, The sending of the task execution queue to the at least two host devices, so that the at least two host devices execute data collection tasks one by one based on the task execution queue, further comprises: When the at least two host devices detect that the current data collection task is paused, obtaining a task execution state of the current data collection task; If the task execution state is a blocking state, pausing the execution of the data collection task of the current target webpage, and executing the data collection tasks of the remaining target webpages other than the current target webpage in sequence based on the task execution queue, until the current data collection task is responded to, and the execution of the data collection task of the current target webpage is continued; If the task execution state is a non-blocking state, waiting for the current data collection task to start, and continuing the execution of the current data collection task.

4. The method of claim 1, wherein, The sending of the task execution queue to the at least two host devices, so that the at least two host devices execute data collection tasks one by one based on the task execution queue, further comprises: The sending of the task execution queue to the at least two host devices, and the storage of the task execution queue in a system path of the at least two host devices; When the data collection task is disconnected, stopping the data collection task, and saving a disconnection node of the data collection task to the system path; When the data collection task is reconnected, extracting the disconnection node from the system path, and continuing the execution of the data collection task from the disconnection node.

5. The method of claim 1, wherein, The parsing and cleaning optimization of the collected data to obtain data collection results, and the storage of the data collection results in a database, comprises: Parsing the collected data to obtain page labels of the target webpages; Cleaning and optimizing the collected data based on the page labels to obtain data collection results, wherein the data collection results comprise field types of the target webpages; According to the field types, structuring the storage of the data collection results in the database.

6. A data acquisition device, characterized by The device comprises: A task acquisition module configured to construct a task list, determine a target site in the task list, and acquire request parameters of each webpage in the target site, wherein the target site comprises at least one webpage, and the request parameters comprise a status code, a response time, and a result quantity; A task generation module configured to extract a target webpage from the target site according to the request parameters, calculate a priority of the target webpage, and generate a task execution queue based on the priority of the target webpage; A task execution module configured to send the task execution queue to at least two host devices, so that the at least two host devices execute data collection tasks one by one based on the task execution queue, and obtain collected data of the target webpage; A data storage module configured to parse and clean and optimize the collected data to obtain data collection results, and store the data collection results in a database; The task generation module is configured to acquire a status code of each webpage in the target site, and compare the status code of each webpage with a preset status code. When the status code of any of the web pages is the same as the preset status code, the web page is marked as a target web page; obtaining the response time and the result quantity of the target web page, and calculating the priority of the target web page based on the response time and the result quantity of the target web page; based on the priority of the target web page, the target web page is sorted according to the sorting rule of the value of the priority from high to low, and a task execution queue is obtained; the task generation module is also used for calculating the priority of the target web page based on the response time and the result quantity of the target web page through a preset priority calculation formula, and the priority calculation formula is: wherein, P is the priority of the target web page, t is the response time of the target web page, and num is the result quantity of the target web page.

7. A storage medium having stored thereon a computer program, characterized in that the computer program is executed by the processor to realize the steps of the method in any one of claims 1 to 5.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, the computer program is executed by the processor to realize the steps of the method in any one of claims 1 to 5.

Images