Method and system for collecting resources

Abstract

The application discloses a collection resource scheduling method, which comprises the following steps: accepting an access request of a collection task system, querying a collection resource in a cache according to the access request, wherein the access request comprises a requested collection resource type; if there is no required collection resource in the cache, querying and acquiring the collection resource in a database, and putting the acquired collection resource into the cache for use by the collection task system. The application further provides a collection resource scheduling system. The application can stably provide effective collection resources for the collection task system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer information technology. More particularly, the present application relates to a collection resource scheduling method and system. BACKGROUND

[0002] The continuous development of innovative technologies such as big data, artificial intelligence and data analysis has increased the demand for data. Artificial intelligence and data analysis technologies rely on the accumulation of data to analyze and train to achieve more accurate and refined development. Internet companies are increasingly focusing on Internet data, and Internet companies use many technical means to protect their data. Some companies identify user identities through user device parameters such as Cookie, Token and other device parameter information, and set up human-computer behavior recognition, parameter expiration and other means to limit collection. When a collection task uses the same user device information to request frequently or carries invalid expired parameters to request, the device information will be marked and identified, causing the collection task to be unable to obtain data. Therefore, a technical solution that can overcome the above-mentioned defects to some extent is needed. SUMMARY

[0003] The purpose of the present application is to provide a collection resource scheduling method and system that can provide effective collection resources for a collection task system in a relatively stable manner.

[0004] In order to achieve these objects and other advantages according to the present application, according to one aspect of the present application, a collection resource scheduling method is provided, comprising: accepting an access request of a collection task system, querying a collection resource in a cache according to the access request, the access request including a requested collection resource type; if there is no required collection resource in the cache, querying and obtaining a collection resource in a database, and putting the obtained collection resource into the cache for use by the collection task system.

[0005] Further, the collection resources in the cache are pre-set with a collection resource expiration time and an interval time for calling a clearing method.

[0006] Further, a query operation is performed on the database to obtain a collection resource queue arranged in ascending order of next available time; the next available time of the collection resource in the collection resource queue is compared with the current time in sequence, if the current time is greater than the next available time, the corresponding collection resource is put into the cache for use by the collection task system; if the current time is less than the next available time, the collection task system is fed back with collection resource unavailable state information and the shortest time required for recovery of the collection resource; wherein the next available time and the shortest time required for recovery of the collection resource are determined according to a pre-set available number of times per unit time of the collection resource and a required sleep time.

[0007] Further, the response information of the collection task system is accepted, if the response information is collection task abnormality, the corresponding abnormal collection resource is put into the cache.

[0008] Further, the number of times of causing collection task abnormality is counted, if the set abnormality number threshold is reached, the unavailability state of the corresponding abnormal collection resource is updated to the database.

[0009] Further, the device corresponding to the abnormal collection resource is accessed, the packet capturing operation is performed on the parameter resource of the device, the updated collection resource is obtained, and is stored in the database.

[0010] According to another aspect of the present application, a collection resource scheduling system is also provided, comprising: a collection scheduling system interface module, configured to accept the access request of a collection task system, and query the collection resource in the cache according to the access request, wherein the access request comprises the requested collection resource type; a collection resource scheduling module, configured to, if there is no required collection resource in the cache, query and obtain the collection resource in the database, and put the obtained collection resource into the cache for the collection task system.

[0011] Further, the query operation is performed on the database, the collection resource queue arranged in ascending order of next available time is obtained, the next available time of the collection resource in the collection resource queue is compared with the current time in sequence, if the current time is greater than the next available time, the corresponding collection resource is put into the cache for the collection task system, if the current time is less than the next available time, the collection resource unavailability state information and the shortest time required for the collection resource recovery are fed back to the collection task system, wherein the next available time and the shortest time required for the collection resource recovery are determined according to the preset available number of collection resources per unit time and the required sleep time.

[0012] Further, the collection resource scheduling module is also configured to accept the response information of the collection task system, if the response information is collection task abnormality, the corresponding abnormal collection resource is put into the cache, the number of times of causing collection task abnormality is counted, if the set abnormality number threshold is reached, the unavailability state of the corresponding abnormal collection resource is updated to the database.

[0013] Further, it further comprises: a collection resource updating module, configured to access the device corresponding to the abnormal collection resource, perform the packet capturing operation on the parameter resource of the device, obtain the updated collection resource, and store it in the database; a collection resource display management module, configured to obtain the state of the collection resource, and display.

[0014] The present application at least comprises the following beneficial effects:

[0015] The application accepts an access request of a collection task system, queries a collection resource in a cache according to the access request, queries and acquires the collection resource in a database if there is no required collection resource in the cache, and puts the acquired collection resource into the cache for use by the collection task system, so that the collection resource use can be reasonably allocated, the collection resource use efficiency is improved, and the collection task system stability is ensured.

[0016] Other advantages, objects, and features of the application will be understood by those skilled in the art from the following description, and will be appreciated by those skilled in the art from the study and practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a framework diagram of the present application;

[0018] Figure 2 is a collection resource cache structure flowchart of the present application. DETAILED DESCRIPTION

[0019] The application will be further described in detail below with reference to the accompanying drawings, so that those skilled in the art can implement it according to the description and the drawings.

[0020] It should be understood that the terms such as "have", "contain" and "include" used herein do not exclude the presence or addition of one or more other elements or combinations thereof.

[0021] Embodiments of the present application provide a collection resource scheduling method, comprising:

[0022] S1: accepting an access request of a collection task system, querying a collection resource in a cache according to the access request, the access request comprising a requested collection resource type; the collection resource is device information required by a crawler task system; the collection resource type comprises information such as required resource site, corresponding list or detail page collection, and whether to bind resource location; optionally, the cached collection resource exists in a thread, and a Tornado coroutine framework task can access; the cache is saved in the form of a class dictionary of a Python class object, the class object integrates insertion, acquisition and clearing methods; first, the collection resource is searched in the cache to avoid directly using a new collection resource, so as to reasonably allocate the collection resource use and improve the collection resource use efficiency;

[0023] S2: if there is no required collection resource in the cache, querying and acquiring the collection resource in a database, and putting the acquired collection resource into the cache for use by the collection task system; the database can be a MongoDB database; the collection resource is loaded into the cache to as long as possible to prolong the survival time of the resource and ensure the stability of the collection task system.

[0024] In some other embodiments, the collection resource in the cache is preset with a collection resource expiration time and an interval time for calling the clearing method to reasonably use the cache, and the collection resource expiration time and the interval time for calling the clearing method are determined according to actual tests.

[0025] In some other embodiments, if the collection task system has a specified collection resource, the specified collection resource in the cache is directly selected, and each time the specified collection resource is used, the expiration time of the specified collection resource is extended.

[0026] In some other embodiments, in S2, a query operation is performed on the database to obtain a collection resource queue arranged in ascending order of next available time; the next available time of the collection resource in the collection resource queue is compared with the current time in sequence, if the current time is greater than the next available time, the corresponding collection resource is put into the cache for use by the collection task system, and the number of collection resources put into the cache is determined according to the access request; if the current time is less than the next available time, the collection task system is fed back with collection resource unavailable state information and the shortest time required for the collection resource to recover; wherein the next available time and the shortest time required for the collection resource to recover are determined according to the preset available number of times per unit time and the required sleep time of the collection resource, specifically, the available number of times per unit time and the required sleep time are set according to the actual test and the use frequency of stable resources, the required sleep time is the time interval between two available times, and the next available time is obtained by adding the required sleep time to the last use time of the current use period; optionally, to avoid multiple coroutine collection tasks concurrent causing the same collection resource to be queried and modified multiple times in a short time by single-threaded query, an asynchronous thread lock is added when the database is queried, and the lock is released when the query is completed; when the collection resource in the MongoDB database is queried, the collection resource is sorted according to the next available time field of the collection resource to obtain the first collection resource; then the available time of the collection resource is compared with the current system time; if the current time is greater than or equal to the available time of the collection resource, it means that the collection resource is available, the available number of times per unit time of the collection resource is configured according to the type of the collection resource, and the collection resource is put into the cache for next use of the collection resource to directly read from the cache; if the current time is less than the available time of the resource, it means that all collection resources of the collection task type are unavailable, and the collection resource scheduling system returns the collection task system with the collection resource unavailable state information of the type and the shortest time required for the resource to recover.

[0027] In some embodiments, the response information of the collection task system is accepted, and if the response information is an abnormal collection task, the corresponding abnormal collection resource is put into the cache; if the collection task system obtains the resource and the response exception after the request, the collection system judges that the exception is caused by the abnormal collection resource, and the interface of the resource management system is called, and the resource management system receives the request, and the collection resource causing the exception is put into the abnormal cache (the structure of the abnormal cache is basically the same as that of the normal resource cache, and will not be repeated here); after being put into the abnormal cache, a new collection task system is still provided for selection and use, because other factors may interfere to cause the request exception, such as network delay, and the request exception has contingency, in order to reduce the interference of the contingency factor and improve the robustness of the collection resource.

[0028] In some embodiments, the number of times of causing the collection task exception is counted, and if the set threshold of the number of exceptions is reached, the unusable state of the corresponding abnormal collection resource is updated to the database; the abnormal collection resource cache type counts the number of collection resource exceptions causing the collection task exception in the cleaning time period; when the set threshold of the number of exceptions is reached, the unusable state of the collection resource exception is updated to the MongoDB library.

[0029] In some embodiments, the corresponding device of the abnormal collection resource is accessed, the parameter resource of the device is executed to obtain the updated collection resource, and the collection resource is stored in the database; specifically, the device corresponding to the collection resource accesses a specific site link, and the MitmProxy tool is used for packet capture. According to the specified site link keyword, the request or response of the site is found, the resource extraction logic corresponding to the site is called to parse the captured request or response content, and the serialized collection resource is saved to the MongoDB library.

[0030] As shown in Figure 1 The embodiments of the present application also provide a collection resource scheduling system, which comprises: a collection scheduling system interface module, configured to accept an access request of a collection task system, query a collection resource in a cache according to the access request, and the access request comprises a requested collection resource type; and a collection resource scheduling module, configured to query and obtain a collection resource in a database if there is no required collection resource in the cache, put the obtained collection resource into the cache, and provide the collection resource for the collection task system to use.

[0031] Specifically, the collection scheduling system interface module is developed by using an asynchronous coroutine framework Tornado; communication with the collection task system is implemented by using an http protocol and a certificate verification method, which ensures the efficiency of resource acquisition and the safety of data transmission, and is coupled with the collection task system. The collection resource scheduling module integrates the functions of a page display management module and a resource scheduling module; the collection resource scheduling module is a core module of the resource system, and the interface module calls functions of the module by carrying information such as site types and task types; the system first queries whether the resource exists in the cache according to the field type, and queries the MongoDB database when the resource does not exist in the cache; when the resource exists in the cache, the scheduling module directly returns the collection resource information;

[0032] The collection task system accesses the collection resource scheduling system interface and sends an Http request; after receiving the request, the collection resource scheduling system interface module calls the scheduling module to query the system cache according to the request collection resource type (such as a required resource site, a list or detail page collection, and whether the resource position is bound and the like) to query the system cache. The cache collection resource exists in a thread, and the Tornado coroutine framework task can access it; as shown in Figure 2 The cache saves a class object dictionary in the form of a Python class, the class object integrates insertion, acquisition and removal methods, and the initialization calls the cache class object to define the collection resource expiration time and the calling removal method interval time; when the system cache does not have the required collection resource, the collection resource needs to be queried from the MongoDB database; in order to avoid multiple coroutine collection tasks from causing the same collection resource to be queried and modified in the database multiple times in a short time, an asynchronous thread lock is added when the database is queried, and the lock is released when the query is completed; when the collection resource is queried in the MongoDB database, the collection resource is sorted according to the next available time field of the collection resource, and the first collection resource is acquired.

[0033] In some other embodiments, a query operation is performed on the database to obtain a collection resource queue arranged in ascending order of next available time;

[0034] The next available time of the collection resource in the collection resource queue is compared with the current time in sequence. If the current time is greater than the next available time, the corresponding collection resource is put into the cache for use by the collection task system. If the current time is less than the next available time, the collection task system is fed back the collection resource unavailable state information and the shortest time required for the collection resource to recover. The next available time and the shortest time required for the collection resource to recover are determined according to the preset available times of the collection resource per unit time and the required sleep time. That is, the available time of the collection resource is compared with the current system time. If the current time is greater than or equal to the available time of the collection resource, it means that the collection resource is available, and the available times of the collection resource per unit time are configured according to the type of the collection resource, and the collection resource is put into the cache so as to be directly read from the cache when the collection resource is used next time. If the current time is less than the available time of the resource, it means that all the collection resources of the collection task type are unavailable, and the collection resource scheduling system returns the collection task system the collection resource unavailable state information and the shortest time required for the resource to recover.

[0035] In some other embodiments, the collection resource scheduling module is also used to accept the response information of the collection task system. If the response information is a collection task exception, the corresponding abnormal collection resource is put into the cache. The number of times of causing the collection task exception is counted. If the set exception number threshold is reached, the unavailable state of the corresponding abnormal collection resource is updated in the database. That is, if the collection task system obtains the resource and gets the response exception after the request, the collection system judges that the exception is caused by the collection resource exception, calls the resource management system interface, and the collection resource management system receives the request, puts the collection resource causing the exception into the cache (the structure of the cache is basically the same as that of the normal resource cache, and is not described again), and the abnormal collection resource cache class counts the number of collection resource exceptions causing the collection task exception in the cleaning time period. When the set exception number threshold is reached, the program triggers the management page to display the collection resource exception, and updates the unavailable state of the collection resource exception to the MongoDB library.

[0036] In some embodiments, the method further comprises: collecting resource updating module, for accessing the device corresponding to the abnormal collection resource, performing packet capture operation on the parameter resource of the device, obtaining the updated collection resource, and storing it in the database; collection resource display management module, for obtaining the state of the collection resource and displaying; specifically, the collection resource display management module: uses the vue front-end framework to integrate the development of the front-end display page. Read the collection resource state information in the MongoDB database, and connect the scheduling module to collect the collection resource usage information, update the page collection resource state. And integrate the collection resource updating module interface, realize the page control trigger resource updating operation. And the page can specify the collection resource binding to a specific node of the collection task system to send a request, avoiding the collection resource from being identified and banned due to frequent ip switching; the collection resource updating module: based on the MitmProxy packet capture tool, integrated business development, targeted configuration and development for the site that needs to use the collection resource, packet capture to obtain the collection resource parameters of the required site and automatically import MongoDB database for persistent storage;

[0037] The page displays that a certain collection resource is in an abnormal state, and the administrator can trigger the updating operation of the abnormal collection resource, and the system will call the device to refresh the parameter resource of the device. The collection resource updating module will receive the task, find the device corresponding to the collection resource, access a specific site link, and capture packets through the MitmProxy tool. According to the specified site link keyword, find the request or response of the site, call the resource extraction logic of the site to parse the captured request or response content, and save the serialized collection resource to the MongoDB database.

[0038] The number of devices and the processing scale described herein are used to simplify the description of the present application. It is obvious to those skilled in the art that the application of the collection resource scheduling method and system of the present application, modification and change is obvious.

[0039] Although the embodiments of the present application have been disclosed as above, it is not limited to the application listed in the specification and the embodiments, and can be fully applied to various fields suitable for the present application, and other modifications can be easily realized by those skilled in the art, therefore, the present application is not limited to specific details and the figures shown and described herein, without departing from the general concept defined by the claims and the equivalent scope.

Claims

1. A resource allocation scheduling method, characterized in that, include: Accept access requests from the data collection task system, and query the data collection resources in the cache according to the access requests. The access requests include the type of data collection resources requested. If the required collection resources are not in the cache, the collection resources are queried and obtained from the database, and then placed into the cache for use by the collection task system. Perform a query operation on the database to obtain the collection resource queue sorted in ascending order by the next available time; The system sequentially compares the next available time of each resource in the resource collection queue with the current time. If the current time is greater than the next available time, the corresponding resource is placed in the cache for use by the resource collection task system. If the current time is less than the next available time, the system sends the resource unavailable status information and the shortest time required to restore the resource to the resource collection task system. The next available time and the minimum time required to recover the collected resources are determined based on the preset number of times the collected resources are available per unit time and the required sleep time.

2. The resource acquisition scheduling method as described in claim 1, characterized in that, The collected resources in the cache have preset expiration times and intervals for calling the cleanup method.

3. The resource acquisition scheduling method as described in claim 1, characterized in that, The system receives the response information from the data acquisition task system. If the response information indicates that the data acquisition task is abnormal, the corresponding abnormal data acquisition resource is placed in the cache.

4. The resource acquisition scheduling method as described in claim 3, characterized in that, The system counts the number of times a data collection task fails. If the number of failures reaches a set threshold, the system updates the database with the unavailability status of the corresponding failed data collection resource.

5. The resource acquisition scheduling method as described in claim 3, characterized in that, Access the device corresponding to the abnormal data collection resource, perform packet capture operation on the parameter resources of the device, obtain the updated data collection resource, and store it in the database.

6. A resource acquisition and scheduling system, characterized in that, include: The data acquisition scheduling system interface module is used to accept access requests from the data acquisition task system and query the data acquisition resources in the cache according to the access requests. The access requests include the type of data acquisition resources requested. The resource acquisition scheduling module is used to query and obtain the required acquisition resources from the database if there are no acquisition resources in the cache, and put the obtained acquisition resources into the cache for use by the acquisition task system. A query operation is performed on the database to obtain a queue of collection resources sorted in ascending order of next available time. The next available time of each collection resource in the queue is compared with the current time. If the current time is greater than the next available time, the corresponding collection resource is placed in the cache for use by the collection task system. If the current time is less than the next available time, the collection task system is fed back with information about the unavailable status of the collection resource and the minimum time required to restore the collection resource. The next available time and the minimum time required to restore the collection resource are determined based on a preset number of times the collection resource can be used per unit time and the required sleep time.

7. The resource acquisition scheduling system as described in claim 6, characterized in that, The resource acquisition scheduling module is also used to receive response information from the acquisition task system. If the response information indicates that the acquisition task is abnormal, the corresponding abnormal acquisition resource is placed in the cache. The system counts the number of times a data collection task fails. If the number of failures reaches a set threshold, the system updates the database with the unavailability status of the corresponding failed data collection resource.

8. The resource acquisition scheduling system as described in claim 6, characterized in that, Also includes: The resource update module is used to access the device corresponding to the abnormal data collection resource, perform packet capture operation on the parameter resources of the device, obtain the updated data collection resource, and store it in the database. The resource collection display and management module is used to obtain and display the status of collected resources.

Images