Information acquisition scheduling method and device, computer equipment and storage medium
By detecting the page to be collected and dynamically adjusting the priority and execution order of the collection tasks, the problem of poor flexibility in information collection and scheduling in the prior art is solved, and more precise timing control and more efficient resource management are achieved.
Patent Information
- Application Number
- CN202411756962.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-03
- Publication Date
- 2025-03-18
AI Technical Summary
In the prior art, information collection and scheduling has problems such as poor flexibility, inability to accurately control execution timing, wasted resources, weak fault recovery capabilities, and insufficient support in complex scheduling scenarios.
Provides an information collection and scheduling method, which generates tasks to be collected by detecting the page to be collected, and executes the collection tasks according to the preset collection task execution strategy. After obtaining the target page information, it generates tasks to be analyzed, and analyzes the target data and stores it according to the preset analysis task execution strategy. The method also includes dynamically adjusting the priority and execution order of the acquisition task to ensure flexibility and efficiency of the acquisition task.
It improves the flexibility of information collection and scheduling, can more accurately control the execution timing of the acquisition task, optimize resource allocation, enhance fault recovery capabilities, and provide better support in complex scheduling scenarios.
Smart Images
Figure CN119645603A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to an information collection and scheduling method, device, computer equipment and storage medium. Background Art
[0002] Information collection scheduling in related technologies mostly adopts the method of scheduled tasks or distributed task queues.
[0003] Scheduled tasks use operating system-level scheduling tools such as cron jobs and Windows Task Scheduler, or built-in task scheduling libraries in programming frameworks such as Quartz and APScheduler. The scheduled task method has the following disadvantages:
[0004] 1. Poor flexibility: Scheduled tasks usually require pre-set execution time intervals, which means that if you want to change the execution frequency or time of the task, you need to manually adjust the configuration, which is not flexible enough in scenarios where the collection frequency needs to be adjusted dynamically.
[0005] 2. Unable to accurately control the execution timing: The execution timing of a scheduled task is limited to the set cycle. If an event that requires immediate processing occurs between two cycles, the scheduled task may not be able to respond immediately.
[0006] 3. Resource waste: During low-load periods, scheduled tasks may occupy unnecessary system resources, and during high-load periods, performance may be degraded due to resource contention.
[0007] 4. Weak fault recovery capability: If a scheduled task fails during execution, additional logic may be required to detect and restart the task, otherwise the task may not recover automatically.
[0008] Distributed task queues such as RabbitMQ, Kafka, and Celery can be used to handle a large number of concurrent crawling tasks, but distributed task queues also have the following disadvantages:
[0009] 1. Increased complexity: The introduction of distributed task queues increases the complexity of the system and requires the maintenance of additional middleware services such as RabbitMQ or Kafka, which increases operation and maintenance costs.
[0010] 2. Data consistency challenge: In a distributed environment, ensuring data consistency on all nodes becomes more difficult, especially when processing transactional operations.
[0011] 3. Performance bottleneck: Although distributed queues can handle a large number of concurrent tasks, if the queue itself becomes a performance bottleneck, such as increased processing latency at high throughput, the efficiency of the entire system will also be affected.
[0012] 4. Fault tolerance and retry mechanisms: Designing effective fault tolerance and retry mechanisms to handle failed tasks is complex, especially in scenarios where transaction integrity needs to be maintained.
[0013] 5. Resource allocation problem: In a large-scale distributed environment, how to reasonably allocate tasks to each working node to avoid some nodes being overloaded while others are idle is also a challenge.
[0014] In summary, scheduled tasks only support the scheduled start and stop of collection tasks, but have limited support for distribution; although distributed task queues solve the problem of distributed collection, they do not support more complex scheduling scenarios well. For example, it is difficult for the caller to adjust the dynamic priority of tasks, and it also increases the threshold for developers of specific collection tasks. Therefore, the information collection scheduling in related technologies is inflexible.
[0015] Currently, no effective solution has been proposed for the inflexibility of information collection and scheduling in related technologies. Summary of the invention
[0016] The purpose of this application is to provide an information collection scheduling method, device, computer equipment and computer-readable storage medium to address the deficiencies in the prior art, so as to at least solve the problem of inflexibility in information collection scheduling in the related art.
[0017] To achieve the above purpose, the technical solution adopted by this application is:
[0018] In a first aspect, an embodiment of the present application provides an information collection scheduling method, the method comprising:
[0019] Detect whether there is a page to be collected;
[0020] If it is detected that the page to be collected exists, generating a task to be collected corresponding to the page to be collected;
[0021] Execute the task to be collected according to the preset collection task execution strategy, collect the page to be collected, and obtain the target page information;
[0022] Generate a task to be parsed corresponding to the target page information;
[0023] Execute the task to be parsed according to the preset parsing task execution strategy to parse the target page information to obtain target data;
[0024] The target data is stored.
[0025] In some embodiments, executing the task to be parsed to parse the target page information to obtain target data includes:
[0026] Detecting whether the information collection of the page to be collected is complete;
[0027] If it is detected that the information of the page to be collected is collected completely, the target page information is parsed to obtain the target data;
[0028] If it is detected that the information collection of the page to be collected is incomplete, the priority of the task to be collected corresponding to the page to be collected is lowered, and the task to be collected corresponding to the page to be collected is regenerated according to the lowered priority.
[0029] In some embodiments, after storing the target data, the method further includes:
[0030] Detecting whether there is an associated page associated with the page to be collected;
[0031] If it is detected that there is the associated page associated with the page to be collected, then an associated collection task corresponding to the associated page is generated.
[0032] In some embodiments, the preset acquisition task execution strategy includes at least one of the following:
[0033] Adjust the execution order of the collection tasks according to the configured priority order of the collection tasks, where the execution order of the collection tasks with higher priority is earlier than the collection tasks with lower priority;
[0034] Adjust the collection frequency of the collection task according to the configured collection task priority, where the collection frequency of the high-priority collection task is higher than that of the low-priority collection task;
[0035] The execution order of the collection tasks is adjusted according to the correlation between the collection tasks, wherein the execution order of a group of collection tasks with correlation is continuous;
[0036] Lower the priority of failed collection tasks.
[0037] In some embodiments, the process of executing the acquisition task according to the preset acquisition task execution strategy includes:
[0038] Check whether the collection queue is empty;
[0039] If it is detected that the acquisition queue is empty, adding a first number of acquisition tasks to the acquisition queue;
[0040] Detecting whether the first number of collection tasks have associated collection tasks;
[0041] If it is detected that the first number of collection tasks have associated collection tasks, adding the associated collection tasks to the collection queue;
[0042] Adjusting the execution order and / or acquisition frequency of the acquisition tasks in the acquisition queue according to the preset acquisition task execution strategy;
[0043] Execute the acquisition tasks in the acquisition queue.
[0044] In some embodiments, the preset parsing task execution strategy includes at least one of the following:
[0045] Adjust the execution order of the parsing tasks according to the configured parsing task priority order, wherein the execution order of the parsing tasks with higher priority is prior to the parsing tasks with lower priority;
[0046] Adjust the parsing frequency of the parsing task according to the configured parsing task priority, wherein the parsing frequency of the high-priority parsing task is higher than that of the low-priority parsing task;
[0047] Adjusting the execution order of the parsing tasks according to the correlation between the parsing tasks, wherein the execution order of a group of parsing tasks with correlation is continuous;
[0048] Execute the parsing tasks in the parsing queue.
[0049] In some embodiments, the process of executing the parsing task according to the preset parsing task execution strategy includes:
[0050] Check whether the parsing queue is empty;
[0051] If it is detected that the parsing queue is empty, adding a second number of parsing tasks to the parsing queue;
[0052] Detecting whether the second number of parsing tasks have associated parsing tasks;
[0053] If it is detected that the second number of parsing tasks have associated parsing tasks, adding the associated parsing tasks to the parsing queue;
[0054] Adjusting the execution order and / or parsing frequency of the parsing tasks in the parsing queue according to the preset parsing task execution strategy;
[0055] Execute the parsing tasks in the parsing queue.
[0056] In a second aspect, an embodiment of the present application provides an information collection and scheduling device, including:
[0057] A first detection unit, used to detect whether there is a page to be collected;
[0058] A first generating unit is configured to generate a task to be collected corresponding to the page to be collected if the page to be collected is detected to exist;
[0059] A first execution unit is used to execute the task to be collected according to a preset collection task execution strategy, collect the page to be collected, and obtain target page information;
[0060] A second generating unit, used to generate a task to be parsed corresponding to the target page information;
[0061] A second execution unit, configured to execute the task to be parsed according to a preset parsing task execution strategy to parse the target page information and obtain target data;
[0062] A storage unit is used to store the target data.
[0063] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the information collection and scheduling method described in the first aspect above is implemented.
[0064] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the information collection and scheduling method as described in the first aspect above.
[0065] This application adopts the above technical solution. Compared with the prior art, the embodiment of this application provides an information collection scheduling method, which includes: detecting whether there is a page to be collected; if it is detected that there is the page to be collected, generating a task to be collected corresponding to the page to be collected; executing the task to be collected according to a preset collection task execution strategy, collecting the page to be collected, and obtaining target page information; generating a task to be parsed corresponding to the target page information; executing the task to be parsed according to a preset parsing task execution strategy to parse the target page information and obtain target data; storing the target data. This method solves the problem of inflexibility in information collection scheduling in related technologies, and achieves the effect of improving the flexibility of information collection scheduling.
[0066] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0068] Figure 1is a flow chart of an information collection and scheduling method according to an embodiment of the present application;
[0069] Figure 2 is a flowchart of a collection task according to an embodiment of the present application;
[0070] Figure 3 is a flowchart of a parsing task according to an embodiment of the present application;
[0071] Figure 4 It is a flowchart of the information collection and scheduling method according to the preferred embodiment of the present application;
[0072] Figure 5 is a structural block diagram of an information collection and scheduling device according to an embodiment of the present application;
[0073] Figure 6 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION
[0074] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0075] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.
[0076] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.
[0077] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.
[0078] This embodiment provides an information collection scheduling method. Figure 1 is a flow chart of the information collection scheduling method according to an embodiment of the present application, such as Figure 1 As shown, the process includes the following steps:
[0079] Step S101, detecting whether there is a page to be collected;
[0080] Step S102, if it is detected that the page to be collected exists, generating a task to be collected corresponding to the page to be collected;
[0081] Step S103, executing the task to be collected according to the preset collection task execution strategy, collecting the page to be collected, and obtaining target page information;
[0082] Step S104, generating a task to be parsed corresponding to the target page information;
[0083] Step S105, executing the task to be parsed according to the preset parsing task execution strategy to parse the target page information and obtain target data;
[0084] Step S106, storing the target data.
[0085] It should be noted that the above-mentioned collection tasks and analysis tasks are all completed by a unified API interface to obtain collection instructions from the scheduling system and push the collected data. The unified interface service reduces the complexity of the system and improves maintainability and scalability.
[0086] In some embodiments, the step S105 executes the task to be parsed to parse the target page information, and the target data obtained includes:
[0087] Detecting whether the information collection of the page to be collected is complete;
[0088] If it is detected that the information of the page to be collected is collected completely, the target page information is parsed to obtain the target data;
[0089] If it is detected that the information collection of the page to be collected is incomplete, the priority of the task to be collected corresponding to the page to be collected is lowered, and the task to be collected corresponding to the page to be collected is regenerated according to the lowered priority.
[0090] In some embodiments, after storing the target data, step S106 further includes:
[0091] Detecting whether there is an associated page associated with the page to be collected;
[0092] If it is detected that there is the associated page associated with the page to be collected, then an associated collection task corresponding to the associated page is generated.
[0093] In some embodiments, the preset acquisition task execution strategy includes at least one of the following:
[0094] Adjust the execution order of the collection tasks according to the configured priority order of the collection tasks, where the execution order of the collection tasks with higher priority is earlier than the collection tasks with lower priority;
[0095] Adjust the collection frequency of the collection task according to the configured collection task priority, where the collection frequency of the high-priority collection task is higher than that of the low-priority collection task;
[0096] The execution order of the collection tasks is adjusted according to the correlation between the collection tasks, wherein the execution order of a group of collection tasks with correlation is continuous;
[0097] Lower the priority of failed collection tasks.
[0098] The embodiment of the present application is a collection task execution strategy of the collection module. By setting the priorities of different tasks, the system can dynamically adjust the collection frequency according to current business needs and resource conditions. For example, when a high-priority task requires more resources, the system can temporarily reduce the execution frequency of low-priority tasks, thereby optimizing resource allocation.
[0099] In some embodiments, the process of executing the collection task according to the preset collection task execution strategy in step S103 includes:
[0100] Check whether the collection queue is empty;
[0101] If it is detected that the acquisition queue is empty, adding a first number of acquisition tasks to the acquisition queue;
[0102] Detecting whether the first number of collection tasks have associated collection tasks;
[0103] If it is detected that the first number of collection tasks have associated collection tasks, adding the associated collection tasks to the collection queue;
[0104] Adjusting the execution order and / or acquisition frequency of the acquisition tasks in the acquisition queue according to the preset acquisition task execution strategy;
[0105] Execute the acquisition tasks in the acquisition queue.
[0106] The embodiment of the present application is the execution process of the collection task of the collection module. By organizing a series of related tasks into a workflow, the complex scheduling logic can be abstracted, making the system easier to understand and maintain.
[0107] In actual application scenarios, the process diagram of the collection task is as follows: Figure 2 As shown, specifically including:
[0108] Check whether the collection queue is empty. If the collection queue is empty, query the 10,000 data with the highest priority in the uncollected data. If the data is extracted, check whether the queried data has a workflow configuration. If there is a workflow configuration, add 10,000 collection subtasks according to the workflow configuration, insert them into the collection queue, and give the first 10,000 data in the queue each time according to the request until the queue is empty. If there is no workflow configuration, directly insert them into the collection queue and give the first 10,000 data in the queue each time according to the request until the queue is empty.
[0109] If the detection collection queue is not empty, the first 10,000 data in the queue will be given each time according to the request until the queue is empty.
[0110] It should be noted that the number of tasks inserted or extracted in the collection queue can be set according to actual needs, and this is only an example.
[0111] In some embodiments, the preset parsing task execution strategy includes at least one of the following:
[0112] Adjust the execution order of the parsing tasks according to the configured parsing task priority order, wherein the execution order of the parsing tasks with higher priority is prior to the parsing tasks with lower priority;
[0113] Adjust the parsing frequency of the parsing task according to the configured parsing task priority, wherein the parsing frequency of the high-priority parsing task is higher than that of the low-priority parsing task;
[0114] Adjusting the execution order of the parsing tasks according to the correlation between the parsing tasks, wherein the execution order of a group of parsing tasks with correlation is continuous;
[0115] Lower the priority of failed parsing tasks.
[0116] The embodiment of the present application is a parsing task execution strategy of the parsing module, which can help the system allocate resources more efficiently through priority management, give priority to ensuring the normal operation of key tasks, and at the same time, when a failure occurs, it can restart high-priority tasks first.
[0117] In some embodiments, the process of executing the parsing task according to the preset parsing task execution strategy in step S105 includes:
[0118] Check whether the parsing queue is empty;
[0119] If it is detected that the parsing queue is empty, adding a second number of parsing tasks to the parsing queue;
[0120] Detecting whether the second number of parsing tasks have associated parsing tasks;
[0121] If it is detected that the second number of parsing tasks have associated parsing tasks, adding the associated parsing tasks to the parsing queue;
[0122] Adjusting the execution order and / or parsing frequency of the parsing tasks in the parsing queue according to the preset parsing task execution strategy;
[0123] Execute the parsing tasks in the parsing queue.
[0124] The embodiment of the present application is the execution process of the parsing task of the parsing module. Through workflow management, more complex scheduling strategies can be implemented, such as conditional task branching, loop execution, parallel processing, etc., so as to meet a wider range of business needs.
[0125] In actual application scenarios, the flowchart of the parsing task is as follows: Figure 3 As shown, specifically including:
[0126] Check whether the parsing queue is empty. If it is detected that the parsing queue is empty, obtain 10,000 data from the collected but unparsed tasks in the database. If the data is obtained, insert it into the parsing queue, and give the first 1,000 data in the queue each time according to the parsing module request until the queue is empty.
[0127] If it is detected that the parsing queue is not empty, the first 1000 data in the queue are given each time according to the parsing module request until the queue is empty.
[0128] It should be noted that the number of tasks inserted or extracted in the parsing queue can be set according to actual needs, and this is only an example.
[0129] Through the above steps, the present application solves the problem of inflexibility in information collection scheduling in the related art, and achieves the effect of improving the flexibility of information collection scheduling.
[0130] The embodiments of the present application are described and illustrated below through preferred embodiments.
[0131] Figure 4 is a flow chart of the information collection and scheduling method according to the preferred embodiment of the present application, such as Figure 4 As shown in the figure, the execution process specifically includes:
[0132] (1) Scheduling Module
[0133] The scheduling module adds a new collection page and generates a task to be collected corresponding to the page to be collected.
[0134] (2) Acquisition Module
[0135] Collecting web pages according to the parameters returned by the scheduling module interface, wherein the parameters returned by the scheduling module interface are the relevant data of the tasks to be collected generated by the scheduling module;
[0136] Call the scheduling module interface to save the collected web page information.
[0137] (3) Parsing module
[0138] Determine whether the target page is fully collected. If it is detected that the information is fully collected, parse the current page and store the target data in the database. Then detect whether a new page to be collected is found. If so, call the scheduling module to insert the new collection page interface and generate the corresponding associated collection task.
[0139] If it is detected that the information collection is incomplete, the scheduling module is called to lower the priority interface and regenerate the tasks to be collected according to the lowered priority.
[0140] The scheduling module in the embodiment of the present application can implement priority management and workflow management, which is specifically described as follows:
[0141] 1. Priority Management
[0142] By setting priorities for different tasks, the system can dynamically adjust the collection frequency based on current business needs and resource conditions; for events or critical tasks that require immediate processing, related tasks can be given higher priorities to ensure that they are executed first, thereby improving the system's response speed and efficiency; when resources are limited, priority management can help the system allocate resources more effectively, prioritize the normal operation of critical tasks, and at the same time, restart high-priority tasks first when failures occur.
[0143] 2. Workflow Management
[0144] Workflow management allows you to define dependencies between tasks and ensure that data is processed in the correct order, which is very useful for scenarios where data needs to be processed in specific steps. By organizing a series of related tasks into workflows, complex scheduling logic can be abstracted, making the system easier to understand and maintain. Workflow management can implement more complex scheduling strategies, such as conditional task branching, loop execution, parallel processing, etc., to meet a wider range of business needs.
[0145] This preferred embodiment is based on an information collection scheduling method, which manages and defines the priority of collected web pages through priority management and workflow management, so that the order of collected web pages is more flexible and closer to business needs.
[0146] This preferred embodiment provides a unified interface service through the scheduling module, which greatly reduces the difficulty of developing the collection and analysis system. All processes, caches, and storage issues are uniformly maintained by the scheduling module to improve the convenience and security of subsequent maintenance.
[0147] The unified interface services include:
[0148] 1. Decoupling and modularization: The collection module and the analysis module obtain collection instructions and push the collected data from the scheduling module through the API. The collection node is decoupled from the back-end data storage, message queue and other components, and the scheduling module operates them uniformly. This reduces the complexity of the system and improves maintainability and scalability.
[0149] 2. Simplify the development process: Developers of collection and analysis tasks do not need to deeply understand the specific implementation details of the underlying data storage or message queues, but only need to focus on the logic of interacting with the system through the API interface. This greatly reduces the difficulty of collection task development, speeds up development, and reduces the possibility of errors.
[0150] 3. Improve security: Directly operating the database or message queue may expose sensitive information or introduce security vulnerabilities. Isolation through APIs can implement stricter security policies such as authentication, permission control, and data encryption, thereby enhancing the overall security of the system.
[0151] 4. Enhanced flexibility and portability: When the system architecture needs to be adjusted, such as changing the database type or message queue service, only the API implementation needs to be modified without changing the code of the collection node. This provides great flexibility for system upgrades and optimizations, and also enhances the portability of the system.
[0152] 5. Centralized management and monitoring: Through the unified API and system cluster communication, the status monitoring and resource management of the collection tasks become more centralized and efficient. System administrators can easily view and control the operation of all collection nodes through the management interface provided by the API.
[0153] 6. Performance optimization: The API layer can serve as a cache or load balancing front end, reducing the pressure on the backend database or message queue by optimizing API calls, thereby improving the overall performance and response speed of the system.
[0154] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0155] This embodiment provides an information collection and scheduling device, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the terms "module", "unit", "subunit", etc. can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0156] Figure 5 is a structural block diagram of an information collection and scheduling device according to an embodiment of the present application, such as Figure 5 As shown, the device comprises:
[0157] A first detection unit 51 is used to detect whether there is a page to be collected;
[0158] The first generating unit 52 is configured to generate a task to be collected corresponding to the page to be collected if the page to be collected is detected to exist;
[0159] The first execution unit 53 is used to execute the task to be collected according to a preset collection task execution strategy, collect the page to be collected, and obtain target page information;
[0160] A second generating unit 54 is used to generate a task to be parsed corresponding to the target page information;
[0161] The second execution unit 55 is used to execute the task to be analyzed according to a preset analysis task execution strategy to analyze the target page information and obtain target data;
[0162] The storage unit 56 is used to store the target data.
[0163] In some embodiments, the second execution unit 55 includes:
[0164] A first detection module is used to detect whether the information collection of the page to be collected is complete;
[0165] A parsing module, for parsing the target page information to obtain target data if it is detected that the information of the page to be collected is complete;
[0166] The adjustment module is used to lower the priority of the task to be collected corresponding to the page to be collected if it is detected that the information collection of the page to be collected is incomplete, and regenerate the task to be collected corresponding to the page to be collected according to the lowered priority.
[0167] In some embodiments, the device further comprises:
[0168] A second detection unit, configured to detect whether there is an associated page associated with the page to be collected after the target data is stored;
[0169] The third generating unit is configured to generate an associated collection task corresponding to the associated page if it is detected that there is an associated page associated with the page to be collected.
[0170] In some embodiments, the preset acquisition task execution strategy includes at least one of the following:
[0171] Adjust the execution order of the collection tasks according to the configured priority order of the collection tasks, where the execution order of the collection tasks with higher priority is earlier than the collection tasks with lower priority;
[0172] Adjust the collection frequency of the collection task according to the configured collection task priority, where the collection frequency of the high-priority collection task is higher than that of the low-priority collection task;
[0173] The execution order of the collection tasks is adjusted according to the correlation between the collection tasks, wherein the execution order of a group of collection tasks with correlation is continuous;
[0174] Lower the priority of failed collection tasks.
[0175] In some embodiments, the device further comprises:
[0176] A third detection unit is used to detect whether the collection queue is empty;
[0177] a first adding unit, configured to add a first number of collection tasks to the collection queue if it is detected that the collection queue is empty;
[0178] A fourth detection unit, used to detect whether there are any associated collection tasks in the first number of collection tasks;
[0179] a second adding unit, configured to add the associated collection tasks to the collection queue if it is detected that the first number of collection tasks have associated collection tasks;
[0180] A first adjustment unit, configured to adjust the execution order and / or acquisition frequency of the acquisition tasks in the acquisition queue according to the preset acquisition task execution strategy;
[0181] The third execution unit is used to execute the collection tasks in the collection queue.
[0182] In some embodiments, the preset parsing task execution strategy includes at least one of the following:
[0183] Adjust the execution order of the parsing tasks according to the configured parsing task priority order, wherein the execution order of the parsing tasks with higher priority is prior to the parsing tasks with lower priority;
[0184] Adjust the parsing frequency of the parsing task according to the configured parsing task priority, wherein the parsing frequency of the high-priority parsing task is higher than that of the low-priority parsing task;
[0185] Adjusting the execution order of the parsing tasks according to the correlation between the parsing tasks, wherein the execution order of a group of parsing tasks with correlation is continuous;
[0186] Lower the priority of failed parsing tasks.
[0187] In some embodiments, the device further comprises:
[0188] A fifth detection unit, used to detect whether the parsing queue is empty;
[0189] a third adding unit, configured to add a second number of parsing tasks to the parsing queue if it is detected that the parsing queue is empty;
[0190] A sixth detection unit, configured to detect whether the second number of parsing tasks has associated parsing tasks;
[0191] a fourth adding unit, configured to add the associated parsing tasks to the parsing queue if it is detected that the second number of parsing tasks have associated parsing tasks;
[0192] A second adjustment unit, configured to adjust the execution order and / or parsing frequency of the parsing tasks in the parsing queue according to the preset parsing task execution strategy;
[0193] The fourth execution unit is used to execute the parsing tasks in the parsing queue.
[0194] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.
[0195] The embodiment provides a computer device. The information collection and scheduling method in accordance with the embodiment of the present application can be implemented by the computer device. Figure 6 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present application.
[0196] The computer device may include a processor 61 and a memory 62 storing computer program instructions.
[0197] Specifically, the processor 61 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0198] Among them, the memory 62 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 62 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 62 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 62 may be inside or outside the data processing device. In a specific embodiment, the memory 62 is a non-volatile memory. In a specific embodiment, the memory 62 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable ProgrammableRead-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable ProgrammableRead-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.
[0199] The memory 62 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 61 .
[0200] The processor 61 implements any one of the information collection and scheduling methods in the above embodiments by reading and executing computer program instructions stored in the memory 62 .
[0201] In some of the embodiments, the computer device may further include a communication interface 63 and a bus 60. Figure 6 As shown, the processor 61, the memory 62, and the communication interface 63 are connected via a bus 60 and communicate with each other.
[0202] The communication interface 63 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication interface 63 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.
[0203] The bus 60 includes hardware, software or both, and couples the components of the computer device to each other. The bus 60 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 60 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of the above. Where appropriate, bus 60 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.
[0204] In addition, in combination with the information collection scheduling method in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the information collection scheduling methods in the above embodiment is implemented.
[0205] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0206] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. An information collection and scheduling method, characterized in that: include: Detect whether there is a page to be collected; If it is detected that the page to be collected exists, generating a task to be collected corresponding to the page to be collected; Execute the task to be collected according to the preset collection task execution strategy, collect the page to be collected, and obtain the target page information; Generate a task to be parsed corresponding to the target page information; Execute the task to be parsed according to the preset parsing task execution strategy to parse the target page information to obtain target data; The target data is stored.
2. The method according to claim 1, characterized in that: The executing of the task to be parsed to parse the target page information to obtain the target data includes: Detecting whether the information collection of the page to be collected is complete; If it is detected that the information of the page to be collected is collected completely, the target page information is parsed to obtain the target data; If it is detected that the information collection of the page to be collected is incomplete, the priority of the task to be collected corresponding to the page to be collected is lowered, and the task to be collected corresponding to the page to be collected is regenerated according to the lowered priority.
3. The method according to claim 2, characterized in that After storing the target data, the method further includes: Detecting whether there is an associated page associated with the page to be collected; If it is detected that there is the associated page associated with the page to be collected, then an associated collection task corresponding to the associated page is generated.
4. The method according to claim 1, characterized in that The preset acquisition task execution strategy includes at least one of the following: Adjust the execution order of the collection tasks according to the configured priority order of the collection tasks, where the execution order of the collection tasks with higher priority is earlier than the collection tasks with lower priority; Adjust the collection frequency of the collection task according to the configured collection task priority, where the collection frequency of the high-priority collection task is higher than that of the low-priority collection task; The execution order of the collection tasks is adjusted according to the correlation between the collection tasks, wherein the execution order of a group of collection tasks with correlation is continuous; Lower the priority of failed collection tasks.
5. The method according to claim 4, characterized in that The process of executing the collection task according to the preset collection task execution strategy includes: Check whether the collection queue is empty; If it is detected that the acquisition queue is empty, adding a first number of acquisition tasks to the acquisition queue; Detecting whether the first number of collection tasks have associated collection tasks; If it is detected that the first number of collection tasks have associated collection tasks, adding the associated collection tasks to the collection queue; Adjusting the execution order and / or acquisition frequency of the acquisition tasks in the acquisition queue according to the preset acquisition task execution strategy; Execute the acquisition tasks in the acquisition queue.
6. The method according to claim 1, characterized in that The preset parsing task execution strategy includes at least one of the following: Adjust the execution order of the parsing tasks according to the configured parsing task priority order, wherein the execution order of the parsing tasks with higher priority is prior to the parsing tasks with lower priority; Adjust the parsing frequency of the parsing task according to the configured parsing task priority, wherein the parsing frequency of the high-priority parsing task is higher than that of the low-priority parsing task; Adjusting the execution order of the parsing tasks according to the correlation between the parsing tasks, wherein the execution order of a group of parsing tasks with correlation is continuous; Execute the parsing tasks in the parsing queue.
7. The method according to claim 6, characterized in that The process of executing the parsing task according to the preset parsing task execution strategy includes: Check whether the parsing queue is empty; If it is detected that the parsing queue is empty, adding a second number of parsing tasks to the parsing queue; Detecting whether the second number of parsing tasks have associated parsing tasks; If it is detected that the second number of parsing tasks have associated parsing tasks, adding the associated parsing tasks to the parsing queue; Adjusting the execution order and / or parsing frequency of the parsing tasks in the parsing queue according to the preset parsing task execution strategy; Execute the parsing tasks in the parsing queue.
8. An information collection and scheduling device, characterized in that: include: A first detection unit, used to detect whether there is a page to be collected; A first generating unit is configured to generate a task to be collected corresponding to the page to be collected if the page to be collected is detected to exist; A first execution unit is used to execute the task to be collected according to a preset collection task execution strategy, collect the page to be collected, and obtain target page information; A second generating unit, used to generate a task to be parsed corresponding to the target page information; A second execution unit, configured to execute the task to be parsed according to a preset parsing task execution strategy to parse the target page information and obtain target data; A storage unit is used to store the target data.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.