Task scheduling and data extraction method and system

Abstract

The invention discloses a task scheduling and data extraction method and system, and the method comprises the steps: automatically configuring the starting and ending time of a task, actuator selection, an alarm condition and a repetition period through a task scheduling center, generating task basic information, creating a task list, and supporting multi-person permission issuing and attachment demand distribution; a task executor scans a task list at regular time, performs attachment demand judgment on a task to be started, directly stores the task into a task database if no attachment exists, triggers a data extraction platform to capture data if the attachment is needed, and pre-generates an empty path address of an OSS storage unit; and the data extraction platform captures task attachment data according to the request, stores a task into a database and fills an attachment to an OSS specified path. According to the method, the problems of low task scheduling efficiency and inaccurate cross-scene data capture in the prior art are solved, the task processing efficiency and the data management capability in a complex scene are improved, and the method is suitable for high-concurrency data processing requirements of multiple cities and multiple persons in charge.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a task scheduling and data extraction method and system. Background Art

[0002] In the field of data processing and management, task scheduling and data extraction are core links supporting the efficient flow and precise application of data. With the acceleration of urban digitalization and the surge in demand for multi-domain data integration, the key challenges in enhancing data-driven decision-making capabilities are how to achieve automated task generation, intelligent allocation, and accurate capture and processing across heterogeneous data sources. Especially in complex scenarios involving multi-city collaboration and multiple stakeholders, task scheduling must balance flexibility and controllability, and data extraction must adapt to differentiated rules. This places higher demands on the technical solution's degree of automation, process optimization capabilities, and scalability.

[0003] At present, although existing technologies can achieve urban data management through cloud resource scheduling, their task scheduling is highly dependent on manual configuration and cannot dynamically adapt to the complex needs of multiple responsible persons and multiple scenarios, and lacks refined control over task decomposition, sub-task reorganization and resource allocation; at the same time, existing technologies are difficult to adapt to the differentiated rules in multi-city and multi-responsible subject scenarios, resulting in a lack of customization capabilities in the capture process of non-security data, and the separation of task execution, alarm triggering and data processing links further aggravates the lag in resource allocation. The disconnection between attachment capture and task allocation logic, the lack of process feedback mechanism and other problems jointly restrict the timeliness and accuracy of data processing, and ultimately it is difficult to meet the high concurrency and multi-dimensional data management needs in modern urban operations.

[0004] Therefore, how to invent and develop a task scheduling and data extraction method that can realize the automatic generation and allocation of tasks, and provide customized data processing services based on the data capture needs of different city leaders has become an urgent problem that needs to be solved. Summary of the Invention

[0005] To this end, the present invention provides a task scheduling and data extraction method and system. By optimizing task scheduling strategies and improving data extraction accuracy, it enables efficient management of complex data processing scenarios. Furthermore, it can provide flexible task allocation and automated data capture processes to meet the data processing needs of multiple cities and multiple responsible individuals, ensuring data accuracy and timeliness, and meeting the urgent needs of various segments for efficient data processing and management.

[0006] In order to achieve the above object, the present invention provides the following technical solution: a task scheduling and data extraction method, comprising:

[0007] The task scheduling center configures the task's start and end time, executor selection, alarm conditions, and repetition period information through automated configuration to generate basic task information. Based on this basic task information, the task generator generates a task list within a set timeframe and assigns multiple people with distribution permissions and attachment requirements to the tasks in the task list.

[0008] The task list is scanned periodically by the task executor to determine whether a task in the task list is about to be started; if the task to be started does not have the attachment requirement, the task to be started is sent to the task database; if the task to be started is accompanied by the attachment requirement, a data extraction request is sent to the data extraction platform, and a file address to be filled with an empty path is generated in the OSS storage unit;

[0009] The data extraction platform extracts the attachment data corresponding to the task to be started according to the data extraction request, and sends the task to be started to the task database; and fills the attachment data into the to-be-filled file address in the OSS storage unit.

[0010] As an optimal solution for task scheduling and data extraction methods, based on the alarm configuration set in the task scheduling center, an alarm event is triggered through xxl-job; the alarm signal is asynchronously notified to the data extraction platform through MQ; the data extraction platform captures the alarm-related data according to the alarm signal, and generates an alarm task through the MQ message queue; and the alarm task is sent to the task database.

[0011] As an optimal solution for task scheduling and data extraction methods, in the process of executing task requirements of multiple cities and multiple responsible persons, the city person in charge information is captured from the Tritium Cloud low-code platform through the LiteFlow flow; the corresponding attachment extraction subtask is generated based on the city person in charge information; and the subtask is logically bound to the task in the task list.

[0012] As a preferred solution for a task scheduling and data extraction method, a snowflake algorithm is used to generate a unique task ID for each task in the task list; operators log in to the task scheduling center through the task ID to view daily, weekly, and monthly task status.

[0013] As a preferred solution for a task scheduling and data extraction method, the LiteFlow flow includes: a liteflow rich client, a process service end, a process persistence layer interface, and a process action maker.

[0014] The present invention further provides a task scheduling and data extraction system, based on the above task scheduling and data extraction method, comprising:

[0015] The task configuration and generation module is used by the task scheduling center to configure the start and end time, executor selection, alarm conditions and repetition period information of the task through automated configuration, and generate basic task information; based on the basic task information, a task generator generates a task list within a set time period, and assigns multiple people's distribution permissions and attachment requirements to the tasks in the task list;

[0016] The task scanning and triggering data extraction module is used to perform a periodic scan of the task list through the task executor to determine whether a task in the task list is about to be started; if the task to be started does not have the attachment requirement, the task to be started is sent to the task database; if the task to be started is accompanied by the attachment requirement, a data extraction request is sent to the data extraction platform, and an empty path file address to be filled is generated in the OSS storage unit;

[0017] The data extraction module is used for the data extraction platform to extract the attachment data corresponding to the task to be started according to the data extraction request, and send the task to be started to the task database; fill the attachment data into the file address to be filled in the OSS storage unit.

[0018] As an optimal solution for a task scheduling and data extraction system, in the task configuration and generation module, based on the alarm configuration set in the task scheduling center, an alarm event is triggered through xxl-job; the alarm signal is asynchronously notified to the data extraction platform through MQ; the data extraction platform captures alarm-related data based on the alarm signal, and generates an alarm task through the MQ message queue; and the alarm task is sent to the task database.

[0019] As an optimal solution for a task scheduling and data extraction system, in the data extraction module, in the process of executing task requirements of multiple cities and multiple responsible persons, the city person in charge information is captured from the Chuanyun low-code platform through the LiteFlow flow; the corresponding attachment extraction subtask is generated based on the city person in charge information; and the subtask is logically bound to the task in the task list.

[0020] As an optimal solution for a task scheduling and data extraction system, in the task configuration and generation module, a unique task ID is generated for each task in the task list through a snowflake algorithm; operators log in to the task scheduling center through the task ID to view the daily, weekly, and monthly task status.

[0021] As a preferred solution for a task scheduling and data extraction system, in the data extraction module, the LiteFlow flow includes: a liteflow rich client, a process service end, a process persistence layer interface and a process action maker.

[0022] The present invention has the following advantages: The present invention achieves efficient management of complex data processing scenarios by optimizing task scheduling strategies and improving the accuracy of data extraction. The present invention can provide flexible task allocation and automated data capture processes for the data processing needs of multiple cities and multiple responsible persons, ensure the accuracy and timeliness of data, and meet the urgent needs of various segments for efficient data processing and management. The present invention realizes intelligent task scheduling and precise data extraction by introducing technologies such as automated configuration, multi-executor collaborative operation, and snowflake algorithm to generate unique task IDs, providing more efficient data processing support for urban operations, network security and other fields. The present invention uses LiteFlow flow technology in combination with a low-code platform to achieve flexible configuration and efficient execution of tasks and data processing, solves the shortcomings of existing technologies in automated process optimization, and provides strong technical support for data-driven modern urban operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can, without inventive effort, derive other implementation drawings based on the provided drawings.

[0024] The structures, proportions, sizes, etc. illustrated in this specification are intended solely to complement the contents disclosed herein and to facilitate understanding and reading by persons skilled in the art. They are not intended to limit the conditions under which the present invention may be implemented and therefore have no substantive technical significance. Any structural modifications, changes in proportions, or adjustments in sizes, without affecting the efficacy and objectives of the present invention, shall remain within the scope of the technical contents disclosed herein.

[0025] Figure 1 A flowchart of a task scheduling and data extraction method provided in Example 1 of the present invention;

[0026] Figure 2 This is a schematic diagram of a specific implementation flow of a task scheduling and data extraction method provided in Example 1 of the present invention;

[0027] Figure 3 This is a schematic diagram of the architecture of a task scheduling and data extraction system provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0028] The following describes the implementation of the present invention using specific embodiments. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. Obviously, the embodiments described are only a portion of the present invention, not all of it. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0029] Example 1

[0030] See also Figure 1 and Figure 2 , embodiment 1 of the present invention provides a task scheduling and data extraction method, comprising the following steps:

[0031] S1. The task scheduling center configures the task's start and end time, executor selection, alarm conditions, and repetition period information through automated configuration to generate basic task information. Based on the basic task information, the task generator generates a task list within a set time period and assigns multiple people's distribution permissions and attachment requirements to the tasks in the task list.

[0032] S2. Performing a periodic scan of the task list by the task executor to determine whether a task in the task list is about to be started; if the task to be started does not have the attachment requirement, sending the task to be started to the task database; if the task to be started has the attachment requirement, sending a data extraction request to the data extraction platform, and generating a file address to be filled with an empty path in the OSS storage unit;

[0033] S3. The data extraction platform extracts the attachment data corresponding to the task to be started according to the data extraction request, and sends the task to be started to the task database; and fills the attachment data into the to-be-filled file address in the OSS storage unit.

[0034] In this embodiment, in step S1, the task scheduling center configures the start and end time, executor selection, alarm conditions, and repetition period information of the task through automated configuration to generate basic task information; based on the basic task information, a task generator generates a task list within a set time period, and assigns multiple people's distribution permissions and attachment requirements to the tasks in the task list;

[0035] Specifically, task parameters, such as the start and end times, executor selection, alarm conditions, and recurrence information, are abstracted into rule-based configurations through the low-code platform's visual interface. Users can compile basic task information through the visual interface, and the task generator uses this basic information to achieve zero-code automated task generation. The task generator assigns a globally unique ID to each task using a snowflake algorithm and uses a time window algorithm (such as a sliding window) to generate a periodic task queue for up to one year into the future, ensuring conflict-free task sequencing.

[0036] In this embodiment, the permissions issued by multiple people are dynamically bound to the executor role and task relationship through the RBAC (role-based access control) model. At the same time, the attachment requirements are associated with the predefined rule base of the data capture platform (such as data source type and capture frequency). To cope with complex scenarios, the system introduces a conflict detection mechanism: when task time overlaps or resources exceed the limit, a dynamic reallocation strategy is triggered (such as queue jumping according to priority);

[0037] In this embodiment, when executing multi-city, multi-responsible-person tasks, LiteFlow captures city leader information from the Chuanyun low-code platform to generate differentiated permission configurations, such as assigning independent data capture rule sets to different cities. Based on the city leader information, corresponding attachment extraction subtasks are generated; these subtasks are logically bound to the tasks in the task list.

[0038] In this embodiment, based on the alarm configuration set in the task scheduling center, an alarm event is triggered through xxl-job; the alarm signal is asynchronously notified to the data extraction platform through MQ; the data extraction platform captures alarm-related data according to the alarm signal, and generates an alarm task through the MQ message queue; the alarm task is sent to the task database.

[0039] Specifically, based on the alarm configuration preset by the task scheduling center (such as resource threshold exceeding, task execution timeout), xxl-job regularly scans the task status. When a trigger condition is detected, xxl-job calls the alarm executor to generate an alarm event, and encapsulates the alarm signal (including task ID, alarm type, trigger time) into an MQ message (such as RocketMQ message body), and asynchronously delivers it to the message queue of the data extraction platform; after the data extraction platform monitors the MQ message, it parses the associated data source information in the alarm signal (such as the database table and log file path corresponding to the abnormal task), and starts the data capture process (through predefined ETL rules or API interface). The system extracts alarm-related data (such as error logs and resource usage snapshots) and binds the captured results with the alarm metadata to generate a structured alarm task (including task type tags, processing priority, and a list of responsible persons), which is then asynchronously pushed to the task database of the task scheduling center through MQ. After receiving the alarm task, the task database writes it into the queue to be allocated based on the transaction mechanism, and dynamically allocates it to the idle executor through the executor resource status detection algorithm (such as weighted polling based on CPU / memory load). At the same time, it updates the task status to "alarm processing", forming a closed-loop process from alarm triggering, data capture to task storage, and realizing the automated linkage between fault response and resource scheduling.

[0040] In this embodiment, in step S2, the task list is scanned periodically by the task executor to determine whether a task in the task list is about to be started; if the task to be started does not have the attachment requirement, the task to be started is sent to the task database; if the task to be started is accompanied by the attachment requirement, a data extraction request is sent to the data extraction platform, and a file address to be filled with an empty path is generated in the OSS storage unit;

[0041] Specifically, the executor utilizes an independent thread pool design. Every minute, it scans the task list using a timestamp comparison algorithm (based on NTP synchronization time and a preset threshold) to select tasks that need to be started within the next minute. For tasks without attachments, their status is directly marked as "pending" and pushed to the task database queue, where database transactions ensure state consistency. For tasks with attachments, the system pregenerates a unique OSS path (e.g., oss: / / task_attachments / {task ID} / data.tmp), uses a hashing algorithm to avoid path conflicts, and reserves storage space to reduce I / O latency.

[0042] To optimize resource utilization, the executor integrates dynamic priority adjustment: when a high-urgency task is detected, it automatically interrupts lower-priority tasks and re-queues them. A resource preemption mechanism also pre-locks the executor's CPU and memory resources to prevent deadlock caused by multi-tasking. If invalid tasks (such as misconfigured or expired tasks) are continuously scanned, the system triggers a circuit breaker, suspends scanning, and sends an alert to the operations team to avoid wasted resources.

[0043] In this embodiment, in step S3, the data extraction platform extracts the attachment data corresponding to the task to be started according to the data extraction request, and sends the task to be started to the task database; and fills the attachment data into the file address to be filled in the OSS storage unit.

[0044] Specifically, the data extraction platform uses the LiteFlow streaming engine to execute a multi-stage processing flow. First, the task ID is resolved based on the associated data source (e.g., the database table for a specific city in the Chuan Cloud platform), and predefined ETL rules (including data cleansing and format conversion) are applied. The processed data is then asynchronously written to a pre-generated address on OSS using a sharded upload technique, with MD5 verification to ensure data integrity. The task database utilizes a master-slave dual-queue design: the master queue stores task metadata (e.g., status, execution logs), while the slave queue stores real-time operation records. Distributed transactions ensure the atomicity of task status and data population. To enhance robustness, the system supports breakpoint-resume uploads. If the crawling process is interrupted, the breakpoint is automatically recorded and incremental synchronization is used to resume. Furthermore, a compression algorithm is dynamically selected based on file size (e.g., GZIP for files ≤ 100MB, Snappy for files > 100MB), reducing storage and transmission costs. Once data population is complete, the task scheduling center is asynchronously notified via MQ to update the task status, triggering the subsequent statistics module to calculate the completion rate (e.g., number of completed tasks per day / total number of completed tasks × 100%) and push the results to the management client, forming a closed-loop feedback loop.

[0045] In a possible embodiment, a specific task scheduling and data extraction example is provided as follows:

[0046] Task scheduling center a is responsible for configuring the time, executor, reminder time, repetition period and other information of tasks t1, t2, t3...tn, and storing this information; data extraction platform b is responsible for obtaining and storing the attachments of tasks t1, t2, t3...tn generated by task scheduling center a.

[0047] Here, a task refers to a work task object created by the message processor b, which represents a work that needs to be performed.

[0048] Tasks can be assigned to specific users or user groups and executed repeatedly according to established rules.

[0049] A large amount of raw data will be generated during the operation of the data center. Data products are formed by processing, classifying and organizing the raw data. Each data product is completed by one or more tasks.

[0050] Each task has a unique ID, including name, status, type, submission time, estimated completion time, CPU usage, memory usage, and operator information. Each task belongs to a project, which has a unique project ID and project name. Tasks are passed one by one to the data extraction platform b in a queue.

[0051] Furthermore, an executor c refers to a compute node in a computing cluster. Each executor c has independent CPU and memory resources. Tasks currently running in an executor c are called active tasks, completed tasks are called completed tasks, and unfinished tasks waiting to be run are called pending tasks.

[0052] Among them, active tasks refer to tasks currently being run by executor c. Such tasks are in an active state and require memory resources to be allocated during their execution.

[0053] A completed task refers to a task that has been completed. From the user's perspective, it is work that has been completed.

[0054] To-do tasks are tasks that have not yet been executed.

[0055] The interaction process between the task scheduling center a and the data extraction platform b is as follows:

[0056] At time t1, data extraction platform b creates task t1 and submits it to task scheduling center a;

[0057] At time t2, the task scheduling center a polls to find an idle executor c and dispatches task t1 to the executor c for execution;

[0058] T21, at time t5, task scheduling center a decomposes task t1 into subtasks t1-1 and t1-2 according to a predefined calculation formula;

[0059] Wherein, t5=t1+(t2-t1)*0.1;

[0060] T22, at time t6, the task scheduling center a recombines the information of subtasks t1-1 and t1-2 into a new subtask t7;

[0061] Wherein, t6=t1-(t2-t1)*0.1;

[0062] T23. At time t7, the task scheduling center a dispatches the new task t7 to the executor c.

[0063] Where, t7 = t1;

[0064] At time t3, after the executor c finishes running, it saves the results to the file system and feeds them back to the task scheduling center a;

[0065] T4. At time t4, the task scheduling center a sends a notification to the data extraction platform b to inform that task t1 has been completed.

[0066] This embodiment also includes a timer d that periodically triggers the task scheduling center a to clean up the database. Visual design modules e, f, g, h, and i represent the product component, layout component, property panel component, style editing component, and chart component, respectively. Draggable components are called product components. Basic elements constitute the foundation of the page.

[0067] Among them, product components include: user-created components, system standard components, and predefined components;

[0068] Layout components combine basic elements to form a page;

[0069] The property panel component is an interface for modifying attribute values ​​such as element style and position;

[0070] The style editor is an interface for defining class names for each element;

[0071] The chart component is an interface for obtaining data from the data warehouse and performing data analysis.

[0072] In this embodiment, users include access users (terminal users) and permission users (administrators), wherein the administrator user can manage the permissions of the access users.

[0073] To sum up, the task scheduling center in the present invention configures the start and end time, executor selection, alarm conditions and repetition period information of the task through automatic configuration to generate basic task information; based on the basic task information, a task list within the set time length is generated by the task generator, and the tasks in the task list are assigned to multiple people with issuance authority and attachment requirements; the task executor performs a regular scan on the task list to determine whether the task in the task list is about to start; if the task to be started does not have the attachment requirement, the task to be started is sent to the task database; if the task to be started is accompanied by the attachment requirement, a data extraction request is sent to the data extraction platform, and an empty path to be filled file address is generated in the OSS storage unit; the data extraction platform extracts the attachment data corresponding to the task to be started according to the data extraction request, and sends the task to be started to the task database; the attachment data is filled into the file address to be filled in the OSS storage unit. Based on the alarm configuration set in the task scheduling center, an alarm event is triggered via xxl-job; the alarm signal is asynchronously notified to the data extraction platform via MQ; the data extraction platform captures the alarm-related data based on the alarm signal and generates an alarm task via the MQ message queue; and the alarm task is sent to the task database. When executing multi-city, multi-responsible person task requirements, the LiteFlow flow captures city manager information from the Chuanyun low-code platform; based on this city manager information, a corresponding attachment extraction subtask is generated; and this subtask is logically bound to the task in the task list. This invention achieves efficient management of complex data processing scenarios by optimizing task scheduling strategies and improving data extraction accuracy. It can provide flexible task allocation and automated data extraction processes for multi-city, multi-responsible person data processing needs, ensuring data accuracy and timeliness, and meeting the urgent needs for efficient data processing and management in various segments. By introducing technologies such as automated configuration, multi-executor collaborative operations, and the snowflake algorithm to generate unique task IDs, this invention achieves intelligent task scheduling and precise data extraction, providing more efficient data processing support for urban operations, network security, and other fields. This invention uses LiteFlow flow technology combined with a low-code platform to achieve flexible configuration and efficient execution of tasks and data processing, solves the shortcomings of existing technologies in automated process optimization, and provides strong technical support for data-driven modern urban operations.

[0074] It should be noted that the method of the embodiments of the present disclosure can be performed by a single device, such as a computer or server. The method of the embodiments of the present disclosure can also be applied in a distributed scenario, where multiple devices cooperate to perform the method. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiments of the present disclosure, and the multiple devices will interact with each other to complete the method.

[0075] It should be noted that the above description is limited to some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0076] Example 2

[0077] See also Figure 3 Embodiment 2 of the present invention further provides a task scheduling and data extraction system, including:

[0078] The task configuration and generation module 001 is used by the task scheduling center to configure the task's start and end time, executor selection, alarm conditions, and repetition period information through automated configuration to generate basic task information. Based on the basic task information, the task generator generates a task list within a set time period and assigns multiple people's distribution permissions and attachment requirements to the tasks in the task list.

[0079] The task scanning and triggering data extraction module 002 is used to perform a periodic scan of the task list through the task executor to determine whether a task in the task list is about to be started; if the task to be started does not have the attachment requirement, the task to be started is sent to the task database; if the task to be started has the attachment requirement, a data extraction request is sent to the data extraction platform, and a file address to be filled with an empty path is generated in the OSS storage unit;

[0080] The data extraction module 003 is used for the data extraction platform to extract the attachment data corresponding to the task to be started according to the data extraction request, and send the task to be started to the task database; fill the attachment data into the file address to be filled in the OSS storage unit.

[0081] In this embodiment, in the task configuration and generation module 001, based on the alarm configuration set in the task scheduling center, an alarm event is triggered through xxl-job; the alarm signal is asynchronously notified to the data extraction platform through MQ; the data extraction platform captures alarm-related data according to the alarm signal, and generates an alarm task through the MQ message queue; and the alarm task is sent to the task database.

[0082] In this embodiment, in the data extraction module 003, in the process of executing the task requirements of multiple cities and multiple responsible persons, the city person in charge information is captured from the Chuanyun low-code platform through the LiteFlow flow; the corresponding attachment extraction subtask is generated based on the city person in charge information; and the subtask is logically bound to the task in the task list.

[0083] In this embodiment, in the task configuration and generation module 001, a unique task ID is generated for each task in the task list through a snowflake algorithm; the operator logs in to the task scheduling center through the task ID to view the daily, weekly, and monthly task status.

[0084] In this embodiment, in the data extraction module 003, the LiteFlow flow includes: a liteflow rich client, a process service end, a process persistence layer interface and a process action maker.

[0085] It should be noted that the information interaction, execution process, etc. between the modules of the above-mentioned system are based on the same concept as the method embodiment in Example 1 of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the present application, and no further details will be given here.

[0086] Example 3

[0087] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, in which a program code of a task scheduling and data extraction method is stored. The program code includes instructions for executing a task scheduling and data extraction method of embodiment 1 or any possible implementation thereof.

[0088] Computer-readable storage media can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0089] Example 4

[0090] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0091] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute a task scheduling and data extraction method of embodiment 1 or any possible implementation thereof.

[0092] Specifically, the processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading software code stored in a memory. The memory can be integrated into the processor or located outside the processor and exist independently.

[0093] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable systems. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode.

[0094] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing system. They can be centralized on a single computing system or distributed across a network of multiple computing systems. Alternatively, they can be implemented using program code executable by a computing system, and thus, they can be stored in a storage system and executed by the computing system. In some cases, the steps shown or described herein can be performed in a different order than that shown, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0095] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made thereto. Therefore, such modifications and improvements, without departing from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. A task scheduling and data extraction method, characterized in that: include: The task scheduling center configures the task's start and end time, executor selection, alarm conditions, and repetition period information through automated configuration to generate basic task information; Based on the basic task information, a task list within a set time period is generated through a task generator, and tasks in the task list are assigned to multiple people with authority to issue the tasks and attachment requirements; The task list is scanned periodically by the task executor to determine whether a task in the task list is about to be started; if the task to be started does not have the attachment requirement, the task to be started is sent to the task database; if the task to be started is accompanied by the attachment requirement, a data extraction request is sent to the data extraction platform, and a file address to be filled with an empty path is generated in the OSS storage unit; The data extraction platform extracts the attachment data corresponding to the task to be started according to the data extraction request, and sends the task to be started to the task database; Fill the attachment data into the to-be-filled file address in the OSS storage unit.

2. A task scheduling and data extraction method according to claim 1, characterized in that: Based on the alarm configuration set in the task scheduling center, trigger the alarm event through xxl-job; the alarm signal is asynchronously notified to the data extraction platform through MQ; The data extraction platform captures alarm-related data according to the alarm signal, generates an alarm task through the MQ message queue, and sends the alarm task to the task database.

3. A task scheduling and data extraction method according to claim 2, characterized in that: In the process of executing multi-city, multi-responsible person task requirements, the LiteFlow flow is used to capture the city person in charge information from the Tritium Cloud low-code platform; based on the city person in charge information, the corresponding attachment extraction subtask is generated; The subtask is logically bound to the task in the task list.

4. A task scheduling and data extraction method according to claim 3, characterized in that: A unique task ID is generated for each task in the task list through a snowflake algorithm; the operator logs into the task scheduling center through the task ID to view the daily, weekly, and monthly task status.

5. A task scheduling and data extraction method according to claim 4, characterized in that: The LiteFlow flow includes: a liteflow rich client, a process server, a process persistence layer interface and a process action maker.

6. A task scheduling and data extraction system, characterized in that: include: The task configuration and generation module is used by the task scheduling center to configure the start and end time, executor selection, alarm conditions and repetition period information of the task through automated configuration, and generate basic task information; based on the basic task information, a task generator generates a task list within a set time period, and assigns multiple people's distribution permissions and attachment requirements to the tasks in the task list; The task scanning and triggering data extraction module is used to perform a periodic scan of the task list through the task executor to determine whether a task in the task list is about to be started; if the task to be started does not have the attachment requirement, the task to be started is sent to the task database; if the task to be started is accompanied by the attachment requirement, a data extraction request is sent to the data extraction platform, and an empty path file address to be filled is generated in the OSS storage unit; A data extraction module is used for the data extraction platform to extract the attachment data corresponding to the task to be started according to the data extraction request, and send the task to be started to the task database; Fill the attachment data into the to-be-filled file address in the OSS storage unit.

7. A task scheduling and data extraction system according to claim 6, characterized in that: In the task configuration and generation module, based on the alarm configuration set in the task scheduling center, the alarm event is triggered through xxl-job; the alarm signal is asynchronously notified to the data extraction platform through MQ; The data extraction platform captures alarm-related data according to the alarm signal, generates an alarm task through the MQ message queue, and sends the alarm task to the task database.

8. A task scheduling and data extraction system according to claim 7, characterized in that: In the data extraction module, when executing multi-city and multi-responsible person task requirements, the city person in charge information is captured from the Tritium Cloud low-code platform through the LiteFlow flow; based on the city person in charge information, the corresponding attachment extraction subtask is generated; The subtask is logically bound to the task in the task list.

9. A task scheduling and data extraction system according to claim 8, characterized in that: In the task configuration and generation module, a unique task ID is generated for each task in the task list through a snowflake algorithm; the operator logs into the task scheduling center through the task ID to view the daily, weekly and monthly task status.

10. A task scheduling and data extraction system according to claim 9, characterized in that: In the data extraction module, the LiteFlow flow includes: a liteflow rich client, a process server, a process persistence layer interface and a process action maker.

Images