Data processing method and system, electronic equipment and storage medium

Abstract

The invention discloses a data processing method and system, electronic equipment and a storage medium, and relates to the technical field of data processing.The method comprises the steps that a data processing task is created based on multiple operation options and managed, and the data processing task comprises at least one data processing logic; and processing the data associated with the data processing task through the data processing task. According to the method, the data processing task can be intelligently created and managed, manual operation is effectively reduced, and the data processing difficulty is reduced.

Description

Background Art

[0002] Since the 21st century, with the rapid development of information technology and the advent of the wave of digital transformation of enterprises, all walks of life have accumulated massive and complex basic data. In particular, in the process of continuously improving the quality of products and services, continuously implementing technological innovation to meet the personalized needs of customers, and enhancing corporate value, enterprises have gathered various types of large-scale and diversified business, product and customer data. These data are often stored in different databases to meet the needs of daily marketing and decision-making of enterprises. Kettle, open sourced by Pentaho, is a commonly used data processing tool in the industry. In the early stage of data technology development, it can support simple data processing within a specific range. However, Kettle can only meet the data processing needs of some specific scenarios. For modern data processing needs, it has the technical problem of "difficult management during data processing and requiring a lot of manual operations." Summary of the invention

[0003] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and specifically provide a data processing method, system, electronic device and storage medium, which are as follows:

[0004] 1) In the first aspect, the present invention provides a data processing method, and the specific technical solution is as follows:

[0005] Based on multiple operation options, create a data processing task and manage the data processing task, wherein the data processing task includes at least one data processing logic;

[0006] Through the data processing task, the data associated with the data processing task is processed.

[0007] The beneficial effects of a data processing method provided by the present invention are as follows:

[0008] It can intelligently create and manage data processing tasks, effectively reduce manual operations, and lower the difficulty of data processing.

[0009] Based on the above solution, a data processing method of the present invention can also be improved as follows.

[0010] Furthermore, it also includes:

[0011] Through the created permission management module, multiple users can create and manage data processing tasks.

[0012] The beneficial effect of adopting the above further scheme is: currently, Kettle does not support multi-user use. The Kettle production environment deployed by the same team can only be operated by one person at the same time. In the present invention, through the created permission management module, multiple users can create and manage data processing tasks, which improves flexibility and practicality, and improves user experience.

[0013] Furthermore, it also includes:

[0014] When creating a data processing task, configure the data processing logic in the data processing task through the created task scheduling module, configure the scheduling logic of the data processing logic in the data processing task, and configure the operation options for executing the statements of each preset database;

[0015] Through data processing tasks, the data associated with the data processing tasks are processed, including:

[0016] According to the scheduling logic, the data processing logic in the data processing task is executed to process the data associated with the data processing task, and the data associated with the data processing task is stored in at least one preset database.

[0017] The beneficial effect of adopting the above further scheme is: currently, Kettle can only schedule a small amount of data processing processes, while the present invention improves the configuration capability and scheduling capability of data processing tasks, supports statements of different databases, and supports DAG (DAG is the abbreviation of Directed Acyclic Graph, which is a directed acyclic graph used to describe the dependencies between tasks) scheduling, further improving practicality and further improving user experience.

[0018] Furthermore, it also includes:

[0019] Monitor the execution status of data processing tasks in real time and issue an alarm when an abnormality occurs.

[0020] The beneficial effect of adopting the above further scheme is: currently, Kettle is unable to monitor the data processing process, while in the present invention, the execution status of the data processing task is monitored in real time. When an abnormality occurs during the execution of the data processing task, an alarm can be issued in time to facilitate subsequent processing.

[0021] Furthermore, it also includes:

[0022] Through the SpringBoot+VUE framework, the process of creating data processing tasks and the process of processing the data associated with the data processing tasks are clustered and deployed.

[0023] The beneficial effect of adopting the above further solution is: currently, Kettle can only be deployed at a single point, but in this application, through the SpringBoot+VUE framework, the process of creating data processing tasks and the process of processing data associated with data processing tasks are clustered and deployed, which has higher practicality, can improve the efficiency of hardware resource utilization and reduce costs.

[0024] 2) In a second aspect, the present invention further provides a data processing system, and the specific technical solution is as follows:

[0025] Including: data processing task creation module and data processing module;

[0026] The data processing task creation module is used to: create a data processing task based on multiple operation options and manage the data processing task, wherein the data processing task includes at least one data processing logic;

[0027] The data processing module is used to process the data associated with the data processing task through the data processing task.

[0028] Based on the above solution, a data processing system of the present invention can also be improved as follows.

[0029] Furthermore, it also includes a permission management module, which is used to enable multiple users to create and manage data processing tasks.

[0030] Further, it also includes: a task scheduling module, the task scheduling module is used to: when creating a data processing task, configure the data processing logic in the data processing task, configure the scheduling logic of the data processing logic in the data processing task, and configure the operation options for executing the statement of each preset database;

[0031] The data processing module is used to: execute the data processing logic in the data processing task according to the scheduling logic, process the data associated with the data processing task, and store the data associated with the data processing task in at least one preset database.

[0032] Furthermore, it also includes a monitoring and alarm module, which is used to:

[0033] Monitor the execution status of data processing tasks in real time and issue an alarm when an abnormality occurs.

[0034] Furthermore, a cluster deployment module is included, and the cluster deployment module is used to:

[0035] Through the SpringBoot+VUE framework, the process of creating data processing tasks and the process of processing the data associated with the data processing tasks are clustered and deployed.

[0036] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, the processor being coupled to a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor so that the electronic device implements any one of the above-mentioned data processing methods.

[0037] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium, in which at least one computer program is stored, and the at least one computer program is loaded and executed by a processor so that the computer implements any of the above-mentioned data processing methods.

[0038] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0040] Figure 1 A schematic diagram of a data processing method according to an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of the interface for creating a data processing task;

[0042] Figure 3 This is a schematic diagram of the interface for the "View" operation option;

[0043] Figure 4 This is a schematic diagram of the interface for the "Modify" operation option;

[0044] Figure 5 This is a schematic diagram of the interface for the "Execute" operation option;

[0045] Figure 6 This is a schematic diagram of the interface for the "polling" operation option;

[0046] Figure 7 This is a schematic diagram of the login interface;

[0047] Figure 8 This is a diagram of the interface of the task monitoring log list;

[0048] Fig. 9 It is a functional architecture diagram;

[0049] Fig.10 This is a technical architecture diagram;

[0050] Fig.11 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0052] like Figure 1 As shown, a data processing method according to an embodiment of the present invention comprises the following steps:

[0053] S1. Based on multiple operation options, create a data processing task and manage the data processing task, wherein the data processing task includes at least one data processing logic;

[0054] S1 implements the task management capability. Multiple operation options (also called capability items) mainly include: "View", "Modify", "Delete", "Activate", "Pause", "Execute", "Configure" and "Log", etc. Multiple operation options can provide the most sophisticated management of data processing tasks. Multiple operation options can create data processing tasks, that is, all related data processing logics are managed through one task (i.e., data processing task). A data processing task is a collection of data processing logics. The interface for creating a data processing task is as follows: Figure 2 shown.

[0055] The specific functions of each operation option are as follows:

[0056] 1) "View" operation option: Click the "View" operation option to obtain metadata information of the data processing task, such as the name of the data processing task, the ID of the data processing task, the system to which the data processing task belongs, the user who created the data processing task, and the creation time of the data processing task. Figure 3 shown.

[0057] 2) "Modify" operation option: Click the "Modify" operation option to modify the name of the data processing task, the execution time of the data processing task, the execution frequency of the data processing task, the execution node of the data processing task, and whether the data processing task is enabled for scheduling. Figure 4 shown.

[0058] 3) “Delete” operation option: Click the “Delete” operation option to delete the currently created data processing task.

[0059] 4) "Activate" operation option: Click the "Activate" operation option to activate the currently created data processing task and add the currently created data processing task to the daily scheduling.

[0060] 5) “Pause” operation option: Click the “Pause” operation option to pause the currently created data processing task in the daily scheduling.

[0061] 6) "Execute" operation option: Click the "Execute" operation option to execute the currently created data processing task, such as Figure 5 shown.

[0062] 7) “Configure” operation option: Click the “Configure” operation option to configure the data processing logic, which will be explained in detail in the subsequent S4.

[0063] 8) "Log" operation option: Click the "Log" operation option to view the running log of the data processing task. The log of each logical node will be fully displayed in the set execution order.

[0064] 9) "Polling" operation option: Click the "Polling" operation option to batch execute the same data processing task according to a preset condition. For example, for the common complement task in data processing, you can schedule the execution of the complement task according to a pre-specified time interval. Figure 6 shown.

[0065] S2. Processing data associated with the data processing task through the data processing task.

[0066] A data processing method according to an embodiment of the present invention can intelligently create and manage data processing tasks, effectively reduce manual operations, and lower the difficulty of data processing.

[0067] Optionally, in the above technical solution, it also includes:

[0068] S3. Through the created permission management module, multiple users can create and manage data processing tasks.

[0069] S3 implements comprehensive permission management capabilities, enabling multiple users to log in with their account and password to create and manage data processing tasks. The login interface is as follows: Figure 7 As shown, users are divided into individuals and tenants (companies, etc.).

[0070] Currently, Kettle does not support multi-user use. The Kettle production environment deployed by the same team can only be operated by one person at the same time. In the present invention, through the created permission management module, multiple users can create and manage data processing tasks, which improves flexibility and practicality and improves user experience.

[0071] The operation options (also called capability items) provided by the permission control module can support multiple people to operate without worrying about mutual interference, which greatly improves the security, reliability and robustness of data processing tasks. Compared with other existing tools in the industry, such as Kettle, it has achieved an improvement from scratch and from 0 to 1.

[0072] Optionally, in the above technical solution, it also includes:

[0073] S4. When creating a data processing task, configure the data processing logic in the data processing task through the created task scheduling module, configure the scheduling logic of the data processing logic in the data processing task, and configure the operation options for executing the statements of each preset database;

[0074] S4 implements the task scheduling capability, which is used to complete the configuration and scheduling of the data processing logic of data processing tasks. It includes 9 logical nodes, including "start", "end", "exception", "judgment", "fail", "shell script", "SQL", "loop" and "synchronization", for users to configure and schedule data processing logic. Through these 9 logic types, the configuration and scheduling of data processing tasks can be completed in a refined manner, and support can be provided for data processing of homogeneous and heterogeneous databases, and the configuration of DAG scheduling logic can be realized. The 9 logical nodes are explained as follows:

[0075] 1) "Start" logic node: When you click the "Start" logic node or the "Execute" logic node, the data processing task is executed in the order configured by the user (the execution order of all data processing logics and all scheduling logics). Specifically, the execution starts from the start node in the data processing task. The start node (the first logic in all data processing logics and all scheduling logics) is the first node of the data processing task. All processing logics (all data processing logics and all scheduling logics) must start with the start node.

[0076] 2) "End" logical node: The data processing task can be terminated at any time. It should be noted that when the end node of the data processing task (the last logic among the data processing logic and all scheduling logics) is executed in the order configured by the user (the execution order of all data processing logics and all scheduling logics), it ends automatically. At this time, the end logical node serves as an identifier to indicate that the data processing task has been completed.

[0077] 3) "Exception" logic node: used to capture the execution exceptions of each node (data processing logic or scheduling logic) so that when an exception occurs in the data processing task, it can be processed through logical control, through the data processing link, or an alarm message can be sent.

[0078] The "Exception" logic node is used to be set after any of the above operation options. For example, if the "Exception" logic node is set after the "Synchronize" operation option, then when the "Synchronize" operation option encounters an exception, it will not affect the data processing task. Instead, the link of the data processing task will be changed to other branches through the "Exception" logic node, and the corresponding reminder message can be sent to the user's client through the "Exception" logic node. The user's client can be enterprise WeChat, etc.

[0079] 4) "Judgment" logic node: used to provide users with logic judgment functions. Specifically, each time a node is executed, the subsequent data processing logic needs to be executed based on the execution result, so that users can evaluate whether the next stage of data processing logic is running based on the execution result obtained each time. So that users can discover some possible data problems in time.

[0080] 5) "Shell script" logic node: When the remote server writes the data processing task into a Shell script, the Shell script logic node can be used to execute the Shell script to perform the data processing task. It can truly achieve remote script calling, while Kettle and other products can only execute local scripts. This logic node realizes the calling of big data components. The "Shell script" logic node is a logic node designed to solve network security and data security.

[0081] 6) "SQL" logical node: used to execute SQL statements of general JDBC database, supporting dozens of heterogeneous databases. If the main database supports JDBC, the SQL logical node can execute SQL statements that conform to its syntax to meet the needs of processing the database data. This node is the core functional point of the present invention, which can connect SQL statements of multiple heterogeneous databases in series or in parallel, and support traditional relational databases, big data databases, information-based databases, etc., such as: mysql, oracle, clickhouse, oceanbase, tbase, hive, etc.

[0082] 7) "Loop" logical node: execute SQL statements in a loop, and call a certain SQL statement repeatedly by passing parameters.

[0083] 8) "Synchronization" logical node: Data synchronization between heterogeneous databases is the core function of the present invention, which can efficiently support dozens of heterogeneous databases, including traditional relational databases, big data databases, and information-based databases.

[0084] In S2, the data associated with the data processing task is processed through the data processing task, including:

[0085] S20. Execute the data processing logic in the data processing task according to the scheduling logic, process the data associated with the data processing task, and store the data associated with the data processing task in at least one preset database.

[0086] At present, Kettle can only schedule a small amount of data processing processes, while the present invention improves the configuration and scheduling capabilities of data processing tasks, supports statements of different databases, and supports DAG (DAG is the abbreviation of Directed Acyclic Graph, which is a directed acyclic graph used to describe the dependencies between tasks) scheduling, further improving practicality and user experience.

[0087] Optionally, in the above technical solution, it also includes:

[0088] S5. Monitor the execution status of data processing tasks in real time and issue an alarm when an abnormality occurs.

[0089] S5 implements monitoring and alarm capabilities, including running status monitoring and alarm information push, a total of two capabilities. The running status monitoring capability is used to monitor the running status of data processing tasks, and the alarm information push capability is used to push alarm information when an abnormality occurs. It can obtain the real-time status of task operation, such as Figure 8 As shown in the figure, as well as the operation logs of the above 9 logical nodes, when problems occur in the operation of each logical node, messages are pushed to users in a timely manner.

[0090] The monitoring and alarming capabilities are capabilities that existing data processing tools in the industry do not have. For data processing tools, this is a creative function created from scratch. For example, at present, Kettle is unable to monitor the data processing process, but in the present invention, the execution status of the data processing task is monitored in real time. When an abnormality occurs during the execution of the data processing task, an alarm can be issued in time to facilitate subsequent processing. Among them, the alarm information is ""createDate:1725810622025,"createUser:"system","jobld:"edbca2f239604549b1fea9e212de334a","nodeId":"2fb772f04f93464f87e81626ce785450","nodeParams":"{intervalNumber-1,statDate=2024-09-07,intervalT ype=D,dateFormat=yyyy-MM-dd,timeout=20000,username=system}","queueld":"e973fdf63f04470f871c7a7bd703cd31","runEndTime":1725810622025,"runLog":"Start node execution completed!","runStartTime:1725810622025,"runTime":0,"status":"su ccess", can also be set according to actual situation.

[0091] Optionally, in the above technical solution, it also includes:

[0092] S6. Through the SpringBoot+VUE framework, the process of creating data processing tasks and the process of processing the data associated with the data processing tasks are clustered and deployed.

[0093] At present, Kettle can only be deployed at a single point. In this application, the SpringBoot+VUE framework is used to cluster the process of creating data processing tasks and processing the data associated with the data processing tasks. This has higher practicality, can improve the efficiency of hardware resource utilization, and reduce costs.

[0094] The present invention can support high-availability multi-link DAG data processing of heterogeneous databases, and on this basis, innovatively proposes the following five core capabilities: ① Complete task management capabilities, greatly reducing manual operations; ② It also has a complete permission management function, supports multi-tenants, and can be operated by multiple people at the same time; ③ A data processing system of the present invention has a complete data processing task scheduling function and supports DAG scheduling; ④ Supporting monitoring functions and alarm message functions are developed for data processing tasks; ⑤ Support clusters, tasks can be executed on different nodes, and have high availability.

[0095] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0096] A data processing system according to an embodiment of the present invention comprises: a data processing task creation module and a data processing module;

[0097] The data processing task creation module is used to: create a data processing task based on multiple operation options and manage the data processing task, wherein the data processing task includes at least one data processing logic;

[0098] The data processing module is used to process the data associated with the data processing task through the data processing task.

[0099] Optionally, the above technical solution further includes a permission management module, which is used to enable multiple users to create and manage data processing tasks.

[0100] Optionally, in the above technical solution, it also includes: a task scheduling module, the task scheduling module is used to: when creating a data processing task, configure the data processing logic in the data processing task, configure the scheduling logic of the data processing logic in the data processing task, and configure the operation options for executing the statement of each preset database;

[0101] The data processing module is used to: execute the data processing logic in the data processing task according to the scheduling logic, process the data associated with the data processing task, and store the data associated with the data processing task in at least one preset database.

[0102] Optionally, in the above technical solution, a monitoring alarm module is also included, and the monitoring alarm module is used to:

[0103] Monitor the execution status of data processing tasks in real time and issue an alarm when an abnormality occurs.

[0104] Optionally, in the above technical solution, a cluster deployment module is further included, and the cluster deployment module is used to:

[0105] Through the SpringBoot+VUE framework, the process of creating data processing tasks and processing the data associated with the data processing tasks are clustered and deployed. Specifically:

[0106] A data processing system of the present invention is constructed through the SpringBoot+VUE framework, with front-end and back-end separation and agile development, which is convenient for adjusting platform functions at any time according to business development. Based on this technology, a data processing system of the present invention can realize clustered deployment. Traditional data processing tools can only support single-node deployment, and the ability to support task concurrency and parallelism is quite limited. It is even more impossible to support scenarios with frequent and demanding data processing tasks. Today, when data processing tasks are becoming more complex and frequent, a data processing system of the present invention can fully provide clustering capabilities, improve the efficiency of hardware resource utilization, and reduce costs.

[0107] In another embodiment, comprising:

[0108] 1) Task management capability module: provides 8 capability items (operation options), so that the data processing tool implemented by the present invention (i.e., a data processing system of the present invention) can provide the most sophisticated data processing task management steps, while traditional tools such as Sqoop, Crontab, and Kettle do not have this capability, and their task management relies on a large amount of manual operations, which is inefficient and has a low fault tolerance rate.

[0109] 2) Permission management module: It provides tenant-level and user-level permission management, so that the data processing system of the present invention can realize multi-tenant / multi-person online real-time operation without affecting each other. However, traditional tools such as Sqoop, Crontab, and Kettle do not have this capability. Their operation can only be performed by one person at a time, which is relatively primitive and no longer meets the data processing requirements of the current industry.

[0110] 3) Task Scheduling Capability Module: It provides 9 logic types for the configuration and scheduling of data processing tasks. It implements all possible logics involved in current data processing, which traditional tools such as Sqoop, Crontab, and Kettle do not have.

[0111] 4) Monitoring and alarm function module: It provides real-time log output and monitoring and alarm capabilities, so that users can immediately know the results of data processing tasks and automatically receive alarm information through social tools when errors occur. However, traditional tools such as Sqoop, Crontab, and Kettle do not have this capability and no longer meet the requirements of modern data processing technology stacks.

[0112] 5) Clustering architecture function module provides clustering deployment capability, so that the data processing system of the present invention can directly meet the modern high-frequency and complex data processing requirements, while traditional tools such as Sqoop, Crontab, Kettle, etc. do not support clustering and are no longer suitable for the current high-frequency and complex data processing requirements.

[0113] The functional architecture and technical architecture required by a data processing system of the present invention are respectively as follows: Fig. 9 and Fig.10 As shown, specifically:

[0114] 1) If Fig. 9 As shown, it is divided into a dispatching center, an executor and a user. The user's client is divided into a management cockpit, a data beacon, a car insurance business assistant, a marketing management robot and a pension data analysis platform. All the user's clients can log in to the dispatching center through a single sign-on service. The user's client executes permission services, that is, creates and executes data processing tasks within the corresponding permissions. The task management of the dispatching center is divided into: task configuration (creation) management, executor management, activation / deactivation management, server list management, dispatching log management, operation report management, failure alarm management (abnormal alarm management) and task dependency management. In the executor management, the scheduler is used to control the executor to perform dispatching queue management and task thread management. According to the dispatching queue and task thread, a log service is generated, a dispatching result is generated according to the log service, and an alarm service is generated according to the dispatching result.

[0115] 2) If Fig.10 As shown, it is divided into client, network layer, technical layer and database. The client can be PC. The network layer includes firewall and reverse proxy. The technical layer includes: SpringBoot, Element-Ul, SpringSecurity, ECharts, MyBatis-Plus, VUE, EasyExcel, TypeScript, Datax, JsPlumb, Python and Shell, etc. The databases include Oracle, Mysql, OceanBase, ClickHouse, Hive, Impala, TDSQL and DAMO database, etc.

[0116] Prior to the present invention, existing data processing tools in the industry include Sqoop, Crontab, Kettle, etc., which have limited functions. Although Sqoop, Crontab, Kettle and other tools have realized the process of "from 0 to 1", their capabilities are too primitive in today's era of rapid technological iteration. In the modern data technology stack, there is an urgent need for capabilities in five areas: task management, authority management, task scheduling, monitoring and alarming, and clustering, which the above tools cannot provide. This makes the productivity of data processing tools unable to match the increasingly complex and high-frequency data processing needs. Based on the above scenarios, a data processing system of the present invention can support high-availability multi-link DAG data processing of heterogeneous databases, realizing the "from 1 to 100" development.

[0117] It should be noted that the beneficial effects of a data processing system provided by the above embodiment are the same as the beneficial effects of the above data processing method, which will not be described in detail here. In addition, when the system provided by the above embodiment realizes its functions, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided by the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be described in detail here.

[0118] like Fig.11 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, the processor 320 is coupled to a memory 310, and the memory 310 stores at least one computer program 330, and the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the above data processing methods, specifically:

[0119] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements any of the data processing methods provided in the above embodiments. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 300 may also include other components for realizing device functions, which will not be described in detail here. The electronic device may specifically be a computer, etc.

[0120] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements any of the above data processing methods.

[0121] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0122] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above-mentioned data processing methods.

[0123] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0124] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0125] Any combination of one or more computer-readable media can be used. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0126] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A data processing method, characterized in that: include: Based on a plurality of operation options, create a data processing task and manage the data processing task, wherein the data processing task includes at least one data processing logic; Through the data processing task, data associated with the data processing task is processed.

2. A data processing method according to claim 1, characterized in that: Also includes: Through the created permission management module, multiple users can create and manage data processing tasks.

3. A data processing method according to claim 2, characterized in that: Also includes: When creating a data processing task, the data processing logic in the data processing task is configured through the created task scheduling module, the scheduling logic of the data processing logic in the data processing task is configured, and the operation options for executing the statements of each preset database are configured; The processing of the data associated with the data processing task through the data processing task includes: According to the scheduling logic, the data processing logic in the data processing task is executed to process the data associated with the data processing task, and the data associated with the data processing task is stored in at least one preset database.

4. A data processing method according to claim 3, characterized in that: Also includes: Monitor the execution status of the data processing task in real time and issue an alarm when an abnormality occurs.

5. A data processing method according to claim 4, characterized in that: Also includes: Through the SpringBoot+VUE framework, the process of creating data processing tasks and the process of processing the data associated with the data processing tasks are clustered and deployed.

6. A data processing system, characterized in that: include: Data processing task creation module and data processing module; The data processing task creation module is used to: create a data processing task based on multiple operation options, and manage the data processing task, wherein the data processing task includes at least one data processing logic; The data processing module is used to process the data associated with the data processing task through the data processing task.

7. A data processing system according to claim 6, characterized in that: It also includes a permission management module, which is used to enable multiple users to create and manage data processing tasks.

8. A data processing system according to claim 7, characterized in that: Also includes: A task scheduling module, the task scheduling module is used to: when creating a data processing task, configure the data processing logic in the data processing task, configure the scheduling logic of the data processing logic in the data processing task, and configure the operation options for executing the statements of each preset database; The data processing module is used to: execute the data processing logic in the data processing task according to the scheduling logic, process the data associated with the data processing task, and store the data associated with the data processing task in at least one preset database.

9. An electronic device, characterized in that: The electronic device includes a processor, the processor is coupled to a memory, the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements a data processing method as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements a data processing method as claimed in any one of claims 1 to 5.

Images