Data processing method and device, equipment, medium and program product

Abstract

The invention provides a data processing method and device, equipment, a storage medium and a program product, and can be applied to the technical field of big data. The data processing method comprises the following steps: in response to a received task processing request, generating a task instance and sending the task instance to a first database; wherein the task processing request comprises task attribute information and execution time; polling operation is executed on the task instances in the first database regularly, and at least one target task instance is obtained; generating a corresponding task thread according to the target task instance; wherein the number of the task threads is the same as that of the target task instances; sending the target task instance to a task execution node based on the task thread; wherein the number of the task execution nodes is not less than 1; and controlling the task execution node to read the target data from the source database based on the received target task instance and create a data snapshot of the target data, and inserting the data snapshot into the target database.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and more specifically to a data processing method, device, equipment, medium and program product. Background Art

[0002] By creating data snapshots regularly, data protection can be effectively achieved. For example, when data is lost or damaged, you can quickly restore the data to the state at a specific point in time by rolling back to the snapshot at that point in time, effectively reducing business interruptions and losses caused by data loss or damage.

[0003] In the prior art, data snapshot tasks are often performed manually. Whenever a data snapshot is required, a technician needs to generate a data snapshot task in real time, which is inefficient and lacks flexibility, and cannot adapt to rapidly changing business needs. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a data processing method, apparatus, device, medium and program product for improving data processing efficiency.

[0005] According to a first aspect of the present disclosure, a data processing method is provided, including: in response to receiving a task processing request, generating a task instance and sending the task instance to a first database; wherein the task processing request includes task attribute information and execution time; periodically performing a polling operation on the task instances in the first database to obtain at least one target task instance; generating corresponding task threads according to the target task instances; wherein the number of task threads is the same as the number of target task instances; sending the target task instance to a task execution node based on the task thread; wherein the number of task execution nodes is not less than 1; controlling the task execution node to read target data from a source database based on the received target task instance and create a data snapshot of the target data, and inserting the data snapshot into the target database.

[0006] According to an embodiment of the present disclosure, the method also includes: obtaining metadata information in a source database; constructing a visualization interface based on the metadata information, wherein the visualization interface is arranged on a front-end device and includes a plurality of event triggers for reading and responding to user interaction actions; generating a task processing request based on the user's interaction actions, wherein the interaction actions include setting a click button, selecting a drop-down box, entering text, etc.

[0007] According to an embodiment of the present disclosure, in response to receiving a task processing request, a task instance is generated and sent to a first database, including: parsing the task processing request to obtain task attribute information and time information; generating a task instance based on the task attribute information and time information; and sending the task instance to a designated location in the first database for storage.

[0008] According to an embodiment of the present disclosure, a task instance is generated based on task attribute information and time information, including: performing structured processing on the task attribute information to obtain task elements; converting the time information into a time series; and generating a task instance based on the task elements and the time series.

[0009] According to an embodiment of the present disclosure, performing a polling operation on a task instance in a first database at a fixed time interval to obtain at least one target task instance includes: traversing each task instance in the first database at a preset time interval to obtain first information of the task instance; wherein the first information includes time information and state information of the task instance; determining whether the first information of the task instance meets a preset condition; and if the first information meets the preset condition, determining the task instance corresponding to the first information as the target task instance. .

[0010] According to an embodiment of the present disclosure, a target task instance is sent to a task execution node based on a task thread, including: sending a request to the task execution node by polling; wherein the request is used to obtain status information of the task execution node; obtaining the status information of the task execution node based on received response information; wherein the response information is generated by the task execution node based on the request; and when the status information of the task execution node is a non-busy state, sending the target task instance to the task execution node.

[0011] According to an embodiment of the present disclosure, a control task execution node reads target data from a source database based on a received target task instance and creates a data snapshot of the target data, and inserts the data snapshot into the target database, including: parsing the target task instance to obtain task configuration information; generating a data read handle and a data write handle based on the task configuration information; reading the target data from the source database based on the data read handle and generating a data snapshot of the target data; and writing the data snapshot into the target database based on the data write handle.

[0012] According to an embodiment of the present disclosure, writing a data snapshot into a target database based on a data write handle includes: writing the data snapshot into an existing file in the target database; or creating a new file in the target database and writing the data snapshot into the new file.

[0013] According to an embodiment of the present disclosure, target data is read from a source database based on a data read handle and a data snapshot of the target data is generated, and it also includes: before reading the target data from the source database, performing a preprocessing operation on the data in the source database; wherein the preprocessing operation includes at least any one of creating a table, filtering data, and updating the data format.

[0014] According to an embodiment of the present disclosure, the method also includes: based on the execution result of the task thread, closing the task thread and updating the status of the target task instance in the first database; and in response to generating a task thread corresponding to the target task instance, changing the status of the target task instance in the first database from a waiting state to an execution state.

[0015] The second aspect of the present disclosure provides a data processing device, including: a first generation module, used to generate a task instance in response to receiving a task processing request and send the task instance to a first database; wherein the task processing request includes task attribute information and execution time; an acquisition module, used to periodically perform polling operations on the task instances in the first database to obtain at least one target task instance; a second generation module, used to generate corresponding task threads according to the target task instance; wherein the number of task threads is the same as the number of target task instances; a sending module, used to send the target task instance to a task execution node based on the task thread; wherein the number of task execution nodes is not less than 1; a read and insert module, used to control the task execution node to read the target data from the source database based on the received target task instance and create a data snapshot of the target data, and insert the data snapshot into the target database.

[0016] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0017] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0018] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0020] Figure 1The application scenario diagram of the data processing method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown;

[0021] Figure 2 A flowchart schematically shows a data processing method according to an embodiment of the present disclosure;

[0022] Figure 3 Schematically shows a flow chart of generating a task processing request according to an embodiment of the present disclosure;

[0023] Figure 4 Schematically illustrates a flow chart of generating a task instance and sending the task instance to a first database in response to receiving a task processing request according to an embodiment of the present disclosure;

[0024] Figure 5 A flowchart of generating a task instance based on task attribute information and time information according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 6 A flowchart schematically shows a method of periodically performing a polling operation on a task instance in a first database to obtain at least one target task instance according to an embodiment of the present disclosure;

[0026] Figure 7 A flowchart of sending a target task instance to a task execution node based on a task thread according to an embodiment of the present disclosure is schematically shown;

[0027] Figure 8 A flowchart schematically shows a control task execution node according to an embodiment of the present disclosure reading target data from a source database based on a received target task instance and creating a data snapshot of the target data, and inserting the data snapshot into a target database;

[0028] Fig. 9 A structural block diagram schematically shows a data processing device according to an embodiment of the present disclosure; and

[0029] Fig.10 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0030] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0031] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0032] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0033] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0034] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0035] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0036] An embodiment of the present disclosure provides a data processing method, including: in response to receiving a task processing request, generating a task instance and sending the task instance to a first database; wherein the task processing request includes target data attributes and execution time; periodically performing a polling operation on the task instances in the first database to obtain a target task instance; generating a corresponding task thread according to the target task instance; reading the target data from the source database based on the task thread and creating a data snapshot of the target data, and inserting the data snapshot into the target database.

[0037] Figure 1The application scenario diagram of the data processing method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown.

[0038] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0039] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0040] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0041] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0042] It should be noted that the data processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the data processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The data processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0043] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0044] The following will be based on Figure 1 The scene described by Figure 2~Figure 8 The data processing method of the disclosed embodiment is described in detail.

[0045] Figure 2 The flowchart of the data processing method according to the embodiment of the present disclosure is schematically shown.

[0046] like Figure 2 As shown, the data processing method of this embodiment includes operations S210 to S250.

[0047] In operation S210, in response to receiving a task processing request, a task instance is generated and sent to a first database; wherein the task processing request includes task attribute information and execution time.

[0048] In some embodiments, the task processing request is automatically generated based on the information input by the user in the visual interface of the front-end device. After receiving the task processing request, the task request is parsed, and a task instance is generated based on the task elements in the task request. The task instance includes the specific execution process of the task request, and the task request is completed by executing the task instance, wherein the task instance is executed at a specified execution time.

[0049] Specifically, after the task instance is generated, the task instance data needs to be sent to the first database based on the database connection, so as to store the task instances uniformly, monitor the status of each task instance in real time, and automatically execute the corresponding task according to the predetermined time or condition. After the task instance is sent to the first database, the status of the task instance is initialized to the waiting state.

[0050] In operation S220, a polling operation is periodically performed on the task instances in the first database to obtain at least one target task instance.

[0051] In some embodiments, by periodically performing a polling operation on the task instances in the first database, a task instance that meets the preset condition is obtained and the task instance is determined as a target task instance. The target task instance refers to a task instance that needs to be executed in the current time period. The time to start the polling is defined by the user, which can be minutes, 10 minutes, hours, days, etc., to filter out the task instances that need to be executed in the current time period.

[0052] Taking polling as an example, assuming that the current time is 2024-01-01 15:31:05, task instances with execution times between 2024-01-01 15:31:00 and 2024-01-01 15:31:59 need to be filtered out as target task instances.

[0053] In operation S230, corresponding task threads are generated according to the target task instance; wherein the number of task threads is the same as the number of target task instances.

[0054] In some embodiments, a corresponding task thread is generated for each target task instance, and the task instance is executed based on the task thread to obtain a task execution result. When the task instance is executed, the task thread corresponding to the task instance is closed or terminated.

[0055] The disclosed embodiment can realize multi-threaded execution of tasks by generating a corresponding task thread for each target task instance, thereby effectively improving the execution efficiency of data processing, reducing task waiting time, and avoiding mutual interference of task keys.

[0056] Specifically, in response to generating a task thread corresponding to the target task instance, the state of the task instance is changed from a waiting state to an executing state.

[0057] In operation S240 , the target task instance is sent to a task execution node based on the task thread; wherein the number of the task execution nodes is not less than one.

[0058] In some embodiments, the task thread polls the application programming interface (API) of the task execution node to obtain the current state of the task execution node, and sends the task instance to the task execution node when the current state of the task execution node meets the preset conditions. The task execution node can be a physical machine, a virtual machine, a container, or any other computing resource that can process the task. By distributing the task instance to different task execution nodes, the resource competition problem that occurs when the task is directly executed based on the task thread can be effectively avoided, and effective isolation of resources can be achieved, and parallel processing of tasks can be achieved, thereby improving the efficiency of data processing.

[0059] In operation S250 , the task execution node reads the target data from the source database based on the task instance and creates a data snapshot of the target data, and inserts the data snapshot into the target database.

[0060] In some embodiments, when the task execution node is in a non-busy state, the control task thread sends the task instance to the task execution node, and the task execution node executes the task based on the task instance.

[0061] During the specific implementation process, the task execution node parses the received task instance data, obtains the task information, and generates a data read handle and a data insert handle based on the task information. The data read handle is used to read the target task from the source data and create a data snapshot of the target data, and the data insert handle is used to insert the data snapshot into the target database.

[0062] The disclosed embodiment performs polling operations on task instances at regular intervals, selects target task instances that need to be executed in the current time period, and acquires and inserts target data snapshots based on the target task instances, thereby achieving scheduled execution of data snapshots and improving the flexibility and efficiency of data snapshots.

[0063] Figure 3 The flowchart of generating a task processing request according to an embodiment of the present disclosure is schematically shown.

[0064] like Figure 3 As shown, the generation of a task processing request in this embodiment includes operations S310 to S330.

[0065] In operation S310, metadata information in a source database is acquired.

[0066] In some embodiments, the metadata information includes table information, field information, relationship information, attribute information, etc. in the source database.

[0067] Specifically, table information includes table name, column name, etc.; field information includes data type and field description of each field; relationship information includes primary key, foreign key, index information, etc., among which primary key and foreign key include primary key definition of the table and external reference relationship with other tables, and index information includes index definition of the table; attribute information includes creation / modification date, permission information, storage information, data model, data dictionary, etc., among which creation / modification date indicates creation and last modification information of data table or field, permission information indicates data table or field accessible to each user, storage information indicates location and space size of data on physical storage medium, data model indicates logical structure and physical structure description of database, and data dictionary contains detailed information of all objects in database (such as tables, views, stored procedures, etc.).

[0068] In operation S320, a visualization interface is constructed based on the metadata information, wherein the visualization interface is set on a front-end device and includes a plurality of event triggers for reading and responding to user interaction actions.

[0069] In some embodiments, a visualization interface is constructed by using the acquired metadata information, and the user can create tasks according to the visualization interface, thereby effectively simplifying the task creation process and improving the user experience and task creation efficiency.

[0070] The visualization interface may include, for example, a navigation menu, a search bar, a drop-down list, etc., so that users can browse and select task parameters within a specific range, thereby achieving flexible definition of data dimensions and task execution time.

[0071] The designated area of ​​the visualization interface includes multiple event triggers, which can read and respond to user interaction actions through the event triggers. Among them, the task parameters may include, for example, task execution time, target data information, etc.

[0072] In operation S330 , a task processing request is generated based on the user's interactive action, wherein the interactive action includes setting a click button, selecting a drop-down box, inputting text, and the like.

[0073] In some embodiments, after the user's interactive action is acquired, the information input by the user under the current interactive action can be parsed through the event processing function, and the information input by the user can be processed to automatically generate a task processing request.

[0074] Specifically, the user can preview the metadata information in the visualization interface by clicking a button and selecting a drop-down box, thereby selecting the target data and determining the execution time, or directly determining the target data and execution time by entering text. When selecting / determining the execution time, a certain moment can be selected as the execution time of the task, or multiple different moments can be selected as the execution time of the task.

[0075] The data processing method provided by the embodiment of the present disclosure builds a visualization interface based on the metadata information in the source database, so that users can flexibly select data dimensions and task execution time through the visualization interface, thereby improving the flexibility of data processing and meeting data processing requirements in different business scenarios.

[0076] Figure 4 The flowchart schematically shows a method of generating a task instance and sending the task instance to a first database in response to receiving a task processing request according to an embodiment of the present disclosure.

[0077] like Figure 4 As shown, in response to receiving a task processing request, generating a task instance and sending the task instance to the first database in this embodiment includes operations S410 to S430.

[0078] In operation S410, the task processing request is parsed to obtain task attribute information and time information.

[0079] In operation S420, a task instance is generated based on the task attribute information and the time information.

[0080] In operation S430 , the task instance is sent to a designated location in the first database for storage.

[0081] In some embodiments, task attribute information and time information are obtained by parsing the task processing request, and a task instance is generated based on the task attribute information and time information. The task instance is stored in the first database through a database connection (such as JDBC, ODBC, ORM framework, etc.), and at this time, the state of the task instance in the first database is the initial state (i.e., the waiting state).

[0082] On the one hand, storing the task instance in the first database can ensure the persistence and reliability of the task instance. For example, when the system is restarted or fails, the task instance information stored in the database will not be lost. The unfinished task can be rescheduled or continued by restoring the task instance information, ensuring the reliability and stability of data processing. On the other hand, storing the task instance in the first database is conducive to the automatic scheduled execution of tasks. Through database query, the execution status, progress results, etc. of the task instance can be monitored in real time, so as to automatically execute the target task according to the predetermined time and conditions, which can effectively improve the flexibility and reliability of task execution.

[0083] Figure 5 The flowchart of generating a task instance based on task attribute information and time information according to an embodiment of the present disclosure is schematically shown.

[0084] like Figure 5 As shown, the generation of a task instance based on task attribute information and time information in this embodiment includes operations S510 to S530.

[0085] In operation S510, structural processing is performed on the task attribute information to obtain task elements.

[0086] In operation S520, the time information is converted into a time series.

[0087] In operation S530 , a task instance is generated based on the task elements and the time sequence.

[0088] In some embodiments, the task attribute information includes source data information, data snapshot information, target database information, etc. The source data information includes the source information and location information of the target data, etc. The target database information includes the database type and database connection information, etc. The data snapshot information includes the data content, snapshot method, snapshot frequency, etc.

[0089] Specifically, the task attribute information is parsed and structured to obtain the task elements, and the task elements and time information are combined to generate a task instance for execution at a specified time.

[0090] Among them, if the time information contains multiple time points, a corresponding time sequence is generated based on the time information (the event sequence includes multiple time points for executing the task), and by combining the task elements with the time sequence, multiple task instances executed at different time points are obtained, and the generated task instances are stored in the first database for scheduling.

[0091] Figure 6 A flowchart schematically illustrates a process of periodically performing a polling operation on a task instance in a first database to obtain a target task instance according to an embodiment of the present disclosure.

[0092] like Figure 6 As shown, this embodiment periodically performs a polling operation on the task instances in the first database to obtain at least one target task instance, including operations S610 to S630.

[0093] In operation S610, each task instance in the first database is traversed at a preset time interval to obtain first information of the task instance.

[0094] In some embodiments, each task instance in the first database can be traversed regularly by the task scheduler to obtain the first information of each task instance. The first information includes the status information and execution time of the task instance. The time interval is customized according to user needs, for example, it can be 1 minute, 30 minutes, 1 hour, etc., corresponding to traversing the task instances in the first database every minute, traversing the task instances in the database every 30 minutes, and traversing the task instances every hour.

[0095] In operation S620, it is determined whether the first information of the task instance satisfies a preset condition.

[0096] In operation S630, when the first information satisfies a preset condition, the task instance corresponding to the first information is determined as a target task instance.

[0097] In some embodiments, the target task instance in the current time period is determined based on the acquired first information. Specifically, the preset conditions include: whether the task instance state is in a waiting state, and whether the execution time is in the current query time period, and the task instance whose task instance state is in a waiting state and whose execution time is in the current query time period is determined as the target task instance. When the first information meets the preset conditions, the task instance corresponding to the first information is determined as the target task instance in the current time period.

[0098] The current query time period is the time period between the current query time and the next query time. For example, if the preset time interval is 2 minutes, if the current query time is 18:00:20, the next query time should be 18:02:20, and the task instances with execution time between 18:00:20 and 18:02:20 are all in the current query time period.

[0099] The embodiment of the present disclosure periodically polls the task instances in the first database at preset time intervals, thereby ensuring that each task instance can be accurately checked at its scheduled execution time, thereby ensuring the accuracy and timeliness of task execution.

[0100] Figure 7 The flowchart of sending a target task instance to a task execution node based on a task thread according to an embodiment of the present disclosure is schematically shown.

[0101] like Figure 7 As shown, this embodiment sends the target task instance to the task execution node based on the task thread, including operations S710 to S730.

[0102] In operation S710, a request is sent to a task execution node; wherein the request is used to obtain status information of the task execution node.

[0103] In some embodiments, after the task thread is created, it enters a polling loop and regularly sends requests to the API interface of each task execution node to obtain the current status of these task execution nodes. The request information is generated based on the identification of each task execution node and the communication protocol, wherein the format of the request information is consistent with the communication protocol.

[0104] The request information should at least include API endpoints, query parameters, etc. The API endpoint is the path part of the API structure, which is used to specify the specific function to be called. The query parameters are used to specify the specific content to be queried.

[0105] In operation S720, state information of the task execution node is obtained based on the received response information; wherein the response information is generated by the task execution node based on the request.

[0106] In some embodiments, after receiving the request sent by the task thread, the task execution node will process the request and return a response message, wherein the response message includes information such as the current status of the task execution node, a list of processed tasks, a list of tasks to be processed, etc.

[0107] In operation S730 , when the status information of the task execution node is a non-busy state, the target task instance is sent to the task execution node.

[0108] In some embodiments, the current state of the task execution node is analyzed based on the received response information. If the state information of the task execution node is a non-busy state, the target task instance is sent to the task execution node, and the task execution node performs subsequent processing operations on the target task instance. If the state information of the current task execution node is a busy state, a request is sent to the next task execution node, and the above operations S710 to S730 are repeated until a task execution node in a non-busy state is found, or the query of all task execution nodes is completed.

[0109] In some embodiments, the task thread queries the API interface of the task execution node one by one to inquire whether the task execution node is in a busy state. If the task execution node is in a non-busy state, the target task instance is sent to the task execution node. If the task execution node is in a busy state, the API interface of the next task execution node is queried. If all task execution nodes are busy, the target task instance distribution fails, the task instance state in the first database is changed to a failed state, and the failure information of the task instance is stored in the first database. If the distribution is successful, the status information of the task instance in the first database is changed to an execution state.

[0110] In some embodiments, if the target task instance is distributed successfully, the task execution node is controlled to execute the target task instance. The task execution node may be a physical machine, a virtual machine, a container or other computing resources, and the number of the task execution nodes is greater than or equal to 1.

[0111] Compared with directly executing task instances based on threads, executing task instances based on task execution nodes can effectively improve the efficiency of task execution and avoid resource competition that may occur when threads execute tasks. Executing task instances based on task execution nodes can also achieve effective resource isolation, improve the privacy and security of task execution, and avoid mutual influence between tasks. In addition, by setting up multiple task execution nodes, tasks can be processed in parallel, thereby improving the execution efficiency of multiple tasks, making full use of multi-core resources, effectively reducing the waiting time of tasks, and improving resource utilization.

[0112] Figure 8 The flowchart schematically shows a control task execution node according to an embodiment of the present disclosure reading target data from a source database based on a received target task instance and creating a data snapshot of the target data, and inserting the data snapshot into a target database.

[0113] like Figure 8As shown, the control task execution node of this embodiment reads target data from the source database based on the received target task instance and creates a data snapshot of the target data, and inserts the data snapshot into the target database, including operations S810 to S840.

[0114] In operation S810, the target task instance is parsed to obtain task configuration information.

[0115] In some embodiments, the task configuration information includes source database information, target information, execution parameters, target database information, etc. Among them, the source database information may include the source database type, source database connection information (such as database name, IP address, port number, file path, etc.). The target information includes the attribute information of the target data (such as the index of the target data, the location information of the target data, the screening conditions of the target data, etc.). The execution parameters include the acquisition strategy of the target data, the snapshot method (such as interrupt mirroring, on-demand replication, virtual image, etc.), the data writing strategy, the snapshot retention strategy, etc. The target database information may include the target database type, the target database connection information (such as database name, IP address, port number, file path, etc.), the target table or storage structure information, etc.

[0116] In operation S820, a data read handle and a data write handle are generated based on the task configuration information.

[0117] In some embodiments, the data read handle is used to read the target information from the source database and create a data snapshot corresponding to the target information, wherein the data read handle includes the object identifier, location information, access rights, etc. of the target data. The data write handle is used to write the data snapshot to the target database, wherein the data write handle should include the target write location, write rights, etc. of the data snapshot.

[0118] In operation S830 , target data is read from the source database based on the data read handle and a data snapshot of the target data is generated.

[0119] In operation S840 , the data snapshot is written into the target database based on the data write handle.

[0120] In some embodiments, the target data is read from the source database through the data read handle, and a data snapshot operation is performed on the read target data to obtain a data snapshot of the target data. The data snapshot is then written to a specified location in the target database through the data write handle to complete the target task instance.

[0121] Specifically, the data snapshot may be written into an existing file in the target database, or a file may be created in the target database based on the data write handle, and the data snapshot may be written into a new file.

[0122] In the specific implementation process, if no abnormal situation occurs during the execution and the data snapshot is successfully written into the target database, the execution result of the target task instance is determined to be successful, and the status of the task instance is updated to a successful state in the first database. If an abnormal situation occurs during the execution, the execution of the target task instance is terminated, the status of the task instance is updated to a failed state in the first database, and the task thread of the task instance is closed.

[0123] In some embodiments, the data read handle may further include pre-execution information. The pre-execution information is used to perform data preparation and environment settings to ensure that correct target information is queried in an executable environment.

[0124] Before reading the target data from the source database based on the data read handle, the source database can also perform preprocessing operations on the data in the source database based on the pre-execution information to improve the correctness of the data snapshot. Among them, the preprocessing operation includes at least one of creating a table, filtering data, and updating the data format. Among them, filtering data includes: cleaning or updating the target data to be snapshotted to ensure the data quality and consistency of the snapshot, and / or filtering out unnecessary records from the database level, and only retaining relevant data within the specified range. Creating a table includes: generating a temporary table or view to facilitate quick access to preprocessed data during query. For example, data from different tables can be integrated into a temporary table, making subsequent data query operations and data snapshot operations simpler and more efficient. Updating the data format includes: converting the format of the target data in the source database based on the data format requirements of the target database, converting the data format in the source database into the format required by the target database, and improving data compatibility and the efficiency of data snapshots.

[0125] The present disclosure generates a data read handle and a data write handle based on a target task instance and completes the query, snapshot, and writing of the target data based on the handle. Compared with directly executing a task instance, the handle provides an abstract method that hides the specific implementation details of the underlying resources. By using the handle, the task implementation process can be effectively simplified and the complexity of resource management can be reduced. Executing tasks through the handle can effectively avoid directly operating memory addresses, which can effectively improve the safety of operations. In addition, the handle provides a unified interface to operate task instances, so that tasks can be executed in different environments and have cross-platform compatibility.

[0126] Based on the above data processing method, the present disclosure also provides a data processing device. Fig. 9 The device is described in detail.

[0127] Fig. 9 The structure block diagram of a data processing device according to an embodiment of the present disclosure is schematically shown.

[0128] like Fig. 9 As shown, the data processing device 900 of this embodiment includes a first generating module 910 , an acquiring module 920 , a second generating module 930 , a sending module 940 and a reading and inserting module 950 .

[0129] The first generation module 910 is used to generate a task instance in response to receiving a task processing request and send the task instance to the first database; wherein the task processing request includes task attribute information and execution time. In one embodiment, the first generation module 910 can be used to perform the operation S210 described above, which will not be repeated here.

[0130] The acquisition module 920 is used to periodically perform a polling operation on the task instances in the first database to obtain at least one target task instance. In one embodiment, the acquisition module 920 can be used to perform the operation S220 described above, which will not be described in detail here.

[0131] The second generation module 930 is used to generate corresponding task threads according to the target task instance; wherein the number of task threads is the same as the number of target task instances. In one embodiment, the second generation module 930 can be used to perform the operation S230 described above, which will not be repeated here.

[0132] The sending module 940 is used to send the target task instance to the task execution node based on the task thread, wherein the number of the task execution node is not less than 1. In one embodiment, the sending module 940 can be used to perform the operation S240 described above, which will not be described in detail here.

[0133] The read and insert module 950 is used to control the task execution node to read the target data from the source database based on the received target task instance and create a data snapshot of the target data, and insert the data snapshot into the target database. In one embodiment, the read and insert module 950 can be used to perform the operation S250 described above, which will not be repeated here.

[0134] According to an embodiment of the present disclosure, any multiple modules of the first generation module 910, the acquisition module 920, the second generation module 930, the sending module 940 and the read insertion module 950 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first generation module 910, the acquisition module 920, the second generation module 930, the sending module 940 and the read insertion module 950 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in a suitable combination of any of them. Alternatively, at least one of the first generating module 910 , the acquiring module 920 , the second generating module 930 , the sending module 940 and the reading and inserting module 950 may be at least partially implemented as a computer program module, which may perform a corresponding function when executed.

[0135] Fig.10 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure is schematically shown.

[0136] like Fig.10 As shown, the electronic device 1000 according to an embodiment of the present disclosure includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 to a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include an onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0137] In RAM 1003, various programs and data required for the operation of electronic device 1000 are stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Processor 1001 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1002 and / or RAM 1003. It should be noted that the program can also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0138] According to an embodiment of the present disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The electronic device 1000 may further include one or more of the following components connected to the input / output (I / O) interface 1005: an input portion 1006 including a keyboard, a mouse, etc.; an output portion 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1008 including a hard disk, etc.; and a communication portion 1009 including a network interface card such as a LAN card, a modem, etc. The communication portion 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output (I / O) interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage portion 1008 as needed.

[0139] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0140] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1002 and / or RAM 1003 described above and / or one or more memories other than ROM 1002 and RAM 1003.

[0141] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data processing method provided by the embodiment of the present disclosure.

[0142] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1001. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0143] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1009, and / or installed from the removable medium 1011. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0144] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0145] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0146] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0147] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0148] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data processing method, characterized in that: The method comprises: In response to receiving a task processing request, generating a task instance and sending the task instance to a first database; wherein the task processing request includes task attribute information and execution time; Periodically performing a polling operation on the task instances in the first database to obtain at least one target task instance; Generate a corresponding task thread according to the target task instance; wherein the number of the task threads is the same as the number of the target task instances; Sending the target task instance to a task execution node based on the task thread; wherein the number of the task execution nodes is not less than 1; The control task execution node reads the target data from the source database based on the received target task instance and creates a data snapshot of the target data, and inserts the data snapshot into the target database.

2. The data processing method according to claim 1, characterized in that: The method further comprises: Obtain metadata information from the source database; Building a visualization interface based on the metadata information, wherein the visualization interface is set on a front-end device and includes a plurality of event triggers for reading and responding to user interaction actions; A task processing request is generated based on the user's interactive action, wherein the interactive action includes setting a click button, selecting a drop-down box, inputting text, etc.

3. The data processing method according to claim 1, characterized in that: In response to receiving the task processing request, generating a task instance and sending the task instance to the first database includes: Parsing the task processing request to obtain task attribute information and time information; Generate a task instance based on the task attribute information and the time information; The task instance is sent to a designated location in the first database for storage.

4. The data processing method according to claim 3, characterized in that: The generating a task instance based on the task attribute information and the time information includes: Performing structured processing on the task attribute information to obtain task elements; Converting the time information into a time series; A task instance is generated based on the task elements and the time sequence.

5. The data processing method according to claim 1, characterized in that: The periodically performing a polling operation on the task instances in the first database to obtain at least one target task instance includes: Traversing each task instance in the first database at a preset time interval to obtain first information of the task instance; wherein the first information includes time information and state information of the task instance; Determining whether the first information of the task instance meets a preset condition; When the first information meets a preset condition, the task instance corresponding to the first information is determined as a target task instance.

6. The data processing method according to claim 1, characterized in that: The step of sending the target task instance to the task execution node based on the task thread includes: Sending a request to the task execution node; wherein the request is used to obtain status information of the task execution node; Obtaining the status information of the task execution node based on the received response information; wherein the response information is generated by the task execution node based on the request; When the status information of the task execution node is a non-busy state, the target task instance is sent to the task execution node.

7. The data processing method according to claim 6, characterized in that: The control task execution node reads target data from a source database based on the received target task instance and creates a data snapshot of the target data, and inserts the data snapshot into the target database, including: Parsing the target task instance to obtain task configuration information; Generate a data read handle and a data write handle based on the task configuration information; Reading target data from the source database based on the data read handle and generating a data snapshot of the target data; The data snapshot is written into a target database based on the data write handle.

8. The data processing method according to claim 7, characterized in that: The step of writing the data snapshot into the target database based on the data write handle includes: Writing the data snapshot to an existing file in the target database; or A new file is created in the target database and the data snapshot is written into the new file.

9. The method according to claim 7, characterized in that: The step of reading the target data from the source database based on the data read handle and generating a data snapshot of the target data further includes: Before reading the target data from the source database, performing a preprocessing operation on the data in the source database; The preprocessing operation includes at least one of creating a table, filtering data, and updating data format.

10. The data processing method according to claim 1, characterized in that: The method further comprises: Based on the execution result of the task thread, closing the task thread and updating the state of the target task instance in the first database; and In response to generating a task thread corresponding to the target task instance, the state of the target task instance in the first database is changed from a waiting state to an executing state.

11. A data processing device, characterized in that: The device comprises: A first generating module, configured to generate a task instance in response to receiving a task processing request and send the task instance to a first database; wherein the task processing request includes task attribute information and execution time; An acquisition module, configured to periodically perform a polling operation on the task instances in the first database to acquire at least one target task instance; A second generating module, configured to generate corresponding task threads according to the target task instance; wherein the number of the task threads is the same as the number of the target task instances; A sending module, used to send the target task instance to a task execution node based on the task thread; wherein the number of the task execution node is not less than 1; The read-insert module is used to control the task execution node to read the target data from the source database based on the received target task instance and create a data snapshot of the target data, and insert the data snapshot into the target database.

12. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images