Data processing system and method

Abstract

The application provides a data processing system, comprising: a Flink execution engine configured to execute a task request; a Key-Value storage device configured to store data; the Flink execution engine is configured to obtain first data from the Key-Value storage device by calling a remote API, execute the task request according to the obtained first data, and store second data generated in the Key-Value storage device. The application also provides a data processing method, a data processing device, a computer device and a computer readable storage medium. The application adopts the above technical scheme, realizes KeyedStateBackend through a remote Key-Value storage device, accesses and calls data in the Key-Value storage device in a network interaction mode, can meet read and write under a super large scale state, facilitates realization of various mixed resources in a native architecture, does not need data migration, and improves applicability and data processing efficiency of the data processing system.

Description

Technical Field

[0001] This application relates to the fields of computer technology and big data technology, and in particular to a data processing system, method, apparatus, computer equipment and computer-readable storage medium. Background Technology

[0002] With the development of computer and big data technologies, data storage and utilization have become key technological focuses. In many application scenarios, the more real-time the data, the more valuable it is. For example, near real-time data is needed for data analysis in scenarios such as live streaming and recommendation systems.

[0003] Flink (a streaming data processing system) is a distributed processing engine and framework for performing state computation on bounded and unbounded data streams. Its KeyedStateBackend is used to carry the state workload. Traditional KeyedStateBackend implementations use either memory-based StateBackend (based on in-memory storage) or RocksDBStateBackend (based on a high-performance embedded key-value database). However, memory-based StateBackend requires maintaining all state in memory, placing high demands on memory space; RocksDBStateBackend suffers from several limitations when dealing with large-scale states / tasks, requiring data migration when adjusting the state distribution of tasks, which takes considerable time and impacts processing efficiency. Summary of the Invention

[0004] This application provides a data processing system and method to solve the problems existing in related technologies. The technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide a data processing system, including:

[0006] The Flink execution engine is used to execute task requests;

[0007] Key-value storage devices are used to store data;

[0008] The Flink execution engine retrieves first data from the key-value storage device by calling a remote API, executes task requests based on the retrieved first data, and stores the resulting second data in the key-value storage device.

[0009] Secondly, embodiments of this application provide a data processing method, including:

[0010] Receive task requests;

[0011] Call the remote API to retrieve the first data from the remote key-value storage device, execute the task request based on the retrieved first data, and store the generated second data in the key-value storage device.

[0012] Thirdly, embodiments of this application provide a data processing apparatus, including:

[0013] The task receiving module is used to receive task requests;

[0014] The task execution module is used to call the remote API to retrieve the first data from the remote key-value storage device, execute the task request based on the retrieved first data, and store the generated second data in the key-value storage device.

[0015] Fourthly, embodiments of this application provide an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the aforementioned data processing method.

[0016] Fifthly, embodiments of this application provide a computer-readable storage medium that stores computer instructions, wherein when the computer instructions are executed on a computer, the methods in any of the above-described embodiments are performed.

[0017] The embodiments of this application adopt the above technical solution, which realizes KeyedStateBackend through a remote Key-Value storage device, and accesses and retrieves data in the Key-Value storage device through network interaction. This can meet the needs of ultra-large-scale state reading and writing, facilitate the implementation of various mixed resources in the native architecture, eliminate the need for data migration, improve the applicability of the data processing system, and increase data processing efficiency.

[0018] The above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of this application will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0019] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0020] Figure 1 A schematic diagram illustrating an environment according to an embodiment of this application is shown.

[0021] Figure 2 This illustration shows a schematic diagram of the architecture of a data processing system according to Embodiment 1 of this application;

[0022] Figure 3 The diagram illustrates the StateBackend hierarchy architecture in the Flink execution engine according to Embodiment 1 of this application.

[0023] Figure 4 A flowchart illustrating the data processing method according to Embodiment 2 of this application is shown schematically.

[0024] Figure 5 A block diagram of a data processing apparatus according to Embodiment 3 of this application is schematically shown; and

[0025] Figure 6 The illustration shows a schematic diagram of the hardware architecture of a computer device suitable for implementing a data processing method according to Embodiment 4 of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the present application and are not intended to limit the present application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0027] It should be noted that the descriptions involving "first," "second," etc., in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0028] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order of the steps, but are only used to facilitate the description of this application and to distinguish each step, and therefore should not be construed as a limitation of this application.

[0029] The following are explanations of some terms used in this application:

[0030] Flink is a distributed real-time computing system used for stateful computation on unbounded and bounded data streams. Flink is designed to run in all common cluster environments, performing computations at memory speeds and any scale. Here, "state" generally refers to the state of a specific task / operator. Stream processing is used to process (e.g., query) streaming data. Streaming data is typically online and has no boundaries.

[0031] Key-Value storage devices refer to a type of NoSQL database system that stores and retrieves data through key / value pairs. It is a distributed storage system with fast query speed, large data storage capacity, and support for high concurrency, making it very suitable for querying by primary key.

[0032] The remote API (Application Programming Interface) allows communication between CoppeliaSim and external applications (i.e., applications running on different processes or machines). It is cross-platform and supports service calls (i.e., blocking calls) and bidirectional data streams.

[0033] HDFS (Hadoop Distributed File System) is the Hadoop Distributed File System.

[0034] To facilitate understanding of the technical solutions provided in the embodiments of this application by those skilled in the art, the relevant technologies are described below:

[0035] In the Flink architecture, the StateBackend is used to store task states. It is divided into two types based on its purpose: OperatorStateBackend and KeyedStateBackend. OperatorStateBackend typically stores data unrelated to the computational logic itself, such as Kafka offsets (sequence numbers). Its size is relatively small and unaffected by data scale or computational logic. KeyedStateBackend, on the other hand, stores state data strongly tied to computational logic, such as intermediate results from aggregation functions / two-stream joins. Its size is influenced by data scale and computational logic; generally, for high-volume, computationally complex tasks, the state can be very large.

[0036] Because OperatorStateBackend is relatively lightweight and mainly uses a list-type data structure, the community only has one in-memory implementation. The data is written to memory and then written to HDFS all at once during checkpointing.

[0037] KeyedStateBackend carries the main state workload and has extremely high requirements for both capacity and performance. Data is stored in a key-value pattern.

[0038] Traditional KeyedStateBackend has two implementation methods:

[0039] 1. MemoryStateBackend: All data is stored in memory. It generally only supports jobs with small states, and the current Flink architecture does not support incremental checkpointing, thus limiting its applicability. Because it maintains the entire state in memory, all data in memory can only be saved at once during a check.

[0040] 2. RocksdbStateBackend manages state based on local SSD (Solid State Drive) disks, enabling large-scale local state read / write operations. However, its drawbacks include CPU (Central Processing Unit) overhead, such as serialization and compression amplification during read / write operations; and it is not well-suited for cloud-native architectures, making it difficult to benefit from mixed-distribution resources. During resource scaling, the underlying KeyGroups are redistributed, data migration is time-consuming, and recovery is slow with large states.

[0041] This application provides an implementation of a Remote statebackend.

[0042] The following provides an exemplary application environment for embodiments of this application. Figure 1 The illustration shows an environmental application diagram according to an embodiment of this application.

[0043] Service platforms (2, 3) can provide services such as data processing, and can be a single server, a server cluster, or a cloud computing service center. Among them, service platforms 2 / 3 have a built-in data processing system 10, which is used to interact with service platforms 3 / 2 and provide real-time or offline processing.

[0044] It should be noted that, Figure 1 The number of servers shown is merely illustrative and is not intended to limit the scope of patent protection in this application.

[0045] The following will provide several embodiments, using the aforementioned service platform 2 as the execution entity, to illustrate the data processing scheme.

[0046] Example 1

[0047] Figure 2An architectural diagram of a data processing system according to Embodiment 1 of this application is illustrated schematically. Figure 2 As shown, the data processing system may include a Flink execution engine 200 and a key-value storage device 300.

[0048] Flink Execution Engine 200 is used to execute task requests. Flink Execution Engine 200 includes features of the native Flink architecture, such as task scheduling, parallel execution of multiple tasks, checkpointing, etc.

[0049] Key-Value storage device 300 is used to store data (such as key-value pairs). It can be a key-value database system that stores and retrieves data according to a key-value storage model. Interconnection with the Flink execution engine 200 can be achieved through a matching SDK (Software Development Kit), allowing the Flink execution engine 200 to access and retrieve data from the key-value storage device 300.

[0050] In this embodiment, the Flink execution engine 200 obtains first data from the Key-Value storage device 300 by calling a remote API, executes a task request based on the obtained first data, and stores the generated second data in the Key-Value storage device 300.

[0051] In this embodiment, the generated second data can be relative to the first data in the Key-Value storage device before the task request is executed, used to distinguish between the two types of data; the data in the Key-Value storage device before the task request is executed is the first data, and the data newly generated based on the task request is the second data. It can be understood that the data stored in the Key-Value storage device includes both the first data and the second data.

[0052] like Figure 2 As shown, the Flink execution engine 200 includes: a partitioning module, used to parse task requests, determine the task execution method based on the parsing results, and divide the task request into multiple subtasks based on the task execution method; each subtask is divided into multiple states; and the multiple states are sliced ​​to generate multiple key groups. Figure 2 (kg1, kg2...kg8).

[0053] Determining the task execution method can involve deciding how many subtasks to divide the task request into, with these subtasks executing completely independently in different threads, physical machines, or containers. It can also involve determining which subtasks will execute in parallel, the execution order of the subtasks, or their execution time points. A subtask can be further divided into multiple states, which can be configured through task configuration; alternatively, a state can be divided into multiple key groups. The subtask is completed when all its key groups are executed sequentially.

[0054] Key-Value storage device 300 includes multiple shards, Figure 2 In the key-value storage device 300 (sh1, sh2...sh8), one shard group corresponds to one key group setting. Specifically, the key-value storage device 300 itself has the concepts of tables and shards. A shard is an indivisible slice of data, that is, the smallest granularity. By matching shards and keygroups one-to-one, i.e., by ensuring that the number of both is consistent, the complex correspondence between data in keygroups and shards can be simplified. When Flink sends changes to the correspondence between keygroups and tasks, it will not send multiple subtasks to operate on the same shard. The concept of tables in the key-value storage device 300 can be mapped to Flink tasks. When a task includes multiple states, they can be identified by adding a key (key-value) prefix to the state.

[0055] In this embodiment, by mapping the pattern partitioning of the Flink execution engine 200 when executing tasks to the table concept and shard concept in the Key-Value storage device 300, data retrieval and reading can be facilitated, avoiding complex calculations during data retrieval and reading, and further improving data processing efficiency.

[0056] The Flink execution engine 200 also includes a recording module for recording the mapping between shard groups and key groups. The recording module can be executed while the key group is being executed. The recording module can also record handles in the data stream to facilitate the retrieval of first and / or second data from the shard.

[0057] The data processing system of this application embodiment, when scaling up or down resources, such as reducing three subtasks to two tasks, can only find the corresponding shard for data processing when executing keyGroup, without the need for data migration operations. Therefore, it can meet the requirement of arbitrarily scaling up or down resources under large-scale state tasks.

[0058] In the data processing system of this application embodiment, the Flink execution engine 200 and the key-value storage device 300 access and call data by sending network requests. There is no need to download or load files, and it does not need to be matched with the native architecture. It can call various hybrid resources in the native architecture, and has wide applicability.

[0059] Flink Execution Engine 200 also includes the following modules to meet various functionalities when executing task requests.

[0060] The inspection module records secondary data and triggers corresponding operations based on the execution status. This could involve recording secondary data during the execution of a key group, triggering a snapshot when a task request is executed, or triggering a restore operation when a subtask needs to be restarted.

[0061] The snapshot module is used to send snapshot requests to the Key-Value storage device 300 so that the Key-Value storage device 300 can perform snapshot operations to obtain snapshot information, which facilitates task recovery in case of user errors.

[0062] The recovery module sends a recovery request to the Key-Value storage device 300 to retrieve snapshot information at the corresponding time point in the event of an exception during task execution. By obtaining the processing progress at the time point prior to the exception, execution can be resumed from that point, improving data processing efficiency.

[0063] The cleanup module is used to send a data deletion request to the Key-Value storage device 300 so that the Key-Value storage device 300 deletes the second data requested to be deleted in the data deletion request.

[0064] In one example, the Flink execution engine can be configured to initiate a task query at preset time intervals (e.g., every hour) to checkpoint information and identify secondary data that will no longer be used. This secondary data, no longer in use, is identified as data to be deleted. Using the table information of the secondary data, the Key-Value storage device 300 is requested to delete the secondary data under the corresponding table. Alternatively, the secondary data can be deleted using the key group information of the secondary data, requesting the Key-Value storage device 300 to delete the secondary data under the corresponding shard group of that key group.

[0065] In one example, the Flink execution engine determines a state where execution no longer needs to continue based on query checkpoint information, and then sends a request to the Key-Value storage device 300 to delete the second data in the corresponding shards of multiple key groups in that state.

[0066] By cleaning and deleting data in key-value storage devices, space can be occupied and resources can be wasted.

[0067] like Figure 3 As shown in this embodiment, the StateBackend of the Flink execution engine 200 may include the following levels:

[0068] StreamOperatorStateHandler (Stream Operator State Handling) Used to operate on StateBackend, it is divided into three parts: Init (initialize StateBackend); Checkpoint (perform snapshot persistence); and Savepoint (perform time node persistence).

[0069] StateBackend: Divided into three parts:

[0070] Cache StateBackend: This extension introduces a Cache Layer to address access to frequently accessed data, while also reducing serialization overhead by caching the data to memory.

[0071] lockDISK ON SSD: Lock the disk.

[0072] Remote StateBackend: Replaces the KeyedState originally stored on Rocksdb with Key-Value storage device 300, enabling large-scale state data processing.

[0073] State: The specific State storage type. Different types of State will be applied to operators in different scenarios of SQL (Structured Query Language).

[0074] CheckpointAPI (Checkpoint Interface): Used to implement checking and recovery.

[0075] Example 2

[0076] Figure 4 A flowchart illustrating a data processing method according to an embodiment of this application is shown. Figure 4 As shown, the data processing method may include:

[0077] S410 receives task requests.

[0078] The Flink execution engine receives task requests to perform stateful computations on unbounded and bounded data streams.

[0079] S420, call the remote API to obtain the first data from the remote Key-Value storage device, execute the task request based on the obtained first data, and store the generated second data in the Key-Value storage device.

[0080] The Flink execution engine calls modules in the Flink architecture to execute the requested task content, including retrieving the first data from a remote key-value storage device and storing the generated second data in the key-value storage device, in order to complete the implementation of the functions of each module.

[0081] In one implementation, before calling the remote API to retrieve the first data from a remote key-value storage device, the process includes: parsing the task request and determining the task execution method;

[0082] Assign task states according to the task execution method; wherein, the assignment of task states includes: dividing a task request into multiple subtasks; wherein each subtask is divided into multiple states; slicing the multiple states to generate multiple key groups.

[0083] After parsing the task request, the task execution method can be configured based on real-time resource conditions; alternatively, the default task execution method can be used. Determining the task execution method includes deciding how many subtasks to divide the task into, and how these subtasks execute completely independently in different threads, physical machines, or containers. The task execution method can also determine which subtasks will execute in parallel, the execution order of the subtasks, or their execution time points. A subtask can contain multiple states; task configuration can be used to divide a subtask into multiple states, or vice versa.

[0084] In one example, each state is typically divided into 1024 key groups.

[0085] In one implementation, executing the task request based on the acquired first data includes:

[0086] The task execution method is changed based on the CPU resource usage information.

[0087] Based on the transformed task execution method, the task allocation state is transformed, wherein the transformation of the task allocation state includes: transforming the number of subtasks, transforming the subtasks corresponding to the state, and / or transforming the state corresponding to the key group. For example, when resources are sufficient, a task request that was originally divided into 3 subtasks can be reduced to 2 tasks.

[0088] When the number of subtasks changes, the first data is obtained through the corresponding shard group of the key group. For example, in the original architecture... Figure 3 In the architecture shown, reducing the original three subtasks to two, it's possible that kg4 corresponds to state1 and kg7 to state2. Therefore, when executing kg4, data only needs to be retrieved from the key-value storage device sh4, without needing to migrate kg4's data to state1, reducing data migration operations and improving data processing efficiency. Similarly, when executing kg7, data only needs to be retrieved from the key-value storage device sh7, without needing to migrate kg7's data to state2.

[0089] In one implementation, the step of calling a remote API to obtain first data from a remote key-value storage device, executing the task request based on the obtained first data, and storing the generated second data in the key-value storage device includes: sending a data retrieval request to the remote key-value storage device based on the key group, and retrieving the first data from the corresponding shard group of the key group.

[0090] The key group is executed based on the acquired first data, and the resulting second data is stored in the corresponding slice group of the key group.

[0091] After configuring and partitioning the task, multiple key groups are executed sequentially to complete the task request. Each key group corresponds to a slice in the key-value storage device. Therefore, when executing a key group, data can be retrieved and read from the corresponding slice, improving execution efficiency.

[0092] In one implementation, the method further includes:

[0093] Record the second data of the executed key group; map the second data to the subtask in which the key group is located;

[0094] A snapshot request is initiated through the subtask, so that multiple corresponding slice groups of multiple key groups in the subtask perform snapshot operations and return the snapshot address to the subtask.

[0095] This execution process can be performed by the checkpoint module. This stage processes a single record, routing the corresponding process data of the keygroup to the appropriate subtask based on the task partition path (keygroup-state-subtask). If a subtask involves state operations, a read / write request will be initiated at the state operation layer to the shard of the key-value storage device mapped to the corresponding keygroup to complete the data read / write. The key-value storage device can return a list of multiple snapshot addresses to send them in parallel, improving transmission efficiency.

[0096] In this embodiment of the application, a snapshot operation is performed through a key-value storage device to record the read / write requests at the current time point. This is to facilitate obtaining the read / write requests at the time point before the exception in case of execution failure or restarting a subtask, so as to continue executing the task from that time point.

[0097] In one embodiment, the data processing method further includes:

[0098] The subtask sends the received list of snapshot addresses to the inspection module.

[0099] The inspection module stores the second data of the snapshot address into HDFS.

[0100] Subtasks submit a list of snapshot addresses as handles to the inspection module. The inspection module collects all snapshots from subtasks and persists this secondary data to HDFS. A snapshot contains information such as the event executed at that point in time, the data requested to execute that event, and the secondary data needed or generated to execute that event. Persistent storage of snapshots is achieved by storing the secondary data at the snapshot addresses in HDFS, allowing for easy retrieval of the secondary data from HDFS upon task restart.

[0101] When the inspection module completes the Snapshot phase, it will simultaneously send a Snapshot ACK (ACKknowledgeCharacter) request to the Key-Value storage device to indicate that the Snapshot is complete. Upon receiving the ACK request, the Key-Value storage device will retain the corresponding Snapshot. If the retention time exceeds a preset time threshold or the retention fails, the corresponding Snapshot will be deleted.

[0102] In one embodiment, the data processing method further includes:

[0103] In the event of a task restart request, the second data is read from the HDFS;

[0104] Based on the second data, the task allocation status is obtained.

[0105] Since the second data information corresponding to the CheckpointId (snapshot address) is stored on HDFS, the restore process is actually the process of reading back the second data from HDFS and redistributing the keygroup to the tasks. The information of the key-value storage device is that the shard is bound to the keygroup, so the allocation of shards to subtasks can follow the principle of one-to-one correspondence.

[0106] In one embodiment, the data processing method further includes:

[0107] In the case of restarting a subtask, a snapshot address and a recovery request are sent to the corresponding slice group of multiple key groups in the subtask. The recovery request includes a recovery time node so that the corresponding slice group can switch to the snapshot of the recovery time node.

[0108] Each subtask is assigned a corresponding keyGroup, i.e., a list of shards, and the snapshot address corresponding to that list. When a subtask is started, a time-node recovery request is sent to the key-value storage device for the task recovery logic. The corresponding shards of multiple key groups in the subtask are switched to the snapshot of the corresponding time node and returned to the subtask so that the subtask or key group can continue to execute the task from the recovery time node.

[0109] In one embodiment, the data processing method further includes:

[0110] Based on a preset time interval, the recorded second data is traversed, and the second data to be deleted is determined from it;

[0111] Based on the second data to be deleted, obtain the corresponding slice group of the second data to be deleted;

[0112] A deletion request is sent to the Key-Value storage device to cause the Key-Value storage device to delete the second data of the corresponding slice group.

[0113] In one example, the Flink execution engine can be configured to initiate a task query at preset time intervals (e.g., every hour) to checkpoint information and identify secondary data that will no longer be used. This secondary data, no longer in use, is identified as data to be deleted. Using the table information of the secondary data, the Key-Value storage device 300 is requested to delete the secondary data under the corresponding table. Alternatively, the secondary data can be deleted using the key group information of the secondary data, requesting the Key-Value storage device 300 to delete the secondary data under the corresponding shard group of that key group.

[0114] In one example, the Flink execution engine determines a state where execution no longer needs to continue based on query checkpoint information, and then sends a request to the Key-Value storage device 300 to delete the second data in the corresponding shards of multiple key groups in that state.

[0115] This application embodiment cleans and deletes data in the key-value storage device by periodically querying and checking information, so as to avoid occupying space and wasting resources.

[0116] Example 3

[0117] Figure 5 A structural block diagram of a data processing apparatus according to an embodiment of this application is shown. Figure 5 As shown, the device may include a task receiving module 510 and a task execution module 520.

[0118] The task receiving module 510 is used to receive task requests;

[0119] The task execution module 520 is used to call a remote API to obtain first data from a remote key-value storage device, execute the task request based on the obtained first data, and store the generated second data in the key-value storage device.

[0120] In one implementation, the task execution module 520 is configured to: parse the task request and determine the task execution method before calling the remote API to retrieve the first data from the remote key-value storage device;

[0121] Assign task states according to the task execution method; wherein, the assignment of task states includes: dividing a task request into multiple subtasks; wherein each subtask is divided into multiple states; slicing the multiple states to generate multiple key groups.

[0122] In one implementation, the task execution module 520 is used for:

[0123] The task execution method is changed according to the CPU resource usage information;

[0124] Based on the transformed task execution mode, the task allocation state is transformed, wherein the transformation of the task allocation state includes: transforming the number of subtasks, transforming the subtasks corresponding to the state, and / or transforming the state corresponding to the key group.

[0125] In one implementation, the task execution module 520 can be used to:

[0126] Based on the key group, a data retrieval request is sent to a remote key-value storage device to retrieve the first data from the corresponding chip group of the key group;

[0127] The key group is executed based on the acquired first data, and the resulting second data is stored in the corresponding slice group of the key group.

[0128] In one implementation, the task execution module 520 is used for:

[0129] Record the second data of the executed key group; map the second data to the subtask in which the key group is located;

[0130] A snapshot request is initiated through the subtask, so that multiple corresponding slice groups of multiple key groups in the subtask perform snapshot operations and return the snapshot address to the subtask.

[0131] In one implementation, the task execution module 520 is used for:

[0132] The subtask sends the received list of snapshot addresses to the inspection module.

[0133] The inspection module stores the second data of the snapshot address into HDFS.

[0134] In one implementation, the task execution module 520 is used for:

[0135] In the event of a task restart request, the second data is read from the HDFS;

[0136] Based on the second data, the task allocation status is obtained.

[0137] In one implementation, the task execution module 520 is used for:

[0138] In the case of restarting a subtask, a snapshot address and a recovery request are sent to the corresponding slice group of multiple key groups in the subtask. The recovery request includes a recovery time node so that the corresponding slice group can switch to the snapshot of the recovery time node.

[0139] In one implementation, the task execution module 520 is used for:

[0140] Based on a preset time interval, the recorded second data is traversed, and the second data to be deleted is determined from it;

[0141] Based on the second data to be deleted, obtain the corresponding slice group of the second data to be deleted;

[0142] A deletion request is sent to the Key-Value storage device to cause the Key-Value storage device to delete the second data of the corresponding slice group.

[0143] The functions of each module in the devices of this application embodiment can be found in the corresponding description in the above data processing method, and will not be repeated here.

[0144] Example 4

[0145] Figure 6 This diagram schematically illustrates a hardware architecture of a computer device suitable for implementing a data processing method according to Embodiment 4 of this application. The computer device 10000 can be a Flink-based service platform 2 or a part thereof. The computer device 10000 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. For example, it can be a rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers), etc. Figure 6 As shown, the computer device 10000 includes, but is not limited to, a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other via a system bus.

[0146] in:

[0147] The memory 10010 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of the computer device 10000, such as the hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 10000. Of course, the memory 10010 may also include both the internal storage module and the external storage device of the computer device 10000. In this embodiment, the memory 10010 is typically used to store the operating system and various application software installed on the computer device 10000, such as program code for data processing methods. Furthermore, the memory 10010 can also be used to temporarily store various types of data that have already been output or will be output.

[0148] In some embodiments, processor 10020 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data interaction or communication with computer device 10000. In this embodiment, processor 10020 is used to run program code stored in memory 10010 or process data.

[0149] Network interface 10030 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and communication connection between computer device 10000 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0150] It should be pointed out that, Figure 6 Only computer devices with components 10010-10030 are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0151] In this embodiment, the data processing method stored in memory 10010 can be further divided into one or more program modules and executed by one or more processors (processor 10020 in this embodiment) to complete this application.

[0152] Example 5

[0153] This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the data processing method in this embodiment.

[0154] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium may include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the data processing method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0155] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.

[0156] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A data processing system, characterized in that, include: The Flink execution engine is used to execute task requests; Key-value storage devices are used to store data; The Flink execution engine retrieves first data from a remote key-value storage device by calling a remote API, executes the task request based on the retrieved first data, and stores the generated second data in the key-value storage device. The Flink execution engine includes: The partitioning module is used to: parse the task request, determine the task execution method based on the parsing result, and divide the task request into multiple subtasks based on the task execution method; wherein each subtask is divided into multiple states; and slice the multiple states to generate multiple key groups; The step of executing the task request based on the acquired first data includes: The task execution method is changed according to the CPU resource usage information; Based on the transformed task execution mode, the task allocation state is transformed, wherein the transformation of the task allocation state includes: transforming the number of subtasks, transforming the subtasks corresponding to the state, and / or transforming the state corresponding to the key group.

2. The system according to claim 1, characterized in that, The key-value storage device includes multiple chip groups, and each chip group corresponds to one key group.

3. The system according to claim 2, characterized in that, The key-value storage device includes: A recording module is used to record the correspondence between the chip group and the key group.

4. The system according to claim 2, characterized in that, The Flink execution engine includes: The inspection module is used to record the second data and trigger corresponding operations based on the execution status; The snapshot module is used to send a snapshot request to the Key-Value storage device so that the Key-Value storage device performs a snapshot operation; The recovery module is used to send a recovery request to the Key-Value storage device to obtain snapshot information at the corresponding time point in the event of an abnormality during task execution; and / or, The cleanup module is used to send a data deletion request to the Key-Value storage device so that the Key-Value storage device deletes the second data requested to be deleted in the data deletion request.

5. A data processing method, characterized in that, include: Receive task requests; Call a remote API to retrieve first data from a remote Key-Value storage device, execute the task request based on the retrieved first data, and store the generated second data into the Key-Value storage device; Before calling the remote API to retrieve the first data from the remote key-value storage device, the following steps are included: The task request is parsed to determine the task execution method; Assign task states according to the task execution method; wherein, the assignment of task states includes: dividing a task request into multiple subtasks; wherein each subtask is divided into multiple states; slicing the multiple states to generate multiple key groups; The step of executing the task request based on the acquired first data includes: The task execution method is changed according to the CPU resource usage information; Based on the transformed task execution mode, the task allocation state is transformed, wherein the transformation of the task allocation state includes: transforming the number of subtasks, transforming the subtasks corresponding to the state, and / or transforming the state corresponding to the key group.

6. The method according to claim 5, characterized in that, The process of calling a remote API to retrieve first data from a remote key-value storage device, executing the task request based on the retrieved first data, and storing the resulting second data in the key-value storage device includes: Based on the key group, a data retrieval request is sent to a remote key-value storage device to retrieve the first data from the corresponding shard group of the key group; The key group is executed based on the acquired first data, and the resulting second data is stored in the corresponding slice group of the key group.

7. The method according to claim 6, characterized in that, Also includes: Record the second data of the executed key group; Map the second data to the subtask in which the key group is located; A snapshot request is initiated through the subtask, so that multiple corresponding slice groups of multiple key groups in the subtask perform snapshot operations and return the snapshot address to the subtask.

8. The method according to claim 7, characterized in that, Also includes: The subtask sends the received list of snapshot addresses to the inspection module. The inspection module stores the second data of the snapshot address into HDFS.

9. The method according to claim 8, characterized in that, Also includes: In the event of a task restart request, the second data is read from the HDFS; Based on the second data, the task allocation status is obtained.

10. The method according to claim 8, characterized in that, Also includes: In the case of restarting a subtask, a snapshot address and a recovery request are sent to the corresponding slice group of multiple key groups in the subtask. The recovery request includes a recovery time node so that the corresponding slice group can switch to the snapshot of the recovery time node.

11. The method according to claim 7, characterized in that, Also includes: Based on a preset time interval, the recorded second data is traversed, and the second data to be deleted is determined from it; Based on the second data to be deleted, obtain the corresponding slice group of the second data to be deleted; A deletion request is sent to the Key-Value storage device to cause the Key-Value storage device to delete the second data of the corresponding slice group.

12. A data processing apparatus, characterized in that, include: The task receiving module is used to receive task requests; The task execution module is used to call a remote API to obtain first data from a remote key-value storage device, execute the task request based on the obtained first data, and store the generated second data into the key-value storage device. The task execution module is used for: Before calling the remote API to retrieve the first data from the remote key-value storage device, the task request is parsed to determine the task execution method; Assign task states according to the task execution method; wherein, the assignment of task states includes: dividing a task request into multiple subtasks; wherein each subtask is divided into multiple states; slicing the multiple states to generate multiple key groups; The task execution method is changed according to the CPU resource usage information; Based on the transformed task execution mode, the task allocation state is transformed, wherein the transformation of the task allocation state includes: transforming the number of subtasks, transforming the subtasks corresponding to the state, and / or transforming the state corresponding to the key group.

13. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 5-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 5-11.

15. A computer program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the method according to any one of claims 5-11.

Images