A data processing method and system

Abstract

Embodiments of the present application provide a data processing method and system. The method comprises: in response to a running request of a workflow, obtaining data kinship of the workflow; running the workflow based on the data kinship of the workflow; obtaining output data of a current operator in the workflow; based on the data kinship, if it is determined that the output data is a final result of the workflow, outputting the output data; if it is determined that the output data is an intermediate result of the workflow, storing the output data in a memory. When the workflow of the embodiments of the present application runs in the memory, the data of the workflow does not go out of the memory, thereby reducing the number of data read and write times, reducing I / O resources, and improving data processing speed.

Description

Technical Field

[0001] This invention relates to big data analytics, and more particularly to a data processing method and system. Background Technology

[0002] With the rapid development of artificial intelligence and big data, using machine learning techniques for model training and then using the trained business models to achieve intelligent processing of big data business has gradually become a common practice in the big data industry. Model training typically involves three main stages: data processing, model building, and model validation. Each stage includes one or more operators, each relatively independent and processing the input data. The input data of the current operator may be the output data of the previous operator, and the output data of the current operator may be the input data of the next operator. Multiple operators are chained together using their input and output data to form a workflow schedule. In existing workflow scheduling models, each operator needs to read input data from the disk via I / O interface before running and write the generated data to the disk file system via I / O interface after running. Since model training uses large datasets, existing workflow scheduling consumes a significant amount of I / O interfaces, resulting in long disk read / write times and slow data processing speeds. Summary of the Invention

[0003] This invention provides a feature warehouse management method and system that solves the problems of existing workflow scheduling requiring a large number of I / O interfaces, long disk read / write times, and slow data processing speed.

[0004] To address the aforementioned technical problems, the present invention provides a data processing method, the method comprising:

[0005] In response to a workflow execution request, obtain the data affinity of the workflow;

[0006] The workflow is executed based on the data affinity relationships within the workflow.

[0007] Obtain the output data of the current operator in the workflow;

[0008] Based on the data affinity, if the output data is determined to be the final result of the workflow, the output data is output; if the output data is determined to be an intermediate result of the workflow, the output data is stored in memory.

[0009] Optionally, the above method further includes: based on the data affinity of the workflow, counting the number N of target operators that call the output data of the current operator; N is an integer greater than 1;

[0010] Create N copies of the output data of the current operator in the memory;

[0011] Running the workflow based on the data affinity of the workflow includes: sending the data copy to the target operator based on the data affinity of the workflow, and running the workflow.

[0012] Optionally, in the above method, the step of obtaining the data affinity of the workflow includes:

[0013] The data affinity relationships between the operators in the workflow are determined based on the reachability analysis algorithm.

[0014] Optionally, the above methods also include:

[0015] If the number of target operators that call the output data of the current operator is 1, then the output data is cleaned up after the target operator has finished executing;

[0016] If the number of target operators that call the output data of the current operator is N, then after all N target operators have been executed, the output data is cleaned up.

[0017] Optionally, the step of cleaning up the output data after all N target operators have been executed includes:

[0018] After the target operator is executed, clean up the data copy referenced by the target operator.

[0019] Optionally, in the above method, before the step of obtaining the output data of the current operator in the workflow, the method further includes:

[0020] Retrieve the target process that is started and in an idle state in memory;

[0021] The target process verifies the computing framework and runtime environment on which the current operator depends;

[0022] If the verification is successful, the target process enters the running state and executes the current operator.

[0023] Optionally, the above method further includes: if the container supports multiple environments, before the step of obtaining the target process that has been started and is in an idle state in memory, the method further includes:

[0024] Determine the operating environment of the current operator in the workflow;

[0025] Based on the process identifier and supported environment pre-recorded by the container, as well as the running environment of the current operator, the target process corresponding to the current operator is determined;

[0026] Start the target process.

[0027] Optionally, in the above method, if the workflow includes parallel operators, the step of the target process entering the running state to execute the current operator includes:

[0028] Based on the parallelism of the current operator in the workflow, multiple threads are started in the target process;

[0029] The parallel operators are executed using the multiple threads respectively.

[0030] Optionally, in the above method, before the step of determining the target process corresponding to the current operator based on the process identifier and supported environment pre-recorded by the container and the environment of the current operator, the method further includes:

[0031] Determine the computational framework upon which the current operator in the workflow depends;

[0032] The container corresponding to the execution of the target process is determined based on the computing framework and runtime environment on which the current operator depends.

[0033] Optionally, in the above method, the data used by each operator during the workflow operation is compressed before use.

[0034] This invention also provides a data processing system, the system comprising:

[0035] The first acquisition module is used to acquire the data affinity of the workflow in response to the workflow's execution request;

[0036] The execution module is used to run the workflow based on the data affinity of the workflow;

[0037] The second acquisition module is used to acquire the output data of the current operator in the workflow;

[0038] An output / storage module is used to, based on the data affinity, output the output data if it is determined that the output data is the final result of the workflow; and store the output data in memory if it is determined that the output data is an intermediate result of the workflow.

[0039] Optionally, the system further includes:

[0040] The statistics module is used to count the number N of target operators that call the output data of the current operator based on the data affinity of the workflow; N is an integer greater than 1.

[0041] A creation module is used to create N data copies of the output data of the current operator in the memory;

[0042] The running module is specifically used to: send the data copy to the target operator based on the data affinity of the workflow, and run the workflow.

[0043] Optionally, in the above system, the first acquisition module is specifically used for:

[0044] The data affinity relationships between the operators in the workflow to be run are determined based on the reachability analysis algorithm.

[0045] Optionally, the system further includes:

[0046] The first cleanup module is used to clean up the output data after the target operator has been executed if the number of target operators that call the output data of the current operator is 1.

[0047] The second cleanup module is used to clean up the output data if the number of target operators that call the output data of the current operator is N, after all N target operators have been executed.

[0048] Optionally, the second cleanup module is further configured to: clean up the data copy referenced by the target operator after the target operator has been executed.

[0049] Optionally, the system further includes:

[0050] The second acquisition module is used to acquire target processes that have been started and are in an idle state in memory;

[0051] The verification module is used by the target process to verify the computing framework and runtime environment on which the current operator depends;

[0052] The execution module is used to execute the current operator when the verification is successful, and the target process enters the running state.

[0053] Optionally, in the above system, if the container supports multiple environments, the system further includes:

[0054] The first determining module is used to determine the operating environment of the current operator in the workflow;

[0055] The second determining module is used to determine the target process corresponding to the current operator based on the process identifier and supported environment pre-recorded by the container and the running environment of the current operator;

[0056] The startup module is used to start the target process.

[0057] Optionally, in the above system, if the workflow includes parallel operators, the startup module includes:

[0058] The startup submodule is used to start multiple threads in the target process based on the parallelism of the current operator.

[0059] An execution submodule is used to execute the parallel operators using the multiple threads respectively.

[0060] Optionally, the system further includes:

[0061] The third determining module is used to determine the computational framework on which the current operator in the workflow depends;

[0062] The fourth determining module is used to determine the container corresponding to the execution of the target process based on the computing framework and runtime environment on which the current operator depends.

[0063] Optionally, in the above system, the data used by each operator during the workflow operation is compressed before use.

[0064] This invention also provides a data processing system, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the data processing method described above.

[0065] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data processing method described above.

[0066] In this embodiment of the invention, the workflow runs in memory, and the workflow data does not leave memory, thereby reducing the number of data read and write operations, reducing I / O resources, and improving data processing speed.

[0067] Furthermore, in this embodiment of the invention, a single data read can be reused by multiple operators, which can further reduce the number of data read and write operations; and it supports the reuse of processes within a container, so that all operators in the workflow can be repeatedly executed in the same process within the same container, thereby reducing the cost of process start-up, shutdown and switching. Attached Figure Description

[0068] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is a flowchart of a data processing method provided in an embodiment of the present invention;

[0070] Figure 2 This is a schematic diagram of a workflow provided in an embodiment of the present invention;

[0071] Figure 3 This is a structural block diagram of a data processing system provided in an embodiment of the present invention. Detailed Implementation

[0072] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0073] See Figure 1 , Figure 1 This is a flowchart of a data processing method provided in an embodiment of the present invention. The feature data processing method is applied to a data processing system, such as... Figure 1 As shown, the data processing method includes the following steps:

[0074] Step 101: In response to the workflow execution request, obtain the data affinity of the workflow.

[0075] Specifically, when a workflow needs to be executed, the user can input a workflow execution request on the data processing system's page; or the data processing system can automatically start and run the workflow after the user completes the workflow definition. In response to the workflow execution request, the original dataset of the workflow to be run is loaded into memory, and the data affinity of the workflow is obtained.

[0076] Optionally, the step of obtaining the data affinity of the workflow includes:

[0077] The data affinity relationships between the operators in the workflow to be run are determined based on the reachability analysis algorithm.

[0078] Specifically, in this embodiment of the invention, the reachability analysis algorithm determines the data affinity between operators based on the operator input and output data streams in the workflow, such as which other operators use the output data of the current operator, and the number of times the output data is used by other operators.

[0079] Reachability analysis can be implemented using a root search algorithm. Starting from the root operator of the workflow to be run, it searches from top to bottom to see if the target operators connected to the root operator are reachable. If the output of a target operator has no reference chain connected to it, it is considered unreachable, meaning that the output data of that target operator is invalid and can be immediately recycled without compression or backup. In this reachability analysis algorithm, only the outputs of operators that can be directly or indirectly connected to the root operator are considered live objects.

[0080] Step 102: Run the workflow based on the data affinity of the workflow.

[0081] Specifically, the data flow direction between operators in the workflow is determined based on the data affinity between them, and then the workflow is run based on the data flow direction between them.

[0082] Optionally, the output data of an operator in the workflow may be used by downstream operators of that operator. When the same set of data is used by multiple operators, in order to avoid the operation of each operator on the set of data affecting the use of other operators, and at the same time to reduce the memory space occupied by the output data, the method includes: according to the data affinity of the workflow, counting the number N of target operators that call the output data of the current operator; N is an integer greater than 1;

[0083] Create N copies of the output data of the current operator in the memory;

[0084] Step 102 of running the workflow based on the data affinity of the workflow includes: sending the data copy to the target operator based on the data affinity of the workflow, and running the workflow.

[0085] Specifically, since at least one output data point calling the current operator exists in the workflow to be run, the output data point calling the current operator can be one or more, depending on the data affinity of the workflow to be run. After determining the target operator for calling the output data of the current operator, the output data of the current operator is sent to the target operator that needs to call the output data of the current operator, and the target operator runs the workflow based on the received data.

[0086] Furthermore, when the current operator is the first operator in the workflow, the number N of target operators calling the original dataset of the workflow is counted according to the reachability analysis algorithm; N data copies of the original dataset are created in memory, and these N data copies are sent to the N target operators respectively. When the current operator is an intermediate operator in the workflow, the number N of target operators calling the output data of the current operator in the workflow is counted according to the reachability analysis algorithm; N data copies of the output data of the current operator are created in memory, and these N data copies are sent to the N target operators respectively. The target operators run the workflow based on the received data. In this embodiment of the invention, creating N data copies of the output data of the current operator in memory allows the output data of the current operator to be used so that if a problem occurs during the workflow execution, the problematic operator can be re-executed based on the output data of the current operator.

[0087] In this embodiment of the invention, each target operator has its own exclusive data copy, which can isolate the operation of multiple target operators, prevent one target operator from polluting the data required by another target operator, ensure that the operators do not interfere with each other, and prevent data from leaving memory, thereby reducing the frequency of disk read and write and improving the running speed.

[0088] Optionally, to reduce the memory space occupied by data in memory, the method further includes:

[0089] If the number of target operators that call the output data of the current operator is 1, then the output data is cleaned up after the target operator has finished executing;

[0090] If the number of target operators that call the output data of the current operator is N, then after all N target operators have been executed, the output data is cleaned up.

[0091] The process of cleaning up the output data after all N target operators have been executed also includes the recycling of a single data copy. The recycling process of a single data copy includes cleaning up the data copy referenced by the target operator after the target operator has been executed.

[0092] Specifically, during reachability analysis, the number of times each operator's output dataset is referenced by other operators is recorded; the count of the dataset is incremented by one each time an operator references it. For example... Figure 2As shown, before workflow execution, reachability analysis is used to obtain data affinity relationships between operators and reference counts for each operator. During workflow execution, based on the data affinity relationships of the workflow, the number of target operators calling the output data of the current operator in the workflow is determined. If the number of target operators calling the output data of the current operator is 1, the output data of the current operator is cleaned up after the target operator has finished executing. That is, the output data of the current operator is no longer compressed (or a low compression ratio is selected) and copied; it is cleaned up immediately after use to release memory resources. If the number of target operators calling the output data of the current operator is N, the output data is cleaned up after all N copies created have been recycled.

[0093] For example, in Figure 2 In the workflow, the first operator has three outputs, but the datasets of the two outputs on the right have no connection references, so these two datasets can be considered unreachable with a reference count of 0. The first output on the left is referenced by the third operator, so this dataset is considered reachable and will not be reclaimed, with a reference count of 1. During workflow execution, if the third operator completes, the reference count of the dataset on the left of the first operator is decremented by one. At this point, the reference count of the dataset on the left of the first operator is 0, and this dataset is about to be reclaimed.

[0094] Based on the data affinity of the workflow and the current running status of the workflow, this invention recycles and manages the output data of the operators during the workflow operation, which can effectively reduce the memory space occupation and achieve more efficient use of memory space.

[0095] In the process of running workflow operators, existing technology requires starting a new process to run the executable code of each operator, and then terminating the process after the operator has finished running. However, in this embodiment of the invention, the operators can be reused during workflow operation. When running a workflow within a container, a resident process (i.e., the target process) is first started to listen for and receive operator execution commands. The resident process is in a standby state. When an operator request is received, the resident process executes the received operator. After the resident process finishes executing the operator, it returns to the standby state.

[0096] Optionally, before the step of obtaining the output data of the current operator in the workflow, the method further includes:

[0097] Retrieve the target process that is started and in an idle state in memory;

[0098] The target process verifies the computing framework and runtime environment on which the current operator depends;

[0099] If the verification is successful, the target process enters the running state and executes the current operator.

[0100] Specifically, during workflow execution, a target process that is already running and in an idle state in memory is retrieved. This target process verifies the computing framework and runtime environment upon which the operator to be executed depends. If the verification passes, the target process enters the running state and executes the operator. Data generated during the execution of this operator is temporarily stored in the memory corresponding to this process.

[0101] It should be noted that if the container containing the memory supports only one environment, the first process can be reused as long as the container passes verification. If the container supports multiple environments, when reusing a process, it is also necessary to verify whether the environment supported by the process matches the environment required by the executed operator. Specifically, if the container supports multiple environments, before the step of obtaining the target process that has been started and is in an idle state in memory, the method includes: determining the running environment of the current operator in the workflow; determining the target process corresponding to the current operator based on the process identifier and supported environments pre-recorded by the container and the running environment of the current operator; and starting the target process. After the target process is started, the identifier of the process and the supported environments will be recorded, so that the corresponding process can be quickly matched based on the process identifier and the supported environments in subsequent workflow processes.

[0102] Optionally, if the workflow includes parallel operators, the target process entering the running state to execute the current operator includes:

[0103] Based on the parallelism of the current operator in the workflow, multiple threads are started in the target process;

[0104] The parallel operators are executed using the multiple threads respectively.

[0105] Specifically, if the running workflow has parallel operators, multiple threads can be started in the target process based on the parallelism of the current operator; the multiple threads are used to execute the parallel operators respectively, thereby realizing the parallel execution of multiple operators.

[0106] Optionally, before the step of determining the target process corresponding to the current operator based on the process identifier and supported environment pre-recorded by the container and the environment of the current operator, the method further includes:

[0107] Determine the computational framework upon which the current operator in the workflow depends;

[0108] The container corresponding to the execution of the target process is determined based on the computing framework and runtime environment on which the current operator depends.

[0109] Specifically, when running a workflow, the system first analyzes the computational frameworks and basic environments required for each operator to run. These frameworks include TensorFlow, Dask, and Python. Based on these frameworks and environments, containers that meet the specified conditions and specifications are identified. If a suitable container is found, the operator execution code is sent to it; otherwise, a new container is started via the k8s-restapi. Once the container receives the operator code, it sends it to the execution process. After execution, the process returns the result to the calling operator in the workflow. This process continues to run with minimal resource consumption, waiting for the next workflow operator call.

[0110] This invention, through process reuse, avoids the time consumed by starting and destroying processes before and after each operator execution, thereby reducing the cost of starting, stopping, and switching processes.

[0111] Optionally, the data used by each operator during the workflow operation is compressed before use.

[0112] Specifically, during the workflow execution, the data used by the workflow operators can be automatically converted to Parquet columnstore file format. When compressing the data, the user can select the compression format through a prompt interface. The provided compression formats include gzip, brotli, and snappy. Alternatively, a default compression format can be preset. When data compression is required, it will be directly compressed according to the preset default compression format. The compressed data is stored in memory, thereby greatly reducing memory space usage.

[0113] Step 103: Obtain the output data of the current operator in the workflow.

[0114] Specifically, when running a workflow, the output data of the current operator in the workflow can be obtained, and the workflow can be determined as to whether it has ended based on the output data of the current operator.

[0115] Step 104: When the output data is determined to be the final result of the workflow based on the data affinity, the output data is output; when the output data is determined to be an intermediate result of the workflow, the output data is stored in memory.

[0116] Specifically, based on the data affinity of the workflow, it is determined whether the output data of the current operator is the final result of the workflow. If it is determined that the output data of the current operator is the final result of the workflow, the output data is output to disk; if it is determined that the output data of the current operator is an intermediate result of the workflow, the output data is stored in memory.

[0117] In this embodiment of the invention, the workflow runs in memory, and the workflow data does not leave memory, thereby reducing the number of data read and write operations, reducing I / O resources, and improving data processing speed.

[0118] Based on the data processing method provided in the above embodiments, this invention also provides a data processing system for implementing the above-described data processing method. Please refer to... Figure 3 The data processing system 300 provided in this embodiment of the invention includes:

[0119] Optionally, the first acquisition module 301 is used to acquire the data affinity of the workflow in response to the workflow's execution request;

[0120] The execution module 302 is used to run the workflow based on the data affinity of the workflow;

[0121] The second acquisition module 303 is used to acquire the output data of the current operator in the workflow;

[0122] The output / storage module 304 is used to output the output data if it is determined that the output data is the final result of the workflow, based on the data affinity; and to store the output data in memory if it is determined that the output data is an intermediate result of the workflow.

[0123] Optionally, the system further includes:

[0124] The statistics module is used to count the number N of target operators that call the output data of the current operator based on the data affinity of the workflow; N is an integer greater than 1.

[0125] A creation module is used to create N data copies of the output data of the current operator in the memory;

[0126] The running module is specifically used to: send the data copy to the target operator based on the data affinity of the workflow, and run the workflow.

[0127] Optionally, the first acquisition module 301 is specifically used for:

[0128] The data affinity relationships between the operators in the workflow to be run are determined based on the reachability analysis algorithm.

[0129] Optionally, the system further includes:

[0130] The first cleanup module is used to clean up the output data after the target operator has been executed if the number of target operators that call the output data of the current operator is 1.

[0131] The second cleanup module is used to clean up the output data if the number of target operators that call the output data of the current operator is N, after all N target operators have been executed.

[0132] Optionally, the second cleanup module is further configured to: clean up the data copy referenced by the target operator after the target operator has been executed.

[0133] Optionally, the system further includes:

[0134] The second acquisition module is used to acquire target processes that have been started and are in an idle state in memory;

[0135] The verification module is used by the target process to verify the computing framework and runtime environment on which the current operator depends;

[0136] The execution module is used to execute the current operator when the verification is successful, and the target process enters the running state.

[0137] Optionally, if the container supports multiple environments, the system may also include:

[0138] The first determining module is used to determine the operating environment of the current operator in the workflow;

[0139] The second determining module is used to determine the target process corresponding to the current operator based on the process identifier and supported environment pre-recorded by the container and the running environment of the current operator;

[0140] The startup module is used to start the target process.

[0141] Optionally, if the workflow includes parallel operators, the execution module includes:

[0142] The startup submodule is used to start multiple threads in the target process based on the parallelism of the current operator.

[0143] An execution submodule is used to execute the parallel operators using the multiple threads respectively.

[0144] Optionally, the system further includes:

[0145] The third determining module is used to determine the computational framework on which the current operator in the workflow depends;

[0146] The fourth determining module is used to determine the container corresponding to the execution of the target process based on the computing framework and runtime environment on which the current operator depends.

[0147] Optionally, the data used by each operator during the workflow operation is compressed before use.

[0148] This invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data processing method described in the above embodiments.

[0149] This invention also provides a readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described data processing method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0150] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0151] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0152] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: In response to a workflow execution request, obtain the data affinity of the workflow; The workflow is executed based on the data affinity relationships within the workflow. Obtain the output data of the current operator in the workflow; Based on the data affinity, if the output data is determined to be the final result of the workflow, the output data is output; if the output data is determined to be an intermediate result of the workflow, the output data is stored in memory. The step of obtaining the data affinity of the workflow includes: Based on the reachability analysis algorithm and the operator input-output data stream in the workflow, the data affinity between each operator is determined. Before the step of obtaining the output data of the current operator in the workflow, the method further includes: Retrieve the target process that is started and in an idle state in memory; The target process verifies the computing framework and runtime environment on which the current operator depends; If the verification passes, the target process enters the running state and executes the current operator; The container supports multiple environments. Before the step of obtaining the target process that has been started and is in an idle state in memory, the method further includes: Determine the operating environment of the current operator in the workflow; Based on the process identifier and supported environment pre-recorded by the container, as well as the running environment of the current operator, the target process corresponding to the current operator is determined; Start the target process.

2. The data processing method according to claim 1, characterized in that, The method further includes: Based on the data affinity of the workflow, count the number N of target operators that call the output data of the current operator; N is an integer greater than 1; Create N copies of the output data of the current operator in the memory; The step of running the workflow based on the data affinity of the workflow includes: sending the data copy to the target operator based on the data affinity of the workflow, and running the workflow.

3. The data processing method according to claim 2, characterized in that, The method further includes: If the number of target operators that call the output data of the current operator is 1, then the output data is cleaned up after the target operator has finished executing; If the number of target operators that call the output data of the current operator is N, then after all N target operators have been executed, the output data is cleaned up.

4. The data processing method according to claim 3, characterized in that, The step of cleaning up the output data after all N target operators have been executed includes: After the target operator is executed, clean up the data copy referenced by the target operator.

5. The data processing method according to claim 1, characterized in that, If the workflow includes parallel operators, the steps for the target process to enter the running state and execute the current operator include: Based on the parallelism of the current operator in the workflow, multiple threads are started in the target process; The parallel operators are executed using the multiple threads respectively.

6. The data processing method according to claim 1, characterized in that, Before the step of determining the target process corresponding to the current operator based on the process identifier and supported environment pre-recorded by the container and the environment of the current operator, the method further includes: Determine the computational framework upon which the current operator in the workflow depends; The container corresponding to the execution of the target process is determined based on the computing framework and runtime environment on which the current operator depends.

7. The data processing method according to any one of claims 1-4, characterized in that, Also includes: The data used by each operator during the workflow process is compressed before use. The compressed data is stored in memory.

8. A data processing system, characterized in that, The system includes: The first acquisition module is used to acquire the data affinity of the workflow in response to the workflow's execution request; The execution module is used to run the workflow based on the data affinity of the workflow; The second acquisition module is used to acquire the output data of the current operator in the workflow; An output / storage module is used to, based on the data affinity, output the output data if it is determined that the output data is the final result of the workflow; and store the output data in memory if it is determined that the output data is an intermediate result of the workflow. The first acquisition module is specifically used for: Based on the reachability analysis algorithm and the operator input-output data stream in the workflow, the data affinity between each operator is determined. The second acquisition module is used to acquire target processes that have been started and are in an idle state in memory; The verification module is used by the target process to verify the computing framework and runtime environment on which the current operator depends; The execution module is used to, if the verification passes, allow the target process to enter a running state and execute the current operator; The container supports multiple environments, and the system also includes: The first determining module is used to determine the operating environment of the current operator in the workflow; The second determining module is used to determine the target process corresponding to the current operator based on the process identifier and supported environment pre-recorded by the container and the running environment of the current operator; The startup module is used to start the target process.

9. The data processing system according to claim 8, characterized in that, The system also includes: The statistics module is used to count the number N of target operators that call the output data of the current operator based on the data affinity of the workflow; N is an integer greater than 1. A creation module is used to create N data copies of the output data of the current operator in the memory; The running module is specifically used to: send the data copy to the target operator based on the data affinity of the workflow, and run the workflow.

10. The data processing system according to claim 9, characterized in that, The system also includes: The first cleanup module is used to clean up the output data after the target operator has been executed if the number of target operators that call the output data of the current operator is 1. The second cleanup module is used to clean up the output data if the number of target operators that call the output data of the current operator is N, after all N target operators have been executed.

11. The data processing system according to claim 10, characterized in that, The second cleanup module is further configured to: clean up the data copy referenced by the target operator after the target operator has been executed.

12. The data processing system according to claim 8, characterized in that, If the workflow includes parallel operators, the execution module includes: The startup submodule is used to start multiple threads in the target process based on the parallelism of the current operator in the workflow. An execution submodule is used to execute the parallel operators using the multiple threads respectively.

13. The data processing system according to claim 8, characterized in that, The system also includes: The third determining module is used to determine the computational framework on which the current operator in the workflow depends; The fourth determining module is used to determine the container corresponding to the execution of the target process based on the computing framework and runtime environment on which the current operator depends.

14. The data processing system according to any one of claims 8-11, characterized in that, The data used by each operator during the workflow operation is compressed before use; the compressed data is stored in memory.

15. A data processing system, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the data processing method as described in any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1-7.

Images