Data processing method, device and storage medium

Abstract

Embodiments of the present application provide a data processing method, device and storage medium. In the data processing method, a created target job instance can dynamically load metadata of one or more data processing models, and run the newly added one or more data processing models according to the dynamically loaded metadata. After running the newly added data processing model, the data in the currently monitored data source can be processed according to the calculation rule of the newly added data processing model. In this implementation, on the one hand, the job instance can dynamically load the data processing model, without the need for customized development of the data processing model when submitting the job, thereby reducing the development cost; on the other hand, one job instance can support the running operation of one or more data processing models, thereby realizing the sharing of resources and greatly improving the utilization rate of resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data computing, and particularly relates to a data processing method, device and storage medium. BACKGROUND

[0002] A streaming process is a continuous processing mode of unbounded data or events, and the process of the streaming process can be described by using a directed acyclic graph (DAG). In the DAG, each edge can represent a data or event stream, and each vertex can represent a data processing model. The data processing model can perform data processing according to a set rule.

[0003] In the prior art, when a streaming data processing framework is used to perform streaming data processing, a job task corresponding to a data processing model can be submitted to the streaming data processing framework. The streaming data processing framework can create a corresponding job instance by using resources, to perform a data processing task corresponding to the data processing model. However, as the number of data processing models increases, the number of job instances also increases, and a large number of job instances will occupy a large amount of resources. Therefore, a new solution needs to be proposed. SUMMARY

[0004] Aspects of the present application provide a data processing method, device and storage medium, to realize sharing of multiple models for a job instance, and improve utilization of resources.

[0005] The present application provides a data processing method, comprising: in response to a model loading event, loading metadata of at least one data processing model from a target storage space by a target job instance; running the at least one data processing model according to the metadata of the at least one data processing model by the target job instance; and processing data in a first data source currently monitored according to a respective calculation rule of the at least one data processing model by the target job instance.

[0006] Further optionally, before loading the model metadata of the at least one data processing model from the specified storage space by the target job instance, the method further comprises: receiving a model submission request; the model submission request carries model metadata of a newly added data processing model and an instance identifier of the target job instance; and storing the model metadata of the newly added data processing model in the target storage space corresponding to the instance identifier.

[0007] Further optionally, before processing the data in the first data source currently monitored according to the respective computing rules of the at least one data processing model, the method further comprises: for any data processing model in the at least one data processing model, determining a second data source corresponding to the data processing model; and synchronizing data in the second data source and processing progress information of the data to the first data source in real time, for reading by the target job instance.

[0008] Further optionally, running the at least one data processing model according to the metadata of the at least one data processing model by the target job instance comprises: determining a plurality of computing units included in the data processing model according to the metadata of the data processing model; identifying a respective computing type of the plurality of computing units; dividing a running process of the data processing model into at least one running stage according to the respective computing type of the plurality of computing units; and running the computing units included in the at least one running stage respectively by using different resources corresponding to the target job instance.

[0009] Further optionally, identifying the respective computing type of the plurality of computing units comprises: for any computing unit in the plurality of computing units, obtaining an execution script of the computing unit from the metadata of the data processing model; parsing the execution script and determining whether the computing unit includes state read-write operations according to a parsing result; and if the computing unit includes state read-write operations, determining that the computing type of the computing unit is stateful computing.

[0010] Further optionally, the target job instance is deployed in a distributed computing system, and the distributed computing system includes: a first type of computing node for performing stateless computing, and a second type of computing node for performing stateful computing; and running the computing units included in the at least one running stage respectively by using different resources corresponding to the target job instance comprises: running, by the target job instance, a computing unit of a stateless computing stage by using a first target node in the first type of computing node, and / or running a computing unit of a stateful computing stage by using a second target node in the second type of computing node.

[0011] Further optionally, processing the data in the first data source currently monitored according to the respective computing rules of the at least one data processing model by the target job instance comprises: performing stateless computing by a computing unit running on the first target node according to the received data to obtain a stateless computing result; and if the data processing model does not include a stateful computing rule, outputting the stateless computing result by the first target node.

[0012] Further optionally, if the data processing model contains stateful computing rules, the first target node sends the stateless computing result to the second target node according to the identifier of the data processing model; and the computing unit running on the second target node performs stateful computing according to the received stateless computing result to obtain a stateful computing result and outputs the stateful computing result.

[0013] The embodiment of the present application further provides a server, comprising a memory and a processor; the memory is used for storing one or more computer instructions; and the processor is used for executing the one or more computer instructions to execute the steps in the method provided by the embodiment of the present application.

[0014] The embodiment of the present application further provides a computer readable storage medium storing a computer program, and the computer program is executed by a processor to implement the steps in the method provided by the embodiment of the present application.

[0015] In the embodiment of the present application, the created target job instance can dynamically load the metadata of one or more data processing models, and run the newly added one or more data processing models according to the dynamically loaded metadata. After running the newly added data processing model, the data in the currently monitored data source can be processed according to the computing rules of the newly added data processing model. In this implementation, on the one hand, the job instance can dynamically load the data processing model, and there is no need to customize the development of the data processing model when submitting the job, thereby reducing the development cost; on the other hand, one job instance can support the running operation of one or more data processing models, thereby realizing the sharing of resources and greatly improving the utilization rate of resources. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the specification and illustrate the illustrative embodiments of the present application and the description thereof, and are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A flowchart of a stream data processing method provided by an exemplary embodiment of the present application;

[0018] Figure 2 A processing logic diagram of a data processing method provided by an exemplary embodiment of the present application;

[0019] Figure 3 A structural schematic diagram of a server provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in connection with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0021] The streaming process is a continuous processing mode for unbounded data or events, and the process of the streaming process can be described by using a directed acyclic graph (DAG). In the DAG, each edge can represent a data or event stream, and each vertex can represent a data processing model. The data processing model can process data according to a set rule.

[0022] In the prior art, when the streaming data processing framework is used for streaming data processing, the server can submit a job instance corresponding to a data processing model to the streaming data processing framework. The streaming data processing framework can create a corresponding job instance by using resources to execute a data processing task corresponding to the data processing model. However, as the number of data processing models increases, the number of job instances also increases, and a large number of job instances will occupy a large amount of resources.

[0023] In order to solve the above technical problems, in some embodiments of the present application, a solution is provided, and the technical solutions provided by the embodiments of the present application will be described in detail below in connection with the drawings.

[0024] Figure 1 The flowchart of the data processing method provided by an exemplary embodiment of the present application is shown in FIG. 1. Figure 1 The method comprises the following steps.

[0025] Step 101, in response to a model loading event, loading metadata of at least one data processing model from a target storage space by a target job instance.

[0026] Step 102, running the at least one data processing model according to the metadata of the at least one data processing model respectively by the target job instance.

[0027] Step 103, processing data in a first data source currently monitored according to the respective calculation rules of the at least one data processing model by the target job instance.

[0028] The execution subject of the embodiment can be one or more servers (hereinafter referred to as servers), any server can be implemented as a conventional server device, a virtual machine, a cloud host or an elastic computing instance on the cloud, and the embodiment does not make any limitation. Among them, the one or more servers run a stream computing application based on a stream computing framework (such as Blink or Flink), which can be used to provide stream computing services.

[0029] In the stream computing framework, a resident computing task can be called a job. When there is a demand to execute a job task, the server can run the stream computing application to create a job instance. In the application, the core components for executing the job task can include a job master component, a task execution component and a resource manager component. Among them, the job master component is used to manage the job; the task execution component is used to execute each task; and the resource manager component is used to manage resources and serve resource requests of the job management component.

[0030] When creating the job instance, the server can allocate certain resources to the job instance, including but not limited to CPU resources, memory resources and network resources, etc.

[0031] Among them, the model loading event can be a timing event or a user event, and the embodiment does not make any limitation. In some cases, the server can trigger the model loading event at a timing, so that the target job instance checks whether there is an updated data processing model at a timing. In other cases, the server can trigger the model loading event according to the operation of the user side (such as the operation of the user submitting a new data processing model), so that the target job instance loads the new data processing model submitted by the user.

[0032] Among them, the data processing model can be integrated by a variety of computing rules, which is used to express the flow and steps of data processing and the specific algorithm in each step. The data processing model can be expressed by a directed acyclic graph composed of a plurality of nodes (or computing units). The node refers to the basic computing unit in the data processing model, which is used to express the computing rule, which can be edited based on the script. The result output by the upstream node in the data processing model can be used as the input of the downstream node.

[0033] In the embodiment, the data processing model can be dynamically configured. When a new data processing model needs to be added in the stream computing framework, the new data processing model can be submitted to the server, and the server can add the new data processing model to the specified storage space. When the model loading event is triggered, the target job instance created on the server can dynamically load the newly added data processing model.

[0034] It is worth mentioning that the target job instance can load a new data processing model in response to a model loading event while running a certain data processing model, or load a new data processing model in response to a model loading event in an idle state. That is, the target job instance can run one or more data processing models at the same time, and the one or more data processing models can share the resources of the target job instance.

[0035] In the process of loading the data processing model by the target job instance, the metadata of the data processing model can be loaded from the target storage space. The target storage space can be located locally on the server or in a remote database, and the present embodiment does not make any limitation. In some embodiments, different job instances can correspond to different target storage spaces, so as to submit data processing models to different job instances. The metadata of the data processing model refers to data used to describe the data processing logic of the data processing model, which can include but is not limited to at least one of the calculation rules, calculation steps, script files, input formats and output formats of the data processing model.

[0036] For any data processing model, after the target job instance obtains the metadata of the data processing model, the server can run the data processing model according to the metadata of the data processing model through the target job instance. When the target job instance loads multiple data processing models from the target storage space, the server can start multiple threads through the target job instance, and use the multiple threads to run the multiple data processing models according to the metadata of the multiple data processing models respectively. Any thread can be used to run a data processing model, so as to realize parallel execution of data processing operations based on multiple data processing models.

[0037] In the process of running the any data processing model by the target job instance, the data processing model can be loaded into the memory, the calculation rules of the data processing model can be parsed, and the target job instance can wait for data to arrive.

[0038] The first data source listened to by the target job instance can be a data source specified for the target job instance when the target job instance is created. The target job instance can fixedly obtain data from the first data source. When data arrives, the target job instance can execute a data processing operation according to the calculation rules of the data processing model.

[0039] In the embodiment, the created target job instance can dynamically load metadata of one or more data processing models, and run the added one or more data processing models according to the dynamically loaded metadata. After running the added data processing model, the data in the currently monitored data source can be processed according to the calculation rule of the added data processing model. In this implementation, on the one hand, the job instance can dynamically load the data processing model, without the need for customized development of the data processing model when submitting the job, thereby reducing the development cost; on the other hand, one job instance can support the running operation of one or more data processing models, thereby realizing the sharing of resources and greatly improving the utilization rate of resources.

[0040] In some exemplary embodiments, the at least one data processing model loaded by the target job instance can be submitted to the server by the user in advance through the client. Alternatively, when the user has the demand of adding a data processing model, if the user does not need to specify the job instance, the client can initiate a model submission request to the server according to the metadata of the added data processing model. After receiving the model submission request, the server can automatically allocate a job instance for the added data processing model, or allocate a job instance with a lower current resource utilization rate for the added data processing model, or newly create a job instance for the added data processing model, which is not limited in the embodiment.

[0041] Alternatively, when the user submits the added data processing model, the user can also specify a specific job instance to run the data processing model. If the user needs to specify the job instance, the client can initiate a model submission request to the server according to the metadata of the data processing model and the identifier of the specified job instance. The instance identifier can be an ID (Identity document, identity identifier), a process number, a number, a name, etc. of the job instance, which is not limited in the embodiment.

[0042] When multiple job instances are created on the server, the storage spaces corresponding to each job instance can be different. When different job instances are created, the server can respectively specify storage spaces for storing data processing models for different job instances, so as to facilitate the dynamic loading of data processing models by the job instance. If the model submission request carries the instance identifier of the target job instance specified by the user, after receiving the model submission request sent by the client, the server can store the model metadata of the added data processing model in the target storage space corresponding to the instance identifier according to the model metadata of the added data processing model and the instance identifier of the target job instance carried in the submission request. Thus, the target job instance can read the added data processing model from the target storage space to meet the demand of the user specifying the job instance.

[0043] Of course, in some cases, if the current resource occupancy of the target job instance specified by the user is high, the server can switch other job instances for the data processing model newly added by the user to ensure the running efficiency of the data processing model. After switching the job instance, the server can send a switching message of the job instance to the user through the client to prompt the user.

[0044] In this implementation, the data processing model can be dynamically configured by the user through the client, without the need for customized development according to the calculation rules of the data processing model when creating the job instance, greatly reducing the development cost. For each job instance, the data processing model running thereon can be loaded in a dynamic manner, greatly improving the flexibility of resource sharing of the job instance.

[0045] Continuing with the target job instance as an example. In some exemplary embodiments, the server can specify the data source (i.e., the first data source) of the target job instance when creating the target job instance. The data source refers to a database in the stream computing framework for receiving external data. The target job instance can listen to the data source in real time to obtain data to be processed during the running process. In some cases, the data sources of different data processing models are different. Therefore, to enable the target job instance to run any data processing model to process data in the data source corresponding to the data processing model, the data source of the data processing model can be synchronized with the first data source in real time.

[0046] Taking the data processing model running on the target job instance as an example, the server can determine the data source corresponding to the data processing model through the target job instance as a second data source, and synchronize the data in the second data source and the processing progress information of the data to the first data source in real time for the target job instance to read. The data processing progress information includes the offset of the data, which is used to avoid repeated processing of the same data. The following will be described in conjunction with specific examples.

[0047] For example, the job instance Al listens to the data source S1 in real time, and the job instance A2 listens to the data source S2 in real time. The data processing model B1 is used to calculate the registration data of a user, which is stored in the data source S3. The data processing model B2 is used to calculate the login data of a user, which is stored in the data source S4. The data processing model B3 is used to calculate the transaction data of a user, which is stored in the data source S5. When the client submits the data processing models B1 and B2, the instance identifier of the job instance Al is carried. When the client submits the data processing model B3, the instance identifier of the job instance A2 is carried. After receiving the submission request of the client, the server can synchronize the data in the data source S3 and the data source S4 and the offset of the data to the data source S1 in real time, for reading by the job instance Al, and can synchronize the data in the data source S5 and the offset of the data to the data source S2 in real time, for reading by the job instance A2.

[0048] In this embodiment, when the created target job instance has a fixed listening data source, by synchronizing the data source of the newly added data processing model to the data source listened by the target job instance, the data readable by the target job instance can be flexibly updated. On the one hand, the running of the target job instance does not need to be paused to modify the listening data source, thereby reducing the influence on the running data processing model. On the other hand, the data to be processed by the newly added data processing model is updated in real time, thereby ensuring the accuracy and reliability of the output result of the data processing model.

[0049] In some exemplary embodiments, when multiple data processing models share the same job instance, in order to further improve the resource utilization, the running process of the multiple data processing models can be divided into stages, and different resources are used in different running stages, so as to reduce the mutual influence of different operation stages in terms of resource use. The following will be exemplarily explained.

[0050] Continuing to take any data processing model as an example, optionally, when the server runs the data processing model according to the metadata of the data processing model, the server can determine a plurality of computing units contained in the data processing model according to the metadata of the data processing model, and identify the computing types of the plurality of computing units respectively. One computing unit can correspond to one operator in a stream computing framework. The computing type refers to the type of the operator corresponding to the computing unit.

[0051] In the stream computing framework, the operators include stateful operators and stateless operators. The stateless operators are mainly used for stateless computation, which means that the operator only needs to process the current data to obtain the computation result. The stateful operators are mainly used for stateful computation, which means that the operator needs to combine some historical states (such as historical computation results, intermediate computation results, etc.) to perform relevant computation to obtain the computation result. The computation speed of the stateful operator is slower, and the demand for resources is larger.

[0052] Based on the above characteristics of the stateful operator, when the server identifies the computation types of the plurality of computation units through the job instance, the server acquires the execution script of any computation unit in the plurality of computation units from the metadata of the data processing model; parses the execution script, and determines whether the computation unit contains state read-write operation according to the parsing result; if the computation unit contains state read-write operation, it is determined that the computation type of the computation unit is stateful computation.

[0053] After determining the computation types of the plurality of computation units, the server can divide the running process of the data processing model into at least one running stage according to the computation types of the plurality of computation units. For example, if the data processing model only includes stateless computation units, the server can only divide the running process of the data processing model into a stateless running stage. If the data processing model includes stateless computation units and stateful computation units, the server can divide the running process of the data processing model into a stateless computation stage and a stateful computation stage.

[0054] After dividing the running stages, the server can run the computation units contained in each of the at least one running stage by using different resources through the target job instance. The resources refer to the resources allocated to the target instance by the server when creating the target instance, including but not limited to CPU, memory, etc.

[0055] Optionally, in this embodiment, the server can allocate different resources to different running stages, and different computation tasks in the same running stage can share the allocated resources, and the resources between the computation tasks in different running stages are in a relatively isolated state, so as to reduce the mutual influence of different running stages in terms of resources and improve the computation efficiency. For example, 2 stateless computation units can share a 2-core CPU and 20G memory, and 2 stateful computation units can share a 4-core CPU and 60G memory.

[0056] Optionally, the resources allocated for different operation stages can include resources such as CPU and memory on a server, or resources such as CPU and memory on different servers in a server cluster. For example, in some embodiments, multiple virtual machines on a server can be allocated to different operation stages respectively. In other embodiments, multiple server devices in a server cluster can be allocated to different operation stages respectively, and the embodiments are not limited in this regard.

[0057] In some example embodiments, the target job instance can be deployed in a distributed computing system including first type of computing nodes for performing stateless computation, and second type of computing nodes for performing stateful computation.

[0058] Based on the above, if the data processing model only contains computation units of stateless computation type, the server can deploy the computation units of stateless computation type to first target nodes in the first type of computing nodes. The first target nodes can be one or more computing nodes in the first type of computing nodes. Accordingly, when the server runs the computation units contained in the at least one operation stage respectively, the target job instance can run the computation units of the stateless computation stage using the first target nodes in the first type of computing nodes.

[0059] If the data processing model only contains computation units of stateful computation type, the server can deploy the computation units of stateful computation type to second target nodes in the second type of computing nodes. The second target nodes can be one or more computing nodes in the second type of computing nodes. Accordingly, when the server runs the computation units contained in the at least one operation stage respectively, the target job instance can run the computation units of the stateful computation stage using the second target nodes in the second type of computing nodes.

[0060] If the data processing model contains computation units of both stateless computation type and stateful computation type, the server can deploy the computation units of stateless computation type to first target nodes in the first type of computing nodes, and deploy the computation units of stateful computation type to second target nodes in the second type of computing nodes. Accordingly, when the server runs the computation units contained in the at least one operation stage respectively, the target job instance can run the computation units of the stateless computation stage using the first target nodes in the first type of computing nodes, and run the computation units of the stateful computation stage using the second target nodes in the second type of computing nodes.

[0061] Accordingly, in some example embodiments, when the server processes the data in the first data source currently monitored according to the calculation rule of the data processing model through the target job instance, the computing unit running on the first target node can perform stateless calculation according to the received data to obtain a stateless calculation result. If the data processing model does not contain a stateful calculation rule, the target job instance can output the stateless calculation result through the first target node.

[0062] Optionally, if the data processing model contains stateful calculation, the second target node needs to perform further calculation according to the stateless calculation result of the first target node. It is worth noting that when performing the calculation task in a distributed manner, for stateful calculation, the correctness of the received intermediate calculation process or accumulated value needs to be ensured. That is, the stateless calculation result of the first target node needs to be accurately sent to the second target node for stateful calculation. When the second type of computing node contains multiple nodes, the first target node needs to accurately determine the node matched with it from the second type of computing node as the second target node.

[0063] Optionally, in this embodiment, the identifier of the data processing model can be added to different computing units in the data processing model when deploying the different computing units in the data processing model to different nodes. Based on this, if the data processing model contains a stateful calculation rule, the stateless calculation result can be sent to the second target node through the first target node according to the identifier of the data processing model. Thus, the computing unit running on the second target node can perform stateful calculation according to the received stateless calculation result to obtain a stateful calculation result, and output the stateful calculation result.

[0064] In this implementation, the data processing model is run in a stage splitting manner, which can run time-consuming stateful calculation and time-consuming stateless calculation using different resources, further rationalize the allocation of resources, and improve the calculation efficiency.

[0065] The following will be further illustrated by combining the Figure 2 The embodiments of the present application are further illustrated.

[0066] As Figure 2As shown, users can dynamically configure data processing models through configuration. Job instances (Blink Jobs or Flink Jobs) can periodically and selectively load and compute the data processing models they need to process. Multiple different data processing models can be loaded for the same job instance, enabling shared computation. This is illustrated in Model 1 and Model 2. When the computation of a data processing model is complex, the job instance can determine whether to compute that model independently or run it in parallel with other data processing models, based on available resources.

[0067] In streaming computing frameworks, jobs are configured with the data sources they need to monitor at startup, such as topics in databases like Kafka and DataHub. The data processing model, however, is dynamically loaded into the job after startup, meaning the data sources the job needs to monitor cannot be determined before startup. Therefore, as... Figure 2 As shown, each job can be configured to listen to a fixed data source (i.e., a unified input), and when the data processing model is loaded, the data listened to by the data processing model and its basic information (such as offsets) are synchronized to this fixed data source. During the job process, data listening and distribution are autonomously implemented, thus solving the problem that jobs in streaming computing frameworks cannot dynamically add or remove data sources.

[0068] Each job primarily involves loading model metadata, parsing computation rules, and performing real-time data computation. Real-time computation includes two types of calculations: stateless computation and stateful computation. Stateful computation (e.g., aggregation computation) requires storing historical information (e.g., cumulative historical values ​​for counts), and these calculations are often quite complex. In the job, the data processing model's computation can be divided into two stages: a MAP (Mapping) stage for stateless computation and an AGG (Aggregation) stage for stateful computation. For example... Figure 2 As shown, in the AGG stage, aggregated data can be stored using the state variable provided by Blink or Flink at the underlying level.

[0069] like Figure 2 As shown, when the data processing model only contains stateless computations, the stateless computation results can be directly output after the MAP stage, as in Model 1. When the data processing model contains stateful computations, the model can be split into two stages starting from the operator where stateful computations occur, and the stateful computation results can be output finally, as in Model 2. When the data processing model contains stateful computations, since the job is distributed, to ensure the correctness of the accumulated values, the nodes in the MAP stage can distribute data according to the data processing model ID, ensuring that data from the same model falls on the same node in AGG. This will not be elaborated further.

[0070] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subject of steps 101 to 104 can be device a; for another example, the execution subject of steps 101 and 102 can be device a, and the execution subject of step 103 can be device b; and the like.

[0071] In addition, in some of the processes described in the above embodiments and the accompanying drawings, a plurality of operations appearing in a certain order are included, but it should be clearly understood that these operations can be executed in the order appearing in the text or in parallel, and the serial numbers of the operations such as 101, 102, etc. are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel.

[0072] It should be noted that the "first", "second", and the like in the text are used to distinguish different messages, devices, modules, etc., and do not represent the order, nor do "first" and "second" represent different types.

[0073] Figure 3 is a structural schematic diagram of a server provided by an exemplary embodiment of the present application, which can be used to execute the data processing method described in the foregoing embodiments. As shown in Figure 3 The server includes a memory 301 and a processor 302.

[0074] The memory 301 is used to store computer programs, and can be configured to store other various data to support operations on the server. Examples of these data include instructions for any application or method operating on the server, contact data, phonebook data, messages, pictures, videos, etc.

[0075] The memory 301 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0076] The processor 302 is coupled with the memory 301, and is configured to execute a computer program in the memory 301, so as to: in response to a model loading event, load metadata of at least one data processing model from a target storage space through a target job instance; run the at least one data processing model according to the metadata of the at least one data processing model respectively through the target job instance; and process data in a first data source currently monitored according to a calculation rule of each of the at least one data processing model through the target job instance.

[0077] Further optionally, before the processor 302 loads the metadata of the at least one data processing model from the specified storage space through the target job instance, the processor 302 is further configured to: receive a model submission request; the model submission request carries metadata of a newly added data processing model and an instance identifier of the target job instance; and store the metadata of the newly added data processing model in the target storage space corresponding to the instance identifier.

[0078] Further optionally, before the processor 302 processes data in the first data source currently monitored according to the calculation rule of each of the at least one data processing model, the processor 302 is further configured to: for any data processing model in the at least one data processing model, determine a second data source corresponding to the data processing model; and synchronize data in the second data source and processing progress information of the data to the first data source in real time, so as to be read by the target job instance.

[0079] Further optionally, when the processor 302 runs the at least one data processing model according to the metadata of the at least one data processing model through the target job instance, the processor 302 is specifically configured to: determine a plurality of calculation units contained in the data processing model according to the metadata of the data processing model; identify a calculation type of each of the plurality of calculation units; divide a running process of the data processing model into at least one running stage according to the calculation type of each of the plurality of calculation units; and run calculation units contained in each of the at least one running stage respectively by using different resources corresponding to the target job instance.

[0080] Further optionally, when the processor 302 identifies the calculation type of each of the plurality of calculation units, the processor 302 is specifically configured to: for any calculation unit in the plurality of calculation units, obtain an execution script of the calculation unit from the metadata of the data processing model; parse the execution script, and determine whether the calculation unit contains a state read-write operation according to a parsing result; and if the calculation unit contains the state read-write operation, determine that the calculation type of the calculation unit is stateful calculation.

[0081] Further optionally, the target job instance is deployed in a distributed computing system, the distributed computing system comprising: a first type of computing node for performing stateless computing, and a second type of computing node for performing stateful computing; the processor 302 is specifically configured to, when running the computing units included in each of the at least one running stage respectively by using different resources corresponding to the target job instance: run, by the target job instance, a computing unit of a stateless computing stage by using a first target node in the first type of computing node, and / or run a computing unit of a stateful computing stage by using a second target node in the second type of computing node.

[0082] Further optionally, the processor 302 is specifically configured to, when processing the data in the first data source currently monitored respectively according to the computing rules of the at least one data processing model by the target job instance: perform stateless computing according to the received data by the computing unit running on the first target node to obtain a stateless computing result; and if the data processing model does not include a stateful computing rule, output the stateless computing result by the first target node.

[0083] Further optionally, the processor 302 is further configured to: if the data processing model includes a stateful computing rule, send, by the first target node, the stateless computing result to the second target node according to the identifier of the data processing model; perform stateful computing according to the received stateless computing result by the computing unit running on the second target node to obtain a stateful computing result, and output the stateful computing result.

[0084] Further, as shown in Figure 3 the server further includes: a communication component 303 and a power component 304, and other components. Figure 3 Some components are only schematically shown in the server, and it does not mean that the server only includes Figure 3 the components shown.

[0085] The communication component 303 is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G or 5G, or a combination thereof. In an example embodiment, the communication component receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component can be implemented based on near field communication (NFC) technology, radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra wide band (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0086] The power supply component 304 provides power to various components of the device in which the power supply component is located. The power supply component can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0087] In this embodiment, the target job instance created on the server can dynamically load the metadata of one or more data processing models, and run the newly added one or more data processing models according to the dynamically loaded metadata. After running the newly added data processing model, the data in the currently monitored data source can be processed according to the calculation rule of the newly added data processing model. In this implementation, on the one hand, the job instance can dynamically load the data processing model, without the need for customized development of the data processing model when submitting the job, thereby reducing the development cost; on the other hand, one job instance can support the running operation of one or more data processing models, thereby realizing the sharing of resources and greatly improving the utilization rate of resources.

[0088] Correspondingly, the embodiment of the application further provides a computer readable storage medium storing a computer program, and the computer program is executed to implement each step that can be executed by the server in the method embodiment.

[0089] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0090] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0091] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0093] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0094] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory, etc. The memory is an example of computer readable media.

[0095] Computer readable media includes permanent and non-permanent, moveable and non- moveable media that can be implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic disks storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that is accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0096] It should also be noted that the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0097] The above description is merely illustrative of the application, and not restrictive. Various modifications and changes can become apparent to those skilled in the art. Incorporating any modification, equivalent substitution, improvement, etc. within the spirit and principle of the application, shall be included in the scope of the claims of the application.

Claims

1. A data processing method, characterized in that, include: In response to a model load event, load the metadata of at least one data processing model from the target storage space using the created target job instance; Using the already created target job instance, the newly added at least one data processing model is run according to the metadata of the at least one data processing model, and the at least one data processing model shares the resources of the target job instance; Using the created target job instance, the data in the currently monitored first data source is processed according to the calculation rules of each of the at least one data processing model.

2. The method according to claim 1, characterized in that, Before loading model metadata for at least one data processing model from the specified storage space using the target job instance, the process also includes: Receive a model submission request; the model submission request carries the model metadata of the newly added data processing model and the instance identifier of the target job instance; The model metadata of the newly added data processing model is stored in the target storage space corresponding to the instance identifier.

3. The method according to claim 1, characterized in that, Before processing the data in the currently monitored first data source according to the respective calculation rules of the at least one data processing model, the process further includes: For any one of the at least one data processing models, determine the second data source corresponding to the data processing model; The data in the second data source and the processing progress information of the data are synchronized to the first data source in real time for the target job instance to read.

4. The method according to claim 3, characterized in that, Using the target job instance, the at least one data processing model is run based on the metadata of the at least one data processing model, including: Based on the metadata of the data processing model, determine the multiple computing units included in the data processing model; Identify the computation type of each of the plurality of computing units; Based on the respective computation types of the multiple computing units, the operation process of the data processing model is divided into at least one operation stage; The computing units included in each of the at least one running stage are run using different resources corresponding to the target job instance.

5. The method according to claim 4, characterized in that, Identifying the computation type of each of the plurality of computing units includes: For any one of the plurality of computing units, the execution script of the computing unit is obtained from the metadata of the data processing model; The execution script is parsed, and the result of the parsing is used to determine whether the computing unit contains state read / write operations. If the computing unit includes state read / write operations, then the computing type of the computing unit is determined to be stateful computing.

6. The method according to claim 4, characterized in that, The target job instance is deployed in a distributed computing system, which includes: a first type of computing node for performing stateless computing, and a second type of computing node for performing stateful computing; Using different resources corresponding to the target job instance, the computing units included in each of the at least one running stage are run, including: Through the target job instance, the computing unit in the stateless computing phase is run by the first target node in the first type of computing nodes, and / or the computing unit in the stateful computing phase is run by the second target node in the second type of computing nodes.

7. The method according to claim 6, characterized in that, Through the target job instance, the data in the currently monitored first data source is processed according to the calculation rules of each of the at least one data processing model, including: The computing unit running on the first target node performs stateless computation based on the received data to obtain the stateless computation result. If the data processing model does not contain stateful computation rules, then the stateless computation result is output through the first target node.

8. The method according to claim 7, characterized in that, Also includes: If the data processing model contains stateful computation rules, then the stateless computation result is sent to the second target node through the first target node according to the identifier of the data processing model; The computing unit running on the second target node performs stateful computation based on the received stateless computation result, obtains the stateful computation result, and outputs the stateful computation result.

9. A server, characterized in that, include: Memory and processor; The memory is used to store one or more computer instructions; The processor is configured to execute one or more computer instructions for performing the steps of the method according to any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it is able to perform the steps of the method described in any one of claims 1-8.

Images