Method and apparatus for deploying data processing units

Abstract

The application provides a data processing unit deployment method and device, wherein the data processing unit deployment method comprises the following steps: acquiring time dimension rules of a plurality of data processing units to be deployed, and determining processing sequences between the plurality of data processing units according to the time dimension rules; determining a first data processing unit in the plurality of data processing units which is not configured with a distributed node identifier; and determining a target distributed node corresponding to the first data processing unit according to a permission attribute of the distributed node and the processing sequences between the plurality of data processing units. In this way, the development and deployment stages of the data processing unit are completely separated, the distributed node to which the data processing unit can be deployed is automatically determined based on the permission attribute of the distributed node and the processing sequences between the plurality of data processing units, the deployment efficiency of the data processing unit is improved, the human cost is saved, and the deployment flexibility is high.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for deploying a data processing unit. This application also relates to a deployment apparatus for a data processing unit, a computing device, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of computer and internet technologies, the amount of data is increasing in all shapes and sizes, prompting the rapid development of data processing technologies. Data processing can be broken down into multiple data processing units, which can be deployed on different distributed nodes.

[0003] In existing technologies, researchers typically develop data processing units and then deploy them on corresponding distributed nodes based on their experience. However, this method involves a complete process of development and deployment of the data processing unit, all manually operated by researchers, resulting in low efficiency, high manpower costs, and limited deployment flexibility. Therefore, an effective method is urgently needed to address these issues. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method for deploying a data processing unit to address the technical deficiencies in the prior art. Embodiments of this application also provide a deployment apparatus for a data processing unit, a computing device, and a computer-readable storage medium.

[0005] According to a first aspect of the embodiments of this application, a method for deploying a data processing unit is provided, comprising:

[0006] Obtain the time dimension rules of multiple data processing units to be deployed, and determine the processing order among the multiple data processing units according to the time dimension rules;

[0007] The first data processing unit among the plurality of data processing units is determined to be one that is not configured with a distributed node identifier;

[0008] The target distributed node corresponding to the first data processing unit is determined based on the permission attributes of the distributed nodes and the processing order among the multiple data processing units.

[0009] Optionally, determining the processing order among the plurality of data processing units according to the time dimension rule includes:

[0010] Determine the endpoint unit among the plurality of data processing units;

[0011] Based on the endpoint unit and the time dimension rule, the processing order between intermediate units and between intermediate units and endpoint units is determined, wherein the intermediate unit is a data processing unit other than the endpoint unit among the plurality of data processing units.

[0012] Optionally, determining the target distributed node corresponding to the first data processing unit based on the permission attributes of the distributed nodes and the processing order among the plurality of data processing units includes:

[0013] Based on the processing order among the plurality of data processing units, the candidate node corresponding to the first data processing unit is determined;

[0014] The processing attribute information of the first data processing unit is determined, and a target distributed node matching the processing attribute information is identified from the distributed nodes.

[0015] Optionally, before obtaining the time dimension rules for the multiple data processing units to be deployed, the method further includes:

[0016] Configure the time dimension rules, data processing rules, and estimated computing resources for the multiple data processing units.

[0017] Optionally, after determining the target distributed node corresponding to the first data processing unit based on the permission attributes of the distributed nodes and the processing order among the plurality of data processing units, the method further includes:

[0018] Based on the processing order among the multiple data processing units and the attribute information of each data processing unit, the data processing indicators of the distributed system are estimated, wherein the data processing indicators include at least one of data flow, computing resource consumption, data transmission volume, and processing time.

[0019] Optionally, determining the data processing metrics of the distributed system based on the processing order among the plurality of data processing units includes at least one of the following:

[0020] The data flow of the distributed system is obtained by superimposing the input data from the multiple data processing units.

[0021] The estimated computing resources of the multiple data processing units are superimposed to obtain the computing resource consumption of the distributed system.

[0022] The estimated computation times of the multiple data processing units are summed to obtain the processing time of the distributed system.

[0023] The connection relationships of the multiple data processing units are traversed to determine the cross-unit data volume between two connected units; the cross-node data volume between each pair of connected units is superimposed to obtain the data transmission volume of the distributed system.

[0024] Optionally, after determining the target distributed node corresponding to the first data processing unit based on the permission attributes of the distributed nodes and the processing order among the plurality of data processing units, the method further includes:

[0025] The first data processing unit and the second data processing unit are deployed on the corresponding target distributed nodes. The second data processing unit is a data processing unit configured with a distributed node identifier among multiple data processing units, and the target distributed node corresponding to the second data processing unit is the distributed node indicated by the distributed node identifier.

[0026] Optionally, after deploying the first data processing unit and the second data processing unit on the corresponding target distributed nodes, the method further includes:

[0027] Receive a data processing request and identify the data attribute information of the data to be processed carried in the data processing request;

[0028] Based on the data attribute information, the data to be processed is input to the corresponding target data processing unit, wherein the target data processing unit is the first data processing unit in the processing stream of the data to be processed.

[0029] Optionally, inputting the data to be processed into the corresponding target data processing unit includes:

[0030] According to the source of the data to be processed, the data to be processed is added to the corresponding first input queue;

[0031] Input elements are obtained from each input queue corresponding to the target data processing unit, and the obtained input elements are combined into an input data item, which is then input into the target data processing unit.

[0032] Optionally, after inputting the input data item into the target data processing unit, the method further includes:

[0033] The output data items of the target data processing unit are added to the corresponding first output queue, wherein the first output queue is the output queue corresponding to the target data processing unit.

[0034] According to a second aspect of the embodiments of this application, a deployment apparatus for a data processing unit is provided, comprising:

[0035] The acquisition module is configured to acquire the time dimension rules of multiple data processing units to be deployed, and determine the processing order among the multiple data processing units according to the time dimension rules;

[0036] The first determining module is configured to determine the first data processing unit among the plurality of data processing units that does not have a distributed node identifier configured;

[0037] The second determining module is configured to determine the target distributed node corresponding to the first data processing unit based on the permission attributes of the distributed node and the processing order among the plurality of data processing units.

[0038] According to a third aspect of the embodiments of this application, a computing device is provided, comprising:

[0039] Memory and processor;

[0040] The memory is used to store computer-executable instructions, and the processor executes the computer-executable instructions to implement the steps of the deployment method of the data processing unit.

[0041] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the deployment method of the data processing unit.

[0042] According to a fifth aspect of the present application, a chip is provided that stores a computer program, which, when executed by the chip, implements the steps of the deployment method of the data processing unit.

[0043] The data processing unit deployment method provided in this application embodiment can obtain the time dimension rules of multiple data processing units to be deployed, and determine the processing order among the multiple data processing units according to the time dimension rules; identify a first data processing unit among the multiple data processing units that is not configured with a distributed node identifier; and determine the target distributed node corresponding to the first data processing unit according to the permission attributes of the distributed node and the processing order among the multiple data processing units. In this case, the processing order among the multiple data processing units can be determined first according to the time dimension rules of the multiple data processing units to be deployed, and then the target distributed node corresponding to the first data processing unit that is not configured with a distributed node identifier can be automatically determined by combining the permission attributes of the distributed node and the processing order among the multiple data processing units. In this way, the development and deployment stages of the data processing unit are completely separated, and the deployable data processing units are automatically determined based on the permission attributes of the distributed node and the processing order among the multiple data processing units, which improves the deployment efficiency of the data processing unit, saves labor costs, and has high deployment flexibility. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the structure of a data processing unit deployment method provided in an embodiment of this application;

[0045] Figure 2 This is a flowchart illustrating a method for deploying a data processing unit according to an embodiment of this application;

[0046] Figure 3a This is a schematic diagram illustrating the processing order among multiple data processing units according to an embodiment of this application;

[0047] Figure 3b This is a schematic diagram of the configuration of a data processing unit provided in an embodiment of this application;

[0048] Figure 3c This is a schematic diagram of a data processing procedure provided in an embodiment of this application;

[0049] Figure 3d This is a schematic diagram of another data processing procedure provided in an embodiment of this application;

[0050] Figure 4a This is a flowchart of a data processing method provided in an embodiment of this application;

[0051] Figure 4b This is a schematic diagram of the deployment of a data processing unit provided in one embodiment of this application;

[0052] Figure 5 This is a schematic diagram of the structure of a deployment device for a data processing unit provided in an embodiment of this application;

[0053] Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this application. Detailed Implementation

[0054] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0055] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a,” “the,” and “the” used in one or more embodiments of this application and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” used in one or more embodiments of this application refers to and includes any or all possible combinations of one or more associated listed items.

[0056] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this application, and similarly, second may also be referred to as first.

[0057] This application provides a method for deploying a data processing unit. This application also relates to a deployment apparatus for a data processing unit, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0058] The execution entity of the data processing unit deployment method provided in this application embodiment can be a server, a terminal, or a terminal and a server working together; this application embodiment does not limit this. Furthermore, the terminal can be any electronic product capable of human-computer interaction, such as a PC (Personal Computer), mobile phone, PPC (Pocket PC), tablet computer, etc. The server can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center; this application embodiment does not limit this.

[0059] Figure 1 This is a schematic diagram of the structure of a data processing unit deployment method provided in an embodiment of this application. First, the processing order between the multiple data processing units to be deployed is determined according to the time dimension rules. Then, the target distributed node corresponding to the first data processing unit that is not configured with a distributed node identifier is automatically determined by combining the permission attributes of the distributed node and the processing order between the multiple data processing units.

[0060] By applying the solution provided in this embodiment, the development and deployment phases of the data processing unit are completely separated. Based on the permission attributes of the distributed nodes and the processing order among multiple data processing units, the data processing units that can be deployed are automatically determined, which improves the deployment efficiency of the data processing units, saves manpower costs, and provides high deployment flexibility.

[0061] Figure 2 A flowchart illustrating a deployment method for a data processing unit according to an embodiment of this application is shown, specifically including the following steps:

[0062] Step 202: Obtain the time dimension rules of the multiple data processing units to be deployed, and determine the processing order among the multiple data processing units according to the time dimension rules.

[0063] Specifically, a data processing unit refers to an execution unit that performs a specific data processing task, such as a data processing unit performing a Laplace transform or a data processing unit performing a weighted summation. The time dimension rule characterizes the processing sequence between data processing units, that is, the dependency relationship between one data processing unit and other data processing units. The processing order refers to the sequence number and order in which the processing is performed.

[0064] In practical applications, the data processing flow can be broken down into multiple data processing units, which can be deployed on different distributed nodes. In other words, the data processing flow can be broken down into data processing units, and then, according to specified deployment constraints and rules, these units can be deployed on corresponding distributed nodes. When a data processing request is received, the corresponding data processing unit is automatically connected, thus forming a data processing flow.

[0065] It should be noted that different data processing units may have a temporal order. Therefore, a time-dimensional rule can be obtained for the multiple data processing units to be deployed. Based on this time-dimensional rule, the processing order among the multiple data processing units can be determined. This time-dimensional rule can specify which data processing unit's output a data processing unit depends on, or which data processing unit's output a data processing unit can supply.

[0066] In one optional implementation of this embodiment, it is possible to distinguish whether a data processing unit is an endpoint unit or an intermediate unit, thereby determining its processing order. That is, based on the time dimension rule, the processing order among the multiple data processing units is determined. The specific implementation process can be as follows:

[0067] Determine the endpoint unit among the plurality of data processing units;

[0068] Based on the endpoint unit and the time dimension rule, the processing order between intermediate units and between intermediate units and endpoint units is determined, wherein the intermediate unit is a data processing unit other than the endpoint unit among the plurality of data processing units.

[0069] Specifically, an endpoint unit is a special type of data processing unit that has only input or output; an intermediate unit is a data processing unit other than the endpoint unit among multiple data processing units, that is, a data processing unit that has both input and output.

[0070] In practical applications, different types of data processing units can have corresponding unit identifiers. Therefore, the endpoint units among multiple data processing units can be listed first based on the endpoint unit identifiers, and then the processing order between intermediate units and between intermediate units and endpoint units can be determined according to the time dimension rules. Specifically, when determining the processing order between intermediate units and between intermediate units and endpoint units according to the time dimension rules, the dependencies can be determined sequentially from the starting endpoint to determine the processing order, or it can start from the ending endpoint.

[0071] For example, the multiple data processing units to be deployed are data processing units 1-10. Assume that among data processing units 1-10, data processing unit 1 and data processing unit 8 are the starting endpoints, and data processing units 6, 7, and 10 are the ending endpoints. Starting from the starting endpoint, for the starting endpoint data processing unit 1, its output is supplied to data processing unit 2; the input of data processing unit 2 depends on data processing unit 1, and its output is supplied to data processing unit 3; the input of data processing unit 3 depends on data processing unit 2, and its output is supplied to data processing units 4 and 5; the input of data processing unit 4 depends on data processing unit 3, and its output is supplied to data processing unit 6; the input of data processing unit 5 depends on data processing unit 3, and its output is supplied to data processing unit 7. For the other starting endpoint data processing unit 8, its output is supplied to data processing units 2 and 9; the input of data processing unit 9 depends on data processing unit 8, and its output is supplied to data processing unit 10; the input of data processing unit 10 depends on data processing unit 9. Based on the above time-dimensional rules, the processing order between each data processing unit can be determined as follows: Figure 3a As shown, Figure 3a This is a schematic diagram illustrating the processing sequence among multiple data processing units according to an embodiment of this application.

[0072] It should be noted that the endpoint units can be listed first, and then the dependencies of the data processing units can be connected through the endpoints to determine the processing order of multiple data processing units. This will allow the system to automatically determine which distributed node the data processing unit should be deployed on based on the processing order.

[0073] In one optional implementation of this embodiment, various rules can be pre-configured to enable the automatic deployment of multiple data processing units. That is, before obtaining the time dimension rules for the multiple data processing units to be deployed, the following steps may also be included:

[0074] Configure the time dimension rules, data processing rules, and estimated computing resources for the multiple data processing units.

[0075] Specifically, during the configuration phase, a configuration interface can be displayed on the front end, allowing users to configure the time dimension rules, data processing rules, and estimated computing resources for each data processing unit. When displaying the configuration interface, a configuration template can be provided, and users can add content to the template to complete the corresponding configuration.

[0076] In practical applications, since the processing order between data processing units requires manual configuration of the input dependencies of the data processing units, time dimension rules for multiple data processing units can be pre-configured. These time dimension rules are used to indicate the dependencies between multiple data processing units. In other words, for each data processing unit, its dependencies with other data processing units can be configured.

[0077] In addition, the estimated computing resources for each data processing unit can be configured, that is, the estimated computational load of each data processing unit, so that the computing resource consumption of the distributed system can be automatically estimated during the configuration phase. This includes statistics on the consumption of resources such as compute resources, memory, and video memory. Computational load refers to the computational complexity calculated using traditional function complexity methods.

[0078] It should be noted that, in addition to the time dimension rules for configuring the data processing units mentioned above, data processing rules for each data processing unit can also be configured. These data processing rules are used to indicate the conditions and constraints for the data processing unit to perform data processing tasks. For example, the data processing rules for each data processing unit may include data acquisition rules, data output rules, data operation rules, etc.

[0079] Data acquisition and output rules can include which data items and their corresponding data types are included in the input data, which data items and their corresponding data types are included in the output data, and the data processing resource consumption (which should be calculated automatically). Data items need to have defined data size (calculated automatically) and security level or permissions (manually configured). The data output of the data processing unit depends on the data input of that unit by default. Both the input and output of the data processing unit have data volume statistics.

[0080] In addition to configuring the data items for input and output data, data acquisition and output rules can also configure corresponding data transmission methods. Data transmission methods can include data communication types, trigger rules, data input rules, and data output rules. Data communication types include: None, where the data is only stored in the input or output queue and requires the other party to use push or pull; Push, the most basic and important data acquisition method, the other two relying on push; Pull, where a request event is first pushed to the data source, and then the data is waited for to be pushed over.

[0081] For example, taking a distributed node as the client, the client has input boxes for entering username and password, as well as a login button. Table 1 below shows the various pre-configured data communication types.

[0082] Table 1. Various Data Communication Types

[0083] push none Can none pull Can none none no push pull Can Two-way triggering, highest timeliness

[0084] It should be noted that in the data processing unit, the input, output, and calculation parts are all configured through trigger rules. The trigger rules are determined based on the data communication type mentioned above, such as triggering by "push", triggering by "pull", or bidirectional triggering (either push or pull is acceptable).

[0085] A data processing unit has only two input methods: pull and wait. If the input uses pull, the data is pulled from the output queue of the relevant data processing unit to its input queue. If the input uses wait, the data waits for the output queue of the upstream data processing unit to push it to the input queue, without performing a pull operation. Similarly, data output has only two output methods: push and wait. If the output uses push, the data is pushed to the input queue of the relevant data processing unit. If the output uses wait, the data is buffered in the output queue without being sent.

[0086] Furthermore, data processing rules are pre-configured. For example, if the input data is not empty and the output queue is not full, the operation is performed immediately, and the result is inserted into the output queue; if the output queue is full, the operation is blocked. Moreover, data types or data items are used for data input and output. Common inputs include constants, timers, calendars, user input, database reads, and file reads. Common outputs include writing to files, user interface output elements, and writing to the database. Network input and output are not configured as endpoints and can be automatically generated by distributed nodes and permissions.

[0087] For example, Table 2 below shows the pre-configured data transmission methods, and Table 3 below shows the pre-configured data types.

[0088] Table 2 Data Transmission Methods

[0089] 0 none Simply add to the queue, do not transmit. 1 push push 2 pull pull

[0090] Table 3 Data Type Table

[0091]

[0092]

[0093] It should be noted that the input data of a data processing unit can be stored in the input queue first, and the output data obtained from the calculation can be stored in the output queue before subsequent operations. Therefore, the input buffer size and output buffer size can be pre-configured for each data processing unit to store the input queue and output queue.

[0094] In practice, the size of the input and output buffers of the data processing unit can be configured, with a default value of 1. The buffers consist of multiple data queues. If a data queue is relied upon by multiple data processing units, it can be provided to different units by replicating the data queue. When the buffer is full, input or output will be blocked.

[0095] In addition, the data processing unit can be divided into endpoint units and intermediate units, so endpoint units can be pre-configured, as shown in Tables 4 and 5 below.

[0096] Table 4. Terminal Unit Configuration Table 1

[0097] 0 constant constant 1 UI.Text text 2 UI.Button.State Interface button status

[0098] Table 5. Terminal Unit Configuration Table 2

[0099] 0 AccountLength constant 2 Account string length 1 PasswordLength constant 3 Password string length 2 SaltLength constant 4 Password random character string length

[0100] Furthermore, after data enters a data processing unit, it can have multiple states, or multiple data perspectives, within that unit. A data perspective refers to how a piece of data is viewed from different viewpoints. By default, data possesses all data perspectives. Data perspectives can be separated and used as inputs and outputs of the data processing unit. For example, a default value is one data perspective, and when used as an endpoint of the data source, it is called a constant. Therefore, data perspectives can also be pre-configured for each data processing unit, as shown in Table 6 below.

[0101] Table 6 DataViewTable

[0102] 0 Default default value 1 UI.Input Interface input 2 UI.Output Interface output 3 UI.Logic Interface Logic

[0103] In addition, the data processing unit relies on functions to perform data processing operations. Therefore, the required functions can be configured for each data processing unit so that the data processing unit can perform the corresponding calculations.

[0104] For example, Table 7 below shows the definition function table for a certain data processing unit, and Table 8 below shows the configuration representation for a certain data processing unit.

[0105] Table 7 defines the function table.

[0106]

[0107] Table 8 Configuration of Data Processing Unit

[0108]

[0109]

[0110] Example, Figure 3b This is a schematic diagram of the configuration of a data processing unit provided in one embodiment of this application, as shown below. Figure 3b As shown, the configuration of the data processing unit includes input, operation and output. Input includes input data volume, trigger rules, etc., operation includes computation volume, storage volume, resource consumption, etc., and output includes output data volume, output rules, etc.

[0111] In this embodiment, information such as time dimension rules, data processing rules, and estimated computing resources for each data processing unit can be pre-configured. Subsequently, based on the time dimension rules, it can be automatically determined which distributed node the data processing unit should be deployed on. Based on the data processing rules, the data to be processed can be processed automatically. Based on the estimated computing resources, the resource consumption of the distributed system can be estimated during the configuration stage to intuitively reflect the system's consumption and provide a reference for data processing.

[0112] Step 204: Identify the first data processing unit among multiple data processing units that is not configured with a distributed node identifier.

[0113] It should be noted that when pre-configuring data processing units, their corresponding distributed nodes can be configured or not. For the second data processing unit configured with a distributed node identifier, the node indicated by the distributed node identifier is directly used as its corresponding target distributed node; that is, the second data processing unit with a distributed node identifier is directly deployed on the configured distributed node. For the first data processing unit without a distributed node identifier, it can be deployed on any distributed node. Therefore, it can be determined that the first data processing unit without a distributed node identifier is among the multiple data processing units. Subsequent analysis can be conducted on the first data processing unit without a distributed node identifier to determine which distributed node it should be deployed on.

[0114] Step 206: Determine the target distributed node corresponding to the first data processing unit based on the permission attributes of the distributed nodes and the processing order among multiple data processing units.

[0115] Specifically, permission attributes refer to using roles to define confidentiality level requirements or permission restrictions.

[0116] It should be noted that a distributed system can include multiple distributed nodes, such as servers and clients. In practice, the deployment of a distributed system is divided into two dimensions: time and space. Time refers to the processing order among multiple data processing units, while space refers to the grouping method among multiple distributed nodes, that is, which nodes are grouped as clients and which nodes are grouped as servers, etc.

[0117] In practical applications, distributed entities such as clients, servers, and databases can be pre-defined; that is, multiple distributed nodes in the distributed system are defined first. Furthermore, since different entities can have different permissions—that is, what operations they are allowed to perform—permission attributes for distributed nodes can also be pre-configured. These permissions are used to limit the attributes that can be set on the data processing units of that distributed node, or to limit the attributes that cannot be set on that distributed node. Specifically, the confidentiality requirements or permissions of distributed nodes can be defined by defining the roles of the distributed nodes.

[0118] For example, Table 9 below shows predefined distributed nodes, and Table 10 below shows predefined permission information tables for distributed nodes.

[0119] Table 9 Distributed Nodes (NodeTable)

[0120] 0 Client 0，1 Client 1 Server 2 server 2 Database 2 database

[0121] Table 10 Permission Information Table

[0122] 0 0 No permission by default 1 1 Direct contact is not possible, but we cannot guarantee against unauthorized clients. 2 2 Server-side verification

[0123] It should be noted that distributed nodes and their permission attributes can be predefined, and the time-dimensional rules, data processing rules, and estimated computing resources for each data processing unit can be pre-configured. Based on the time-dimensional rules, the connection order between the data processing units can be determined. Then, for the first data processing unit without a configured distributed node identifier, the target distributed node corresponding to the first data processing unit can be determined by combining the permission attributes of the distributed nodes and the processing order between the data processing units, thus achieving automatic deployment of the data processing unit. In this way, the development and integration / deployment phases of the data processing unit are completely separated. The same processing flow can satisfy different deployment schemes, as shown in Table 11 below, which lists various deployment schemes.

[0124] Table 11 Multiple Deployment Schemes

[0125]

[0126]

[0127] In one optional implementation of this embodiment, the target distributed node corresponding to the first data processing unit is determined based on the permission attributes of the distributed nodes and the processing order among the multiple data processing units. The specific implementation process can be as follows:

[0128] Based on the processing order among the plurality of data processing units, the candidate node corresponding to the first data processing unit is determined;

[0129] The processing attribute information of the first data processing unit is determined, and a target distributed node matching the processing attribute information is determined from the distributed nodes based on the permission attributes of the distributed nodes.

[0130] It should be noted that the candidate nodes corresponding to the first data processing unit can be determined first based on the processing order among multiple data processing units, and then the target distributed nodes that match the processing attribute information of the first data processing unit can be selected based on the permission attributes of each candidate node.

[0131] Continuing with the previous example, the data processing units are 1-10. Assuming that data processing unit 3 is located after data processing unit 2 and before data processing units 4 and 5, data processing unit 3 should be deployed among the distributed nodes between data processing units 2, 4, and 5. Therefore, the distributed nodes between data processing units 2 and 4, and between data processing units 2 and 5, can be considered as candidate nodes, namely candidate node 1, candidate node 2, candidate node 3, candidate node 4, and candidate node 5.

[0132] Assuming that the processing attribute information of data processing unit 3 is to modify data, while candidate node 2 is a read-only database and does not match the processing attribute information of data processing unit 3, and the other candidate nodes match the processing attribute information of data processing unit 3, then candidate node 1, candidate node 3, candidate node 4, and candidate node 5 can be used as the target distributed nodes of data processing unit 3.

[0133] In this embodiment, candidate nodes corresponding to the first data processing unit can be screened based on the processing order among multiple data processing units, and then candidate nodes with incompatible permissions can be screened out to determine the target distributed node that matches the first data processing unit. Subsequently, the first data processing unit can be deployed based on the determined target distributed node to achieve automatic deployment of the data processing unit and completely separate the development and deployment stages of the data processing unit.

[0134] In an optional implementation of this embodiment, after determining the target distributed node corresponding to the first data processing unit based on the permission attributes of the distributed nodes and the processing order among the plurality of data processing units, the method further includes:

[0135] The first data processing unit and the second data processing unit are deployed on the corresponding target distributed nodes. The second data processing unit is a data processing unit configured with a distributed node identifier among multiple data processing units, and the target distributed node corresponding to the second data processing unit is the distributed node indicated by the distributed node identifier.

[0136] In practical applications, for a first data processing unit that is not configured with a distributed node identifier, there may be multiple target distributed nodes. The final deployment node can be determined based on the path weights of each path. The path weights can be calculated based on parameters such as data traffic, transmission volume, and computational load. Each path can have its corresponding path weight calculated based on these parameters, facilitating the determination of the final deployment node for the first data processing unit based on the path weights of each path, thereby saving processing resources in the distributed system.

[0137] It's important to note that path weights are only used during automatic deployment. If all endpoint and intermediate units reside on a single node, no weight calculation is necessary. Path weights are configurable and can be applied to different deployment environments, such as clients with varying computing capabilities.

[0138] In addition to selecting the final deployment node of the first data processing unit based on path weight in practical applications, the GOAP (Goal-Oriented Action Planning) path planning algorithm can also be used. This algorithm mainly determines the deployment method by evaluating the cost of deployment at different nodes, and determines the connectivity between the input and output data items of the data processing unit, thereby determining the final deployment node of the first data processing unit. Of course, other shortest path algorithms can also be used to determine the deployment node of the first data processing unit, and this application embodiment does not impose any limitations on this.

[0139] It should be noted that the second data processing unit, configured with a distributed node identifier, can be directly deployed to the target distributed node indicated by the distributed node identifier. The first data processing unit, without a distributed node identifier, can be deployed to the finally determined deployment node. This means that a deployment chain for each data processing unit can be automatically generated, and subsequently, each data processing unit can be automatically deployed to its corresponding distributed node based on this chain. Specifically, the functions configured for each data processing unit can be embedded into the corresponding distributed node, allowing the distributed node to perform the corresponding data processing tasks by calling the appropriate function interfaces.

[0140] In an optional implementation of this embodiment, during the configuration phase, after estimating the processing metrics of the distributed system—that is, determining the target distributed node corresponding to the first data processing unit based on the permission attributes of the distributed nodes and the processing order among the multiple data processing units—the method further includes:

[0141] Based on the processing order among the multiple data processing units and the attribute information of each data processing unit, the data processing indicators of the distributed system are estimated, wherein the data processing indicators include at least one of data flow, computing resource consumption, data transmission volume, and processing time.

[0142] Specifically, attribute information refers to information such as the nature and characteristics of the data processing unit, including data transmission volume, set resource consumption, and set computation time. Data processing indicators refer to the estimated amount of resources, transmission volume, and processing time required for data processing.

[0143] It's important to note that a change to a single data item can trigger a series of calculations, leading to subsequent changes to other specific data items. These changes can include temporary data variables and data changes initially reflected outside the system (final data changes). The number of subsequent temporary and final data changes caused by a change in a single data item, along with the timing of these changes, can directly reflect the magnitude of the change. If the resource consumption of the triggered changes is measurable, and it's clear which data is sensitive, various useful data processing metrics can be estimated when configuring the data processing unit.

[0144] In this embodiment, the processing order among multiple data processing units and the attribute information of each data processing unit can be analyzed to determine the data processing indicators of the distributed system. In this way, the data processing indicators can be automatically estimated during the configuration stage, providing indicator references for data processing.

[0145] In one optional implementation of this embodiment, the data processing metrics include at least one of data flow, computing resource consumption, data transmission volume, and processing time. Therefore, the data processing metrics of the distributed system are determined according to the processing order among the plurality of data processing units, including at least one of the following:

[0146] The data flow of the distributed system is obtained by superimposing the input data from the multiple data processing units.

[0147] The estimated computing resources of the multiple data processing units are superimposed to obtain the computing resource consumption of the distributed system.

[0148] The estimated computation times of the multiple data processing units are summed to obtain the processing time of the distributed system.

[0149] The connection relationships of the multiple data processing units are traversed to determine the cross-unit data volume between two connected units; the cross-node data volume between each pair of connected units is superimposed to obtain the data transmission volume of the distributed system.

[0150] Specifically, data flow refers to the amount of data input to the data processing unit; computing resource consumption refers to the amount of resources consumed by the data processing unit during computation; processing time refers to the time consumed by the data processing unit during computation; and data transmission volume refers to the amount of data transmitted across nodes.

[0151] It should be noted that the input data volume, estimated computing resources, and estimated computing time of each data processing unit can all be pre-configured. By traversing each data processing unit and summing them up, the data flow, computing resource consumption, and processing time of the distributed system can be obtained.

[0152] In addition, the amount of data transmitted between two connected data processing units is the cross-unit data amount between the two connected units. By traversing the connection relationships of multiple data processing units and superimposing the cross-node data amounts between each connected pair of units, the data transmission volume of the distributed system can be obtained.

[0153] In this embodiment, after automatically deploying the data processing units to the corresponding distributed nodes, the data usage, computational load, and estimated processing time of each data processing unit can be calculated and accumulated according to the connectivity relationships between the data processing units to obtain the data traffic, computational resource consumption, and processing time of the distributed system. Alternatively, the connection relationships can be traversed, and the data processing units can be traversed according to the connection relationships to calculate the data usage across each node and accumulate to calculate the data transmission volume of the distributed system. Thus, determining data processing indicators from a more granular perspective can improve the accuracy of data processing indicators, and data traffic, computational resource consumption, processing time, and data transmission volume can be automatically estimated during the configuration phase for reference.

[0154] In one optional implementation, the computational resource consumption may include resource usage and processing complexity; accordingly, the estimated computational resources of the multiple data processing units are summed to obtain the computational resource consumption of the distributed system. The specific implementation process can be as follows:

[0155] The estimated resource usage of multiple data processing units is summed to obtain the total resource usage of the distributed system.

[0156] The total processing complexity of the distributed system is obtained by summing the processing complexities of multiple data processing units.

[0157] Specifically, the estimated resource usage and processing complexity of each data processing unit can be pre-configured. It should be noted that the processes of determining the total resource usage and the total processing complexity can be performed simultaneously or in a specific order; this embodiment does not impose any limitation on this. Thus, determining data processing metrics from a more granular perspective can further improve the accuracy of the data processing metrics.

[0158] In one optional implementation of this embodiment, after deploying each data processing unit on the corresponding distributed node, the corresponding data processing unit can be connected to process the data. That is, after deploying the first data processing unit and the second data processing unit on the corresponding target distributed node, the implementation further includes:

[0159] Receive a data processing request and identify the data attribute information of the data to be processed carried in the data processing request;

[0160] Based on the data attribute information, the data to be processed is input to the corresponding target data processing unit, wherein the target data processing unit is the first data processing unit in the processing stream of the data to be processed.

[0161] Specifically, the data to be processed refers to the data that needs to be processed. This can be modified data, such as newly added data, filtered data, data with partially replaced content, etc., or it can be data representing the user's desired processing results, such as monthly workload. A data processing request is a request issued to process the data to be processed, and it carries the data to be processed. Data processing requests can be initiated by the user through the client based on data requirements, or they can be generated by the executing entity based on a trigger (such as a timer reaching a set time). Data attribute information can include the data's character type and processing type. Character types include strings, integers, etc., and can represent the data's corresponding usage type, such as workload, cost, sales revenue, etc., or other types. The processing type can be the constraints or rules for processing the data. In practice, the corresponding processing rules can be directly obtained from the rule base based on the identifier or type of the data to be processed.

[0162] It should be noted that when the executing entity receives a data processing request carrying data to be processed, it needs to respond to the request and identify the data category of the data to be processed. Identification can be based on the identifier of the data to be processed or on the source information of the data.

[0163] In practical applications, the required data processing units can be determined based on the data attribute information of the data to be processed. Connecting these data processing units allows for the processing of the data. Specifically, the data processing units corresponding to the data to be processed can be connected into a unit queue according to the dependency relationship indicated by the time dimension rule. The first data processing unit in the unit queue is then used as the target data processing unit. After the data to be processed is input into the target data processing unit, it can be automatically processed by the corresponding unit queue to obtain the corresponding processing result. Here, a unit queue refers to multiple data processing units arranged in a certain order, that is, a set of processing units that process data with a certain type of data attribute information.

[0164] In one possible implementation, the data processing units corresponding to the data to be processed are connected into a unit queue according to the dependency relationship indicated by the time dimension rules. This can be achieved through the following steps: determine the processing order among the data processing units corresponding to the data to be processed according to the time dimension rules, and determine the group identifier corresponding to each data processing unit according to the grouping rules of the distributed nodes; then parse the processing order and group identifier to determine the connection order among multiple data processing units, and connect the data processing units corresponding to the data to be processed according to the connection order to obtain the corresponding unit queue.

[0165] It should be noted that the processing order and group identifiers are parsed to determine the connection order between the various data processing units corresponding to the data to be processed. Finally, the data processing units corresponding to the data to be processed are connected according to the connection order to obtain the corresponding unit queue. In this way, determining the connection order from both time and space dimensions can improve the accuracy of the connection order, thereby improving the robustness of the unit queue determined based on the connection order.

[0166] Optionally, the group identifier may include a client identifier and a server identifier; correspondingly, the specific implementation process of parsing the processing order and the group identifier to determine the connection order among the various data processing units corresponding to the data to be processed can be as follows:

[0167] The processing order of each third data processing unit in the processing sequence is determined as the first connection order, wherein the third data processing unit is the data processing unit whose group identifier is the client identifier;

[0168] The processing order of each fourth data processing unit in the processing sequence is determined as the second connection order, wherein the fourth data processing unit is the data processing unit whose group identifier is the server identifier;

[0169] The processing order between the first data processing unit and the second data processing unit in the processing sequence is determined as the third connection sequence.

[0170] Specifically, client-side identifier representation nodes or data processing units should be deployed on the client side, while server-side identifier representation nodes or data processing units should be deployed on the server side.

[0171] In practical applications, the processing order of data processing units grouped with client identifiers is determined as the first connection order; the processing order of data processing units grouped with server identifiers is determined as the second connection order; and the processing order between data processing units grouped with client identifiers and data processing units grouped with server identifiers is determined as the third connection order. In this way, determining the connection order according to different distributions can improve the reliability and accuracy of the connection sequence.

[0172] In practical implementation, the third data processing units can be connected according to a first connection order to obtain a first initial unit queue; the fourth data processing units can be connected according to a second connection order to obtain a second initial unit queue; and the first and second initial unit queues can be connected according to a third connection order to obtain the target unit queue. In this way, multiple data processing units on each node (with the same group identifier) ​​are connected, and data processing units with connection relationships on different nodes are connected. This improves connection efficiency and the reliability of the unit queues.

[0173] In practical applications, the data to be processed is input into the connected unit queue for processing to obtain the data processing result. Specifically, starting from the first data processing unit in the unit queue, the output result of the previous data processing unit is input into the current data processing unit for processing to obtain the output result of the current data processing unit, until the last data processing unit outputs the data processing result.

[0174] In this embodiment, all data processing units that use relevant data can be automatically aggregated and processed to obtain the corresponding processing results. This enables data processing using a unified processing device, and different unit queues can be generated for processing data with different data attributes, achieving targeted processing. Furthermore, it eliminates the need for dedicated equipment for data with different attributes, significantly reducing data processing costs.

[0175] In one optional implementation of this embodiment, since each data processing unit generates lazy computation triggered by push or pull based on the data deployment and its own configuration, that is, the data processing unit only performs calculations, pulls or pushes data when triggered, the input and output data of the data processing unit can be temporarily stored in a data queue. That is, the data to be processed is input to the corresponding target data processing unit. The specific implementation process may include:

[0176] According to the source of the data to be processed, the data to be processed is added to the corresponding first input queue;

[0177] Input elements are obtained from each input queue corresponding to the target data processing unit, and the obtained input elements are combined into an input data item, which is then input into the target data processing unit.

[0178] It should be noted that a data processing unit can correspond to at least one category of input data, and the input data of each category can be added to the corresponding input queue. In practical applications, the data to be processed can be added to the corresponding first input queue according to its source. If input elements exist in all input queues corresponding to the target data processing unit, input elements can be obtained from each input queue, combined into an input data item, and then input into the target data processing unit for data processing. If no input element exists in any of the input queues corresponding to the target data processing unit, it is necessary to wait for an input element to be written to that input queue before combining it into an input data item for processing.

[0179] Example, Figure 3c This is a schematic diagram of a data processing procedure provided in an embodiment of this application. Figure 3c The scenario illustrates a data processing unit that depends on only one input. The input data is a queue of input data, which is sequentially fed into a unit queue. The unit queue is comprised of all the data processing units that process the input data queue, and each data processing unit only needs to configure its prerequisites.

[0180] Another example, Figure 3d This is a schematic diagram of another data processing procedure provided in an embodiment of this application. Figure 3d This illustrates a scenario where the data processing unit relies on three inputs: input queue 1, input queue 2, and input queue 3. Figure 3d As shown, the first input element in input queue 1, input queue 2, and input queue 3 can be combined to obtain the corresponding first input data item; the second input element in input queue 1, input queue 2, and input queue 3 can be combined to obtain the corresponding second input data item; and the third input element in input queue 1, input queue 2, and input queue 3 can be combined to obtain the corresponding third input data item. If any input element is missing in any input queue, the system must wait to receive the corresponding input element before data processing can proceed. Each input data item can be input to the data processing unit, processed from state VI to state VO, and the output data item can be stored in the corresponding output queue. That is, the first, second, and third input data items are processed sequentially to obtain the first, second, and third output data items, which are then stored sequentially in their respective output queues.

[0181] In this embodiment, the input data of any data processing unit can be temporarily stored in the corresponding input queue, and the operation is performed once all the input data is ready. In this way, a data processing unit can rely on the data of multiple data processing units simultaneously, improving the flexibility of data processing.

[0182] In an optional implementation of this embodiment, the output data may also be temporarily stored in the output queue for use by subsequent data processing units. That is, after the input data items are input into the target data processing unit, the following may also be included:

[0183] The output data items of the target data processing unit are added to the corresponding first output queue, wherein the first output queue is the output queue corresponding to the target data processing unit.

[0184] Specifically, the first output queue refers to the output queue corresponding to the target data processing unit.

[0185] In practical applications, the output data items of the target data processing unit can be added to the corresponding first output queue. Subsequently, the first output queue can be copied to other data processing units that depend on the target data processing unit, serving as the input queue for those other data processing units.

[0186] In addition, the output queue can be used to control the execution of the data processing unit. Specifically, when the output queue is full, the data processing unit can be blocked from performing operations; when the output queue is not full, it can wait until all the corresponding input queues have input elements, then combine the corresponding input data items and input them into the target data processing unit for operation.

[0187] In practice, any data processing unit in the unit queue can refer to the execution process of the target data processing unit, and both input and output data are temporarily stored in the corresponding data queue.

[0188] It's worth noting that version numbers can also be assigned to each input and output data. That is, the input and output data of the data processing unit are managed by two version numbers: an input data change version number and an output data change version number. When input data changes, the input data change version number increments. After processing and filling in the output data, the input data change version number is assigned to the output data change version number. This allows for monitoring of input data and its changes, enabling timely detection of data changes.

[0189] The data processing unit deployment method provided in this application first determines the processing order among multiple data processing units based on time-dimensional rules. Then, combining the permission attributes of the distributed nodes and the processing order among the multiple data processing units, it automatically determines the target distributed node corresponding to the first data processing unit that is not configured with a distributed node identifier. In this way, the development and deployment phases of the data processing unit are completely separated. Based on the permission attributes of the distributed nodes and the processing order among the multiple data processing units, the method automatically determines the deployable data processing units, improving deployment efficiency, saving labor costs, and offering high deployment flexibility.

[0190] Figure 4a The flowchart shown is a data processing method according to an embodiment of this application, which specifically includes the following steps:

[0191] Step 402: Configure time dimension rules, data processing rules, and estimated computing resources for multiple data processing units.

[0192] Step 404: Obtain the time dimension rules for each data processing unit to be deployed, determine the endpoint units in each data processing unit, and determine the processing order between intermediate units and between intermediate units and endpoint units based on the endpoint units and the time dimension rules.

[0193] The intermediate unit is a data processing unit other than the endpoint unit among multiple data processing units.

[0194] Step 406: Identify the first data processing unit among the data processing units that is not configured with a distributed node identifier, and determine the candidate node corresponding to the first data processing unit according to the processing order among the data processing units; then, determine the processing attribute information of the first data processing unit, and identify the target distributed node that matches the processing attribute information from the distributed nodes.

[0195] Step 408: Determine the deployment node corresponding to the first data processing unit based on the path weight calculated in advance based on data traffic and computational load.

[0196] Step 410: Based on the target distributed node indicated by the distributed node identifier configured in the second data processing unit and the deployment node corresponding to the first data processing unit, automatically generate the deployment link of each data unit.

[0197] Example, Figure 4b This is a schematic diagram of the deployment of a data processing unit provided in one embodiment of this application, as shown below. Figure 4bAs shown, data processing units 2, 0, 8, and 9 are deployed on the first node (e.g., the client), data processing units 1 and 3 are deployed on the second node (e.g., the database), and data processing units 6, 5, 7, and 4 are deployed on the third node (e.g., the server). The output data of data processing unit 0 is supplied to data processing units 1 and 3; the output data of data processing unit 1 is supplied to data processing unit 2; the output data of data processing unit 3 is supplied to data processing unit 4; and the output data of data processing unit 4 is supplied to data processing units 8 and 9. Additionally, the output data of data processing unit 6 is supplied to data processing unit 5; the output data of data processing unit 5 is supplied to data processing unit 4; and the output data of data processing unit 7 is supplied to data processing unit 4.

[0198] Step 412: Deploy each data processing unit on the corresponding distributed node according to the deployment link of each data unit.

[0199] Step 414: Based on the processing order among the data processing units and the attribute information of each data processing unit, estimate at least one of the following in the distributed system: data flow, computing resource consumption, data transmission volume, and processing time.

[0200] Among them, the data processing indicators of the distributed system are determined according to the processing order among multiple data processing units, including at least one of the following:

[0201] The data flow of a distributed system is obtained by superimposing the input data from multiple data processing units.

[0202] The estimated computing resources of multiple data processing units are superimposed to obtain the computing resource consumption of the distributed system.

[0203] The estimated computation times of multiple data processing units are summed to obtain the processing time of the distributed system.

[0204] Traverse the connection relationships of multiple data processing units to determine the cross-unit data volume between two connected units; sum up the cross-node data volume between each pair of connected units to obtain the data transmission volume of the distributed system.

[0205] Step 416: Receive a data processing request, identify the data attribute information of the data to be processed carried in the data processing request, obtain the corresponding candidate processing units according to the data attribute information, and connect the candidate processing units into a unit queue.

[0206] Step 418: For any data processing unit in the unit queue, obtain input elements from each input queue corresponding to the data processing unit, combine the obtained input elements into an input data item, input the input data item into the data processing unit, and add the output data item of the data processing unit to the corresponding output queue.

[0207] The data processing unit deployment method provided in this application first determines the processing order among multiple data processing units based on time-dimensional rules. Then, combining the permission attributes of the distributed nodes and the processing order among the multiple data processing units, it automatically determines the target distributed node corresponding to the first data processing unit that is not configured with a distributed node identifier. In this way, the development and deployment phases of the data processing unit are completely separated. Based on the permission attributes of the distributed nodes and the processing order among the multiple data processing units, the method automatically determines the deployable data processing units, improving deployment efficiency, saving labor costs, and offering high deployment flexibility.

[0208] Furthermore, it enables the use of unified processing equipment for data processing, generating different unit queues for processing based on data with different data attributes, thus achieving targeted processing; and it eliminates the need to set up dedicated equipment for data with different data attributes, greatly reducing data processing costs.

[0209] Corresponding to the above method embodiments, this application also provides an embodiment of a deployment apparatus for a data processing unit. Figure 5 A schematic diagram of the structure of a deployment apparatus for a data processing unit according to an embodiment of this application is shown. Figure 5 As shown, the device includes:

[0210] The acquisition module 502 is configured to acquire the time dimension rules of multiple data processing units to be deployed, and determine the processing order among the multiple data processing units according to the time dimension rules;

[0211] The first determining module 504 is configured to determine the first data processing unit among the plurality of data processing units that does not have a distributed node identifier configured.

[0212] The second determining module 506 is configured to determine the target distributed node corresponding to the first data processing unit based on the permission attributes of the distributed node and the processing order among the plurality of data processing units.

[0213] Optionally, the acquisition module 502 is further configured as follows:

[0214] Determine the endpoint unit among the plurality of data processing units;

[0215] Based on the endpoint unit and the time dimension rule, the processing order between intermediate units and between intermediate units and endpoint units is determined, wherein the intermediate unit is a data processing unit other than the endpoint unit among the plurality of data processing units.

[0216] Optionally, the second determining module 506 is further configured to:

[0217] Based on the processing order among the plurality of data processing units, the candidate node corresponding to the first data processing unit is determined;

[0218] The processing attribute information of the first data processing unit is determined, and a target distributed node matching the processing attribute information is identified from the distributed nodes.

[0219] Optionally, the device further includes a configuration module configured to:

[0220] Configure the time dimension rules, data processing rules, and estimated computing resources for the multiple data processing units.

[0221] Optionally, the device also includes a prediction module configured to:

[0222] Based on the processing order among the multiple data processing units and the attribute information of each data processing unit, the data processing indicators of the distributed system are estimated, wherein the data processing indicators include at least one of data flow, computing resource consumption, data transmission volume, and processing time.

[0223] Optionally, the prediction module is further configured to be at least one of the following:

[0224] The data flow of the distributed system is obtained by superimposing the input data from the multiple data processing units.

[0225] The estimated computing resources of the multiple data processing units are superimposed to obtain the computing resource consumption of the distributed system.

[0226] The estimated computation times of the multiple data processing units are summed to obtain the processing time of the distributed system.

[0227] The connection relationships of the multiple data processing units are traversed to determine the cross-unit data volume between two connected units; the cross-node data volume between each pair of connected units is superimposed to obtain the data transmission volume of the distributed system.

[0228] Optionally, the device also includes a deployment module configured to:

[0229] The first data processing unit and the second data processing unit are deployed on the corresponding target distributed nodes. The second data processing unit is a data processing unit configured with a distributed node identifier among multiple data processing units, and the target distributed node corresponding to the second data processing unit is the distributed node indicated by the distributed node identifier.

[0230] Optionally, the device further includes an input module configured to:

[0231] Receive a data processing request and identify the data attribute information of the data to be processed carried in the data processing request;

[0232] Based on the data attribute information, the data to be processed is input to the corresponding target data processing unit, wherein the target data processing unit is the first data processing unit in the processing stream of the data to be processed.

[0233] Optionally, the input module is further configured as follows:

[0234] According to the source of the data to be processed, the data to be processed is added to the corresponding first input queue;

[0235] Input elements are obtained from each input queue corresponding to the target data processing unit, and the obtained input elements are combined into an input data item, which is then input into the target data processing unit.

[0236] Optionally, the device also includes an addition module configured to:

[0237] The output data items of the target data processing unit are added to the corresponding first output queue, wherein the first output queue is the output queue corresponding to the target data processing unit.

[0238] The data processing unit deployment apparatus provided in this application embodiment can first determine the processing order among multiple data processing units based on the time dimension rules of the multiple data processing units to be deployed. Then, combining the permission attributes of the distributed nodes and the processing order among the multiple data processing units, it automatically determines the target distributed node corresponding to the first data processing unit that is not configured with a distributed node identifier. In this way, the development and deployment phases of the data processing unit are completely separated. Based on the permission attributes of the distributed nodes and the processing order among the multiple data processing units, the deployable data processing units are automatically determined, which improves the deployment efficiency of the data processing units, saves labor costs, and provides high deployment flexibility.

[0239] The above is an illustrative scheme of a data processing unit deployment apparatus according to this embodiment. It should be noted that the technical solution of this data processing unit deployment apparatus and the technical solution of the data processing unit deployment method described above belong to the same concept. For details not described in detail in the technical solution of the data processing unit deployment apparatus, please refer to the description of the technical solution of the data processing unit deployment method described above.

[0240] Furthermore, the components in the device embodiment should be understood as functional modules necessary to implement each step of the program flow or method, and these functional modules are not actual functional divisions or separations. A device claim defined by such a set of functional modules should be understood as a functional module architecture that primarily implements the solution through the computer program described in the specification, and not as a physical device that primarily implements the solution through hardware.

[0241] Figure 6 A structural block diagram of a computing device according to an embodiment of this application is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0242] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0243] In one embodiment of this application, the aforementioned components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0244] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 600 can also be a mobile or stationary server.

[0245] The processor 620 is configured to execute computer-executable instructions for the deployment method of the data processing unit.

[0246] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data processing unit deployment method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data processing unit deployment method described above.

[0247] One embodiment of this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, are used for a deployment method of a data processing unit.

[0248] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the deployment method of the data processing unit described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the deployment method of the data processing unit described above.

[0249] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0250] One embodiment of this application also provides a chip that stores a computer program, which, when executed by the chip, implements the steps of the deployment method of the data processing unit.

[0251] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0252] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0253] The preferred embodiments disclosed above are merely illustrative of this application. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this application. These embodiments are selected and specifically described in this application to better explain the principles and practical applications of this application, thereby enabling those skilled in the art to better understand and utilize this application. This application is limited only by the claims and their full scope and equivalents.

Claims

1. A method for deploying a data processing unit, characterized in that, include: Obtain the time dimension rules of multiple data processing units to be deployed, and determine the processing order among the multiple data processing units according to the time dimension rules; The first data processing unit among the plurality of data processing units is determined to be one that is not configured with a distributed node identifier; Based on the permission attributes of the distributed nodes and the processing order among the multiple data processing units, the target distributed node corresponding to the first data processing unit is determined, wherein the permission attributes of the distributed nodes are used to characterize the types of data processing units that the distributed nodes are allowed to deploy.

2. The method according to claim 1, characterized in that, Determining the processing order among the multiple data processing units according to the time dimension rule includes: Determine the endpoint unit among the plurality of data processing units; Based on the endpoint unit and the time dimension rule, the processing order between intermediate units and between intermediate units and endpoint units is determined, wherein the intermediate unit is a data processing unit other than the endpoint unit among the plurality of data processing units.

3. The method according to claim 1, characterized in that, The step of determining the target distributed node corresponding to the first data processing unit based on the permission attributes of the distributed nodes and the processing order among the multiple data processing units includes: Based on the processing order among the plurality of data processing units, the candidate node corresponding to the first data processing unit is determined; The processing attribute information of the first data processing unit is determined, and a target distributed node matching the processing attribute information is identified from the distributed nodes.

4. The method according to claim 1, characterized in that, Before obtaining the time dimension rules for the multiple data processing units to be deployed, the process also includes: Configure the time dimension rules, data processing rules, and estimated computing resources for the multiple data processing units.

5. The method according to claim 4, characterized in that, After determining the target distributed node corresponding to the first data processing unit based on the permission attributes of the distributed nodes and the processing order among the multiple data processing units, the method further includes: Based on the processing order among the multiple data processing units and the attribute information of each data processing unit, the data processing indicators of the distributed system are estimated, wherein the data processing indicators include at least one of data flow, computing resource consumption, data transmission volume, and processing time.

6. The method according to claim 5, characterized in that, The determination of data processing metrics for the distributed system based on the processing order among the plurality of data processing units includes at least one of the following: The data flow of the distributed system is obtained by superimposing the input data from the multiple data processing units. The estimated computing resources of the multiple data processing units are superimposed to obtain the computing resource consumption of the distributed system. The estimated computation times of the multiple data processing units are summed to obtain the processing time of the distributed system. The connection relationships of the multiple data processing units are traversed to determine the cross-unit data volume between two connected units; the cross-node data volume between each pair of connected units is superimposed to obtain the data transmission volume of the distributed system.

7. The method according to any one of claims 1-6, characterized in that, After determining the target distributed node corresponding to the first data processing unit based on the permission attributes of the distributed nodes and the processing order among the multiple data processing units, the method further includes: The first data processing unit and the second data processing unit are deployed on the corresponding target distributed nodes. The second data processing unit is a data processing unit configured with a distributed node identifier among multiple data processing units, and the target distributed node corresponding to the second data processing unit is the distributed node indicated by the distributed node identifier.

8. The method according to claim 7, characterized in that, After deploying the first data processing unit and the second data processing unit on the corresponding target distributed nodes, the method further includes: Receive a data processing request and identify the data attribute information of the data to be processed carried in the data processing request; Based on the data attribute information, the data to be processed is input to the corresponding target data processing unit, wherein the target data processing unit is the first data processing unit in the processing stream of the data to be processed.

9. The method according to claim 8, characterized in that, The step of inputting the data to be processed into the corresponding target data processing unit includes: According to the source of the data to be processed, the data to be processed is added to the corresponding first input queue; If there are input elements in each input queue corresponding to the target data processing unit, the input elements are obtained from each input queue corresponding to the target data processing unit, the obtained input elements are combined into an input data item, and the input data item is input into the target data processing unit.

10. The method according to claim 9, characterized in that, After inputting the input data item into the target data processing unit, the method further includes: The output data items of the target data processing unit are added to the corresponding first output queue, wherein the first output queue is the output queue corresponding to the target data processing unit.

11. A deployment apparatus for a data processing unit, characterized in that, include: The acquisition module is configured to acquire the time dimension rules of multiple data processing units to be deployed, and determine the processing order among the multiple data processing units according to the time dimension rules; The first determining module is configured to determine the first data processing unit among the plurality of data processing units that does not have a distributed node identifier configured; The second determining module is configured to determine the target distributed node corresponding to the first data processing unit based on the permission attributes of the distributed node and the processing order among the plurality of data processing units, wherein the permission attributes of the distributed node are used to characterize the types of data processing units that the distributed node is allowed to deploy.

12. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the deployment method of the data processing unit according to any one of claims 1 to 10.

13. A computer-readable storage medium storing computer instructions, characterized in that, When executed by the processor, this instruction implements the steps of the deployment method of the data processing unit according to any one of claims 1 to 10.

14. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the deployment method of the data processing unit according to any one of claims 1 to 10.

Images