Code topology cross-cluster conversion method and device, computer equipment and storage medium
By analyzing and traversing the code topology of the JStorm cluster, determining the upstream operator and converting it into corresponding operators on the Flink cluster, the problem of inefficient manual conversion in the existing technology is solved and efficient code topology conversion is achieved.
Patent Information
- Application Number
- CN202311375669.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-20
- Publication Date
- 2025-04-22
AI Technical Summary
When the existing technology supports code encoded by JStorm technology on a Flink cluster, developers need to perform manual conversion, which is inefficient.
Provide a cross-cluster conversion method for code topology. By obtaining and parsing the code topology of the first cluster, traversing each operator, determining the upstream operator based on directed edges, ensuring that all operators upstream of the target operator are converted, and avoiding repeated traversal of processed operators.
The algorithm complexity is reduced, the conversion efficiency is improved, and the integrity and efficiency of the conversion process is ensured.
Smart Images
Figure CN119861934A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of communication technology, and in particular to a method, device, computer equipment and storage medium for code topology cross-cluster conversion. Background Art
[0002] JStorm is a distributed real-time computing engine. JStorm is a system similar to Hadoop MapReduce. Users implement a task according to the specified interface, and then hand over the task to the JStorm system. JStorm runs the task and runs it 24 / 7. If a worker fails unexpectedly, the scheduler immediately assigns a new worker to replace the failed worker. For user code, commonly used coding interfaces include topology, spout, bolt, etc.
[0003] Apache Flink is a distributed processing engine and framework for stateful computation of unbounded and bounded data streams. It can run in common cluster environments and execute at memory speed and at any scale. Common interfaces for users include source, transform, sink / operator, etc.
[0004] Currently, there is technology in the industry that supports the code encoded with jstorm technology in the flink cluster, but it often requires developers to perform manual conversion, which is inefficient. Summary of the invention
[0005] Based on this, it is necessary to provide a code topology cross-cluster conversion method, device, computer equipment and storage medium to address the above technical problems. When traversing to a certain target first operator, it can ensure that the first operators upstream of the target first operator are all converted, and the first operators that have been processed will no longer be traversed repeatedly, thereby reducing the algorithm complexity and improving the conversion efficiency.
[0006] In a first aspect, the present application provides a code topology cross-cluster conversion method, including: obtaining a first code topology of a first cluster to be converted; parsing the first code topology to obtain directed edges between each first operator and each first operator; traversing each first operator, and when a target first operator in each first operator has not been converted into a second operator of a second cluster, determining an upstream first operator of the target first operator according to the directed edge; if an upstream second operator of the second cluster converted by the upstream first operator is obtained, obtaining a target second operator of the second cluster converted by the target first operator according to the target first operator and the upstream second operator; and obtaining a second code topology of the second cluster according to the target second operator and the directed edge.
[0007] In some embodiments of the present application, the operators include a data source operator and a data processing operator. When the target first operator in each first operator is not converted into the second operator of the second cluster, the upstream first operator of the target first operator is determined according to the directed edge, including: when the target first operator in each first operator is not converted into the second operator of the second cluster, if the target first operator is determined to be the first data source operator according to the directed edge, the target first operator is converted into the target second operator of the second cluster; when the target first operator in each first operator is not converted into the second operator of the second cluster, if the target first operator is determined to be the first data processing operator according to the directed edge, the upstream first operator of the target first operator is determined according to the directed edge.
[0008] In some embodiments of the present application, when the target first operator in each first operator has not been converted into the second operator of the second cluster, after determining the upstream first operator of the target first operator according to the directed edge, it also includes: if the upstream first operator has not been converted into the second operator of the second cluster, determining the first operator upstream of the upstream first operator according to the directed edge, until the first operator of the converted second operator of the second cluster is obtained, or, until the obtained first operator is an unconverted first data source operator.
[0009] In some embodiments of the present application, the first code topology is parsed to obtain directed edges between each first operator, including: parsing the first code topology to obtain a main data stream input into each first operator, a business logic module of each first operator, and directed edges between each first operator; determining a sub-data stream obtained after processing by the first operator according to the main data stream and the business logic module of the first operator; determining a downstream first operator of the first operator according to the directed edges; determining a grouping flow strategy from the first operator to the downstream first operator according to the sub-data stream of the first operator and the main data stream of the downstream first operator, and determining it as the grouping flow strategy of the downstream first operator.
[0010] In some embodiments of the present application, a target second operator is obtained according to a business logic module of a target first operator, a target main data flow, a target group flow strategy, and a target sub-data flow of a target second operator, including: obtaining environmental information of a second cluster; encapsulating the business logic module of the target first operator according to the environmental information to obtain the business logic module of the target second operator; obtaining the target second operator according to the business logic module of the target second operator, the target main data flow, the target group flow strategy, and the target sub-data flow of the target second operator.
[0011] In a second aspect, the present application provides a code topology cross-cluster conversion device, including: an information acquisition module, used to obtain a first code topology of a first cluster to be converted; a code parsing module, which is communicated and connected to the information acquisition module, and is used to parse the first code topology to obtain directed edges between each first operator and each first operator; a code conversion module, which is communicated and connected to the code parsing module, and is used to traverse each first operator, and when the target first operator in each first operator is not converted into the second operator of the second cluster, determine the upstream first operator of the target first operator according to the directed edge; if the upstream second operator of the second cluster converted by the upstream first operator is obtained, then the target second operator of the second cluster converted by the target first operator is obtained according to the target first operator and the upstream second operator; the second code topology of the second cluster is obtained according to the target second operator and the directed edge.
[0012] In some embodiments of the present application, the operators include data source operators and data processing operators, and the code conversion module is also used for, when the target first operator in each first operator is not converted into the second operator of the second cluster, if the target first operator is determined to be the first data source operator according to the directed edge, then the target first operator is converted into the target second operator of the second cluster; when the target first operator in each first operator is not converted into the second operator of the second cluster, if the target first operator is determined to be the first data processing operator according to the directed edge, then the upstream first operator of the target first operator is determined according to the directed edge.
[0013] In some embodiments of the present application, the code conversion module is also used to determine the first operator upstream of the upstream first operator according to the directed edge if the upstream first operator is not converted into the second operator of the second cluster, until the first operator of the converted second operator of the second cluster is obtained, or until the obtained first operator is an unconverted first data source operator.
[0014] In some embodiments of the present application, the code parsing module is also used to parse the first code topology to obtain the main data flow input to each first operator, the business logic module of each first operator, and the directed edges between each first operator; determine the sub-data flow obtained after processing by the first operator according to the main data flow and the business logic module of the first operator; determine the downstream first operator of the first operator according to the directed edge; determine the grouping flow strategy from the first operator to the downstream first operator according to the sub-data flow of the first operator and the main data flow of the downstream first operator, and determine it as the grouping flow strategy of the downstream first operator.
[0015] In some embodiments of the present application, the code conversion module is also used to obtain environmental information of the second cluster; encapsulate the business logic module of the target first operator according to the environmental information to obtain the business logic module of the target second operator; obtain the target second operator according to the business logic module of the target second operator, the target main data flow of the target second operator, the target grouping flow strategy and the target sub-data flow.
[0016] In a third aspect, the present application also provides a computer device, comprising: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement a code topology cross-cluster transformation method.
[0017] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and the computer program is loaded by a processor to execute the steps in the code topology cross-cluster transformation method.
[0018] The above-mentioned code topology cross-cluster conversion method, device, computer equipment and storage medium, when traversing to the target first operator that has not converted the second operator, recursively traverses the upstream operators upward according to the directed edges of the first code topology, and converts the upstream operators that have not been converted before, that is, when traversing to a certain target first operator, it can ensure that the first operators upstream of the target first operator are all converted, and the first operators that have been processed are no longer repeatedly traversed, thereby reducing the algorithm complexity and improving the conversion efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0020] Figure 1 It is a flowchart of a code topology cross-cluster conversion method in an embodiment of the present application;
[0021] Figure 2 is a schematic diagram of a cache layer of a second code topology in another embodiment of the present application;
[0022] Figure 3 is a flow chart of a code topology cross-cluster conversion method in another embodiment of the present application;
[0023] Figure 4 It is a schematic diagram of the structure of the code topology cross-cluster conversion device in an embodiment of the present application;
[0024] Figure 5 It is a schematic diagram of the structure of a computer device in an embodiment of the present application. DETAILED DESCRIPTION
[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0026] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.
[0027] In the description of the present application, the word "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.
[0028] The current algorithm for converting jstorm topology to flink topology is as follows: 1. Traverse the spout operators of jstormtopology and generate the corresponding flink source operator for each operator; 2. If the number of remaining operators is greater than 0, traverse the remaining bolt operators in the jstorm topology; 2.1. If all upstream operators of this bolt operator are ready, generate the flink operator operator corresponding to this operator, and then remove the bolt from the remaining operators; 2.2. If all upstream operators of this bolt operator are not ready yet, jump to step 2.
[0029] There is a problem with this algorithm. When traversing the bolt operator of the jstorm topology to generate the flinkoperator operator, a brute force algorithm is used. The complexity of the algorithm is O(n^2). When there are many operator data, this conversion algorithm will be slow.
[0030] See also Figure 1 , the embodiment of the present application provides a code topology cross-cluster conversion method, the method includes steps S101 to S105, which are as follows:
[0031] S101: Obtain a first code topology of a first cluster to be converted.
[0032] Specifically, this embodiment is applied to converting a first code topology of a first cluster into a second code topology of a second cluster. The code frameworks of the first cluster and the second cluster are different, so the code topologies of the two cannot be directly run under the cluster framework of the other party. For example, the first cluster is set to a jstorm cluster, and the second cluster is set to a flink cluster. The first code topology of the first cluster to be converted is the jstorm topology.
[0033] S102: Parse the first code topology to obtain directed edges between each first operator.
[0034] Specifically, the first code topology is parsed to obtain directed edges between each first operator and each first operator in the first code topology, and the directed edges point to the data flow between the two connected first operators. If the data of the first operator A flows to the first operator B, then the first operator A is the upstream operator of the first operator B, and the first operator B is the downstream operator of the first operator A. An upstream operator can have one or more downstream operators, and a downstream operator can also have one or more upstream operators.
[0035] In one embodiment, this step includes: S201, parsing the first code topology to obtain the main data flow input to each first operator, the business logic module of each first operator and the directed edges between the first operators; S202, determining the sub-data flow obtained after processing by the first operator according to the main data flow and the business logic module of the first operator; S203, determining the downstream first operator of the first operator according to the directed edge; S204, determining the grouping flow strategy from the first operator to the downstream first operator according to the sub-data flow of the first operator and the main data flow of the downstream first operator, and determining it as the grouping flow strategy of the downstream first operator.
[0036] Specifically, the first code topology is parsed to obtain the main data stream input to each first operator, the business logic module of each first operator, and the directed edges between each first operator. The business logic module of the first operator processes the input main data stream to obtain a sub-data stream, and the number of sub-data streams can be 0, one, or more. The downstream first operator of the first operator is determined according to the directed edge, and the grouping flow strategy from the first operator to the downstream first operator is determined according to the sub-data stream belonging to the first operator in the main data stream input to the downstream first operator. The grouping flow strategy from the first operator to the downstream first operator determines in what grouping form the sub-data stream of the first operator is distributed to the downstream first operator. For ease of calling, the grouping flow strategy from the first operator to the downstream first operator is determined as the grouping flow strategy of the downstream first operator. Various types of data are divided into main data streams, sub-data streams, and grouping flow strategies, and are cached in different cache layers respectively, so that during the operator iteration process, it can be quickly compared to confirm whether it has been iterated, thereby speeding up the processing speed.
[0037] For example, the first operator outputs sub-data stream A1, sub-data stream A2 and sub-data stream A3, and the first operator is connected to the downstream first operator B1, the downstream first operator B2 and the downstream first operator B3. Based on different first operator to downstream first operator grouping flow strategies, sub-data stream A1, sub-data stream A2 and sub-data stream A3 can be sent to the downstream first operator B1, the downstream first operator B2 and the downstream first operator B3 for execution respectively, or sub-data stream A1, sub-data stream A2 and sub-data stream A3 can be sent to the downstream first operator B1 for execution, or the three subtasks in the downstream first operator B1 can process sub-data stream A1 in parallel. The grouping flow strategy can be set based on different requirements, and this embodiment does not make specific limitations.
[0038] In addition, after each first operator of the first code topology of the first cluster is converted into a second operator of the second cluster, the converted second operator is stored in the cache layer. A mainstream cache layer, a shunt cache layer, and a packet stream cache layer can be set separately, the mainstream cache layer is used to store the main data stream of each second operator, the shunt cache layer is used to store the sub-data stream of each second operator, and the packet stream cache layer is used to store a plurality of packet streams divided by each second operator based on the packet stream strategy from the second operator to each downstream second operator.
[0039] like Figure 2 As shown in the figure, the first layer is called the mainstream cache layer. This layer contains the processing logic of each processing node in the second code topology. The nodes generated by the spout operator or bolt operator in the jstorm topology after being converted into the flink operator will be cached in this layer. A "circle" represents a processing node and also a node mainstream. A hash table structure is used to represent the mainStream node mainstream cache layer:
[0040] <mainStreamId,mainStream>
[0041] Among them, the key mainStreamId is a string (equivalent to the name of this mainStream), and the value mainStream is the representation of the mainstream data structure.
[0042] The second layer is called the stream cache layer. A node in the main stream cache layer can output 0 or more streams, and the streams are recorded in this stream cache layer. We use a two-layer nested hash table to record the mapping information of the streams:
[0043] <mainStreamId,<splitStreamId,splitStream> >
[0044] The meaning of the key mainStreamId is the same as that of the mainStreamId in the first layer, and the value<splitStreamId,splitStream> It is also a hash structure. The key splitStreamId is a string identifier, which represents the id of a diversion. It is not globally unique, but the diversion id combined with the mainstream id can uniquely determine the diversion. The value splitStream is the data structure of the diversion.
[0045] The third layer is called the packet flow cache layer. The role of the packet flow is to further subdivide the split flow and determine the packet form in which the data in the split flow is distributed to its downstream nodes. We use a three-layer nested hash structure to record the mapping information of the packet flow:
[0046] <mainStreamId,<splitStreamId,<groupStreamId,groupStream> >
[0047] Among them, mainStreamId and splitStreamId have been explained in the first and second layers, groupStreamId is a string identifier, indicating the id of the split stream, and groupStream is a data structure representation of the grouped stream.
[0048] S103 , traversing the first operators, and when a target first operator among the first operators has not been converted into a second operator of the second cluster, determining an upstream first operator of the target first operator according to the directed edge.
[0049] Specifically, each first operator in the first code topology is transformed into a second operator of the second cluster one by one, and a target first operator among each first operator is used to illustrate the transformation process, and the transformation process of each first operator is the same as the transformation process of the target first operator.
[0050] When converting the first operator, it is generally necessary to convert the first operators of each layer in sequence according to the data flow in the first code topology, that is, it is necessary to convert the downstream operator according to the upstream operator. When the target first operator is not converted into the second operator of the second cluster, that is, the second operator converted by the target first operator is not queried in the cache layer, the upstream first operator of the target first operator is determined according to the directed edge, so as to convert the target first operator according to the upstream first operator.
[0051] In one embodiment, the operator includes a data source operator and a data processing operator, and this step includes: S301, when the target first operator among the first operators has not been converted into the second operator of the second cluster, if the target first operator is determined to be the first data source operator according to the directed edge, the target first operator is converted into the target second operator of the second cluster; S302, when the target first operator among the first operators has not been converted into the second operator of the second cluster, if the target first operator is determined to be the first data processing operator according to the directed edge, the upstream first operator of the target first operator is determined according to the directed edge.
[0052] Specifically, operators include data source operators and data processing operators. The data source operators and data processing operators have different positions and functions in the code topology. The data source operator is used to obtain data streams, and the data processing operator is used to process data streams.
[0053] When the target first operator is not converted into the second operator of the second cluster, the target first operator is determined to be the first data source operator according to the directed edge, that is, the target first operator has no upstream operator, and therefore the target first operator is directly converted into the target second operator of the second cluster.
[0054] When the target first operator is not converted to the second operator of the second cluster, the target first operator is determined to be the first data processing operator according to the directed edge, and the upstream first operator of the target first operator is determined according to the directed edge. The upstream first operator may be the first data source operator or the first data processing operator. When the first cluster is a jstorm cluster, the first data source operator is a spout operator, and the first data processing operator is a bolt operator. When the second cluster is a flink cluster, the second data source operator is a source operator, and the second data processing operator is an operator operator. The spout operator is converted to the source operator, and the bolt operator is converted to the operator operator. Operators of different types are converted accordingly based on the cluster topology structure so that the second cluster after conversion has the same topological logic as the first cluster.
[0055] In one embodiment, when the target first operator among the first operators has not been converted into the second operator of the second cluster, after determining the upstream first operator of the target first operator according to the directed edge, the step also includes: S401, if the upstream first operator has not been converted into the second operator of the second cluster, determining the first operator upstream of the upstream first operator according to the directed edge, until the first operator that has been converted into the second operator of the second cluster is obtained, or, until the obtained first operator is an unconverted first data source operator.
[0056] Specifically, if the upstream first operator is also not converted into the second operator of the second cluster, the first operator upstream of the upstream first operator is determined again according to the directed edge. If the upstream first operator is the first data source operator, that is, the upstream first operator has no upstream operator, the upstream first operator is directly converted into the second operator of the second cluster. If the upstream first operator is the first data processing operator, it is determined again whether the first operator upstream of the upstream first operator is converted into the second operator of the second cluster. And so on, recursively upward along the directed edge until the upstream operator obtained is the first operator that has been converted into the second operator of the second cluster (it can be the first data source operator or the first data processing operator), or the upstream operator obtained is the first data source operator that has not been converted, then the upstream operator obtained is directly converted into the second operator of the second cluster.
[0057] In this embodiment, operators are compared from bottom to top based on the topological structure until the first operator of the second operator of the converted second cluster or the unconverted first data source operator is determined as the starting point of operator conversion, thereby avoiding repeated conversion of the same operator and avoiding missing operators during the conversion process.
[0058] S104: If the upstream second operator of the second cluster transformed by the upstream first operator is obtained, a target second operator of the second cluster transformed by the target first operator is obtained according to the target first operator and the upstream second operator.
[0059] Specifically, if the upstream first operator of the acquired target first operator has an upstream second operator that transforms the second cluster, the main data stream, sub-data stream, and business logic module of the upstream second operator can be obtained. The target second operator of the second cluster transformed by the target first operator is obtained according to the target first operator and the upstream second operator.
[0060] In one embodiment, this step includes: S501, if the upstream second operator of the second cluster transformed by the upstream first operator is obtained, then the sub-data stream of the upstream second operator is obtained; S502, obtaining at least one initial grouping flow strategy of the target first operator, the business logic module of the target first operator and the sub-data stream of the target first operator; S503, according to the at least one initial grouping flow strategy, the business logic module of the target first operator, the sub-data stream of the target first operator and the sub-data stream of the upstream second operator, obtaining the target second operator of the second cluster transformed by the target first operator.
[0061] Specifically, if the acquired upstream first operator of the target first operator has an upstream second operator that transforms the second cluster, where the upstream second operator may be one or more second operators, then the sub-data streams of each upstream second operator are acquired.
[0062] The upstream operator of the target first operator may be one or more. The initial group flow strategy of the target first operator is the strategy for sending the diverted data of the upstream operator of the target first operator. Therefore, the initial group flow strategy is also one or more. By parsing the first code topology, at least one initial group flow strategy of the target first operator, the business logic module of the target first operator, and the diverted data flow of the target first operator are obtained. According to at least one initial group flow strategy, the business logic module of the target first operator, the diverted data flow of the target first operator, and the diverted data flow of the upstream second operator, the target second operator of the second cluster transformed by the target first operator is obtained.
[0063] In one embodiment, step S503, obtaining the target second operator of the second cluster transformed by the target first operator according to the at least one initial group flow strategy, the business logic module of the target first operator, the sub-data flow of the target first operator and the sub-data flow of the upstream second operator, includes: S601, obtaining the target group flow strategy and the target main data flow of the target second operator according to the at least one initial group flow strategy and the sub-data flow of the upstream second operator; S602, obtaining the target sub-data flow of the target second operator according to the main data flow of the target first operator, the sub-data flow of the target first operator and the target main data flow of the target second operator; S603, obtaining the target second operator according to the business logic module of the target first operator, the target main data flow of the target second operator, the target group flow strategy and the target sub-data flow.
[0064] Specifically, after the first operator of the first cluster is converted into the second operator of the second cluster, its main data flow, sub-data flow and business processing logic correspond one to one. The initial grouping flow strategy of the target first operator is the strategy for sending the sub-data of the upstream operator of the target first operator. The main data flow of the target first operator can be obtained based on the initial grouping flow strategy of the target first operator and the sub-data flow of each upstream operator of the target first operator.
[0065] The grouping flow strategy of the target first operator and the grouping flow strategy of the target second operator have the same grouping flow logic, and the allocated sub-data flows have a corresponding relationship. Therefore, the grouping flow strategy of the target second operator is determined according to the grouping flow strategy of the target first operator. For example, the upstream operator of the target first operator includes sub-data flows A1, A2, and A3, and the upstream operator of the target second operator includes sub-data flows B1, B2, and B3. The sub-data flows A1, A2, and A3 correspond to the sub-data flows B1, B2, and B3 in sequence. The grouping flow strategy of the target first operator is to send the sub-data flows A1 and A2 to the target first operator for execution, and the grouping flow strategy of the target second operator is to send the sub-data flows B1 and B2 corresponding to the sub-data flows A1 and A2 to the target second operator for execution.
[0066] The target main data stream of the target second operator is determined according to the sub-data stream of the upstream second operator and the target grouping stream strategy of the target second operator. According to the correspondence between the main data stream and the sub-data stream of the target first operator, the target sub-data stream of the target second operator is obtained in combination with the target main data stream of the target second operator. Finally, the target second operator is obtained according to the business logic module of the target first operator, the target main data stream of the target second operator, the target grouping stream strategy and the target sub-data stream.
[0067] In one embodiment, step S603, obtaining the target second operator according to the business logic module of the target first operator, the target main data flow of the target second operator, the target group flow strategy and the target sub-data flow, includes: S701, obtaining the environmental information of the second cluster; S702, encapsulating the business logic module of the target first operator according to the environmental information to obtain the business logic module of the target second operator; S703, obtaining the target second operator according to the business logic module of the target second operator, the target main data flow of the target second operator, the target group flow strategy and the target sub-data flow.
[0068] Specifically, the environment information of the second cluster includes but is not limited to the submission class environment information of the second cluster, cluster interface information, etc. The business logic module of the target first operator is encapsulated according to the environment information to obtain the business logic module of the target second operator, that is, the framework of the second cluster wraps the business logic module of the target first operator. The target second operator is obtained according to the business logic module of the target second operator, the target main data flow of the target second operator, the target grouping flow strategy and the target sub-data flow. Then the target second operator still processes the target main data flow through the business logic module of the target first operator to obtain the target sub-data flow. The encapsulation of the business logic module of the target first operator in the target second operator is to ensure that the transmission of data flows between the second operators meets the framework requirements of the second cluster.
[0069] S105: Obtain a second code topology of the second cluster according to the target second operator and the directed edge.
[0070] Specifically, each first operator is traversed one by one, and converted into a second operator of the second cluster in the manner of the above embodiment, and then each second operator is connected according to directed edges to finally obtain the second code topology of the second cluster.
[0071] In this embodiment, when traversing to a target first operator that has not been converted to a second operator, the upstream operator is recursively traversed upward according to the directed edge of the first code topology to convert the upstream operator that has not been converted before. That is, when traversing to a certain target first operator, it can be ensured that the first operator upstream of the target first operator is converted, and the first operator that has been processed is no longer repeatedly traversed, thereby reducing the complexity of the algorithm and improving the conversion efficiency.
[0072] like Figure 3 As shown, the embodiment of the present application provides a code topology cross-cluster conversion method. When the first cluster is a jstorm cluster, the first data source operator is a spout operator, the first data processing operator is a bolt operator, and the second cluster is a flink cluster. The second data source operator is a source operator, and the second data processing operator is an operator operator. The spout operator is correspondingly converted to a source operator, and the bolt operator is correspondingly converted to an operator operator, including the following steps:
[0073] S1: Extract the set of bolt operators from jstorm topology (first code topology), recorded as the set {boltMainStreamid};
[0074] S2: Set the bolt operator iterator to point to the set {boltMainStreamid} to check whether all the bolt operators in the set {boltMainStreamid} have been traversed;
[0075] Wherein, step S2 includes sub-steps S21 to S23:
[0076] S21: If the bolt operator iterator is traversed completely, it means that all the first operators in the first code topology are transformed, and the transformation ends;
[0077] S22: If the bolt operator iterator has not traversed all the bolt operators, the bolt operator iterator points to the next bolt operator;
[0078] S23: Take out the main data stream id of the next bolt operator pointed to by the bolt operator iterator: boltMainStreamId;
[0079] S3: Search the main stream cache layer to see whether the main stream (main data stream) corresponding to the next bolt operator exists according to boltMainStreamId;
[0080] Wherein, step S3 includes sub-steps S31 to S32:
[0081] S31: If mainStream already exists, it means that the mainStream corresponding to the next bolt operator has been created, then jump to step [S2];
[0082] S32: If mainStream does not exist, it means that the mainStream corresponding to the next bolt operator has not been created, then extract all upstream operators of the next blot1 operator and input them into the dictionary set of the packet flow strategy of the next blot1 operator, which is recorded as: <mainStreanId,<splitStreamId,grouping> >
[0083] S4: Extract the iterator pointing to the dictionary set of packet flow policies <mainStreanId,<splitStreamId,grouping> >, collection <mainStreanId,<splitStreamId,grouping> >Whether the internal packet flow strategy has been traversed;
[0084] Wherein, step S4 includes sub-steps S41 to S44:
[0085] S41: If the packet flow policy iterator is traversed, it means that the sub-data flows of the upstream operator of the next blotl operator in all the packet flow policies in the dictionary set of the packet flow policy exist in the shunt cache layer, and the packet flows input to the next blotl operator by the upstream operator of the next blotl operator based on the packet flow policies in the dictionary set of the packet flow policy exist in the packet flow cache layer, so the main data flow of the next blotl operator is created according to the packet flows input to the next blotl operator by the upstream operator of the next blotl operator;
[0086] S42: If the dictionary set iterator of the packet flow policy has not been traversed completely, the packet flow policy iterator points to the next packet flow policy;
[0087] S43: Take out the next packet flow policy pointed to by the packet flow policy iterator: (mainStreamId, splitStreamId, groupId) set, extract the main data stream id: mainStreamId, the split data stream id: splitStreamId and the grouping information of the next packet flow policy, wherein the main data stream id and the split data stream id of the next packet flow policy are the data streams of the upstream operator of the next blot1 operator;
[0088] S44: searching in the packet flow cache layer whether a packet flow corresponding to the next packet flow policy exists;
[0089] Wherein, step S44 includes sub-steps S441 to S442:
[0090] S441: If the packet flow corresponding to the next packet flow strategy (the packet flow of the second cluster converted from the packet flow of the upstream operator of the next blot1 operator) exists in the packet flow cache layer, jump to step [S4];
[0091] S442: If the packet flow corresponding to the next packet flow policy does not exist in the packet flow cache layer, search the split data flow corresponding to the packet flow policy in the split flow cache layer;
[0092] Wherein, step S442 includes sub-steps S4421 to S4422:
[0093] S4421: If the sub-data flow corresponding to the grouping flow policy (the sub-data flow of the second cluster converted from the sub-data flow of the upstream operator of the next blot1 operator) exists in the flow distribution cache layer, jump to step [S44226];
[0094] S4422: If the sub-data flow corresponding to the packet flow policy does not exist in the sub-flow cache layer, search for the main data flow corresponding to the sub-data flow in the main flow cache layer;
[0095] Wherein, step S4422 includes sub-steps S44221 to S44226:
[0096] S44221: If the main data stream corresponding to the sub-data stream (the main data stream of the second cluster converted from the main data stream of the upstream operator of the next blot1 operator) exists in the mainstream cache layer, jump to step [S44225];
[0097] S4.4.2.2.2: If the main data stream corresponding to the sub-data stream does not exist in the mainstream cache layer, identify the identifier of the main data stream (the main data stream of the second cluster converted from the main data stream of the upstream operator of the next blot1 operator);
[0098] S44223: If the identifier of the main data stream determines that the upstream operator of the next blot1 operator is the blot1 operator, recursively jump to step [S3];
[0099] S44224: If the identifier of the main data stream determines that the upstream operator of the next blot1 operator is a spout operator, create a main data stream of the source operator of the second cluster according to the main data stream of the spout operator in the first code topology, and store it in the main cache layer;
[0100] S44225: Create a sub-data stream of the second cluster converted by the upstream operator of the next blot1 operator according to the main data stream of the second cluster converted by the upstream operator of the next blot1 operator, and store the converted sub-data stream in the shunt cache layer;
[0101] S44226: According to the data flow and packet flow strategy of the second cluster of the upstream operator of the next blotl operator that has been converted in the diversion cache layer, create a packet flow that the upstream operator of the next blotl operator sends to the second cluster of the next blotl operator, and store the converted packet flow in the packet flow cache layer.
[0102] like Figure 4 As shown, the embodiment of the present application provides a code topology cross-cluster conversion device 900, including:
[0103] An information acquisition module 910 is used to acquire a first code topology of a first cluster to be converted;
[0104] A code parsing module 920, which is in communication with the information acquisition module 910 and is used to parse the first code topology to obtain directed edges between each first operator and each first operator;
[0105] The code conversion module 930 is communicatively connected with the code parsing module 920, and is used for traversing the first operators. When the target first operator among the first operators is not converted into the second operator of the second cluster, the upstream first operator of the target first operator is determined according to the directed edge; if the upstream second operator of the second cluster converted by the upstream first operator is obtained, the target second operator of the second cluster converted by the target first operator is obtained according to the target first operator and the upstream second operator; and the second code topology of the second cluster is obtained according to the target second operator and the directed edge.
[0106] In some embodiments of the present application, the operator includes a data source operator and a data processing operator, and the code conversion module 930 is also used for, when the target first operator among the first operators is not converted into the second operator of the second cluster, if the target first operator is determined to be the first data source operator according to the directed edge, then converting the target first operator into the target second operator of the second cluster; when the target first operator among the first operators is not converted into the second operator of the second cluster, if the target first operator is determined to be the first data processing operator according to the directed edge, then determining the upstream first operator of the target first operator according to the directed edge.
[0107] In some embodiments of the present application, the code conversion module 930 is also used to determine the first operator upstream of the upstream first operator according to the directed edge if the upstream first operator is not converted into the second operator of the second cluster, until the first operator that has converted the second operator of the second cluster is obtained, or until the obtained first operator is an unconverted first data source operator.
[0108] In some embodiments of the present application, the code parsing module 920 is also used to parse the first code topology to obtain the main data flow input to each first operator, the business logic module of each first operator and the directed edges between the first operators; determine the sub-data flow obtained after processing by the first operator according to the main data flow and the business logic module of the first operator; determine the downstream first operator of the first operator according to the directed edge; determine the grouping flow strategy from the first operator to the downstream first operator according to the sub-data flow of the first operator and the main data flow of the downstream first operator, and determine it as the grouping flow strategy of the downstream first operator.
[0109] In some embodiments of the present application, the code conversion module 930 is also used to obtain the sub-data stream of the upstream second operator if the upstream second operator of the second cluster converted by the upstream first operator is obtained; obtain at least one initial grouping flow strategy of the target first operator, the business logic module of the target first operator, and the sub-data stream of the target first operator; and obtain the target second operator of the second cluster converted by the target first operator based on the at least one initial grouping flow strategy, the business logic module of the target first operator, the sub-data stream of the target first operator, and the sub-data stream of the upstream second operator.
[0110] In some embodiments of the present application, the code conversion module 930 is also used to obtain the target packet flow strategy and the target main data flow of the target second operator according to the at least one initial packet flow strategy and the sub-data flow of the upstream second operator; obtain the target sub-data flow of the target second operator according to the main data flow of the target first operator, the sub-data flow of the target first operator and the target main data flow of the target second operator; obtain the target second operator according to the business logic module of the target first operator, the target main data flow of the target second operator, the target packet flow strategy and the target sub-data flow.
[0111] In some embodiments of the present application, the code conversion module 930 is also used to obtain environmental information of the second cluster; encapsulate the business logic module of the target first operator according to the environmental information to obtain the business logic module of the target second operator; obtain the target second operator according to the business logic module of the target second operator, the target main data flow of the target second operator, the target grouping flow strategy and the target sub-data flow.
[0112] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0113] For example, Figure 5 The computer device shown can be Figure 4The information acquisition module 910 in the code topology cross-cluster conversion device 900 shown executes step S101. The computer device can execute step S102 through the code parsing module 920. The computer device can execute steps S103, S104 and S105 through the code conversion module 930. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external computer device through a network connection. When the computer program is executed by the processor, a code topology cross-cluster conversion method is implemented.
[0114] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0115] In some embodiments of the present application, a computer device is provided, comprising one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor in the steps of the above-mentioned code topology cross-cluster conversion method. The steps of the code topology cross-cluster conversion method here may be the steps of the code topology cross-cluster conversion method in the above-mentioned embodiments.
[0116] In some embodiments of the present application, a computer-readable storage medium is provided, which stores a computer program, and the computer program is loaded by a processor, so that the processor executes the steps of the above-mentioned code topology cross-cluster conversion method. Here, the steps of the code topology cross-cluster conversion method can be the steps of the code topology cross-cluster conversion method in each of the above-mentioned embodiments.
[0117] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0118] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0119] The above is a detailed introduction to a code topology cross-cluster conversion method, device, computer equipment and storage medium provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A code topology cross-cluster conversion method, characterized in that: include: Obtain a first code topology of a first cluster to be converted; Parsing the first code topology to obtain directed edges between each first operator and each first operator; Traversing the first operators, when a target first operator among the first operators has not been converted into a second operator of the second cluster, determining an upstream first operator of the target first operator according to the directed edge; If the upstream second operator of the second cluster transformed by the upstream first operator is obtained, then the target second operator of the second cluster transformed by the target first operator is obtained according to the target first operator and the upstream second operator; A second code topology of the second cluster is obtained according to the target second operator and the directed edge.
2. The code topology cross-cluster conversion method according to claim 1, characterized in that: The operator includes a data source operator and a data processing operator, and when the target first operator in each of the first operators is not converted into the second operator of the second cluster, determining the upstream first operator of the target first operator according to the directed edge includes: When the target first operator among the first operators has not been converted into the second operator of the second cluster, if it is determined according to the directed edge that the target first operator is the first data source operator, converting the target first operator into the target second operator of the second cluster; When a target first operator among the first operators is not converted into a second operator of the second cluster, if the target first operator is determined to be the first data processing operator according to the directed edge, an upstream first operator of the target first operator is determined according to the directed edge.
3. The code topology cross-cluster conversion method according to claim 2, characterized in that: When the target first operator among the first operators is not converted into the second operator of the second cluster, after determining the upstream first operator of the target first operator according to the directed edge, the method further includes: If the upstream first operator has not been converted into the second operator of the second cluster, the first operator upstream of the upstream first operator is determined according to the directed edge, until the first operator that has been converted into the second operator of the second cluster is obtained, or until the obtained first operator is an unconverted first data source operator.
4. The code topology cross-cluster conversion method according to claim 1, characterized in that: The parsing the first code topology to obtain directed edges between each first operator and each first operator includes: Parsing the first code topology to obtain a main data stream input to each first operator, a business logic module of each first operator, and directed edges between each first operator; Determine, according to the main data stream and the business logic module of the first operator, a sub-data stream obtained after being processed by the first operator; Determine a first downstream operator of the first operator according to the directed edge; According to the sub-data stream of the first operator and the main data stream of the downstream first operator, a grouping flow strategy from the first operator to the downstream first operator is determined, and is determined as the grouping flow strategy of the downstream first operator.
5. The code topology cross-cluster conversion method according to claim 4, characterized in that: If the upstream second operator of the second cluster transformed by the upstream first operator is obtained, obtaining the target second operator of the second cluster transformed by the target first operator according to the target first operator and the upstream second operator includes: If the upstream second operator of the second cluster transformed by the upstream first operator is obtained, then obtaining the sub-data stream of the upstream second operator; Acquire at least one initial packet flow strategy of the target first operator, a service logic module of the target first operator, and a sub-data flow of the target first operator; A target second operator of the second cluster converted from the target first operator is obtained according to the at least one initial grouping flow strategy, the service logic module of the target first operator, the sub-data flow of the target first operator, and the sub-data flow of the upstream second operator.
6. The code topology cross-cluster conversion method according to claim 5, characterized in that: The step of obtaining the target second operator of the second cluster converted by the target first operator according to the at least one initial grouping flow strategy, the service logic module of the target first operator, the sub-data flow of the target first operator, and the sub-data flow of the upstream second operator includes: Obtaining a target packet flow strategy and a target main data flow of the target second operator according to the at least one initial packet flow strategy and the sub-data flow of the upstream second operator; Obtaining a target sub-data stream of the target second operator according to the main data stream of the target first operator, the sub-data stream of the target first operator, and the target main data stream of the target second operator; The target second operator is obtained according to the business logic module of the target first operator, the target main data flow of the target second operator, the target packet flow strategy and the target sub-data flow.
7. The code topology cross-cluster conversion method according to claim 6, characterized in that: The step of obtaining the target second operator according to the service logic module of the target first operator, the target main data flow of the target second operator, the target packet flow strategy, and the target sub-data flow comprises: Acquire the environment information of the second cluster; Encapsulating the business logic module of the target first operator according to the environment information to obtain the business logic module of the target second operator; The target second operator is obtained according to the service logic module of the target second operator, the target main data flow of the target second operator, the target packet flow strategy and the target sub-data flow.
8. A code topology cross-cluster conversion device, characterized in that: include: An information acquisition module, used to acquire a first code topology of a first cluster to be converted; A code parsing module, which is in communication with the information acquisition module and is used to parse the first code topology to obtain directed edges between each first operator and each first operator; a code conversion module, which is in communication connection with the code parsing module, and is used for traversing the first operators, and when the target first operator among the first operators is not converted into the second operator of the second cluster, determining the upstream first operator of the target first operator according to the directed edge; if the upstream second operator of the second cluster converted by the upstream first operator is obtained, obtaining the target second operator of the second cluster converted by the target first operator according to the target first operator and the upstream second operator; A second code topology of the second cluster is obtained according to the target second operator and the directed edge.
9. A computer device, characterized in that: The computer device comprises: one or more processors; A memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the code topology cross-cluster conversion method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to execute the steps in the code topology cross-cluster conversion method according to any one of claims 1 to 7.