Multi-engine visualization data flow implementation method, device, equipment and storage medium

Abstract

The present application provides a method, device, equipment and computer-readable storage medium for implementing a multi-engine visualized data flow. The multi-engine visualized data flow implementation method includes: obtaining a configuration file and target data; parsing the configuration file to obtain corresponding configuration attribute information; wherein the configuration attribute information at least includes the identification of at least two open source computing engines; according to the identification of at least two open source computing engines, accessing the corresponding at least two open source computing engines; and using at least two open source computing engines to process the target data. According to the embodiments of the present application, the flexibility of data processing can be improved.

Description

Technical Field

[0001] The present application belongs to the technical field of multi-engine visualization data flow, and in particular, relates to a method, device, equipment and computer-readable storage medium for implementing multi-engine visualization data flow. Background Art

[0002] Visual data flow engines generally rely on open source big data computing engines to complete computing tasks. The operation process is as follows: Figure 1 As shown in the figure, the front-end generates the configuration first, and then the back-end passes the configuration to the visual data flow engine. The visual data flow engine performs the following steps in sequence: (1) obtains and parses the configuration; (2) generates an open source computing engine job based on the configuration; (3) submits and manages the open source computing engine job. Finally, the open source computing engine is used to run the job.

[0003] However, the current visualization data flow solution has at least the following disadvantages: it can only access one open source computing engine, but cannot access multiple open source computing engines, which limits the flexibility of data processing. Therefore, how to improve the flexibility of data processing is a technical problem that technicians in this field need to solve urgently. Summary of the invention

[0004] The embodiments of the present application provide a method, device, equipment and computer-readable storage medium for implementing a multi-engine visualized data flow, which can improve the flexibility of data processing.

[0005] In a first aspect, an embodiment of the present application provides a method for implementing a multi-engine visualized data flow, including:

[0006] Get profile and target data;

[0007] Parse the configuration file to obtain corresponding configuration attribute information; wherein the configuration attribute information includes at least two identifiers of open source computing engines;

[0008] According to the identifiers of the at least two open source computing engines, access the corresponding at least two open source computing engines;

[0009] At least two open source computing engines are used to process the target data.

[0010] Furthermore, the configuration attribute information also includes identifications of at least two data structures; after parsing the configuration file to obtain the corresponding configuration attribute information, the method further includes:

[0011] Based on the identifiers of the at least two data structures, determining the corresponding at least two data structures;

[0012] At least two data structures are used to perform data analysis on the target data.

[0013] Furthermore, the configuration attribute information also includes a data flow identifier, a data flow name, and a data flow step;

[0014] Each data flow step includes an input identifier of the step, an output identifier of the step, an identifier of an open source computing engine used in the step, and an identifier of a data structure used in the step;

[0015] The identifier of the open source computing engine used in the step is one of at least two identifiers of the open source computing engine; the identifier of the data structure used in the step is one of at least two identifiers of the data structure.

[0016] Further, using at least two open source computing engines to process the target data and using at least two data structures to analyze the target data include:

[0017] In the first data flow step, the first target data is processed by using the first open source computing engine, and the processed first target data is analyzed by using the first data structure;

[0018] In the second data flow step, the second target data is processed by using a second open source computing engine, and the processed second target data is analyzed by using a second data structure;

[0019] The first open source computing engine and the second open source computing engine are different types of open source computing engines; the first data structure and the second data structure are different types of data structures.

[0020] Furthermore, the first open source computing engine and the second open source computing engine are different types of open source computing engines such as Spark engine and Flink engine.

[0021] Furthermore, the first data structure and the second data structure are different types of data structures such as DataFrame data structure and DataSet data structure.

[0022] Furthermore, obtain the configuration file, including:

[0023] After the front-end device generates the configuration file, pass the configuration file through the back-end device to obtain the configuration file.

[0024] In a second aspect, an embodiment of the present application provides a multi-engine visualization data flow implementation device, including:

[0025] The acquisition module is used to obtain configuration files and target data;

[0026] A parsing module, used to parse the configuration file to obtain corresponding configuration attribute information; wherein the configuration attribute information at least includes identifications of at least two open source computing engines;

[0027] An access module, used to access the corresponding at least two open source computing engines according to the identifiers of the at least two open source computing engines;

[0028] The data processing module is used to process the target data using at least two open source computing engines.

[0029] Furthermore, the configuration attribute information also includes identifications of at least two data structures; and the device also includes:

[0030] A determination module, configured to determine at least two corresponding data structures based on identifiers of at least two data structures;

[0031] The data analysis module is used to perform data analysis on target data using at least two data structures.

[0032] Furthermore, the configuration attribute information also includes a data flow identifier, a data flow name, and a data flow step;

[0033] Each data flow step includes an input identifier of the step, an output identifier of the step, an identifier of an open source computing engine used in the step, and an identifier of a data structure used in the step;

[0034] The identifier of the open source computing engine used in the step is one of at least two identifiers of the open source computing engine; the identifier of the data structure used in the step is one of at least two identifiers of the data structure.

[0035] Furthermore, the data processing module and the data analysis module are used to:

[0036] In the first data flow step, the first target data is processed by using the first open source computing engine, and the processed first target data is analyzed by using the first data structure;

[0037] In the second data flow step, the second target data is processed by using a second open source computing engine, and the processed second target data is analyzed by using a second data structure;

[0038] The first open source computing engine and the second open source computing engine are different types of open source computing engines; the first data structure and the second data structure are different types of data structures.

[0039] Furthermore, the first open source computing engine and the second open source computing engine are different types of open source computing engines such as Spark engine and Flink engine.

[0040] Furthermore, the first data structure and the second data structure are different types of data structures such as DataFrame data structure and DataSet data structure.

[0041] Furthermore, a module is obtained for:

[0042] After the front-end device generates the configuration file, pass the configuration file through the back-end device to obtain the configuration file.

[0043] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a processor and a memory storing computer program instructions;

[0044] When the processor executes the computer program instructions, the multi-engine visualization data flow implementation method shown in the first aspect is implemented.

[0045] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the multi-engine visualization data flow implementation method as shown in the first aspect is implemented.

[0046] The multi-engine visualization data flow implementation method, device, equipment and computer-readable storage medium of the embodiments of the present application can improve the flexibility of data processing.

[0047] The multi-engine visualized data flow implementation method includes: obtaining a configuration file and target data; parsing the configuration file to obtain corresponding configuration attribute information; wherein the configuration attribute information at least includes identifications of at least two open source computing engines; accessing the corresponding at least two open source computing engines according to the identifications of the at least two open source computing engines; and processing the target data using the at least two open source computing engines.

[0048] It can be seen that the method accesses at least two corresponding open source computing engines according to the identifiers of at least two open source computing engines, and uses at least two open source computing engines to process the target data. That is, by integrating multiple open source computing engines, a data stream can use multiple open source computing engines to process data at the same time, which greatly improves the flexibility of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 It is a schematic diagram of the operation process of a visual data flow engine in the prior art;

[0051] Figure 2 It is a flowchart of a method for implementing a multi-engine visual data flow provided by an embodiment of the present application;

[0052] Figure 3 It is a structural diagram of a multi-engine visual data flow implementation device provided by an embodiment of the present application;

[0053] Figure 4 It is a structural diagram of the xx device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0054] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0055] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0056] Visual data flow engines generally rely on open source big data computing engines to complete computing tasks. The operation process is as follows: Figure 1 As shown in the figure, the front-end generates the configuration first, and then the back-end passes the configuration to the visual data flow engine. The visual data flow engine performs the following steps in sequence: (1) obtains and parses the configuration; (2) generates an open source computing engine job based on the configuration; (3) submits and manages the open source computing engine job. Finally, the open source computing engine is used to run the job.

[0057] However, the current visualization data flow solution has at least the following disadvantages: 1) It can only access one open source computing engine, but not multiple open source computing engines, which limits the flexibility of data processing. 2) It can only rely on one data structure for data analysis, but not any data structure, which limits the flexibility of data analysis.

[0058] In order to solve the problems of the prior art, the embodiments of the present application provide a method, device, equipment and computer-readable storage medium for implementing multi-engine visualization data flow. The following first introduces the method for implementing multi-engine visualization data flow provided by the embodiments of the present application.

[0059] Figure 2 FIG. 1 is a flow chart showing a method for implementing a multi-engine visualization data flow according to an embodiment of the present application. Figure 2 As shown, the multi-engine visualization data flow implementation method includes:

[0060] S201, obtaining configuration files and target data;

[0061] In one embodiment, obtaining a configuration file includes:

[0062] After the front-end device generates the configuration file, pass the configuration file through the back-end device to obtain the configuration file.

[0063] Users can generate configuration files on the front-end device according to user needs, and then transmit the configuration files through the back-end device.

[0064] S202, parsing the configuration file to obtain corresponding configuration attribute information; wherein the configuration attribute information at least includes identifications of at least two open source computing engines;

[0065] S203. Accessing at least two corresponding open source computing engines according to the identifiers of at least two open source computing engines;

[0066] S204: Process the target data using at least two open source computing engines.

[0067] It can be seen that the method accesses at least two corresponding open source computing engines according to the identifiers of at least two open source computing engines, and uses at least two open source computing engines to process the target data. That is, by integrating multiple open source computing engines, a data stream can use multiple open source computing engines to process data at the same time, which greatly improves the flexibility of data processing.

[0068] As can be seen from the above, the current visualization data flow solution still has at least the following shortcomings: data analysis can only be performed based on one data structure, and data analysis cannot be performed with any data structure, which limits the flexibility of data analysis.

[0069] In order to improve the flexibility of data analysis, in one embodiment, the configuration attribute information further includes identifications of at least two data structures; after parsing the configuration file to obtain the corresponding configuration attribute information, the method further includes:

[0070] Based on the identifiers of the at least two data structures, determining the corresponding at least two data structures;

[0071] At least two data structures are used to perform data analysis on the target data.

[0072] This embodiment uses at least two data structures to perform data analysis on target data, which can improve the flexibility of data analysis.

[0073] Further, in one embodiment, the configuration attribute information also includes a data flow identifier, a data flow name, and a data flow step;

[0074] Each data flow step includes an input identifier of the step, an output identifier of the step, an identifier of an open source computing engine used in the step, and an identifier of a data structure used in the step;

[0075] The identifier of the open source computing engine used in the step is one of at least two identifiers of the open source computing engine; the identifier of the data structure used in the step is one of at least two identifiers of the data structure.

[0076] In order to improve the flexibility of data processing and data analysis at the same time, in one embodiment, at least two open source computing engines are used to process the target data and at least two data structures are used to analyze the target data, including:

[0077] In the first data flow step, the first target data is processed by using the first open source computing engine, and the processed first target data is analyzed by using the first data structure;

[0078] In the second data flow step, the second target data is processed by using a second open source computing engine, and the processed second target data is analyzed by using a second data structure;

[0079] The first open source computing engine and the second open source computing engine are different types of open source computing engines; the first data structure and the second data structure are different types of data structures.

[0080] Each data flow step in this embodiment uses a corresponding open source computing engine and data structure. Moreover, different open source computing engines and data structures are used between different data flow steps, so the flexibility of data processing and data analysis can be improved at the same time.

[0081] In order to further improve the flexibility of data processing, in one embodiment, the first open source computing engine and the second open source computing engine are different types of open source computing engines such as Spark engine and Flink engine.

[0082] The Spark engine is an open source cluster computing environment similar to Hadoop, but there are some differences between the two. These useful differences make the Spark engine superior in certain workloads. In other words, the Spark engine enables memory distributed data sets. In addition to being able to provide interactive queries, it can also optimize iterative workloads. The Spark engine is implemented in the Scala language, and it uses Scala as its application framework. Unlike Hadoop, the Spark engine and Scala are tightly integrated, and Scala can operate distributed data sets as easily as local collection objects. Although the Spark engine was created to support iterative jobs on distributed data sets, it is actually a complement to Hadoop and can run in parallel in the Hadoop file system.

[0083] This behavior is supported by a third-party cluster framework called Mesos. Spark engine, which can be used to build large-scale, low-latency data analysis applications.

[0084] The Spark engine has three main features: First, the high-level API strips away the focus on the cluster itself, and Spark engine application developers can focus on the calculations that the application needs to do.

[0085] Secondly, the Spark engine is very fast and supports interactive computing and complex algorithms.

[0086] Finally, the Spark engine is a general engine that can be used to perform a variety of operations, including SQL queries, text processing, machine learning, etc. Before the emergence of the Spark engine, we generally needed to learn a variety of engines to handle these requirements separately.

[0087] Performance characteristics of the Spark engine:

[0088] (1) Faster speed:

[0089] In-memory computing, the Spark engine is 100 times faster than Hadoop.

[0090] (2) Ease of use:

[0091] The Spark engine provides more than 80 high-level operators.

[0092] (3) Versatility:

[0093] Spark provides a large number of libraries, including SQL, DataFrames, MLlib, GraphX, and Spark Streaming. Developers can seamlessly combine these libraries in the same application.

[0094] (4)Support multiple resource managers:

[0095] The Spark engine supports Hadoop YARN, Apache Mesos, and its own independent cluster manager Spark engine ecosystem.

[0096] Features of the Flink engine:

[0097] (1)Support high throughput, low latency, and high performance simultaneously;

[0098] (2) Support the concept of event time:

[0099] Most window calculations use system time (Process Time), which is the current time of the system host when the event is transmitted to the computing framework. The Flink engine can support window calculations based on the semantics of event time (Event Time), that is, the time when the event is generated. This time-driven mechanism allows the stream system to calculate accurate results even if the events arrive out of order, maintaining the original time sequence of the time. Try to avoid the influence of network transmission or hardware system.

[0100] (3) Support stateful computing:

[0101] The Flink engine implements state management. The so-called state means that the intermediate result data of the operator is stored in the memory or file system during the streaming computing process. When the operator is entered next time, the current result can be calculated from the intermediate result of the previous state, so there is no need to calculate the results based on all the original data every time. This method greatly improves the performance of the system and reduces the consumption of resources in the computing process. For streaming computing scenarios with large data volumes and very complex computing logic, state plays a very important role.

[0102] (4) Support highly flexible window operations:

[0103] In stream processing applications, data is continuous and needs to be aggregated and calculated within a certain range through windows. For example, to count how many users clicked on a web page in the past minute, we must define a window to collect data from the last minute and perform statistical calculations on the data in the window. The Flink engine divides windows into window operations based on time, count, session, and data-driven. Windows can be customized with flexible trigger conditions to support complex stream transmission modes. Users can define different window trigger mechanisms to meet different needs.

[0104] (5) Fault tolerance based on lightweight distributed snapshots (CheckPoint):

[0105] The Flink engine can run distributedly on thousands of nodes, breaking down the process of a large computing task into small computing processes, and then distributing the tasks to parallel nodes for processing. During the execution of tasks, it can automatically discover errors in the event processing process that lead to data inconsistency. For example: node downtime, network transmission problems, or the restart of computing services due to user upgrades or repairs. In these cases, the state information of the execution process is persistently stored through CheckPoints based on distributed snapshot technology. Once the task stops abnormally, the Flink engine can automatically recover the task from the Checkpoint to ensure the consistency of data during the processing process (Exactly-ONCE).

[0106] (6) Independent memory management based on JVM:

[0107] Memory management is an important consideration for all computing frameworks. Especially for computing scenarios with large amounts of computing, how to manage data in memory is crucial. For memory management, the Flink engine implements its own memory management mechanism to minimize the impact of JVM GC on the system. In addition, the Flink engine converts all data objects into binary data for storage in memory through serialization / deserialization methods, reducing the risk of performance degradation or task exceptions caused by GC. Therefore, the Flink engine is more stable than other distributed processing frameworks and will not affect the operation of the entire application due to problems such as JVM GC.

[0108] (5)Save Point:

[0109] For streaming applications that run 24 / 7, data sources are continuously connected. Termination of applications within a period of time may lead to data loss or inaccurate calculation results, such as cluster version upgrades, downtime operation and maintenance operations, etc. It is worth mentioning that the Flink engine saves snapshots of task execution on storage media through Save Points technology. When the task is restarted, it can directly restore the original calculation state from the saved Save Points, so that the task continues to run according to the state before the shutdown. The Save Point technology allows users to better manage and maintain.

[0110] Based on the characteristics and performance of the above-mentioned Spark engine and Flink engine, the flexibility of data processing can be further improved.

[0111] In order to further improve the flexibility of data analysis, in one embodiment, the first data structure and the second data structure are different types of data structures such as DataFrame data structure and DataSet data structure.

[0112] To support the processing of structured data, Spark SQL provides a new data structure, DataFrame. DataFrame is a dataset consisting of named columns. It is conceptually equivalent to a table in a relational database or a data frame in R / Python. Since Spark SQL supports development in multiple languages, each language defines the abstraction of DataFrame.

[0113] DataFrame has a clear Scheme structure inside, that is, the column names and column field types are known. This has the benefit of reducing data reading and better optimizing the execution plan, thereby ensuring query efficiency.

[0114] Dataset is also a distributed data set. It was introduced in Spark 1.6. It integrates the advantages of RDD and DataFrame, has strong typing characteristics, and supports Lambda functions, but can only be used in Scala and Java. After Spark 2.0, in order to facilitate developers, Spark merged the APIs of DataFrame and Dataset and provided a structured API (StructuredAPI), that is, users can complete operations on both through a set of standard APIs.

[0115] Based on the characteristics and performance of the above-mentioned DataFrame data structure and DataSet data structure, the flexibility of data analysis can be further improved.

[0116] Specifically, in one embodiment, a visualization data stream is described by a configuration file. The following JSON string describes a visualization data stream:

[0117]

[0118]

[0119] Field Description: flowId is the data flow identifier, name is the data flow name, steps is the data flow step, each step has 5 attributes, stepId is the step identifier, inputLanes is the input identifier of the step, outputLanes is the output identifier of the step, engine is the identifier of the open source engine used in the step, and datastructure is the data structure identifier of the step.

[0120] Engine and datastructure are unique attributes of this application. These two attributes are used to specify the engine and data structure used in this step. The visual data flow engine decides which engine and data structure to use to run the logic contained in this step based on these two attributes. The above data flow can use two open source computing engines and two data structures at the same time. Step 1 will use the Spark engine for data processing and the DataFrame data structure for data analysis. Step 2 will use the Flink engine for data processing and the DataSet data structure for data analysis.

[0121] This embodiment integrates multiple open source computing engines, and one data stream can use multiple open source computing engines to process data at the same time, which greatly improves the flexibility of data processing. It allows developers to perform data analysis with any data structure, and one data stream can use multiple data structures to analyze data, which greatly improves the flexibility of data analysis.

[0122] Figure 3 : is a schematic diagram of a structure of a multi-engine visualized data flow implementation device provided by an embodiment of the present application, the multi-engine visualized data flow implementation device comprising:

[0123] An acquisition module 301 is used to acquire configuration files and target data;

[0124] The parsing module 302 is used to parse the configuration file to obtain corresponding configuration attribute information; wherein the configuration attribute information at least includes identifications of at least two open source computing engines;

[0125] An access module 303, configured to access the corresponding at least two open source computing engines according to the identifiers of the at least two open source computing engines;

[0126] The data processing module 304 is used to process the target data using at least two open source computing engines.

[0127] In one embodiment, the configuration attribute information further includes identifications of at least two data structures; and the apparatus further includes:

[0128] A determination module, configured to determine at least two corresponding data structures based on identifiers of at least two data structures;

[0129] The data analysis module is used to perform data analysis on target data using at least two data structures.

[0130] In one embodiment, the configuration attribute information also includes a data flow identifier, a data flow name, and a data flow step;

[0131] Each data flow step includes an input identifier of the step, an output identifier of the step, an identifier of an open source computing engine used in the step, and an identifier of a data structure used in the step;

[0132] The identifier of the open source computing engine used in the step is one of at least two identifiers of the open source computing engine; the identifier of the data structure used in the step is one of at least two identifiers of the data structure.

[0133] In one embodiment, the data processing module 304 and the data analysis module are used to:

[0134] In the first data flow step, the first target data is processed by using the first open source computing engine, and the processed first target data is analyzed by using the first data structure;

[0135] In the second data flow step, the second target data is processed by using a second open source computing engine, and the processed second target data is analyzed by using a second data structure;

[0136] The first open source computing engine and the second open source computing engine are different types of open source computing engines; the first data structure and the second data structure are different types of data structures.

[0137] In one embodiment, the first open source computing engine and the second open source computing engine are different types of open source computing engines such as Spark engine and Flink engine.

[0138] In one embodiment, the first data structure and the second data structure are different types of data structures such as DataFrame data structure and DataSet data structure.

[0139] In one embodiment, the acquisition module 301 is used to:

[0140] After the front-end device generates the configuration file, pass the configuration file through the back-end device to obtain the configuration file.

[0141] Figure 3 Each module in the device shown has the function of realizing Figure 2 The functions of each step in the process can achieve the corresponding technical effects, which will not be described in detail here for the sake of brevity.

[0142] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown.

[0143] The electronic device may include a processor 401 and a memory 402 storing computer program instructions.

[0144] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0145] Memory 402 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, memory 402 may include a removable or non-removable (or fixed) medium. In appropriate cases, memory 402 may be inside or outside of an electronic device. In a particular embodiment, memory 402 may be a non-volatile solid-state memory.

[0146] In one embodiment, the memory 402 may be a read-only memory (ROM). In one embodiment, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0147] The processor 401 implements any one of the multi-engine visualization data flow implementation methods in the above embodiments by reading and executing computer program instructions stored in the memory 402 .

[0148] In one example, the electronic device may further include a communication interface 403 and a bus 410. Figure 4 As shown, the processor 401, the memory 402, and the communication interface 403 are connected via a bus 410 and communicate with each other.

[0149] The communication interface 403 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0150] Bus 410 includes hardware, software or both, and the parts of electronic equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 410 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0151] In addition, in combination with the multi-engine visualization data flow implementation method in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any multi-engine visualization data flow implementation method in the above embodiment is implemented.

[0152] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0153] The functional modules shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable media" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0154] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0155] The above reference is according to the method of the embodiment of the present application, the flow chart of the device (system) and the computer program product and / or the block diagram described various aspects of the present application.It should be understood that each square box in the flow chart and / or the block diagram and the combination of each square box in the flow chart and / or the block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more square boxes of the flow chart and / or the block diagram.Such a processor can be but is not limited to a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit.It can also be understood that each square box in the block diagram and / or the flow chart and the combination of the square boxes in the block diagram and / or the flow chart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and computer instructions.

[0156] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A method for realizing multi-engine visualization data flow, characterized in that: include: Get configuration files and target data; Parse the configuration file to obtain corresponding configuration attribute information; wherein the configuration attribute information at least includes identifiers of at least two open source computing engines, identifiers of at least two data structures, a data flow identifier, a data flow name, and a data flow step; each of the data flow steps includes an input identifier of the step, an output identifier of the step, an identifier of the open source computing engine used in the step, and an identifier of the data structure used in the step; the identifier of the open source computing engine used in the step is one of the identifiers of the at least two open source computing engines; the identifier of the data structure used in the step is one of the identifiers of the at least two data structures; the identifiers of the open source computing engines used in the steps of different data flow steps and the identifiers of the data structures used in the steps are different; Based on the identifiers of the at least two data structures, determining at least two corresponding data structures; Performing data analysis on the target data using the at least two data structures; Accessing the corresponding at least two open source computing engines according to the identifiers of the at least two open source computing engines; The target data is processed using the at least two open source computing engines.

2. The method for realizing multi-engine visualization data flow according to claim 1, characterized in that: The using the at least two open source computing engines to process the target data and the using the at least two data structures to analyze the target data include: In the first data flow step, the first target data is processed by using the first open source computing engine, and the processed first target data is analyzed by using the first data structure; In the second data flow step, the second target data is processed by using a second open source computing engine, and the processed second target data is analyzed by using a second data structure; The first open source computing engine and the second open source computing engine are different types of open source computing engines; the first data structure and the second data structure are different types of data structures.

3. The method for realizing multi-engine visualization data flow according to claim 2, characterized in that: The first open source computing engine and the second open source computing engine are different types of open source computing engines such as Spark engine and Flink engine.

4. The method for realizing multi-engine visualization data flow according to claim 2, characterized in that: The first data structure and the second data structure are different types of data structures such as DataFrame data structure and DataSet data structure.

5. The method for realizing multi-engine visualization data flow according to any one of claims 1 to 4, characterized in that: Obtaining the configuration file includes: After the front-end device generates the configuration file, the configuration file is transmitted through the back-end device to obtain the configuration file.

6. A multi-engine visual data flow implementation device, characterized in that: include: The acquisition module is used to obtain configuration files and target data; A parsing module, used to parse the configuration file to obtain corresponding configuration attribute information; wherein the configuration attribute information at least includes identifications of at least two open source computing engines, identifications of at least two data structures, a data flow identification, a data flow name, and at least two data flow steps; each of the data flow steps includes an input identification of the step, an output identification of the step, an identification of the open source computing engine used in the step, and an identification of the data structure used in the step; the identification of the open source computing engine used in the step is one of the identifications of the at least two open source computing engines; the identification of the data structure used in the step is one of the identifications of the at least two data structures; the identifications of the open source computing engines used in different data flow steps and the identifications of the data structures used are different; A determination module, configured to determine at least two corresponding data structures based on the identifiers of the at least two data structures; A data analysis module, used for performing data analysis on target data using the at least two data structures; An access module, used to access the corresponding at least two open source computing engines according to the identifiers of the at least two open source computing engines; A data processing module is used to process the target data using the at least two open source computing engines.

7. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the multi-engine visualization data flow implementation method as described in any one of claims 1-5 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the multi-engine visualization data flow implementation method according to any one of claims 1 to 5 is implemented.

Images