A construction method of a multi-mode data generator based on Flink

Abstract

The present invention discloses a construction method of a multi-mode data generator based on Flink, which includes the following steps: (1) constructing a multi-mode data dictionary; (2) constructing a multi-mode data generator; (3) configuring and deploying the multi-mode data generator based on Apache Flink; (4) implementing monitoring of the resource usage status of the multi-mode data generator. The multi-mode data generator based on Flink disclosed by the present invention can generate data including multiple models such as relational, key-value, graph, Json, and XML. At the same time, users can generate data of the required size according to their own needs. The multi-mode data generator is implemented and deployed based on the Flink platform, utilizes the parallel processing and active memory management mechanisms of Flink to achieve its high efficiency and scalability, and also integrates Prometheus to perform more fine-grained resource utilization.

Description

Technical Field

[0001] The present invention relates to the technical field of software function measurement and database benchmarking, and is a method for constructing a multi-model data generator based on Flink. Background Art

[0002] In the big data era, researchers have paid increasing attention to the processing and utilization of data, and various big data tools and platforms have emerged and developed continuously. Currently, an urgent big data problem to be solved is how to conduct benchmarking on numerous big data tools and platforms using large-scale multi-model data, which is crucial for the selection and optimization of different big data tools and platforms. However, in the real world, it is very difficult to obtain large-scale and real multi-model data. First, due to the confidentiality of proprietary systems and the privacy of commercial contracts and records, most business data owners are reluctant to share their real data. Second, in terms of the inherent scarcity of multi-model data, it is difficult to obtain a representative data set containing various types (structured, semi-structured or unstructured). Finally, if large-scale multi-model data is obtained through the Internet or large data sets are obtained through intelligent devices, it will require high cost.

[0003] To solve the above problems, a practical and feasible method is to use small-scale and real multi-model data as seed data, and generate the required data set by constructing a multi-model data generator. From the current research and application status in the industry, the existing data generators are not complete, often only including one data mode, lacking generators focusing on generating multi-model big data, and there is also room for research in the parallel computing ability and monitoring mechanism of multi-model data generation. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for constructing a multi-model data generator based on Apache Flink, which can use a small-scale real data set as seed data, and utilize the parallel processing and active memory management mechanisms of Apache Flink to quickly generate large-scale multi-model synthetic data within linear total time.

[0005] To solve the above technical problems, the present invention provides a method for constructing a multi-model data generator based on Apache Flink, including the following steps:

[0006] (1) Construct a multi-model data dictionary;

[0007] (2) Construct a multi-model data generator;

[0008] (3) Configure and deploy the multi-model data generator based on Apache Flink;

[0009] (4) Implement resource usage monitoring of multi-modal data generators.

[0010] Preferably, in step (1), the process of constructing a multimodal data dictionary can be described as: obtaining multimodal source data (Customer, Vendor, RegUser, Product, Order, Invoice, Review) from Amazon, LDBC and DBpedia. The data set originates from the social business scenario (social network combined with e-commerce environment) simulated by UniBench (http: / / udbms.cs.helsinki.fi / ?projects / ubench). Among them, Customer, Vendor, and RegUser are relational data types, representing customers, suppliers, and registered users; Product and Order are Json data types, representing product and order information; Invoice represents invoice information and is in XML format; Review represents customer feedback and is Key-value type data; SocialNetwork represents social network data, and its data type is graph data. Data preprocessing is performed on the above data to remove null value data and erroneous data, and the association relationship between the data is further checked to construct a multimodal data dictionary, which constitutes the seed data of the multimodal data set. The data structure of the multimodal data dictionary is listed in Table 1 below. The data dictionary can be further extended, such as adding customer interest data and supplier location information data.

[0011] Table 1 Data structure of multi-mode data dictionary

[0012]

[0013] Preferably, in step (2), the steps of constructing a multimodal data generator are as follows:

[0014] (21) Setting the multi-mode data generator user parameter λ. The user can adjust the size of λ according to the data size to control the data size generated by the data generator, with GB as the unit of measurement;

[0015] (22) Based on the multi-mode data dictionary constructed above, a Java program is implemented to generate various data (Customer, Vendor, RegUser, Product, Order, Invoice, Review) of fixed size using a pseudo-random data generation algorithm. The size of various data (except graph data) is proportional to λ;

[0016] (23) Implement a Java program to traverse randomly generated Product and Order data based on the map operator of Apache Flink and convert it into the Json data format;

[0017] (24) Implement a Java program to traverse randomly generated Invoice data based on the map operator of Apache Flink and convert it into the XML data format;

[0018] (25) Implement a Java program to perform multi-table and multi-relationship queries on randomly generated Customer and RegUser data and generate the required SocialNetwork graph data based on the Table&SQL API of Apache Flink;

[0019] (26) Write a Java program to implement multiple data storage interfaces (HDFS, multi-model database, and local disk or hard drive) for data storage and reading.

[0020] Preferably, in step (3), construct a multi-model data generator module based on Flink:

[0021] (31) Package the runnable program of the multi-model data generator into a jar package;

[0022] (32) Use 3 machines to build a Flink cluster, enter the bin directory of Flink in the client, and input. / start-cluster to successfully start the Flink cluster;

[0023] (33) Submit the jar package packaged in step (31) to the Flink cluster from the client or the Flink Web UI. When submitting this jar package, set the user parameter λ. Table 2 shows the relationship between the parameter λ, data size, and consumption time.

[0024] Table 2 Relationship table of parameter λ, data size, and consumption time

[0025]

[0026] (4) Construct a resource utilization monitoring module for the multi-model data generator:

[0027] (41) Integrate Prometheus, configure and enable Prometheus in the client;

[0028] (42) Enter the two query conditions of "100–(avg by(instance)(irate(node_cpu{instance="xxx",mode="idle"}[5s]))*100)" and "((node_memory_MemTotal_bytes-node_memory_Buffers_bytes-node_memory_Cached_bytes-node_memory_MemFree_bytes-node_memory_Slab_bytes) / node_memory_MemTotal_bytes)*100" in the Prometheus Web UI to obtain the CPU and memory utilization rates of this data generator, so as to optimize the performance of the multi-mode data generator in the follow-up.

[0029] The beneficial effects of the present invention are as follows: Aiming at the problems of single data mode of the existing data generator and difficulty in obtaining large-scale multi-mode data, this data generator can generate various model data, and at the same time, users can generate data of a fixed size according to their own needs. This multi-mode data generator is implemented based on the Flink platform and utilizes the parallel processing and active memory management mechanisms of Flink to achieve its high efficiency and scalability; this data generator also integrates Prometheus to perform more fine-grained resource utilization. Brief Description of the Drawings

[0030] Figure 1 It is a schematic diagram of the method flow of the present invention. Detailed Embodiments

[0031] A construction method of a multi-mode data generator based on Flink, as Figure 1 shown, mainly includes the following steps:

[0032] (1) Construct a data dictionary module:

[0033] Multimodal source data (Customer, Vendor, RegUser, Product, Order, Invoice, Review) are obtained from Amazon, LDBC and DBpedia. This dataset is derived from the social business scenario (social network combined with e-commerce environment) simulated by UniBench (http: / / udbms.cs.helsinki.fi / ?projects / ubench). Among them, Customer, Vendor, and RegUser are relational data types, representing customers, suppliers, and registered users; Product and Order are Json data types, representing product and order information; Invoice represents invoice information and is in XML format; Review represents customer feedback and is a Key-value type of data; SocialNetwork represents social network data, and its data type is graph data. Data preprocessing is performed on the above data to remove null value data and error data, and further check the association between data to construct a multimodal data dictionary, which constitutes the seed data of the multimodal dataset. The data structure of the multimodal data dictionary is listed in Table 1 below. The data dictionary can be expanded, such as adding customer interest data and supplier location information data.

[0034] Table 1 Data structure of multi-mode data dictionary

[0035]

[0036]

[0037] (2) Constructing a multi-modal data generator module:

[0038] (21) Setting the multi-mode data generator user parameter λ. The user can adjust the size of λ according to the data size to control the data size generated by the data generator, with GB as the unit of measurement;

[0039] (22) Based on the multi-mode data dictionary constructed above, a Java program is implemented to generate various data (Customer, Vendor, RegUser, Product, Order, Invoice, Review) of fixed size using a pseudo-random data generation algorithm. The size of various data (except graph data) is proportional to λ;

[0040] (23) Implement a Java program based on Apache Flink's map operator to traverse the randomly generated Product and Order data and convert them into Json data format;

[0041] (24) Implement a Java program to traverse randomly generated Invoice data based on the map operator of Apache Flink and convert it into XML data format;

[0042] (25) Implement a Java program to perform multi-table and multi-relationship queries on randomly generated Customer and RegUser data and generate the required SocialNetwork graph data based on the Table&SQL API of Apache Flink;

[0043] (26) Write a Java program to implement multiple data storage interfaces (HDFS, multi-model database, and local disk or hard drive) for data storage and reading.

[0044] (3) Build a multi-model data generator module based on Flink:

[0045] Build a multi-model data generator module based on Flink:

[0046] (31) Package the runnable program of the multi-model data generator into a jar package;

[0047] (32) Use 3 machines to build a Flink cluster. Enter the bin directory of Flink in the client and input. / start-cluster to successfully start the Flink cluster;

[0048] (33) Submit the jar package packaged in step (31) to the Flink cluster from the client or the Flink Web UI. Set the user parameter λ when submitting this jar package. Table 2 shows the relationship between the parameter λ, data size, and consumption time.

[0049] Table 2 Relationship table of parameter λ, data size, and consumption time

[0050]

[0051]

[0052] (4) Build a resource utilization monitoring module for the multi-model data generator:

[0053] (41) Integrate Prometheus, configure and enable Prometheus in the client;

[0054] Enter the two query conditions of "100–(avg by(instance)(irate(node_cpu{instance=\"xxx\",mode=\"idle\"}[5s]))*100)" and "((node_memory_MemTotal_bytes-node_memory_Buffers_bytes-node_memory_Cached_bytes-node_memory_MemFree_bytes-node_memory_Slab_bytes) / node_memory_MemTotal_bytes)*100" in the Prometheus Web UI to obtain the CPU and memory utilization rates of this data generator, so as to optimize the performance of the multi-mode data generator subsequently.

Claims

1. A construction method of a multi-mode data generator based on Flink, characterized in that The steps include: (1) Construct a multimodal data dictionary; obtain multimodal source data Customer, Vendor, RegUser, Product, Order, Invoice, and Review from Amazon, LDBC, and DBpedia. The data set is derived from the social business scenario simulated by UniBench. Among them, Customer, Vendor, and RegUser are relational data types, representing customers, suppliers, and registered users; Product and Order are Json data types, representing product and order information; Invoice represents invoice information and is in XML format; Review represents customer feedback and is a Key-value type of data; SocialNetwork represents social network data and its data type is graph data; perform data preprocessing on the above data, remove null value data and error data, and further check the association relationship between data objects, so as to construct a multimodal data dictionary and constitute the seed data of the multimodal data set. The data structure of the multimodal data dictionary is listed in Table 1 below. The data dictionary is further expanded; Table 1 Data structure of multi-mode data dictionary (2) Construct a multi-modal data generator; specifically, the following steps are included: (21) Setting the multi-mode data generator user parameter λ. The user adjusts the value of λ according to the data size to control the data size generated by the data generator, with GB as the unit of measurement; (22) Based on the multi-mode data dictionary constructed above, implement a Java program and use a pseudo-random data generation algorithm to generate various data of fixed size Customer, Vendor, RegUser, Product, Order, Invoice, and Review. The size of each data is proportional to λ; (23) Implement a Java program based on Apache Flink's map operator to traverse the randomly generated Product and Order data and convert them into Json data format; (24) Implement a Java program based on Apache Flink's map operator to traverse the randomly generated Invoice data and convert it into XML data format; (25) Implement a Java program to perform multi-table and multi-relationship queries on randomly generated Customer and RegUser data, and generate the required SocialNetwork graph data based on Apache Flink's Table&SQLAPI; (26) Write Java programs to implement various data storage interfaces for data storage and reading; (3) Configure and deploy a multi-mode data generator based on Apache Flink; (4) Monitor the resource usage of the multimodal data generator.

2. The construction method of the multi-mode data generator based on Flink according to claim 1, characterized in that, In step (3), configuring and deploying a multi-mode data generator based on Apache Flink specifically includes the following steps: (31) Package the program run by the multi-mode data generator into a jar package; (32)Successfully start the Flink cluster on the client; (33)Submit the jar package packed in step (31) to the Flink cluster from the client or the Flink Web UI.

3. The construction method of the multi-mode data generator based on Flink according to claim 2, wherein, In step (31), when implementing the Java functions of the multi-modal data generator, the map operator and Table&SQL API of Apache Flink are used to generate multi-modal data.

4. The construction method of the multi-mode data generator based on Flink according to claim 1, characterized in that In step (4), monitoring the resource utilization of the multi-modal data generator specifically includes the following steps: (41)Integrate Prometheus, configure and enable Prometheus on the client; (42)Enter "100–(avg by(instance)(irate(node_cpu {instance="xxx",mode="idle"}[5s]))*100)” and "((node_memory_MemTotal_bytes -node_memory_Buffers_bytes-node_memory_Cached_bytes -node_memory_MemFree_bytes-node_memory_Slab_bytes) / node_memory_MemTotal_bytes)*100” in the Prometheus Web UI to obtain the CPU and memory utilization rates of the data generator, so as to optimize the performance of the multi-modal data generator subsequently.

Images