Json file data synchronization method and device, equipment and storage medium

Abstract

The invention discloses a json file data synchronization method and device, equipment and a storage medium, and relates to the technical field of data processing.The json file data synchronization method comprises the steps that connection information of a database to be processed is maintained in a global configuration file; according to the to-be-processed database table name, a corresponding Hive table name is generated according to a layering rule, and a table building statement is generated according to the to-be-processed database type, the metadata information and the mapping relation of the corresponding Hive field type; establishing a Hive table according to the Hive table name and the table establishment statement; and generating a json file according to the file format requirement of the DataX and the connection information and the metadata information of the database to be processed, calling a synchronization command of the DataX, and finishing data synchronization by using the json file. According to the method, the existing data synchronization development steps can be simplified, and the automatic data synchronization function is realized.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a JSON file data synchronization method, device, equipment and storage medium. Background Art

[0002] DataX is a framework for data synchronization that can synchronize data from different data sources. DataX abstracts the Reader plug-in that reads data from the source data source and the Writer plug-in that writes data to the target end, and is committed to achieving stable and efficient data synchronization between various heterogeneous data sources including relational databases (MySQL, Oracle, etc.), HDFS, Hive, ODPS, HBase, FTP, etc.

[0003] DataX synchronization requires the maintenance of a json file, which contains the configuration of source data and target data, column field mapping and other relationships. To achieve json file data synchronization, Apache Hive, HiveServer2 and data stratification are usually used.

[0004] Apache Hive and HiveServer2 are open source tools for data warehousing and data query. Apache Hive provides a query and analysis tool for large-scale data processing, while HiveServer2 is a remote service of Apache Hive that allows access to Apache Hive functions over the network.

[0005] Data stratification is an important strategy for managing data in the field of big data. By stratifying data, data can be better organized and managed, making it easier to query, analyze and use. In data stratification, the ODS (Operational Data Store) layer is the first layer, which is usually used to store raw data obtained from business systems.

[0006] In order to complete the ods layer data synchronization processing, the usual development steps are:

[0007] 1. Complete Hive table design and manually create tables in the big data warehouse;

[0008] 2. Manually fill in the connection between the source database and the target data platform;

[0009] 3. Manually fill in the field mapping of source data and target data;

[0010] 4. Manually fill in the target data storage path, storage format and other information;

[0011] 5. Create a json file with the above information in the format required by DataX;

[0012] 6. Manually execute the DataX command to complete the synchronization of business database data to Hive database table data;

[0013] However, the above development process is rather cumbersome, and data synchronization is also complex and difficult, which seriously affects development efficiency and work efficiency. Summary of the invention

[0014] The present application provides a JSON file data synchronization method, device, equipment and storage medium, which can simplify the existing data synchronization development steps and realize the function of automated data synchronization.

[0015] In a first aspect, an embodiment of the present application provides a JSON file data synchronization method, the JSON file data synchronization method comprising:

[0016] Maintain the connection information of the database to be processed in the global configuration file;

[0017] Generate the corresponding Hive table name according to the database table name to be processed according to the hierarchical rules, and generate the table creation statement according to the mapping relationship between the database type and metadata information to be processed and the corresponding Hive field type;

[0018] Create a Hive table according to the Hive table name and table creation statement;

[0019] Generate a json file according to the file format requirements of DataX and the connection information and metadata information of the database to be processed, call the synchronization command of DataX and use the json file to complete data synchronization.

[0020] In combination with the first aspect, in one implementation, generating a corresponding Hive table name according to a hierarchical rule based on a database table name to be processed, and generating a table creation statement according to a mapping relationship between a database type to be processed and metadata information and a corresponding Hive field type, includes:

[0021] Generate the corresponding Hive table name based on the data layer number, database type, database name and table name information;

[0022] Get Hive column name, data type and column comment;

[0023] Get table comments from the metadata database information_schema.TABLES;

[0024] Generate a table creation script according to the Hive column name, data type, column comment and the table comment.

[0025] In combination with the first aspect, in one implementation, generating a json file according to the file format requirements of DataX and the connection information and metadata information of the database to be processed includes:

[0026] Get the column name of the data table according to the metadata information of the database to be processed, and form the content of the reader->parameter->column parameter in the JSON file;

[0027] Generate the contents of the writer->parameter->column parameter in the json file based on the metadata information of the database to be processed and the corresponding Hive data type mapping;

[0028] Generate a json file according to the connection information of the database to be processed, the json file format of the DataX configuration template, and the reader->parameter->column parameter and writer->parameter->column parameter.

[0029] In combination with the first aspect, in one implementation, calling the synchronization command of DataX and using the json file to complete data synchronization includes:

[0030] Implement a python script that calls DataX;

[0031] Implement a shell script that calls the python script;

[0032] Data synchronization is completed by executing shell scripts through Java.

[0033] In a second aspect, an embodiment of the present application provides a json file data synchronization device, the json file data synchronization device comprising:

[0034] A configuration module, which is used to maintain connection information of the database to be processed in a global configuration file;

[0035] The metadata processing module generates the corresponding Hive table name according to the database table name to be processed according to the hierarchical rules, and generates a table creation statement according to the mapping relationship between the database type to be processed and the metadata information and the corresponding Hive field type;

[0036] The metadata processing module also generates a json file according to the file format requirements of DataX, as well as the connection information and metadata information of the database to be processed;

[0037] A HiveServer service module, which creates a Hive table according to the Hive table name and table creation statement;

[0038] The DataX data synchronization module is used to call the synchronization command of DataX and use the json file to complete data synchronization.

[0039] In conjunction with the second aspect, in one implementation, the metadata processing module generates a corresponding Hive table name according to the database table name to be processed according to the hierarchical rule, and generates a table creation statement according to the mapping relationship between the database type to be processed and the metadata information and the corresponding Hive field type, including:

[0040] Generate the corresponding Hive table name based on the data layer number, database type, database name and table name information;

[0041] Get Hive column name, data type and column comment;

[0042] Get table comments from the metadata database information_schema.TABLES;

[0043] Generate a table creation script according to the Hive column name, data type, column comment and the table comment.

[0044] In conjunction with the second aspect, in one implementation, the metadata processing module generates a json file according to the file format requirements of DataX and the connection information and metadata information of the database to be processed, including:

[0045] Get the column name of the data table according to the metadata information of the database to be processed, and form the content of the reader->parameter->column parameter in the JSON file;

[0046] Generate the contents of the writer->parameter->column parameter in the json file based on the metadata information of the database to be processed and the corresponding Hive data type mapping;

[0047] Generate a json file according to the connection information of the database to be processed, the json file format of the DataX configuration template, and the reader->parameter->column parameter and writer->parameter->column parameter.

[0048] In conjunction with the second aspect, in one implementation, the DataX data synchronization module calls a synchronization command of DataX and uses the json file to complete data synchronization, including:

[0049] Implement a python script that calls DataX;

[0050] Implement a shell script that calls the python script;

[0051] Data synchronization is completed by executing shell scripts through Java.

[0052] In a third aspect, an embodiment of the present application provides a JSON file data synchronization device, which includes a processor, a memory, and a JSON file data synchronization program stored on the memory and executable by the processor, wherein when the JSON file data synchronization program is executed by the processor, the steps of the JSON file data synchronization method described in any one of the above are implemented.

[0053] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a JSON file data synchronization program is stored, wherein when the JSON file data synchronization program is executed by a processor, the steps of the JSON file data synchronization method as described in any one of the above are implemented.

[0054] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0055] The json file data synchronization method in the present application maintains the connection information of the database to be processed in the global configuration file; generates the corresponding Hive table name according to the database table name to be processed according to the hierarchical rule, and generates a table creation statement according to the mapping relationship between the database type and metadata information to be processed and the corresponding Hive field type; establishes a Hive table according to the Hive table name and the table creation statement; generates a json file according to the file format requirements of DataX, as well as the connection information and metadata information of the database to be processed, calls the synchronization command of DataX and uses the json file to complete data synchronization.

[0056] This simplifies the existing data synchronization development steps and realizes the function of automatic data synchronization. You only need to configure the data table name to be synchronized in the business database to realize the automatic table creation of the data warehouse Hive, automatically generate the complex and misconfigured Datax json file, and automatically synchronize the data, making data synchronization automatic. This avoids the need for users to be familiar with the syntax and type mapping relationship between different libraries, the complex configuration parameters, and the manual method that is prone to configuration errors and various abnormal errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of an embodiment of a JSON file data synchronization method in this application;

[0058] Figure 2 This is a schematic diagram of data stratification;

[0059] Figure 3 This is a table definition diagram of the department table in the original database in this application;

[0060] Figure 4 This is a schematic diagram of the data results obtained in this application;

[0061] Figure 5 This is a functional module diagram of an embodiment of a json file data synchronization device of the present application;

[0062] Figure 6 This is a system flow diagram of the json file data synchronization device for this application;

[0063] Figure 7 This is a schematic diagram of the hardware structure of the JSON file data synchronization device involved in the embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0066] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0067] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0068] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0069] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0070] In a first aspect, an embodiment of the present application provides a JSON file data synchronization method.

[0071] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of the json file data synchronization method of this application. Figure 1 As shown, the json file data synchronization method includes:

[0072] S1. Maintain the connection information of the database to be processed in the global configuration file;

[0073] It is worth mentioning that you can obtain its configuration information by selecting the database to be processed on the page.

[0074] S2. Generate the corresponding Hive table name according to the database table name to be processed according to the hierarchical rules, and generate a table creation statement according to the mapping relationship between the database type and metadata information to be processed and the corresponding Hive field type;

[0075] Specifically, in step S2, first generate the corresponding Hive table name according to the layer number, database type, library name and table name information of the data, and the generation rule is: ods_database type (such as mysql)_library name_table name; then obtain the Hive column name, data type and column comment; then obtain the table comment according to the metadata library information_schema.TABLES; finally, generate a table creation script according to the Hive column name, data type and column comment and the table comment.

[0076] See also Figure 2 As shown, generally speaking, in a data warehouse based on big data technology, in order to better manage data, the data will be layered. The ODS layer (source layer) is the first layer of the layer, which is mainly used to synchronize business database data to the data warehouse. The table structure and data of the ODS layer data table and the business database data table remain consistent.

[0077] S3. Create a Hive table according to the Hive table name and table creation statement;

[0078] Specifically, the table creation statement in step S2 will be sent to HiveServer2 to execute the table creation operation and complete the table creation at the ODS layer.

[0079] It is worth mentioning that Apache Hive is an open source data warehouse system built on Hadoop. It can map structured and semi-structured data files stored in Hadoop files into a database table, and provides a SQL-like query model based on the table, called Hive Query Language (HQL), which is used to access and analyze large data sets stored in Hadoop files.

[0080] Hiveserver2 is part of the Apache Hive project. It is a remote service for Hive that provides support for SQL queries and interactive queries for Hive. Hiveserver2 uses Thrift as a cross-language communication framework and can interact with multiple programming languages, such as Java, Python, Ruby, etc.

[0081] Hiveserver2 provides a client-server architecture where the client sends SQL queries to the server, which executes the queries and returns the results. This architecture makes it possible to connect to Hiveserver2 from anywhere and execute queries without having to install Hive locally.

[0082] S4. Generate a json file according to the file format requirements of DataX and the connection information and metadata information of the database to be processed, call the synchronization command of DataX and use the json file to complete data synchronization.

[0083] Therefore, through the above steps, the present application greatly reduces the complexity and difficulty of data synchronization by simplifying the six steps of data synchronization into one visual operation, reduces the operation steps, and improves work efficiency.

[0084] It can also automatically complete a series of tasks in the data synchronization process, including masking the syntax and field type differences between different databases, processing the data type mapping relationship with the Hive database, automatically creating tables, generating DataX configuration files, etc., avoiding problems that may be caused by manual operations and improving accuracy.

[0085] In addition, through automated processing and visual operation, errors caused by manual operation are reduced, the possibility of abnormal error reporting is reduced, and development efficiency is further improved. Due to the simplification of the operation process and the realization of automated processing, data synchronization is faster, the development cycle is shorter, and development efficiency is improved.

[0086] The following is a specific example to further illustrate the above steps:

[0087] For example, create an ODS layer table of the department (department information) table under the auth_user of the MySQL database. The department table in the original database can be found in Figure 3 As shown:

[0088] (1) Generate the table name: ods_mysql_auth_user_department according to the naming rule of the ods layer described in step S2 above. The hdfs path is generated according to the naming rule / ods / table name, that is, / ods / ods_mysql_auth_user_department;

[0089] 2) Obtain the final hive column name, data type, and column comment through the code;

[0090] It is well known to those skilled in the art to obtain column names, data types, and column comments through code, and will not be described in detail here. For the obtained data results, see Figure 4 shown.

[0091] 3) Get table comments based on information_schema.TABLES;

[0092] information_schema.TABLES is a built-in metadata database in MySQL. It provides a way to access database metadata. Metadata is data about data, such as database name or table name, column data type, or access permissions.

[0093] 4) Generate a table creation script based on the information in steps 2) and 3).

[0094] After obtaining the table creation statement, you can send HiveServer2 to execute the table creation operation to complete the table creation at the ODS layer. Then, according to the DataX configuration template json file format, the data synchronization json file is automatically generated based on the database connection information, database table, table fields and other information.

[0095] Specifically:

[0096] First, obtain the column name of the data table according to the metadata information of the database to be processed, and form the content of the reader->parameter->column parameter in the JSON file;

[0097] Generate the contents of the reader->parameter->column parameter:

[0098] "column":["id","department_name","type","level","owner_id","owner_name","source","related_wx_department","parent_id","related_type","related_id","shop_t ype","org_company_id","org_company_name","create_time","create_user_id","create_user_name","update_time","update_user_id","update_user_name","del_flag"].

[0099] Then, based on the metadata information of the database to be processed and the corresponding Hive data type mapping, generate the content of the writer->parameter->column parameter in the json file;

[0100] Generate the contents of the writer->parameter->column parameter:

[0101] "column":[{"name":"id","type":"int"},{"name":"department_name","type":"stri ng"},{"name":"type","type":"int"},{"name":"level","type":"int"},{"name":"owner_id","type":"int"},{"name":"owner_name","type":"string"},{"name":"source","type":"i nt"},{"name":"related_wx_department","type":"int"},{"name":"parent_id","type":"int"},{"name":"related_type","type":"int"},{"name":"related_id","type":"int"},{"nam e":"shop_type","type":"int"},{"name":"org_company_id","type":"int"},{"name":"or g_company_name","type":"string"},{"name":"create_time","type":"timestamp"},{"name":"create_user_id","type":"int"},{"name":"create_user_name","type":"string"}, {"name":"update_time","type":"timestamp"},{"name":"update_user_id","type":"int"},{"name":"update_user_name","type":"string"},{"name":"del_flag","type":"int"}].

[0102] Finally, a json file is generated according to the connection information of the database to be processed, the json file format of the DataX configuration template, and the reader->parameter->column parameter and writer->parameter->column parameter.

[0103] In addition, the synchronization command of DataX is called and the data synchronization is completed using the json file, including:

[0104] 1) Implement a python script that calls datax, such as run_datax.py, where the json file path is passed in as a parameter;

[0105] 2) Implement a shell script that calls the above python script (run_datax.py), for example, run_datax_via_python.sh;

[0106] 3) Java uses Runtime.exec() to execute the shell script run_datax_via_python.sh to complete data synchronization.

[0107] At this point, the function of automatic data synchronization has been completed.

[0108] The specific codes in the above steps are well known to those skilled in the art and will not be described in detail in this application.

[0109] To summarize, the json file data synchronization method in the present application maintains the connection information of the database to be processed in the global configuration file; generates the corresponding Hive table name according to the hierarchical rules based on the database table name to be processed, and generates a table creation statement based on the mapping relationship between the database type and metadata information to be processed and the corresponding Hive field type; establishes a Hive table according to the Hive table name and the table creation statement; generates a json file according to the file format requirements of DataX, as well as the connection information and metadata information of the database to be processed, calls the synchronization command of DataX and uses the json file to complete data synchronization.

[0110] This simplifies the existing data synchronization development steps and realizes the function of automatic data synchronization. You only need to configure the data table name to be synchronized in the business database to realize the automatic table creation of the data warehouse Hive, automatically generate the complex and misconfigured Datax json file, and automatically synchronize the data, making data synchronization automatic. This avoids the need for users to be familiar with the syntax and type mapping relationship between different libraries, the complex configuration parameters, and the manual method that is prone to configuration errors and various abnormal errors.

[0111] In a second aspect, an embodiment of the present application also provides a json file data synchronization device.

[0112] In one embodiment, referring to Figure 5 and Figure 6 As shown, Figure 5 This is a functional module diagram of an embodiment of a json file data synchronization device of the present application. Figure 6 is the corresponding system flow chart. Figure 5 As shown, the json file data synchronization device includes:

[0113] A configuration module, which is used to maintain connection information of the database to be processed in a global configuration file;

[0114] The metadata processing module generates the corresponding Hive table name according to the database table name to be processed according to the hierarchical rules, and generates a table creation statement according to the mapping relationship between the database type to be processed and the metadata information and the corresponding Hive field type;

[0115] The metadata processing module also generates a json file according to the file format requirements of DataX, as well as the connection information and metadata information of the database to be processed;

[0116] A HiveServer service module, which creates a Hive table according to the Hive table name and table creation statement;

[0117] The DataX data synchronization module is used to call the synchronization command of DataX and use the json file to complete data synchronization.

[0118] Further, in one embodiment, the metadata processing module generates a corresponding Hive table name according to the database table name to be processed according to the hierarchical rule, and generates a table creation statement according to the mapping relationship between the database type to be processed and the metadata information and the corresponding Hive field type, including:

[0119] Generate the corresponding Hive table name based on the data layer number, database type, database name and table name information;

[0120] Get Hive column name, data type and column comment;

[0121] Get table comments from the metadata database information_schema.TABLES;

[0122] Generate a table creation script according to the Hive column name, data type, column comment and the table comment.

[0123] Furthermore, in one embodiment, the metadata processing module generates a json file according to the file format requirements of DataX and the connection information and metadata information of the database to be processed, including:

[0124] Get the column name of the data table according to the metadata information of the database to be processed, and form the content of the reader->parameter->column parameter in the JSON file;

[0125] Generate the contents of the writer->parameter->column parameter in the json file based on the metadata information of the database to be processed and the corresponding Hive data type mapping;

[0126] Generate a json file according to the connection information of the database to be processed, the json file format of the DataX configuration template, and the reader->parameter->column parameter and writer->parameter->column parameter.

[0127] Furthermore, in one embodiment, the DataX data synchronization module calls the synchronization command of DataX and uses the json file to complete data synchronization, including:

[0128] Implement a python script that calls DataX;

[0129] Implement a shell script that calls the python script;

[0130] Data synchronization is completed by executing shell scripts through Java.

[0131] Among them, the functional implementation of each module in the above-mentioned JSON file data synchronization device corresponds to the various steps in the above-mentioned JSON file data synchronization method embodiment, and its functions and implementation processes will not be repeated here one by one.

[0132] In a third aspect, an embodiment of the present application provides a json file data synchronization device, which may be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0133] Reference Figure 7 , Figure 7 Schematic diagram of the hardware structure of the JSON file data synchronization device involved in the embodiment of the present application. In the embodiment of the present application, the JSON file data synchronization device may include a processor, a memory, a communication interface and a communication bus.

[0134] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0135] The communication interface includes input / output (I / O) interface, physical interface and logical interface, etc., which are used to realize the interconnection of devices inside the json file data synchronization device, and the interface used to realize the interconnection between the json file data synchronization device and other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.; the user device can be a display (Display), a keyboard (Keyboard), etc.

[0136] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0137] The processor may be a general-purpose processor, and the general-purpose processor may call the json file data synchronization program stored in the memory and execute the json file data synchronization method provided in the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the json file data synchronization program is called may refer to the various embodiments of the json file data synchronization method of the present application, and will not be repeated here.

[0138] Those skilled in the art will understand that Figure 7 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0139] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium.

[0140] A json file data synchronization program is stored on the computer-readable storage medium of the present application, wherein when the json file data synchronization program is executed by a processor, the steps of the json file data synchronization method as described above are implemented.

[0141] Among them, the method implemented when the json file data synchronization program is executed can refer to the various embodiments of the json file data synchronization method of the present application, and will not be repeated here.

[0142] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0143] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0144] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A json file data synchronization method, characterized in that: The json file data synchronization method includes: Maintain the connection information of the database to be processed in the global configuration file; Generate the corresponding Hive table name according to the database table name to be processed according to the hierarchical rules, and generate the table creation statement according to the mapping relationship between the database type and metadata information to be processed and the corresponding Hive field type; Create a Hive table according to the Hive table name and table creation statement; Generate a json file according to the file format requirements of DataX and the connection information and metadata information of the database to be processed, call the synchronization command of DataX and use the json file to complete data synchronization.

2. The JSON file data synchronization method according to claim 1, characterized in that: The method generates a corresponding Hive table name according to the database table name to be processed according to the hierarchical rules, and generates a table creation statement according to the mapping relationship between the database type and metadata information to be processed and the corresponding Hive field type, including: Generate the corresponding Hive table name based on the data layer number, database type, database name and table name information; Get Hive column name, data type and column comment; Get table comments from the metadata database information_schema.TABLES; Generate a table creation script according to the Hive column name, data type, column comment and the table comment.

3. The JSON file data synchronization method according to claim 2, characterized in that: The json file is generated according to the file format requirements of DataX and the connection information and metadata information of the database to be processed, including: Get the column name of the data table according to the metadata information of the database to be processed, and form the content of the reader->parameter->column parameter in the JSON file; Generate the contents of the writer->parameter->column parameter in the json file based on the metadata information of the database to be processed and the corresponding Hive data type mapping; Generate a json file according to the connection information of the database to be processed, the json file format of the DataX configuration template, and the reader->parameter->column parameter and writer->parameter->column parameter.

4. The JSON file data synchronization method according to claim 3, characterized in that: The calling of the DataX synchronization command and the use of the json file to complete data synchronization include: Implement a python script that calls DataX; Implement a shell script that calls the python script; Data synchronization is completed by executing shell scripts through Java.

5. A json file data synchronization device, characterized in that: The json file data synchronization device comprises: A configuration module, which is used to maintain connection information of the database to be processed in a global configuration file; The metadata processing module generates the corresponding Hive table name according to the database table name to be processed according to the hierarchical rules, and generates a table creation statement according to the mapping relationship between the database type to be processed and the metadata information and the corresponding Hive field type; The metadata processing module also generates a json file according to the file format requirements of DataX, as well as the connection information and metadata information of the database to be processed; A HiveServer service module, which creates a Hive table according to the Hive table name and table creation statement; The DataX data synchronization module is used to call the synchronization command of DataX and use the json file to complete data synchronization.

6. The JSON file data synchronization device according to claim 5, characterized in that: The metadata processing module generates a corresponding Hive table name according to the database table name to be processed according to the hierarchical rules, and generates a table creation statement according to the mapping relationship between the database type to be processed and the metadata information and the corresponding Hive field type, including: Generate the corresponding Hive table name based on the data layer number, database type, database name and table name information; Get Hive column name, data type and column comment; Get table comments from the metadata database information_schema.TABLES; Generate a table creation script according to the Hive column name, data type, column comment and the table comment.

7. The JSON file data synchronization device according to claim 6, characterized in that: The metadata processing module generates a json file according to the file format requirements of DataX and the connection information and metadata information of the database to be processed, including: Get the column name of the data table according to the metadata information of the database to be processed, and form the content of the reader->parameter->column parameter in the JSON file; Generate the contents of the writer->parameter->column parameter in the json file based on the metadata information of the database to be processed and the corresponding Hive data type mapping; Generate a json file according to the connection information of the database to be processed, the json file format of the DataX configuration template, and the reader->parameter->column parameter and writer->parameter->column parameter.

8. The JSON file data synchronization device according to claim 7, characterized in that: The DataX data synchronization module calls the DataX synchronization command and uses the json file to complete data synchronization, including: Implement a python script that calls DataX; Implement a shell script that calls the python script; Data synchronization is completed by executing shell scripts through Java.

9. A json file data synchronization device, characterized in that: The JSON file data synchronization device includes a processor, a memory, and a JSON file data synchronization program stored in the memory and executable by the processor, wherein when the JSON file data synchronization program is executed by the processor, the steps of the JSON file data synchronization method as described in any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a json file data synchronization program, wherein when the json file data synchronization program is executed by a processor, the steps of the json file data synchronization method according to any one of claims 1 to 4 are implemented.

Images