An intelligently optimized data type synchronization fusion system

Abstract

The invention relates to an intelligently optimized data type synchronous fusion system, which belongs to the field of data fusion and comprises a data adapter, which is used for collecting the field types of a remote destination table and related information, and finally transmitting an execution statement to the destination according to the operation logic; a type ladder escape module, which comprises a type compatibility comparison table between databases and hierarchies the types; information conversion is performed on source information, and a new parameter set is passed to a data element analysis module; the data element analysis module analyzes the data and judges the multi-dimensional information of the data type: whether to redefine the destination field specification, and passes the parameters into a data definition language compilation module; the data definition language compilation module comprises a set of configuration files, and when the destination structure needs to be adjusted, the script language corresponding to the destination table and the corresponding operation is automatically generated using the relevant parameters and the data definition language configuration file, and is executed at the destination.

Description

Technical Field

[0001] The invention belongs to the field of data fusion and relates to an intelligently optimized data type synchronous fusion system. Background Art

[0002] In the process of building the data center of the group hospital, there is a need to aggregate multi-source data into a unified storage for data analysis and utilization. In each institution, the unified production system has been changed due to business requirements, resulting in differences in the originally consistent table structure of the same production system, such as adjusting the precision of floating-point fields and modifying the length of character fields. These differences often cause synchronization failures and data deadlocks when the data center merges data due to the incompatibility of the newly included data source with the existing storage data type; the traditional processing mode is to manually adjust the data type or expand the length precision and other attributes of the data type as much as possible when creating a table on the destination side, so as to avoid errors as much as possible. However, this method cannot completely avoid hidden dangers. When incompatibility occurs, it is still necessary to wait until the program reports an error or captures an exception before manually processing; the second method is to ignore the business source field type and uniformly process the fields of the destination table structure into highly compatible types, such as long text types or long string types; this processing method consumes storage and, most of the time, will reduce the query performance of the data. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide an intelligently optimized data type synchronization fusion system, which performs automated compatibility analysis on the data type information of the source and destination ends, and makes necessary adjustments, thereby ensuring that the synchronization of the source data to the destination end is always successful.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] An intelligently optimized data type synchronous fusion system, comprising: a data adapter, a type ladder comparison table, a data logic analysis module, and a data definition language compilation module;

[0006] The data adapter is used to collect the table field types and related information of the remote destination, and finally transmit the execution statement to the destination according to the operation logic;

[0007] The type ladder escape module includes a type compatibility comparison table between databases, and stratifies types according to their compatibility; performs information conversion on the incoming source information, and passes a new parameter set containing the source field type information after the transfer into the data element analysis module;

[0008] The data element analysis module analyzes the data passed in from the type ladder escape module, determines the multi-dimensional information of the data type: whether to redefine the destination field specification, and passes the parameters into the data definition language compilation module;

[0009] The data definition language compilation module includes a set of configuration files that record statements for operating the library table structure definition of various database types. When the target end structure needs to be adjusted, the relevant parameters and the data definition language configuration file are used to automatically generate the script language corresponding to the target end table and the corresponding operation, and execute it on the target end.

[0010] Furthermore, when the data synchronization task reaches the stage where data type synchronization is compatible, the necessary parameters in the data stream are pulled through the data adapter, assembled into query statements, and query requests are sent to the source and destination ends to obtain the table structure information of the source and destination ends corresponding to the data stream, thereby preparing for the subsequent steps of the data synchronization task.

[0011] Furthermore, the type ladder escape module extracts the corresponding field type of the destination end according to the type of the source end table field, in preparation for subsequent comparison. The extraction steps are as follows:

[0012] 1) Find the "read" page of the source database from the ladder table file, and find the level code LEVEL corresponding to the corresponding field type;

[0013] 2) Find the "write" page of the destination database from the ladder table file, and find the destination field type of the corresponding level of the corresponding field type;

[0014] 3) Pack the extracted information to obtain a parameter package.

[0015] Furthermore, the type ladder escape module is also used to compare whether the source field exists in the destination; when the source field does not exist in the destination, the parameter package is passed into the data definition language compilation module; when the source field exists in the destination structure, the parameter package is passed into the data element analysis module.

[0016] Further, the data element analysis module compares the source and destination data information to determine whether the destination table structure needs to be adjusted. The determination steps are as follows:

[0017] Step 1: Determine whether the compatibility level code value obtained after the source field is escaped is greater than the compatibility level code value of the existing field type on the destination side. If it is greater than the existing compatibility level code value of the destination field, adjust the data type of the destination field; if it is less than or equal to the compatibility level code value, proceed to step 2;

[0018] Step 2: Compare the length and precision of the source data type before escaping with the existing length and precision of the corresponding field on the destination side. If any element does not meet the constraints of the destination field data structure, adjust the destination field data type, otherwise release it.

[0019] Step 3: Pack the parameter information adjusted in steps 1 and 2 and pass it to the data definition language compilation module.

[0020] Furthermore, the data definition language compilation module automatically generates a data definition language script corresponding to the target library using the parameters according to the parameter package passed in by the data element analysis module, and sends an execution instruction to the target data through the data adapter.

[0021] The beneficial effects of the present invention are as follows: by performing data synchronization through the present invention, an intelligent balance is achieved in the compatibility of data types and the consumption of data space. The synchronization work between different databases by users no longer requires manual intervention, and the difference analysis of database tables can be completed quickly, and the structural compatibility transformation operation of the destination database can be automatically executed.

[0022] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0024] Figure 1 This is a working diagram of the module of the intelligently optimized data type synchronization fusion system of the present invention;

[0025] Figure 2 Analyze the module logic diagram for data elements. DETAILED DESCRIPTION

[0026] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0027] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0028] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0029] The purpose of the present invention is achieved through the following technical solutions: an intelligently optimized data type synchronization fusion system, including a data adapter, a type ladder comparison table, a data logic analysis module, and a data definition language configuration file.

[0030] The data adapter is responsible for collecting the field types and related information of the remote destination table, and finally transmitting the execution statement to the destination according to the operation logic.

[0031] The type ladder escape module contains a set of type compatibility comparison tables between databases, and stratifies types according to their compatibility. It converts the incoming source information and passes a new parameter set containing the source field type information after the transfer into the data element analysis module.

[0032] The data element analysis module analyzes the data passed in from the type ladder escape module, automatically determines whether to redefine the destination field specification based on the multi-dimensional information of the data type, such as length, precision, etc., and passes the parameters into the data definition language compilation module.

[0033] The data definition language compilation module contains a set of configuration files that record statements for operating the library table structure definition for various database types. When the target end structure needs to be adjusted, the relevant parameters and data definition language configuration files will be used by this module to automatically generate the script language corresponding to the target end table and the corresponding operation, and execute it on the target end.

[0034] like Figure 1As shown in the figure, when the data synchronization task proceeds to the "Data Type Synchronization Compatibility" step, the data adapter is triggered and started. By pulling the necessary parameters in the data flow, the adapter assembles a query statement, sends a query request to the source and destination ends, and obtains the table structure information of the source and destination ends corresponding to the data flow, preparing for subsequent steps.

[0035] After the necessary information is collected, the type ladder escape module will extract the corresponding field type of the destination according to the type of the source table field, in preparation for subsequent comparison. Combined with the structure introduction of Table 1, the extraction strategy is: 1: Find the source database from the ladder table file, and the level code LEVEL corresponding to the field type in the "read" page; 2: Find the destination database from the ladder table file, and the destination field type corresponding to the level on the "write" page. The extracted information is packaged for use in the next step.

[0036] Table 1

[0037]

[0038] During execution, the program will compare whether the source field exists in the destination. When the source field does not exist in the destination, the program will go directly to the data definition language compilation module. When the source field exists in the destination structure, the program will pass the parameter package to the data element analysis module. This module compares the source and destination data information to determine whether the destination table structure needs to be adjusted. The determination operation is divided into two steps, combined with Figure 2 The logical flow is explained as follows:

[0039] The first step is to determine whether the compatibility level code value obtained after escaping the source field is greater than the compatibility level value of the existing field type of the destination field. If it is greater than the existing compatibility level of the destination field, the data type of the destination field must be adjusted; if it is less than or equal to, proceed to the second step.

[0040] In the second step, the length and precision of the source data type before escaping are compared with the existing length and precision of the corresponding field on the destination side. When any element does not meet the constraints of the destination field data structure, the program will be triggered to package the parameters and pass the information to the subsequent steps.

[0041] The data definition language compilation module automatically generates a data definition language script corresponding to the target library based on the parameter package passed in the previous step, and sends execution instructions to the target data through the data adapter.

[0042] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. An intelligently optimized data type synchronization fusion system, characterized by: Including data adapter, type ladder comparison table, data logic analysis module, data definition language compilation module; The data adapter is used to collect the table field types and related information of the remote destination, and finally transmit the execution statement to the destination according to the operation logic; The type ladder escape module contains a type compatibility comparison table between databases, which stratifies types according to their compatibility. Convert the incoming source information and pass a new parameter set containing the transferred source field type information to the data element analysis module; The data element analysis module analyzes the data passed in from the type ladder escape module, determines the multi-dimensional information of the data type: whether to redefine the destination field specification, and passes the parameters into the data definition language compilation module; The data definition language compilation module includes a set of configuration files that record statements for operating the library table structure definition of various database types. When the target end structure needs to be adjusted, the relevant parameters and the data definition language configuration file are used to automatically generate the script language corresponding to the target end table and the corresponding operation, and execute it on the target end.

2. The intelligent optimized data type synchronization fusion system according to claim 1, characterized in that: When the data synchronization task reaches the stage where data type synchronization is compatible, the necessary parameters in the data stream are pulled through the data adapter, assembled into query statements, and query requests are sent to the source and destination ends to obtain the table structure information of the source and destination ends corresponding to the data stream, in preparation for the subsequent steps of the data synchronization task.

3. The intelligent optimized data type synchronization fusion system according to claim 1, characterized in that: The type ladder escape module extracts the corresponding field type of the destination end according to the type of the source end table field, in preparation for subsequent comparison. The extraction steps are as follows: 1) Find the "read" page of the source database from the ladder table file, and find the level code LEVEL corresponding to the corresponding field type; 2) Find the "write" page of the destination database from the ladder table file, and find the destination field type of the corresponding level of the corresponding field type; 3) Pack the extracted information to obtain a parameter package.

4. The intelligent optimized data type synchronization fusion system according to claim 1, characterized in that: The type ladder escape module is also used to compare whether the source end field exists in the destination end; when the source end field does not exist in the destination end, the parameter package is passed into the data definition language compilation module; when the source end field exists in the destination end structure, the parameter package is passed into the data element analysis module.

5. The intelligent optimized data type synchronization fusion system according to claim 1, characterized in that: The data element analysis module compares the source and destination data information to determine whether the destination table structure needs to be adjusted. The determination steps are as follows: Step 1: Determine whether the compatibility level code value obtained after the source field is escaped is greater than the compatibility level code value of the existing field type on the destination side. If it is greater than the existing compatibility level code value of the destination field, adjust the data type of the destination field; if it is less than or equal to the compatibility level code value, proceed to step 2; Step 2: Compare the length and precision of the source data type before escaping with the existing length and precision of the corresponding field on the destination side. If any element does not meet the constraints of the destination field data structure, adjust the destination field data type, otherwise release it. Step 3: Pack the parameter information adjusted in steps 1 and 2 and pass it to the data definition language compilation module.

6. The intelligent optimized data type synchronization fusion system according to claim 5, characterized in that: The data definition language compilation module automatically generates a data definition language script corresponding to the target library using the parameters according to the parameter package passed in by the data element analysis module, and sends an execution instruction to the target data through the data adapter.

Images