A method and device for converting document model data into relational model data

Abstract

The present invention relates to the technical field of data synchronization, and provides a method and device for converting document model data into relational model data. The present invention dynamically updates mapping configuration information according to historical document model data, uses the mapping configuration information as a reference, extracts data from current document model data to obtain row data, and loads the row data into a corresponding relational table to obtain relational model data; wherein, the mapping configuration information is obtained based on historical document model data in the evolution process of the current document model data, so that the mapping configuration information covers as much of the table structure and dependency relationship in the current document model data as possible, thereby ensuring that there is no large amount of data redundancy or unclear table structure in the row data obtained by matching the mapping configuration information, which greatly improves the practicality of the data synchronization system and solves the problem of affecting upper-level applications due to data redundancy and unclear table structure.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization, and in particular to a method and device for converting document model data into relational model data. Background Art

[0002] The traditional database management system (DBMS) is a relational database, which is designed and implemented based on relational data theory and is the mainstream database on the market. In order to meet the flexible storage requirements of massive data, modern database management systems often use specialized databases for specific vertical application fields, such as document databases, in conjunction with relational databases.

[0003] The existing data synchronization system focuses on relational model data. Simple document model data without multiple layers of nesting can be directly mapped to relational model data, that is, one document model data corresponds to one relational model data. For document model data with multiple layers of nesting, a "flattening" method is used to "flatten" the multi-layer document model data into a "wide table" of relational data. The relational model data obtained by conversion in the prior art has data redundancy and unclear relational table structure, resulting in low practicality of the data synchronization system and affecting the upper-level application program. Therefore, there is an urgent need for a general method for converting document model data into relational model data.

[0004] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method and device for converting document model data into relational model data. Its purpose is to provide a general method for converting document model data into relational model data, converting multi-level nested document model data into row data based on mapping configuration information, and solving the problem that the relational model data obtained by conversion in the prior art has data redundancy and unclear relational table structure, resulting in low practicality of the data synchronization system and affecting the upper-level application program.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for converting document model data into relational model data, comprising:

[0008] Dynamically update mapping configuration information based on historical document model data;

[0009] According to the mapping configuration information, data is extracted from the current document model data to obtain row data;

[0010] The row data is loaded into the corresponding relational table to obtain relational model data.

[0011] Further, the dynamically updating mapping configuration information according to the historical document model data includes:

[0012] Determine at least one historical document model data whose structural difference is within a preset range;

[0013] When the amount of the historical document model data is greater than the document amount threshold, sampling the historical document model data to obtain a document data subset; when the amount of the historical document model data is less than the document amount threshold, using the historical document model data as a document data subset;

[0014] A corresponding template mapping-linked list nested body is obtained according to the document data subset, and mapping configuration information is generated according to the template mapping-linked list nested body.

[0015] Furthermore, the sampling of the historical document model data to obtain a document data subset includes:

[0016] Determining the generation time of the historical document model data in the upper layer application;

[0017] When the frequency of adjustment of the historical document model data by the upper layer application is less than the first frequency, determining the running period of the upper layer application as the early stage of system running;

[0018] When the adjustment frequency is greater than the first frequency and less than the second frequency, the operation period is determined as the middle period of system operation;

[0019] When the adjustment frequency is greater than or equal to the second frequency, determining the operation period as a system operation stable period;

[0020] According to the generation time corresponding to the historical document model data in the early stage of the system operation, the middle stage of the system operation, the stable stage of the system operation and each period, the historical document model data required for sampling is determined to obtain a document data subset.

[0021] Furthermore, the historical document model data to be sampled is determined according to the generation time corresponding to the historical document model data in the early stage of the system operation, the middle stage of the system operation, the stable stage of the system operation and each period, and the document data subset obtained includes:

[0022] According to the number of iterations of the historical document model data in the early stage of the system operation, an early time is configured for the early stage of the system operation; for the historical document model data generated at the early stage of the system operation, sampling is performed at intervals of the early time each time to obtain an early number of document model data;

[0023] According to the number of iterations of the historical document model data in the mid-term of the system operation, a mid-term time is configured for the mid-term of the system operation; for the historical document model data generated at the mid-term of the system operation, sampling is performed at intervals of the mid-term time each time to obtain a mid-term amount of document model data;

[0024] According to the number of iterations of the historical document model data in the system operation stable period, configuring the stable period time for the system operation stable period; sampling the historical document model data generated in the system operation stable period at intervals of the stable period time each time to obtain the document model data of the stable period quantity;

[0025] The early amount of document model data, the mid-term amount of document model data and the stable period amount of document model data are integrated to obtain a document data subset.

[0026] Further, the obtaining a corresponding template mapping-linked list nested body according to the document data subset, and generating mapping configuration information according to the template mapping-linked list nested body includes:

[0027] Representing the document data subset in JSON format as a template mapping-linked list nested body of a tree structure;

[0028] Obtain each mapping body in the template mapping-linked list nested body, and use the name of the mapping body as the name of the relationship table configuration information;

[0029] Determine the key name of each key-value pair in the mapping body as the column name of the column in the relationship table configuration information; determine the data type of the value of the key-value pair as the column data type of the column in the relationship table configuration information;

[0030] Determine first table information to be merged with the same name from the relationship table configuration information, merge the column fields of the first table information to be merged to obtain second table information to be merged; merge the columns with the same column names in the second table information to be merged to obtain merged table information; and obtain mapping configuration information based on the merged table information and information other than the first table information to be merged in the relationship table configuration information.

[0031] Further, the extracting the current document model data according to the mapping configuration information to obtain the row data includes:

[0032] The current document model data is represented in JSON format as a tree-structured nested body of a mapping-linked list to be converted;

[0033] Using the table information in the mapping configuration information to match the mapping-linked list nested body to be transferred, and obtaining a relationship table to be merged according to the mapping body in the mapping-linked list nested body to be transferred;

[0034] Determine redundant relationship tables with the same table name from the relationship tables to be merged, and determine the relationship tables to be merged other than the redundant relationship tables as non-redundant relationship tables;

[0035] The column fields of the redundant relationship table are merged to obtain a merged relationship table; the columns with the same column name in the merged relationship table are merged to obtain merged row data; and the row data is obtained according to the merged row data and the non-redundant relationship table.

[0036] Further, the using table information in the mapping configuration information to match the to-be-transferred mapping-linked list nested body, and obtaining the to-be-merged relationship table according to the mapping body in the to-be-transferred mapping-linked list nested body includes:

[0037] According to the table information, extract the name of each mapping body in the mapping-linked list nested body to be converted, and initialize the extracted name as the table name of the relationship table to be merged corresponding to the mapping body;

[0038] Determine the key name of each key-value pair in the mapping body as the column name of the relationship table to be merged;

[0039] Determine the data type of the value of the key-value pair as the column data type of the column of the to-be-merged relational table;

[0040] The table structure is obtained according to the column names and column data types of the columns of the to-be-merged relational table; the value at the corresponding position in the table structure is determined according to the value of the key-value pair in the mapping body; and the dependency relationship between different to-be-merged relational tables is established according to the parent-child relationship between different mapping bodies to obtain the to-be-merged relational table.

[0041] Further, the establishing of dependency relationships between different to-be-merged relationship tables according to the parent-child relationships between different mapping bodies to obtain the to-be-merged relationship tables includes:

[0042] According to the mapping configuration information, a parent-child relationship between each mapping body in the nested body of the mapping-linked list to be transferred is obtained;

[0043] According to the parent-child relationship, a first foreign key column is added to the to-be-merged relationship table corresponding to the mapping body of the child node, and the value of the first foreign key column is set to the primary key column value of the to-be-merged relationship table corresponding to the parent node;

[0044] A second foreign key column is added to the to-be-merged relation table corresponding to the parent node, and the value of the second foreign key column is set as the primary key column of the to-be-merged relation table corresponding to the child node, so as to establish a dependency relationship between different to-be-merged relation tables.

[0045] In a second aspect, the present invention further provides a device for converting document model data into relational model data, which is used to implement the method for converting document model data into relational model data in the first aspect, and comprises a document reading component, a document line conversion component and at least one table loading component; wherein:

[0046] The document reading component is used to obtain historical document model data, and the document line conversion component is used to dynamically update mapping configuration information according to the historical document model data;

[0047] The document reading component is further used to obtain current document model data; the document line conversion component is further used to extract data from the current document model data according to the mapping configuration information to obtain line data;

[0048] The table loading component is used to load the received row data into the corresponding relational table to obtain relational model data.

[0049] Furthermore, it also includes a document conversion component; wherein:

[0050] The document conversion component is used to perform content conversion on the received row data to obtain processed row data; and transmit the processed row data to the corresponding table loading component to obtain the relational model data through the table loading component.

[0051] Furthermore, the device comprises:

[0052] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the method of converting document model data into relational model data as described in the first aspect.

[0053] In a third aspect, the present invention further provides a non-volatile computer storage medium storing computer executable instructions, which are executed by one or more processors to complete the method of converting document model data into relational model data as described in the first aspect.

[0054] Different from the prior art, the present invention has at least the following beneficial effects:

[0055] The present invention dynamically updates mapping configuration information according to historical document model data, uses the mapping configuration information as a reference, extracts data from current document model data to obtain row data, and loads the row data into a corresponding relationship table to obtain relationship model data; wherein, mapping configuration information is obtained based on historical document model data in the evolution process of current document model data, so that the mapping configuration information covers as much table structure and dependency relationship in the current document model data as possible, thereby ensuring that there will not be a large amount of data redundancy or unclear table structure in the row data obtained by matching the mapping configuration information, thereby greatly improving the practicality of the data synchronization system and solving the problem of affecting upper-level applications due to data redundancy and unclear table structure.

[0056] Furthermore, corresponding document model data are sampled in the early stage of system operation, the middle stage of system operation and the stable stage of system operation to generate document data subsets; the document data subsets are processed to construct mapping configuration information, which solves the problem in the prior art that fixed document data patterns can only be extracted after relying on manual efforts to clearly understand the evolution history of the document data pattern. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 is a schematic diagram of a relational model data provided by an embodiment of the present invention;

[0059] Figure 2 It is a schematic diagram of the overall process of a method for converting document model data into relational model data provided by an embodiment of the present invention;

[0060] Figure 3 is a flow chart of step 10 of an embodiment of the present invention;

[0061] Figure 4 is a flow chart of step 102 of an embodiment of the present invention;

[0062] Figure 5 is a schematic diagram of an operation period and a sampling time provided by an embodiment of the present invention;

[0063] Figure 6 is a flow chart of step 1025 of an embodiment of the present invention;

[0064] Figure 7 is a flow chart of step 103 of an embodiment of the present invention;

[0065] Figure 8 is a schematic diagram of a mapping-linked list nested body provided by an embodiment of the present invention;

[0066] Fig. 9 is a flow chart of step 20 of an embodiment of the present invention;

[0067] Fig.10 It is a schematic diagram of a tree structure of a mapping-linked list nested body provided by an embodiment of the present invention;

[0068] Fig.11 is a schematic diagram of document model data provided by an embodiment of the present invention;

[0069] Fig.12 is a schematic diagram of mapping configuration information provided by an embodiment of the present invention;

[0070] Fig.13 is a flow chart of step 202 of an embodiment of the present invention;

[0071] Fig.14 is a flow chart of step 2024 of an embodiment of the present invention;

[0072] Fig.15 It is a schematic diagram of an ETL system for performing relational model data conversion provided by an embodiment of the present invention;

[0073] Fig.16 is a schematic diagram of another ETL system for performing relational model data conversion provided by an embodiment of the present invention;

[0074] Fig.17 It is a schematic diagram of the architecture of an apparatus for converting document model data into relational model data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0076] In the description of the present invention, the terms "inside", "outside", "longitudinal", "lateral", "upper", "lower", "top", "bottom" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and do not require that the present invention must be constructed and operated in a specific orientation. Therefore, they should not be understood as limitations on the present invention.

[0077] The terms "first", "second", etc. in the present invention are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0078] In the present invention, unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection or an indirect connection through an intermediate medium.

[0079] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0080] Embodiment 1:

[0081] In order to more clearly illustrate the solution of this embodiment, a relational database and a document database are briefly introduced here.

[0082] Relational databases use a relational model to organize data, storing data in the form of rows and columns to facilitate user understanding. In a relational database, a table consists of rows and columns, and users retrieve data from the relational database by querying the table. The relational model can be simply understood as a two-dimensional table model consisting of rows and columns.

[0083] The relational model uses a "fixed key-value pair" model to model real-world objects. Data exists in a simple two-dimensional table. Data in the same table is forced to have the same "key", that is, column field information. One table is used to model the characteristics of one level of objects. However, objects in the real world have multi-attribute and multi-level characteristics. It is necessary to combine multiple data tables to achieve a complete description of objects. Tables are associated with each other through associated columns (foreign keys). When querying data, use the "join" operation to splice multiple tables into a large table, and return complete description information of specific real-world objects in the database world. "Join" is an expensive database operation, and the relational model cannot avoid using "join".

[0084] The use of relational databases requires the pre-definition of the table structure of the relational model. Each record added to the relational database table (i.e., row data) must be added, deleted, modified, and queried strictly in accordance with the pre-defined table structure. In addition, in the face of massive growth in data, in order to obtain complete description information of specific real things in the database world, when using relational databases, multiple relational database tables have to be spliced ​​into one large table through the "connection" operation to establish associations between multiple database tables.

[0085] In recent years, a large number of non-relational databases have emerged, such as graph databases, time series databases, document databases and other specialized databases for specific vertical application fields, which are used to replace relational databases. In particular, document databases use document-type data models to represent, operate, store and manage complete information describing real-world objects. A document model data has multi-attribute and multi-level characteristics, and there is no need to use a "join" operation when querying data.

[0086] The function of the data synchronization system is mainly to extract, transform and load data (Extract-Transform-Loading, abbreviated as ETL), mainly for data in the two-dimensional table form of the relational model; the "transformation" here is a low-level conversion, which only converts the form of data into another form, and does not rise to the conversion of the data model level. The emergence of non-relational databases has put forward new requirements for data synchronization systems, requiring semantic equivalence conversion of data under different data models, such as conversion between relational model data and document model data, without losing the description information of things.

[0087] In terms of representation, relational model data is in the form of a two-dimensional table, such as Figure 1 As shown, the header of the relational table is a row of fixed number of column information "Col1, Col2, ..., Coli", which is a predefined table structure, which specifies the data type of the corresponding column in the relational table; the body of the relational table is composed of multiple rows of data, each row of data has multiple column values ​​(i.e., field values), each row of data describes a real thing, and each field value describes a feature of the thing. In an upper-level application, multiple relational tables are usually required to model each active subject in the entire upper-level application. The relevant data of each active subject is often distributed in multiple different relational tables. The relational database returns the complete (or most) information of the active subject through the "connection" operation of the relational table.

[0088] The document model data stores the data of each activity subject as much as possible. The document model data is a multi-level nested format in terms of its presentation. The data of each level is a key-value set. The data of the next level constitutes one item of a key-value pair in the previous level. The industry has specially designed the JavaScript Object Notation (JSON) format for this purpose. A legal JSON instance is: "{"a":1,"b":[1,2,3]}". When querying, the document database can directly return the complete or most of the information of the activity subject, avoiding the use of "join" operations.

[0089] In the prior art, the application demand for document model data conversion to relational model data is mainly in data analysis applications. The data objects analyzed by existing data analysis algorithms are mainly structured relational data sets, and data mining analysis such as classification, clustering, and association rule discovery cannot be performed directly on document model data. It is necessary to extract the information of document model data and convert it into equivalent structured relational model data before data analysis can be performed on document model data.

[0090] The data synchronization system in the prior art often directly expands the multi-level document model data and converts each level of the document model data into multi-layer nested relational model data. Although the result of this conversion retains the relationship between the data in the original format of the document model data, due to the inevitable "connection" operation when generating the relational table, the relationship between the data is very complicated and there are many unnecessary nested relationships. That is, this conversion method artificially introduces data redundancy and unclear table structure of the relational model data, resulting in poor practicality of the data synchronization system. Using this redundant relational model data to model each active subject in the entire upper-level application will greatly increase the complexity of the upper-level application implementation logic.

[0091] In order to solve the above problems, Figure 2 As shown, an embodiment of the present invention provides a method for converting document model data into relational model data, including:

[0092] Step 10: Dynamically update mapping configuration information based on historical document model data.

[0093] As time goes by, the historical document model data may continue to increase, so the mapping configuration information needs to be dynamically updated according to the evolution of the historical document model data.

[0094] For one or a batch of document model data that are constantly adjusted and modified, when the current document model data specified this time is the same as the document model data specified at a certain time, since the mapping configuration information has been generated for the document model data, the mapping configuration information can be used for this conversion; when the current document model data specified this time evolves from the document model data specified at a certain time, the document model data specified at a certain time is part of the historical document model data of the current document model data specified this time, and this time, based on the historical document model data, new mapping configuration information for the document model data is generated, that is, the mapping configuration information corresponding to the document model data is dynamically updated.

[0095] Step 20: According to the mapping configuration information, data is extracted from the current document model data to obtain row data.

[0096] The mapping configuration information is a template for converting document model data into relational model data. According to the document model data that needs to be converted during the operation of the data synchronization system (i.e., the ETL system), the mapping configuration information is used to convert the document model data into relational model data. The row data finally converted is one or more records based on the relational model; the row data includes the table structure of each relational table itself (such as column data type, etc.) and the values ​​corresponding to each column in the relational table. When the relational table corresponding to the row data has a dependency relationship with other relational tables (such as primary key, foreign key, etc.), the row data also includes information related to the dependency relationship.

[0097] Step 30: Load the row data into the corresponding relational table to obtain relational model data.

[0098] After the row data of the relational table is extracted, it is written into the corresponding position in the relational database according to the table structure to obtain the relational model data in the relational database.

[0099] The present invention dynamically updates mapping configuration information according to historical document model data, uses the mapping configuration information as a reference, extracts data from current document model data to obtain row data, and loads the row data into a corresponding relationship table to obtain relationship model data; wherein, mapping configuration information is obtained based on historical document model data in the evolution process of current document model data, so that the mapping configuration information covers as much table structure and dependency relationship in the current document model data as possible, thereby ensuring that there is no large amount of data redundancy or unclear table structure in the row data obtained by matching the mapping configuration information, thereby greatly improving the practicality of the data synchronization system and solving the problem of affecting upper-level applications due to data redundancy and unclear table structure.

[0100] In relational model data, each relational table has fixed column information, and different row data in the same column must be of the same data type. Document model data does not have fixed schema constraints. The document data schema of different document model data in the same collection can be completely different, with different key-value pairs, and the values ​​corresponding to the same key can have different data types. The conversion of free-schema document model data to relational model data with fixed schema constraints requires that the schema of the document model data is relatively fixed and cannot be arbitrarily free. The document data schema allows a certain degree of evolution, and appropriately increases or decreases key-value pairs.

[0101] Faced with document model data, it is difficult to manually extract a suitable fixed document data model (table, column information, etc.) from it. If the document data has a multi-level nested structure, it is even more difficult to manually write mapping configuration information, which requires the user to have a clear understanding of the evolution history of the document data model to achieve this task. In order to solve the above problems, an embodiment of the present invention provides a solution for automatically generating mapping configuration information, generating a rough configuration template (mapping configuration information) so that the user can fine-tune it for data conversion tasks.

[0102] The following is an explanation of the process of dynamically updating mapping configuration information. Figure 3 As shown, the step 10 includes:

[0103] Step 101: Determine at least one piece of historical document model data whose structural difference is within a preset range.

[0104] The preset range is determined by those skilled in the art based on empirical analysis in combination with specific usage scenarios and is not limited here. Since the structure of the current document model data that is designated to be converted together each time is basically similar, the structure of the corresponding historical document model data is also basically similar and all belong to the same type of document model data; whether they are the same type of document model data is determined by the user when configuring the document model data in the document database.

[0105] Step 102: When the amount of the historical document model data is greater than the document amount threshold, the historical document model data is sampled to obtain a document data subset; when the amount of the historical document model data is less than the document amount threshold, the historical document model data is used as a document data subset.

[0106] The document quantity threshold is determined by those skilled in the art based on empirical analysis in combination with specific usage scenarios and is not limited here; in an optional embodiment, the document quantity threshold may be specified with reference to the quantity of document model data in a document database.

[0107] When the amount of document data reaches a certain scale (i.e., the threshold of the number of documents), if all the historical document model data from the beginning of the operation of the upper-layer application to the sampling are used to generate the mapping configuration information, since there are basically similar structures in a large amount of historical document model data, it will produce great redundancy. Even if this redundancy can be solved by merging, the corresponding merging operation will consume huge computing resources, resulting in too low generation efficiency. Therefore, the embodiment of the present invention selects part of the historical document model data to determine the document data subset through the sampling method, and uses the document data subset as a reference for generating the mapping configuration information.

[0108] If the amount of document data is relatively small, all document model data under the specified collection in the document database are sampled for automatic generation of mapping configuration information. If the number of all document model data under the specified collection is less than the document quantity threshold, all document model data under the collection are sampled, so that the relationship table that can be mapped and the column fields in the relationship table can be fully obtained.

[0109] Step 103: Obtain a corresponding template mapping-linked list nested body according to the document data subset, and generate mapping configuration information according to the template mapping-linked list nested body.

[0110] The readability of general document model data is poor, especially in scenarios with many nested levels, or when the user is unfamiliar with the document model data. It is extremely difficult and impractical for the user to design matching rules (equivalent to the mapping configuration information in the embodiment of the present invention) based on their own understanding and needs of the document model data. The embodiment of the present invention provides a method for assisting the user in generating mapping configuration information. The user only needs to check the mapping configuration information and can selectively fine-tune the mapping configuration information as needed.

[0111] When the amount of historical document model data is greater than the document amount threshold, the embodiment of the present invention first determines the generation time of the historical document model data in the upper layer application and the running period of the upper layer application, so as to sample the historical document model data with reference to the generation time and the running period. Specifically, Figure 4 As shown, in step 102, the historical document model data is sampled to obtain a document data subset including:

[0112] Step 1021: Determine the generation time of the historical document model data in the upper layer application.

[0113] Step 1022: When the frequency of adjustment of the historical document model data by the upper layer application is less than the first frequency, the running period of the upper layer application is determined as the early stage of system operation.

[0114] Among them, the first frequency is obtained by technicians in this field based on empirical analysis in combination with specific usage scenarios. It should be noted that when the value of the selected first frequency is small, the period of time determined to belong to the early stage of system operation is longer, which may cause the subsequently generated mapping configuration information to contain more information from the early stage of system operation, but this problem can be solved by adjusting in combination with other sampling factors; and because the early stage of system operation accounts for a high proportion, the finally generated mapping configuration information may lack data mode information from the mid-term and stable periods of system operation, which will reduce the efficiency and effect of conversion using the mapping configuration information to a certain extent, and vice versa.

[0115] In an optional embodiment, the user determines the time parameter t1 according to the first frequency, and when the running time of the upper layer application is less than the time parameter t1, the running time is determined as the early stage of system operation.

[0116] Step 1023: When the adjustment frequency is greater than the first frequency and less than the second frequency, the operation period is determined as the middle period of system operation.

[0117] Among them, the second frequency is obtained by technicians in this field based on empirical analysis in combination with specific usage scenarios. Similarly, when the value of the selected second frequency is small, the period determined to belong to the mid-term of system operation is longer, which may cause the subsequently generated mapping configuration information to contain more information about the mid-term of system operation, but it can be adjusted in combination with other sampling factors (such as the first frequency) to solve this problem. In an optional embodiment, the user determines the time parameter t2 based on the second frequency, and within the range where the running time of the upper-level application is greater than the time parameter t1 and less than the time parameter t2, the running time is determined as the mid-term of system operation.

[0118] Step 1024: When the adjustment frequency is greater than or equal to the second frequency, the operation period is determined as a system operation stable period.

[0119] In an optional embodiment, when the running time of the upper layer application is greater than or equal to the time parameter t2, the running time is determined as the system running stable period.

[0120] Step 1025: According to the generation time corresponding to the historical document model data in the early stage of the system operation, the middle stage of the system operation, the stable stage of the system operation and each period, determine the historical document model data to be sampled, and obtain the document data subset. Each historical document model data corresponds to a generation time, and the historical document model data in each period refers to all historical document model data whose generation time is respectively in the early stage of the system operation, the middle stage of the system operation and the stable stage of the system operation.

[0121] The embodiment of the present invention determines the sampling generation method of the document data subset based on the design and evolution rules of the upper layer application. Figure 5As shown, it is generally more likely that the document data model will be adjusted in the early stage of system operation, and the document data model will generally not be modified after the system is running stably. If the historical document model data is randomly sampled from the beginning of the corresponding document model data from the beginning of the upper-level application to all the historical document model data before sampling, it is completely impossible to control whether the generated mapping configuration information completely contains the relationship tables that can be mapped and the column fields in the relationship tables and other information; and if all the historical document model data are sampled on average, then referring to the generated mapping configuration information, the adjustment of the document model data in the early stage of system operation that can be obtained is less, and the adjustment in the stable period of system operation is more, and it is impossible to completely contain the relationship tables that can be mapped and the column fields in the relationship tables and other information. In order to solve the above problems, specifically, as Figure 6 As shown, the step 1025 includes:

[0122] Step 10251: According to the number of iterations of the historical document model data in the early stage of the system operation, configure an early time for the early stage of the system operation; for the historical document model data generated in the early stage of the system operation, sample each time at intervals of the early time to obtain an early number of document model data.

[0123] The number of iterations refers to the number of times the target document model data is modified. The early number is obtained by those skilled in the art based on empirical analysis in combination with specific usage requirements. In an optional embodiment, the early time is one day and the early number is 1.

[0124] The target document model data is continuously modified from the beginning of the operation of the upper-layer application to before sampling. Such modification may include modification of specific instances (e.g., specific values) in the document model data, or may include adjustment of the document data model; each time the current historical document model data is modified, new historical document model data is obtained at the corresponding generation time. Since the modification frequency in different periods (e.g., early system operation or mid-system operation) is often different, the embodiment of the present invention specifies the corresponding sampling scheme in different periods; since it is not convenient to determine whether each data in a large amount of data has modified the data model one by one in the same period of the upper-layer application (e.g., early system operation), the embodiment of the present invention performs average sampling of the corresponding historical document model data at a certain time interval in the same period, so as to obtain the data model of each modification as much as possible, and then obtain the relationship table corresponding to the target document model data in each period and the column fields in the relationship table and other information.

[0125] Step 10252: According to the number of iterations of the historical document model data in the mid-term of the system operation, configure the mid-term time for the mid-term of the system operation; for the historical document model data generated in the mid-term of the system operation, sample each time at intervals of the mid-term time to obtain a mid-term amount of document model data.

[0126] The mid-term quantity is obtained by those skilled in the art based on empirical analysis in combination with specific usage requirements. In an optional embodiment, the mid-term time is one week and the mid-term quantity is 1.

[0127] Step 10253: According to the number of iterations of the historical document model data in the system operation stable period, configure the stable period time for the system operation stable period; sample the historical document model data generated during the system operation stable period at intervals of the stable period time to obtain the document model data of the stable period quantity.

[0128] The number of stable periods is obtained by those skilled in the art based on empirical analysis in combination with specific usage requirements. In an optional embodiment, the stable period is one month and the number of stable periods is one.

[0129] Step 10254: Integrate the early quantity of document model data, the mid-term quantity of document model data, and the stable period quantity of document model data to obtain a document data subset.

[0130] like Figure 5 As shown, based on this observation, the sampling frequency of document model data in the early stage of system operation should be larger, and the sampling frequency of document model data should be smaller after the system runs stably; that is, the stable period is generally longer than the mid-term period, and the mid-term period is generally longer than the early period; the mid-term quantity is generally greater than or equal to the stable period quantity, and the early quantity is generally greater than or equal to the mid-term quantity. For example, in the early stage of system operation, document model data is extracted daily, and one document model data is extracted every day; in the mid-term of system operation, document model data is extracted weekly, and one document model data is extracted every week; in the stable period of system operation, document model data is extracted monthly, and one document model data is extracted every month.

[0131] After obtaining the document data subset, the mapping configuration information is generated based on the document data subset; specifically, Figure 7 As shown, step 103 includes:

[0132] Step 1031: Represent the document data subset in JSON format as a template mapping-linked list nested body of a tree structure.

[0133] Among them, Figure 8 As shown, the form of the map-linked list nested body is multi-layer nested (MAP / LIST(MAP)), presenting a tree structure, that is, a nested body of key-value pairs and linked lists. Except for the innermost key-value pairs, the values ​​of the key-value pairs at other levels are linked lists LIST. In the process of generating the mapping configuration information of the current document model data, the historical document model data is represented as a template map-linked list nested body.

[0134] Step 1032: Obtain each mapping body in the template mapping-linked list nested body, use the name of the mapping body as the name MAP of the relationship table configuration information, and use the name of the mapping body as the name of the relationship table configuration information.

[0135] Combination Figure 8 In the template mapping-linked list nested body, one JSON level corresponds to one relational table, and multiple relational tables are used to completely store all the data information contained in a JSON level; the mapping configuration information defines how to extract data information from the mapping-linked list nested body and convert it into row data.

[0136] A mapping body (MAP) corresponds to a JSON level and is a key-value pair set, where key is the column name and value is the column value. The mapping body (MAP) of one or more data in an attached level constitutes a linked list (LIST(MAP)), corresponding to the array object in the JSON object, which is the value part of a key-value pair item in its parent mapping body. Similarly, the main level can have multiple attached levels, and an attached level can have its own attached levels.

[0137] Step 1033: determine the key name key of each key-value pair in the mapping body as the column name of the column in the relationship table configuration information; determine the data type of the value value of the key-value pair as the column data type of the column in the relationship table configuration information.

[0138] Step 1034: Determine the first table information to be merged with the same name from the relationship table configuration information, merge the column fields of the first table information to be merged to obtain the second table information to be merged; merge the columns with the same column names in the second table information to be merged to obtain merged table information; obtain mapping configuration information based on the merged table information and the information in the relationship table configuration information except the first table information to be merged.

[0139] The column field is a specific attribute of an entity in a relational table, including a column name and a value corresponding to the column.

[0140] by Fig.10 For example, the table corresponding to the telephone number node is a specific instance of the first table information to be merged. The entities in the first table information to be merged include an entity with an ID of 16, and the attribute name (NUMBER) and attribute value (179****5577) of the entity are column fields.

[0141] Since the current document model data of the embodiment of the present invention often has an evolution process, that is, there is more than one template mapping-linked list nested body used to generate mapping configuration information, and the table structure is similar, it is very likely that the first table information to be merged with the same name and similar table structure will be parsed from the template mapping-linked list nested body with similar table structure. When there are two relationship table configuration information named TEL_NUMS, in order to reduce data redundancy as much as possible, the column fields of the two are merged; for example, the first first table information to be merged named TEL_NUMS includes multiple columns and corresponding column fields, and the second first table information to be merged named TEL_NUMS includes multiple columns and corresponding column fields. The column names and column data types of the two columns overlap to a high degree. In order to reduce this part of redundant data, the two first table information to be merged are merged.

[0142] After solving the data redundancy problem of the same type of name, since multiple template mapping-linked list nested bodies often have similar table structures, after merging the column fields into one table (i.e., a second table to be merged information), there will often be multiple columns with the same column name, and these columns need to be merged.

[0143] When merging, if the columns with the same column name have different column data types in both columns, the column data type of the column in the merged mapping configuration information is oriented to an upwardly compatible type. The specific upward compatibility rules are determined by those skilled in the art according to the database used. For example, in a MySQL database, an integer type int can be implicitly converted to a floating point type float or double.

[0144] The embodiment of the present invention merges relational table configuration information with the same name into one relational table configuration information, that is, all the column fields therein are merged into the same relational table configuration information; in order to further avoid data redundancy in the second table information to be merged after the merger, the columns with the same column names in the second table information to be merged are also merged into the same column; the merged result is used as the automatically generated mapping configuration information; since the subsequent conversion of the current document model data is based on the mapping configuration information, reducing the data redundancy of the table structure in the mapping configuration information can directly and greatly reduce the data redundancy after the conversion, thereby avoiding the impact on the upper-level application.

[0145] It should be noted that the automatic generation of mapping configuration information using the embodiment of the present invention greatly improves efficiency, but it is impossible to determine whether the automatically generated mapping configuration information completely contains all table information and column information. At this time, user intervention is generally required to make fine adjustments based on the automatically generated mapping configuration information before using it to convert the current document model data.

[0146] The following is an explanation of fine-tuning:

[0147] Under the same document model data, when there are multiple data in the child node linked list, the key sets of different data are different. The nodes in the tree structure (i.e., template mapping-linked list nested body), the linked list of values ​​in the nodes, and the elements of the linked list are all composed of multiple key-value pairs. All elements in this linked list belong to the same relational table. The number of columns in the relational table and the data type of each column are fixed, while the key-value pairs of each element in the linked list may not be fixed; among them, the application scenario generally requires basic consistency and should contain the main keys; based on the key-value pair set in the sampling mapping configuration information, the extracted table structure may not contain all columns, and there may be columns that are completely unnecessary in the actual application, so the user needs to adjust it to add or delete certain columns.

[0148] In multiple document model data, the key sets in the elements of the linked lists under the corresponding nodes are also different. The basic principle of the sampling method is to use all the keys that appear in the sampled data as the column names of the relationship table to generate mapping configuration information, but this setting is likely to be inconsistent with actual needs and needs to be adjusted by the user.

[0149] In the prior art, simple document model data without multi-layer nesting can be directly mapped to relational model data; however, complex document model data with multi-layer nesting greatly increases the difficulty of conversion, both in terms of readability and the difficulty of obtaining the evolutionary process of multiple adjustments and modifications. Even if the conversion can be achieved by users who have a clear understanding of the evolutionary process, it is impossible to accurately grasp the relationship between each table structure in large amounts of data. Therefore, data redundancy is always artificially introduced, resulting in unclear table structure of relational model data, resulting in poor practicality of the data synchronization system.

[0150] The embodiment of the present invention extracts the table structure and generates row data according to the mapping configuration information. The process is described below. Fig. 9 As shown, the step 20 includes:

[0151] Step 201: The current document model data is represented in JSON format through JsonPath as a tree-structured nested body of a mapping-linked list to be converted.

[0152] Among them, JsonPath is an information extraction library, through which JSON document data is converted into a document path object with a tree structure. In the process of obtaining row data according to mapping configuration information, the current document model data is represented as a nested body of mapping-linked lists to be converted, and the structure of the nested body of mapping-linked lists to be converted is the same as that of the template mapping-linked lists.

[0153] like Figure 8As shown, the document model data is represented as a nested body of mapping linked lists (MAP / LIST(MAP)) after conversion within the ETL system; the term "main-subordinate level" is used to describe the document data format of multi-level nested representation. A main level has 0, 1 or more subordinate levels, and the subordinate level may also have its own subordinate levels, forming a multi-level nested structure.

[0154] A JSON level in the mapping-linked list nested body corresponds to a relational table, and multiple relational tables are used to completely store all the data information contained in a JSON level; the mapping configuration information defines how to extract data information from the mapping-linked list nested body and convert it into row data.

[0155] A mapping body (MAP) corresponds to a JSON level and is a key-value pair set, where key is the column name and value is the column value. The mapping body (MAP) of one or more data in an attached level constitutes a linked list (LIST(MAP)), corresponding to the array object in the JSON object, which is the value part of a key-value pair item in its parent mapping body. Similarly, the main level can have multiple attached levels, and an attached level can have its own attached levels.

[0156] The embodiment of the present invention provides a nested body of a mapping-linked list to be transferred such as Fig.10 As shown. Fig.11 The document model data is shown below. Fig.10 The personal information data is presented in the form of a tree structure. Fig.10 In the value1, value2 is "2023-10-10T10:24:34.000Z", value3 is "2023-10-10T10:33:31.000Z", value4 is "2023-10-11T02:33:31.000Z", value5 is "2023-10-11T02:53:30.000Z", and the corresponding values ​​of value6 and value7 are in Fig.11It is not shown in . Among them, this example uses a tree structure composed of a person's basic information, phone numbers, call records, and bank card information to describe the constituent elements of the document model data, including four types of entities and three levels of association relationships. The reality depicted by this document model data is that a person can have 0, 1, or multiple phone numbers, each phone number can have 0, 1, or multiple call records (limited to a certain time range), and a person can also have 0, 1, or multiple bank cards. A certain system design organizes this information centrally in a document model data for upper-layer application programs to query and obtain. When the main level at the JSON level is the basic node of personal information, the key-value pairs in the basic node of personal information are represented in the form of "key-value" as "ID-10", "NAME-Zhang San", "GENDER-male", "CARDS-list_c", and "TEL_NUMS-list_t"; in the key-value pair "CARDS-list_c", the value is list_c, and list_c corresponds to the bank card node (mapping body), which is the next-level JSON level; in the attached bank card node, "ID-15", "PERSON_ID-10", "CARD-value1", "BANK_NAME-First Bank" form a linked list body, and so on.

[0157] The mapping-linked list nested body is essentially a tree structure. Each level corresponds to a node of the tree, and each node can correspond to a table of relational model data. The keys in the mapping body form the columns of the relational table, and the hierarchical relationship between the main and attached levels is represented by the primary and foreign keys of the relational table.

[0158] Step 202: Use the table information in the mapping configuration information to match the to-be-converted mapping-linked list nested body, and obtain the to-be-merged relational table according to the mapping body in the to-be-converted mapping-linked list nested body.

[0159] After each current document model data is read into the ETL system and converted into a to-be-converted mapping-linked list nested body (MAP / LIST(MAP)) in a tree structure, each node (i.e., the mapping body) in this tree structure corresponds to a relational table. The keys in the mapping body (i.e., MAP) serve as the columns of the relational table, and the data type of the value serves as the column data type in the relational table. Each document model data can generate one or more relational tables.

[0160] Among them, the table information refers to the field corresponding to the table name in the mapping configuration information; for example, as Fig.12 shown is a specific example of mapping configuration information implemented in the YAML (Yet Another Markup Language) format. "table:" etc. are used to mark the table name, and the table information is "PERSON", "PHONE", "RECORD", or "CARDS".

[0161] Step 203: Determine redundant relationship tables with the same table name from the relationship tables to be merged, and determine the relationship tables to be merged other than the redundant relationship tables as non-redundant relationship tables.

[0162] Step 204: merge the column fields of the redundant relationship table to obtain a merged relationship table; merge the columns with the same column name in the merged relationship table to obtain merged row data; obtain the row data according to the merged row data and the non-redundant relationship table.

[0163] The merging process for merging table information when generating mapping configuration information is the same as the merging process for merging relationship tables when obtaining row data. The following example illustrates the merging process when obtaining row data: Fig.12 The mapping configuration information shown is similar to Fig.10 After the nested mapping-linked list to be converted is matched, since the structure of the current document model data that is designated to be converted together each time is basically similar, it is very likely that the relationship tables to be merged with the same table name and similar table structure will be parsed from the nested mapping-linked list to be converted with similar table structure. When there are two relationship tables to be merged and both have the table name TEL_NUMS, that is, when two telephone number tables corresponding to telephone number nodes are generated, in order to reduce data redundancy as much as possible, the column fields in the two are merged. This process is equivalent to merging each record in the two into the same merged relationship table; and the merged relationship table often has columns with the same column name (for example, columns with the name "NUMBER"), and then the columns with the same column name are merged. Similarly, when merging, the column data types are compatible types.

[0164] The embodiment of the present invention uses mapping configuration information for matching, first obtains the table structure, and then fills it based on the table structure to obtain the relationship table to be merged, which greatly improves the clarity of the table structure and has greatly reduced data redundancy. When obtaining row data, data redundancy is further reduced by merging, solving the problem of affecting upper-level applications due to data redundancy and unclear table structure.

[0165] The specific process of obtaining the relationship table to be merged is described below: Fig.13 As shown, step 202 includes:

[0166] Step 2021: extract the name of each mapping body in the mapping-linked list nested body to be converted according to the table information, and initialize the extracted name as the table name of the relationship table to be merged corresponding to the mapping body.

[0167] For example, the mapping-linked list nested body to be transferred includes a personal basic information node (mapping body), and the name PERSON of the mapping body is initialized to the table name PERSON.

[0168] Step 2022: Determine the key name of each key-value pair in the mapping body as the column name of the relationship table to be merged.

[0169] For example, the personal basic information node (mapping body) includes a key-value pair set of personal basic information, that is, a key-value pair set with key names of ID, NAME, GENDER, CARDS and TEL_NUMS, and ID, NAME, GENDER, CARDS and TEL_NUMS are used as column names of columns in the personal basic information table respectively.

[0170] Step 2023: Determine the data type of the value of the key-value pair as the column data type of the column of the to-be-merged relational table.

[0171] The data type of the list values ​​corresponding to ID, NAME, GENDER, CARDS, and TEL_NUMS is used as the column data type of the column corresponding to the corresponding column name. For example, in the personal basic information node, the list value corresponding to NAME (such as Fig.10 The data type of "Zhang San" shown in the figure is String. In the personal basic information table, the column data type of the column named NAME is String.

[0172] When the list value is List, the data type of the primary key in List is used as the corresponding column data type. For example, the mapping body corresponding to the personal basic information node and the bank card node is MAP1(List_c(MAP2)); MAP1 is the personal basic information node; List_c is the bank card node, and List_c includes the following: Fig.10 The multiple bank card information with ID 15 is shown; MAP2 is a key-value pair set in the bank card node, and the key name is a key-value pair set of ID, PERSON_ID, CARD and BANK_NAME in the bank card information with ID 15. Among them, the key of the mapping body is CARDS in the personal basic information node, and CARDS is used as the column name of the personal basic information (PERSON) table; because the list value is List_c, the primary key in List_c is ID, and the data type of the ID is used as the column data type of the column named CARDS in the generated personal basic information table.

[0173] Step 2024: Obtain the table structure according to the column names and column data types of the columns of the to-be-merged relational table; determine the values ​​at corresponding positions in the table structure according to the values ​​of the key-value pairs in the mapping body; establish dependency relationships between different to-be-merged relational tables according to the parent-child relationships between different mapping bodies to obtain the to-be-merged relational table.

[0174] After initializing the relationship table, the embodiment of the present invention fills the table structure of the relationship table according to the corresponding parts in the nested body of the mapping-linked list to be converted, and generates the corresponding position, column name and column data type of each column in the relationship table, but the primary key or foreign key of the relationship table is not set here.

[0175] The embodiment of the present invention represents the current document model data with a nested body (tree structure) of a mapping to be converted-linked list, and specifies the node in the tree structure through JSONPATH, corresponding to a table of the relational model, and the data in the node is organized together with a linked list, each element in the linked list is a set of multiple key-value pairs, and there are one or more elements in the linked list; extract the key of the key-value pair (key-value) in the linked list element as the column name of the relational table, extract the corresponding value to infer the data type of the column, and define the table structure of the relational table with the column name and column type. The structure of one or more tables can be obtained by mapping the configuration information, and the document model has been converted into a relational model at this step.

[0176] After obtaining the table structure, the value of the key-value pair in each element of the linked list under the node is extracted through JSONPATH to form a row of data in a relational data model; the "path" attribute of each relational table specifies the node corresponding to the relational table (such as CARDS or TEL_NUMS), and the "columns" attribute specifies the columns that need to be filled in the relational table. Refer to the above structure to obtain the list value to fill the constructed relational table. The dependency relationship between data in different relational tables (primary and foreign keys, etc.) can be directly obtained according to the mapping configuration information when extracting data. The following is an explanation of the process of building dependency relationships:

[0177] In order to construct information at the relational table level, the embodiment of the present invention obtains the dependency relationship between relational tables based on the parent-child relationship between each node (i.e., level) of the tree structure; for example, in the personal basic information table, list_c is the parent node of each node in the bank card node, the column named ID is the primary key, which is used to uniquely identify the individual, and the column named CARDS is the foreign key. The table corresponding to the foreign key is the bank card table, and the primary key of the bank card table is the column named ID in the table. Specifically, Fig.14 As shown, in step 2024, the dependency relationship between different to-be-merged relationship tables is established according to the parent-child relationship between different mapping bodies to obtain the to-be-merged relationship table, including:

[0178] Step 20241: According to the mapping configuration information, obtain the parent-child relationship between each mapping body in the nested body of the mapping-linked list to be converted.

[0179] For example, according to Fig.12 The table structure generated by the mapping configuration information shown in the figure shows that in the PERSON table, there are columns named "ID", "NAME" and "GENDER". The values ​​corresponding to "ID", "NAME" and "GENDER" are obtained from the mapping body, which are 10, Zhang San and Male respectively. According to the mapping configuration information, the column name of the primary key column in the PERSON table is ID (such as Fig.12 , sets the primary key of the PERSON table. According to the association relationship between each relationship table in the mapping configuration information, the parent-child relationship between the personal basic information node and the bank card node is obtained. For example, in the personal basic information node with the mapping body name PERSON, the key is CARDS, and the corresponding value is list_c. The personal basic information node is the parent node, and the bank card node is the child node. The key-value pair in the child node is list_c. Among them, the way of storing the association relationship between each relationship table in the mapping configuration information is determined by those skilled in the art based on empirical analysis in combination with specific usage scenarios, and is not limited here; Fig.12 As shown, in an optional embodiment, the association relationship is represented by indentation of fields in the mapping configuration information.

[0180] Step 20242: According to the parent-child relationship, a first foreign key column is added to the to-be-merged relationship table corresponding to the mapping body of the child node, and the value of the first foreign key column is set to the primary key column value of the to-be-merged relationship table corresponding to the parent node.

[0181] like Fig.10 As shown, according to the parent-child relationship, the first foreign key column PERSON_ID is added to the CARDS table, and the value of the column is set to 10 (ie, the value of the primary key "ID" of the PERSON table).

[0182] Step 20243: Add a second foreign key column to the to-be-merged relationship table corresponding to the parent node, and set the value of the second foreign key column to the primary key column of the to-be-merged relationship table corresponding to the child node, so as to establish a dependency relationship between different to-be-merged relationship tables.

[0183] Add a second foreign key column CARDS to the PERSON table and set the value of this column as the primary key column of the CARDS table, that is, add a foreign key constraint to the column, according to the format of "alter table CARDS table add foreign key (ID) references PERSON (ID)" in Structured Query Language (SQL).

[0184] According to the association between each relational table in the mapping configuration information, the dependency relationship between the personal basic information table and the bank card table is obtained. For example, the column named CARDS is a foreign key, the table corresponding to the foreign key is the bank card table, and the primary key of the bank card table is the column named ID in the bank card table.

[0185] For example, the four relationship tables obtained by modeling the personal information nodes are:

[0186] PERSON(ID,NAME,GENDER) - Personal basic information table, the fields include ID (here ID is the person's ID), name NAME, and gender GENDER.

[0187] PHONE(ID,PERSON_ID,NUMBER) - Phone number table, fields include ID (here ID is the ID of the phone number), person ID, and phone number.

[0188] RECORD(ID,PHONE_ID,START_TIME,END_TIME) - call record table, the fields include ID (here ID is the ID of the call record), phone ID, start time, and end time.

[0189] CARDS (ID, PERSON_ID, CARD, BANK_NAME) - bank card table, fields include ID (here ID is the ID of the bank card), person ID, card number, and bank name.

[0190] The relationship between the tables is: the personal basic information table is the main table, and one piece of data represents a person's basic information (ID, name and gender); the phone number table is one of the appendices of the personal basic information table, which represents the phone numbers (ID, person ID and number) owned by a person; the call record table is an appendix of the phone number table, which records the call records of the phone number (ID, phone ID, call start time and call end time); the bank card table is one of the appendices of the personal basic information table, which represents the bank cards owned by a person (ID, person ID, card number and bank name). A person has 0, 1 or more phone numbers; each phone number must belong to one person, and each phone number has 0, 1 or more call records, and each call record must be associated with a phone number; a person has 0, 1 or more bank cards, and each bank card data must be associated with one person. The row data in the above relationship tables has three levels and two sets of dependencies.

[0191] Embodiment 2:

[0192] This embodiment provides a device for converting document model data into relational model data based on the method for converting document model data into relational model data in Embodiment 1. In a componentized ETL system, a data conversion task is constructed.

[0193] like Fig.15 Specifically, the basic components required to complete the data conversion task include a document reading component, a document line conversion component and at least one table loading component; wherein:

[0194] The document reading component is used to obtain historical document model data, and the document line conversion component is used to dynamically update mapping configuration information according to the historical document model data.

[0195] The document reading component is also used to obtain the current document model data; the document conversion component is also used to extract data from the current document model data according to the mapping configuration information to obtain row data; in order to distribute the row data to the loading component, the loading component to which each row data needs to be distributed is determined based on the table structure and dependency relationship contained in the current document model data, and the row data is sent to the corresponding loading component. Fig.15 In the ETL system shown, if the document-to-row component sends a row of data Row1 to the table loading 01 component, and the row of data Row1 is determined to be the i-th row in the bank card table according to the table structure, when the row data in the bank card table is sent, it is sent to the table loading 01 component.

[0196] The table loading component is used to load the received row data into the corresponding relational table to obtain the relational model data. The row data is written into the corresponding relational table in the relational database by the table loading component, or the relational model data in the form of a file is generated, and the data after the table loading operation is used as the corresponding relational model data. For example, Fig.15 As shown, the document-to-row component sends the row data Row01 to the corresponding table loading 01 component, and the table loading 01 component loads it to the corresponding position in the relational database according to the table structure and dependency relationship.

[0197] In one embodiment, a document reading component is used to transfer current document model data to a document row conversion component; the document row conversion component is used to generate mapping configuration information based on a template mapping-linked list nested body; it is also used to use the table information in the mapping configuration information to match the mapping-linked list nested body to be converted, and to obtain the table structure of the relationship table to be merged based on the mapping body in the mapping-linked list nested body to be converted, so as to obtain row data; and the row data is transmitted to a table loading component.

[0198] In actual application scenarios, data transmission between different components still maintains the form of structured row data, so as to facilitate the expansion of the function of converting multi-level nested document model data into relational model data in the traditional ETL system, and utilize the existing conversion components in the ETL system to convert the content of row data.

[0199] like Fig.16 As shown, when the required relational model data needs to be converted in content (such as data desensitization, etc.), a document conversion component is also included; wherein:

[0200] The document conversion component is used to perform content conversion on the received row data to obtain processed row data; transmit the processed row data to the corresponding table loading component to obtain the relational model data through the table loading component;

[0201] For example, the conversion 01 component performs content conversion on the row data Row1 output by the document conversion component according to the requirements and the table structure, such as data desensitization, to obtain the processed row data, and the conversion 01 component sends it to the corresponding table loading 01 component.

[0202] Operation details not mentioned in the embodiments of the present invention are determined by those skilled in the art according to specific usage scenarios without paying any creative work and by referring to the prior art, and are not elaborated here.

[0203] Embodiment 3:

[0204] like Fig.17, is a schematic diagram of the architecture of an apparatus for converting document model data into relational model data according to an embodiment of the present invention. The apparatus for converting document model data into relational model data according to this embodiment includes one or more processors 31 and a memory 32. Fig.17 A processor 31 is taken as an example.

[0205] The processor 31 and the memory 32 may be connected via a bus or other means. Fig.17 The example of connecting through bus is taken in the following.

[0206] The memory 32 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the method for converting document model data into relational model data in Embodiment 1. The processor 31 executes the method for converting document model data into relational model data by running the non-volatile software programs and instructions stored in the memory 32.

[0207] The memory 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 32 may optionally include a memory remotely arranged relative to the processor 31, and these remote memories may be connected to the processor 31 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0208] The program instructions / modules are stored in the memory 32, and when executed by the one or more processors 31, execute the method of converting document model data into relational model data in the above-mentioned embodiment 1, for example, execute the various steps shown in the method of converting document model data into relational model data described above.

[0209] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0210] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0211] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for converting document model data into relational model data, characterized in that: include: Dynamically update mapping configuration information based on historical document model data; According to the mapping configuration information, data is extracted from the current document model data to obtain row data; wherein, according to the mapping configuration information, data is extracted from the current document model data to obtain row data, including: representing the current document model data in JSON format as a tree-structured mapping-linked list nested body to be converted; using the table information in the mapping configuration information to match the mapping-linked list nested body to be converted, and obtaining a relationship table to be merged according to the mapping body in the mapping-linked list nested body to be converted; determining redundant relationship tables with the same table name from the relationship table to be merged, and determining the relationship tables to be merged other than the redundant relationship table as non-redundant relationship tables; merging the column fields of the redundant relationship table to obtain a merged relationship table; merging the columns with the same column name in the merged relationship table to obtain merged row data; obtaining the row data according to the merged row data and the non-redundant relationship table; The row data is loaded into the corresponding relational table to obtain relational model data.

2. The method for converting document model data into relational model data according to claim 1, characterized in that: The dynamically updating mapping configuration information according to the historical document model data includes: Determine at least one historical document model data whose structural difference is within a preset range; When the amount of the historical document model data is greater than the document amount threshold, sampling the historical document model data to obtain a document data subset; when the amount of the historical document model data is less than the document amount threshold, using the historical document model data as a document data subset; A corresponding template mapping-linked list nested body is obtained according to the document data subset, and mapping configuration information is generated according to the template mapping-linked list nested body.

3. The method for converting document model data into relational model data according to claim 2, characterized in that: The sampling of the historical document model data to obtain a document data subset includes: Determining the generation time of the historical document model data in the upper layer application; When the frequency of adjustment of the historical document model data by the upper layer application is less than the first frequency, determining the running period of the upper layer application as the early stage of system running; When the adjustment frequency is greater than the first frequency and less than the second frequency, the operation period is determined as the middle period of system operation; When the adjustment frequency is greater than or equal to the second frequency, determining the operation period as a system operation stable period; According to the generation time corresponding to the historical document model data in the early stage of the system operation, the middle stage of the system operation, the stable stage of the system operation and each period, the historical document model data required for sampling is determined to obtain a document data subset.

4. The method for converting document model data into relational model data according to claim 3, characterized in that: The historical document model data to be sampled is determined according to the generation time corresponding to the historical document model data in the early stage of the system operation, the middle stage of the system operation, the stable stage of the system operation and each period, and the document data subset obtained includes: According to the number of iterations of the historical document model data in the early stage of the system operation, an early time is configured for the early stage of the system operation; for the historical document model data generated at the early stage of the system operation, sampling is performed at intervals of the early time each time to obtain an early number of document model data; According to the number of iterations of the historical document model data in the mid-term of the system operation, a mid-term time is configured for the mid-term of the system operation; for the historical document model data generated at the mid-term of the system operation, sampling is performed at intervals of the mid-term time each time to obtain a mid-term amount of document model data; According to the number of iterations of the historical document model data in the system operation stable period, configuring the stable period time for the system operation stable period; sampling the historical document model data generated in the system operation stable period at intervals of the stable period time each time to obtain the document model data of the stable period quantity; The early amount of document model data, the mid-term amount of document model data and the stable period amount of document model data are integrated to obtain a document data subset.

5. The method for converting document model data into relational model data according to claim 2, characterized in that: The obtaining of a corresponding template mapping-linked list nested body according to the document data subset and generating mapping configuration information according to the template mapping-linked list nested body comprises: Representing the document data subset in JSON format as a template mapping-linked list nested body of a tree structure; Obtain each mapping body in the template mapping-linked list nested body, and use the name of the mapping body as the name of the relationship table configuration information; Determine the key name of each key-value pair in the mapping body as the column name of the column in the relationship table configuration information; determine the data type of the value of the key-value pair as the column data type of the column in the relationship table configuration information; Determine first table information to be merged with the same name from the relationship table configuration information, merge the column fields of the first table information to be merged to obtain second table information to be merged; merge the columns with the same column names in the second table information to be merged to obtain merged table information; and obtain mapping configuration information based on the merged table information and information other than the first table information to be merged in the relationship table configuration information.

6. The method for converting document model data into relational model data according to claim 1, characterized in that: The using table information in the mapping configuration information to match the mapping-linked list nested body to be transferred, and obtaining the relationship table to be merged according to the mapping body in the mapping-linked list nested body to be transferred includes: According to the table information, extract the name of each mapping body in the mapping-linked list nested body to be converted, and initialize the extracted name as the table name of the relationship table to be merged corresponding to the mapping body; Determine the key name of each key-value pair in the mapping body as the column name of the relationship table to be merged; Determine the data type of the value of the key-value pair as the column data type of the column of the to-be-merged relational table; The table structure is obtained according to the column names and column data types of the columns of the to-be-merged relational table; the value at the corresponding position in the table structure is determined according to the value of the key-value pair in the mapping body; and the dependency relationship between different to-be-merged relational tables is established according to the parent-child relationship between different mapping bodies to obtain the to-be-merged relational table.

7. The method for converting document model data into relational model data according to claim 6, characterized in that: The step of establishing dependency relationships between different to-be-merged relationship tables according to the parent-child relationships between different mapping bodies to obtain the to-be-merged relationship tables includes: According to the mapping configuration information, a parent-child relationship between each mapping body in the nested body of the mapping-linked list to be transferred is obtained; According to the parent-child relationship, a first foreign key column is added to the to-be-merged relationship table corresponding to the mapping body of the child node, and the value of the first foreign key column is set to the primary key column value of the to-be-merged relationship table corresponding to the parent node; A second foreign key column is added to the to-be-merged relation table corresponding to the parent node, and the value of the second foreign key column is set as the primary key column of the to-be-merged relation table corresponding to the child node, so as to establish a dependency relationship between different to-be-merged relation tables.

8. A device for converting document model data into relational model data, characterized in that: The method for converting document model data into relational model data according to any one of claims 1 to 7 comprises a document reading component, a document line conversion component and at least one table loading component; wherein: The document reading component is used to obtain historical document model data, and the document line conversion component is used to dynamically update mapping configuration information according to the historical document model data; The document reading component is also used to obtain current document model data; the document-to-row component is also used to extract data from the current document model data according to the mapping configuration information to obtain row data; wherein, the document-to-row component is also used to represent the current document model data in JSON format as a tree-structured mapping-linked list nested body to be converted; the table information in the mapping configuration information is used to match the mapping-linked list nested body to be converted, and a relationship table to be merged is obtained according to the mapping body in the mapping-linked list nested body to be converted; redundant relationship tables with the same table name are determined from the relationship tables to be merged, and the relationship tables to be merged other than the redundant relationship tables are determined as non-redundant relationship tables; the column fields of the redundant relationship tables are merged to obtain a merged relationship table; the columns with the same column name in the merged relationship table are merged to obtain merged row data; the row data is obtained according to the merged row data and the non-redundant relationship table; The table loading component is used to load the received row data into the corresponding relational table to obtain relational model data.

9. The device for converting document model data into relational model data according to claim 8, characterized in that: Also included is a document conversion component; wherein: The document conversion component is used to perform content conversion on the received row data to obtain processed row data; and transmit the processed row data to the corresponding table loading component to obtain the relational model data through the table loading component.

Images