Data model structure mapping method, system and terminal

Abstract

The application provides a data model structure mapping method, system and terminal, which synchronizes and supplements metadata of a to-be-mapped data model structure, generates a corresponding original operation data storage model through the supplemented metadata of the to-be-mapped data model structure, performs a mapping operation on the to-be-mapped data model structure based on a pre-constructed table field mapping large language model, maps the to-be-mapped data model structure to a preset standard data model structure, generates a corresponding data model mapping relationship and a data model mapping script, thereby replacing an artificial mapping mode, reducing mapping time consumption, reducing mapping cost, improving mapping efficiency and accuracy, and solving the technical problems of large artificial workload, high cost, low efficiency, dependence on artificial experience, poor accuracy, reliability and traceability in the prior art of mapping each data model structure by using an artificial mapping mode.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data model structure mapping method, system, and terminal. Background Technology

[0002] With the rapid development of modern technology in digitalization, informatization, and intelligentization, most medical institutions currently collect, store, process, and transfer multi-modal, heterogeneous, and high-dimensional medical data generated throughout the entire lifecycle of healthcare activities, including disease prevention, diagnosis, treatment, rehabilitation, and health management, through different business systems. This enables all core business processes of medical institutions, such as daily operations, clinical diagnosis and treatment, and management decision-making. In processing medical data, it is necessary to organize medical data from multiple business systems of different medical institutions and store it in the operational data storage layer of a data warehouse. However, because the data model structures of different medical institutions and business systems are inconsistent, when storing them in a unified operational data storage layer, it is necessary to first map each data model structure to a standard data model structure.

[0003] Existing technologies typically involve manually identifying the mapping relationships between various data model structures and standard data model structures, and then developing data model structure mapping scripts accordingly. However, this manual approach has many drawbacks.

[0004] ① Extremely inefficient and difficult to scale

[0005] It requires mapping a massive number of tables and fields one by one, which is labor-intensive, time-consuming, and prone to lag in scenarios with multiple data sources. Moreover, the repetitive work of mapping is redundant, and the same business logic mapping needs to be manually operated repeatedly. There is no reuse mechanism, which is inefficient and difficult to scale.

[0006] ②Accuracy fluctuates greatly, quality is difficult to guarantee, and traceability is poor.

[0007] Manual data processing is prone to errors due to fatigue or negligence (such as confusion about field meanings or type deviations); differences in experience can also lead to inconsistent mapping standards, resulting in inconsistent mapping results for the same data source's data model structure; and the lack of standardized logs in the mapping process makes it impossible to find evidence when troubleshooting errors later.

[0008] ③ High reliance on knowledge and high cost of knowledge transfer

[0009] Relying on the experience of senior staff can lead to fragmented experience accumulation, and a "knowledge gap" can easily occur when staff leave; while newcomers need to familiarize themselves with business terminology, historical rules, etc., which takes a long time to get started.

[0010] ④ Weak ability to adapt to change and insufficient scalability

[0011] If new data sources or new business functions are added, the mapping logic needs to be re-examined, and historical experience cannot be quickly reused; knowledge base updates rely on manual input, which lags behind business iterations.

[0012] ⑤ High labor costs and low long-term cost-effectiveness.

[0013] It requires long-term deployment of experienced personnel, resulting in a high proportion of human resource costs; repetitive tasks cannot free up manpower, making it difficult to invest in high-value work such as rule optimization. Summary of the Invention

[0014] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a data model structure mapping method, system and terminal to solve the technical problems of large workload, high cost, low efficiency, reliance on human experience and poor accuracy, reliability and traceability of the existing manual mapping of various data model structures.

[0015] To achieve the above and other related objectives, a first aspect of this application provides a data model structure mapping method, comprising: acquiring a target heterogeneous data source data model structure to be mapped and metadata of the target heterogeneous data model structure, performing a quality assessment operation on the metadata of the target heterogeneous data model structure to generate complete metadata of the target heterogeneous data model structure; generating an original operational data storage model of the target heterogeneous data model structure based on the complete metadata of the target heterogeneous data model structure; performing a mapping operation on the target heterogeneous data model structure based on a pre-built table field mapping language model and the original operational data storage model to map the target heterogeneous data model structure to a preset standard data model structure, and generating a data model mapping relationship of the target heterogeneous data model structure; and generating a data model mapping script of the target heterogeneous data model structure based on the data model mapping relationship, so as to store the heterogeneous data of the target heterogeneous data source defined by the target heterogeneous data model structure to the target data detail layer of the target data warehouse.

[0016] In some embodiments of the first aspect of this application, the method of performing a quality assessment operation on the metadata of the data model structure to be mapped includes: determining the interpretability completeness of the metadata of the data model structure to be mapped according to preset metadata quality control rules; if the metadata interpretation is incomplete, supplementing the metadata to generate complete metadata of the data model structure to be mapped; if the metadata interpretation is complete, using the metadata as the complete metadata of the data model structure to be mapped.

[0017] In some embodiments of the first aspect of this application, the method of supplementing the metadata of the data model structure to be mapped includes: obtaining the target vendor table structure rules of the target vendor of the target heterogeneous data source based on a pre-built vendor table structure knowledge base, and obtaining the target organization table structure rules of the target heterogeneous data source based on a pre-built organization table structure knowledge base; performing equivalence ratio matching on multiple data tables to be mapped and multiple data fields of each data table defined by the data model structure to be mapped according to the target vendor table structure rules and the target organization table structure rules, to obtain annotation information of one or more matched data tables and one or more matched data fields to supplement the metadata of the data model structure to be mapped; and performing semantic similarity matching on each data table and each data field to be mapped based on a pre-built table field annotation language model, according to the target vendor table structure rules and the target organization table structure rules, to supplement the annotation information of each data table and each data field to be mapped, to further supplement the metadata of the data model structure to be mapped, and obtain the complete metadata of the data model structure to be mapped.

[0018] In some embodiments of the first aspect of this application, after supplementing the metadata of the data model structure to be mapped, the target organization table structure rules are updated based on the complete metadata of the data model structure to be mapped, and the organization table structure knowledge base is updated.

[0019] In some embodiments of the first aspect of this application, the method for generating the data model mapping relationship of the data model structure to be mapped includes: obtaining a standard data model structure and its metadata based on a pre-built standard data model knowledge base; inputting the metadata of the standard data model structure and the metadata of the data model structure to be mapped into the table field mapping large language model to perform a mapping operation on the data model structure to be mapped, thereby generating the data model mapping relationship of the data model structure to be mapped.

[0020] In some embodiments of the first aspect of this application, the method of performing mapping operations on the data model structure to be mapped includes: performing table mapping operations on multiple data tables defined by the data model structure to be mapped based on the metadata of the standard data model structure and the metadata of the data model structure to be mapped, to establish table mapping relationships between each data table to be mapped and multiple standard data tables defined by the standard data model structure; performing field mapping operations on multiple data fields of each data table to be mapped based on the metadata of the standard data model structure, the metadata of the data model structure to be mapped, and the table mapping relationships, to establish field mapping relationships between each mapped data field and multiple standard data fields of each standard data table defined by the standard data model structure; and establishing a data model mapping relationship between the data model structure to be mapped and the standard data model structure based on the table mapping relationships and the field mapping relationships.

[0021] In some embodiments of the first aspect of this application, the method of performing a mapping operation on the data model structure to be mapped further includes: obtaining instance heterogeneous data of the target heterogeneous data source defined by the data model structure to be mapped, and inputting the table field mapping large language model to understand the data content of the instance heterogeneous data, and updating the mapping relationship of each table and the mapping relationship of each field accordingly, and updating the data model mapping relationship.

[0022] In some embodiments of the first aspect of this application, the data model structure mapping method further includes: constructing a data model mapping relationship knowledge base based on the data model mapping relationship; and constructing a data model mapping script knowledge base based on the data model mapping script.

[0023] To achieve the above and other related objectives, a second aspect of this application provides a data model structure mapping system, comprising: a metadata synchronization module, configured to acquire the data model structure to be mapped from a target heterogeneous data source and the metadata of the mapped data model structure, and perform a quality assessment operation on the metadata of the data model structure to be mapped to generate complete metadata of the data model structure to be mapped; a model generation module, connected to the metadata synchronization module, configured to generate an original operational data storage model of the data model structure to be mapped based on the complete metadata of the data model structure to be mapped; a model mapping module, connected to the model generation module, configured to perform a mapping operation on the data model structure to be mapped based on a pre-built table field mapping language model and the original operational data storage model, mapping the data model structure to be mapped to a preset standard data model structure, and generating a data model mapping relationship for the data model structure to be mapped; and a script generation module, connected to the model mapping module, configured to generate a data model mapping script for the data model structure to be mapped based on the data model mapping relationship, so as to store the heterogeneous data of the target heterogeneous data source defined by the data model structure to be mapped into the target data detail layer of the target data warehouse.

[0024] To achieve the above and other related objectives, a third aspect of this application provides a data model structure mapping terminal, the data model structure mapping terminal comprising: a memory and a processor; the memory for storing a computer program; and the processor for executing the computer program stored in the memory, so that the terminal implements the data model structure mapping method as described in any of the above embodiments.

[0025] As described above, this application has the following beneficial effects: This application provides a data model structure mapping method, system, and terminal. By synchronizing and supplementing the metadata of the data model structure to be mapped, a corresponding original operational data storage model is generated. Based on a pre-built table field mapping language model, a mapping operation is performed on the data model structure to be mapped, mapping the data model structure to be mapped to a preset standard data model structure, generating corresponding data model mapping relationships and data model mapping scripts. This replaces the manual mapping method, reduces mapping time, lowers mapping costs, and improves mapping efficiency and accuracy. It solves the technical problems of existing manual mapping of various data model structures, such as large workload, high cost, low efficiency, reliance on human experience, and poor accuracy, reliability, and traceability. Attached Figure Description

[0026] Figure 1 The diagram shown is a flowchart illustrating a data model structure mapping method in one embodiment of this application.

[0027] Figure 2 The diagram shown is a flowchart illustrating the quality assessment operation in one embodiment of this application.

[0028] Figure 3 The diagram shows a flowchart of supplementing metadata for the data model structure to be mapped in one embodiment of this application.

[0029] Figure 4 The diagram shown is a flowchart illustrating the process of generating a data model mapping relationship in one embodiment of this application.

[0030] Figure 5 The diagram shown is a flowchart illustrating the mapping operation in one embodiment of this application.

[0031] Figure 6 The diagram shown is a schematic representation of a data model structure mapping system in one embodiment of this application.

[0032] Figure 7 The diagram shown is a schematic representation of the data model structure mapping terminal in one embodiment of this application. Detailed Implementation

[0033] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0034] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect, without limiting their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0035] To address the problems mentioned above, this application provides a data model structure mapping method, system, and terminal. The aim is to generate a corresponding original operational data storage model from the metadata of the data model structure to be mapped, and to perform mapping operations on the data model structure to be mapped based on a pre-built table field mapping language model, mapping the data model structure to be mapped to a preset standard data model structure. This solves the technical problems of existing manual mapping methods, such as high workload, high cost, low efficiency, reliance on human experience, and poor accuracy, reliability, and traceability.

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application are further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0037] like Figure 1 The diagram illustrates a flowchart of a data model structure mapping method according to an embodiment of this application. The data model structure mapping method in this embodiment mainly includes the following steps.

[0038] Step S1: Obtain the data model structure to be mapped from the target heterogeneous data source and the metadata of the data model structure to be mapped, and perform a quality assessment operation on the metadata of the data model structure to be mapped to generate complete metadata of the data model structure to be mapped.

[0039] The target heterogeneous data source can be a business system of a medical institution. The data model structure to be mapped from the target heterogeneous data source is used to define the data content, data type, data format, data constraints, business meaning, and relationships between data in the medical data of that business system. By obtaining the data model structure to be mapped from the target heterogeneous data source and its corresponding metadata, the data model structure to be mapped is mapped to a preset standard data model structure. This allows the medical data of that business system of the medical institution to be stored in a unified operational data storage layer using a standard data model structure, thus completing the processing of polymorphic, heterogeneous, and high-dimensional medical data. However, it should be noted that the target heterogeneous data source can also be other business systems or data sources, and this application is not limited to them.

[0040] In one embodiment, such as Figure 2 As shown, the method for performing quality assessment operations on the metadata of the data model structure to be mapped includes the following steps.

[0041] Step S11: Determine the completeness of the metadata interpretation of the data model structure to be mapped according to the preset metadata quality control rules.

[0042] The metadata quality control rules include multiple metadata quality control conditions used to evaluate the completeness of metadata interpretation, such as whether the meaning of all data tables and all data fields is explained, whether the data types of all data fields are explained, whether the value range of all data fields is explained, and whether the business rules of all data fields are explained.

[0043] Step S12: If the metadata interpretation is incomplete, supplement the metadata to generate complete metadata for the data model structure to be mapped.

[0044] In one embodiment, such as Figure 3 As shown, the method of supplementing the metadata of the data model structure to be mapped includes the following steps.

[0045] Step S121: Based on the pre-built vendor table structure knowledge base, obtain the target vendor table structure rules of the target heterogeneous data source, and based on the pre-built organization table structure knowledge base, obtain the target organization table structure rules of the target heterogeneous data source.

[0046] The vendor table structure knowledge base includes vendor table structure rules from multiple vendors. It should be noted that "vendor" refers to the IT vendor of the target heterogeneous data source, such as the IT vendor of each business system in various medical institutions. The vendor table structure rules originate from the table structure documents generated by the vendors when designing their business systems, including design documents for all data table structures and fields. Each time the target heterogeneous data source's data model structure is mapped, the corresponding target vendor's table structure documents can be collected and accumulated synchronously to construct or update the target vendor table structure rules and update the vendor table structure knowledge base.

[0047] The institutional table structure knowledge base includes: customized table structure documents generated by the vendor when designing multiple business systems for a specified medical institution, including design documents for all data table structures and fields. Each time a target heterogeneous data source's data model structure is mapped, the corresponding target institution's table structure documents can be collected and accumulated synchronously to construct or update the target institution's table structure rules and update the institutional table structure knowledge base.

[0048] Therefore, when mapping the target heterogeneous data source data model structure, the vendor table structure knowledge base and the organization table structure knowledge base, constructed based on historical table structure documents, can be obtained to obtain the target vendor table structure rules and the target organization table structure rules, which are used to supplement the metadata of the target data model structure.

[0049] Step S122: Based on the target manufacturer table structure rules and the target organization table structure rules, perform equal-value ratio matching on multiple data tables to be mapped and multiple data fields to be mapped in the data model structure definition to be mapped, and obtain the annotation information of one or more matched data tables and one or more data fields to be mapped, so as to supplement the metadata of the data model structure to be mapped.

[0050] It should be understood that equivalence matching refers to the process of associating and pairing data in different datasets based on one or more key fields with equal values. Specifically, it involves obtaining multiple design data tables and their respective design data fields defined in the target manufacturer table structure rules and the target organization table structure rules, and obtaining multiple data tables to be mapped and their respective data fields defined in the data model structure to be mapped. Equivalence matching is then performed between each design data table and each data table to be mapped, and between each design data field and each data field to be mapped. This yields one or more data tables and one or more data fields to be mapped that have undergone equivalence matching. Furthermore, annotation information is obtained based on the target manufacturer table structure rules and the target organization table structure rules to supplement the metadata of the data model structure to be mapped.

[0051] The annotation information includes, but is not limited to: the meaning, purpose, data type, value range, constraints, and business rules of the data table or data field.

[0052] Step S123: Based on the pre-built table field annotation large language model, according to the target manufacturer table structure rules and the target organization table structure rules, perform semantic similarity matching on each data table to be mapped and each data field to be mapped, supplement the annotation information of each data table to be mapped and each data field to be mapped, so as to further supplement the metadata of the data model structure to be mapped, and obtain the complete metadata of the data model structure to be mapped.

[0053] It should be understood that the large language model for table field annotations is obtained by training based on a large language model (LLM). The model structure of the large language model for table field annotations is a deep neural network composed of multiple layers of stacked Transformer blocks. The model parameters can be set by the user according to their needs, and this application does not specifically limit them.

[0054] In one embodiment, the table field annotation large language model is trained using federated learning technology. The large language model is initialized on the target central server to construct an initial table field annotation large language model. The target central server then encrypts and distributes the initial table field annotation large language model to target clients of various vendors, allowing each target client to train the initial table field annotation large language model based on table structure documentation stored locally on the vendor's machine. Multiple converged optimized table field annotation large language models are then encrypted and sent to the target central server. The target central server uses an aggregation algorithm to fuse the optimized table field annotation large language models to obtain the final table field annotation large language model.

[0055] In this embodiment, the present application introduces federated learning technology, which enables various vendors to jointly train the recognition capability of the large language model of the table field annotations without sharing the original table structure documents, thereby achieving intelligent supplementation of cross-vendor metadata.

[0056] For other unmatched data tables and fields to be mapped, this application inputs the target manufacturer table structure rules and the target organization table structure rules into the table field annotation language model to understand the design data tables and design data fields defined by the target manufacturer table structure rules and the target organization table structure rules, and learns their field meanings and naming patterns. Simultaneously, each data table and each field to be mapped is input into the table field annotation language model to understand the data tables and fields to be mapped, and learn their field meanings and naming patterns. The two learning results are then further analyzed and learned. Semantic similarity matching is performed on each data table and each field to be mapped to establish mapping relationships between each data table to be mapped and each design data table, as well as mapping relationships between each data field to be mapped and each design data field. Based on the target manufacturer table structure rules and the target organization table structure rules, annotation information is obtained to further supplement the metadata of the data model structure to be mapped, resulting in complete metadata including complete annotation information for all data tables and fields to be mapped, i.e., complete metadata of the data model structure to be mapped.

[0057] This application automatically supplements metadata by annotating the large language model in the table fields, reducing the workload of manually organizing and completing metadata, lowering labor costs, improving metadata synchronization efficiency, and helping to improve the overall mapping efficiency of the data model structure.

[0058] In one embodiment, after supplementing the metadata of the data model structure to be mapped, the target organization table structure rules are updated based on the complete metadata of the data model structure to be mapped, and the organization table structure knowledge base is updated, thereby accumulating the target manufacturer table structure rules and the target organization table structure rules. This helps to ensure the accuracy and completeness of subsequent automatic supplementation of metadata, ensures the interpretation completeness of the complete metadata of the data model structure to be mapped, and ensures the accuracy of mapping the data model structure to be mapped.

[0059] Step S13: If the metadata is fully interpreted, then use the metadata as the complete metadata of the data model structure to be mapped.

[0060] Step S2: Generate the original operational data storage model of the data model structure to be mapped based on the complete metadata of the data model structure to be mapped.

[0061] The original operational data storage model includes multiple data tables to be mapped, conforming to the structure definition of the data to be mapped model, and multiple data fields to be mapped for each data table. Specifically, the original operational data storage model is a subject-oriented, integrated, variable, current or near-current heterogeneous data set of the target heterogeneous data source, used to store this heterogeneous data in the target operational data storage layer of the target warehouse, providing a high-quality, integrated data source for the target data warehouse.

[0062] Step S3: Based on the pre-built table field mapping large language model, according to the original operation data storage model, perform a mapping operation on the data model structure to be mapped, map the data model structure to be mapped to the preset standard data model structure, and generate the data model mapping relationship of the data model structure to be mapped.

[0063] It should be understood that the model structure of the table field annotation large language model is a deep neural network composed of multiple stacked Transformer blocks, which can be trained based on a Large Language Model (LLM). Specifically, multiple historical data models to be mapped can be obtained to construct training, validation, and test sets, and the large language model can be trained to obtain a converged large language model for mapping the table fields. It should be noted that the model parameters and model training methods can be set by the user according to their needs, and this application does not specifically limit them.

[0064] In one embodiment, such as Figure 4 As shown, step S3 includes the following steps.

[0065] Step S31: Based on the pre-built standard data model knowledge base, obtain the standard data model structure and the metadata of the standard data model structure.

[0066] The standard data model knowledge base defines a standard data model structure, which can be understood as the data model structure of the target data detail layer of the target warehouse. It defines the conditions that data stored in the target data detail layer must meet, including the number, type, and content of data tables, as well as the meaning, type, content, constraints, and rules of the data fields in each table. These defined conditions are represented through the metadata of the standard data model structure.

[0067] This application defines a standard data model structure, which can avoid the defects of inconsistent mapping standards caused by differences in human experience, and unify the mapping standards, which is also necessary to ensure the accuracy of the mapping results.

[0068] Step S32: Input the metadata of the standard data model structure and the metadata of the data model structure to be mapped into the table field mapping language model to perform a mapping operation on the data model structure to be mapped, and generate the data model mapping relationship of the data model structure to be mapped.

[0069] In one embodiment, such as Figure 5 As shown, the method for performing mapping operations on the data model structure to be mapped includes the following steps.

[0070] Step S321: Based on the metadata of the standard data model structure and the metadata of the data model structure to be mapped, perform table mapping operations on the multiple data tables to be mapped defined in the data model structure to be mapped, and establish table mapping relationships between each data table to be mapped and the multiple standard data tables defined in the standard data model structure.

[0071] Specifically, multiple standard data tables defined by the standard data model structure are obtained; based on the table field mapping language model, according to the metadata of the standard data model structure and the metadata of the data model structure to be mapped, one or more data tables to be mapped that semantically match each standard data table are selected, thereby establishing a table mapping relationship between each data table to be mapped and each standard data table, that is, each standard data table corresponds to one or more data tables to be mapped that have a table mapping relationship.

[0072] Step S322: Based on the metadata of the standard data model structure, the metadata of the data model structure to be mapped, and the mapping relationships of each table, perform field mapping operations on multiple data fields of each data table to be mapped as defined in the data model structure to be mapped, and establish field mapping relationships between each mapped data field and multiple standard data fields of each standard data table as defined in the standard data model structure.

[0073] Specifically, multiple standard data fields of each standard data table defined by the standard data model structure are obtained; based on the table field mapping big language model, according to the metadata of the standard data model structure and the metadata of the data model structure to be mapped, for each standard data table, a unique data field to be mapped that semantically matches each standard data field is selected from one or more data tables to be mapped that have a table mapping relationship with the standard data table, thereby establishing a field mapping relationship between each data field to be mapped and each standard data field, that is, each standard data field corresponds to a unique data field to be mapped with a field mapping relationship.

[0074] In one embodiment, the table mapping relationship of the corresponding standard data table can also be updated according to the field mapping relationship of each standard data field of each standard data table.

[0075] Step S323: Based on the mapping relationships of each table and each field, establish the data model mapping relationship between the data model structure to be mapped and the standard data model structure.

[0076] In one embodiment, step S3 further includes step S33.

[0077] Step S33: Obtain instance heterogeneous data of the target heterogeneous data source defined by the data model structure to be mapped, and input the table field mapping language model to understand the data content of the instance heterogeneous data, and update the mapping relationship of each table and the mapping relationship of each field accordingly, and update the data model mapping relationship.

[0078] The heterogeneous data in this context originates from the target heterogeneous data source and conforms to the definition of the data model structure to be mapped. For example, if medical data from a target medical institution's target business system is stored in a target data warehouse, and the data model structure of that target business system needs to be mapped to a standard data model structure, then during the mapping operation, a large language model can be mapped based on the table fields. Based on the metadata of the standard data model structure and its metadata, a data model mapping relationship between the data model structure and the standard data model structure can be established. Furthermore, instance medical data can be introduced as instance heterogeneous data to update the established data model mapping relationship.

[0079] In this embodiment, by inputting the heterogeneous instance data into the table field mapping large language model, and by learning and understanding the data content in the heterogeneous instance data, the meaning, purpose, business rules of each data table to be mapped, as well as the meaning, purpose, data type, value range, constraints, business rules, etc. of each data field to be mapped, are automatically identified. This updates the mapping relationships of each table and each field, and updates the data model mapping relationship between the data model structure to be mapped and the standard data model structure. This further ensures the accuracy of the data model mapping relationship and the accuracy of mapping the data model structure to be mapped, reducing errors such as confusion of field meaning and type matching deviation caused by human fatigue and negligence.

[0080] This application achieves intelligent mapping of data model structures through the aforementioned table field mapping language model, replacing manual mapping of numerous data tables and fields one by one. This significantly reduces mapping time, improves work efficiency, and frees up manpower, allowing experienced personnel to devote themselves to other more valuable tasks. Furthermore, based on the learning capabilities of the aforementioned table field mapping language model, it can efficiently adapt to various data sources and complex scenarios with large numbers of fields, making it suitable for large-scale processing.

[0081] Step S4: Based on the data model mapping relationship, generate a data model mapping script for the data model structure to be mapped, so as to store the heterogeneous data of the target heterogeneous data source defined by the data model structure to be mapped into the target data detail layer of the target data warehouse.

[0082] Preferably, the data model mapping script can be an SQL (Structured Query Language) script, used to store the heterogeneous data in the target data detail layer according to the standard data model structure based on the data model mapping relationship.

[0083] It should be noted that the data model mapping script can also be written in other computer languages, and this application does not specifically limit it.

[0084] In one embodiment, the data model structure mapping method further includes: constructing a data model mapping relationship knowledge base based on the data model mapping relationship; and constructing a data model mapping script knowledge base based on the data model mapping script. During subsequent mapping operations, the data model mapping relationship knowledge base and the data model mapping script knowledge base can be updated and iteratively optimized. This allows for the accumulation of data model structure mapping knowledge and experience. Data model structure mapping for the same business logic can directly reuse existing results, reducing the manpower and time costs of repetitive development. Furthermore, new data model mapping scripts can be quickly developed when new data sources or new business functions are added, resulting in high work reuse efficiency. Simultaneously, it also avoids the recurrence of the same errors.

[0085] like Figure 6 The diagram shows a schematic representation of a data model structure mapping system 600 according to an embodiment of this application. The data model structure mapping system 600 in this embodiment mainly includes: a metadata synchronization module 601, a model generation module 602, a model mapping module 603, and a script generation module 604 connected in sequence.

[0086] The metadata synchronization module 601 is used to obtain the data model structure to be mapped from the target heterogeneous data source and the metadata of the data model structure to be mapped, and to perform a quality assessment operation on the metadata of the data model structure to be mapped to generate complete metadata of the data model structure to be mapped.

[0087] The model generation module 602 is used to generate the original operational data storage model of the data model structure to be mapped based on the complete metadata of the data model structure to be mapped.

[0088] The model mapping module 603 is used to map a large language model based on a pre-built table field mapping, and to perform a mapping operation on the data model structure to be mapped according to the original operation data storage model, mapping the data model structure to be mapped to a preset standard data model structure, and generating a data model mapping relationship for the data model structure to be mapped.

[0089] The script generation module 604 is used to generate a data model mapping script for the data model structure to be mapped according to the data model mapping relationship, so as to store the heterogeneous data of the target heterogeneous data source defined by the data model structure to be mapped into the target data detail layer of the target data warehouse.

[0090] It should be understood that the module division in the embodiments of this application is illustrative and only represents a logical functional division. In actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor or functional module, exist as separate physical entities, or be divided into more functional modules. The integrated modules or units described above can be implemented in hardware or as software functional modules.

[0091] It should also be understood that the data model structure mapping system and the data model structure mapping method provided in the above embodiments belong to the same concept. The specific implementation methods and steps of each functional module are detailed in the method embodiments, and will not be repeated here.

[0092] Figure 7 This is a schematic diagram of the data model structure mapping terminal 700 provided in an embodiment of this application. For example... Figure 7 As shown, the data model structure mapping terminal 700 includes at least one processor 701, a memory 702, at least one network interface 703, and a user interface 705. The various components in the terminal are coupled together via a bus system 704 to implement the data model structure mapping method described in the above embodiments, replacing the manual sorting of mapping relationships between various data model structures and standard data model structures, and the development of data model structure mapping scripts. It is understood that the bus system 704 is used to implement the connection and communication between these components. In addition to a data bus, the bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 7 The general will label all buses as bus systems.

[0093] The user interface 705 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0094] It is understood that memory 702 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable categories of memory.

[0095] In this embodiment, the memory 702 is used to store various types of data to support the operation of the data model structure mapping terminal 700. Examples of this data include any executable program that operates on the data model structure mapping terminal 700, such as operating system 7021 and application program 7022. Operating system 7021 includes various system programs, such as framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Application program 7022 may include various applications, such as media player, browser, etc., used to implement various application services. The data model structure mapping method provided in this embodiment can be included in application program 7022.

[0096] The data model structure mapping method disclosed in the above embodiments of this application can be applied to processor 701, or implemented by processor 701. Processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the data model structure mapping method can be completed by the integrated logic circuit of the hardware in processor 701 or by instructions in the form of software. The processor 701 mentioned above may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 701 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor 701 may be a microprocessor or any conventional processor, etc. The steps of the data model structure mapping method provided in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0097] In an exemplary embodiment, the data model structure mapping terminal 700 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0098] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented using computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0099] In the embodiments provided in this application, the computer-readable and writable storage medium may include read-only memory, random access memory, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, flash memory, USB flash drive, portable hard drive, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Additionally, any connection may be appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. However, it should be understood that computer-readable and writable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are intended for non-transient, tangible storage media. The disks and optical discs used in the application include compact optical discs (CDs), laser optical discs, optical discs, digital multifunction optical discs (DVDs), floppy disks, and Blu-ray discs, where disks typically copy data magnetically, while optical discs use lasers to copy data optically.

[0100] In summary, this application provides a data model structure mapping method, system, and terminal. By synchronizing and supplementing the metadata of the data model structure to be mapped, a corresponding original operational data storage model is generated. Based on a pre-built table field mapping language model, a mapping operation is performed on the data model structure to be mapped, mapping the data model structure to be mapped to a preset standard data model structure. This generates corresponding data model mapping relationships and data model mapping scripts, thereby replacing manual mapping methods, reducing mapping time, lowering mapping costs, and improving mapping efficiency and accuracy. This solves the technical problems of existing manual mapping methods, such as high workload, high cost, low efficiency, reliance on human experience, and poor accuracy, reliability, and traceability.

[0101] Therefore, this application effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0102] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A data model structure mapping method, characterized in that, include: Obtain the data model structure to be mapped from the target heterogeneous data source and the metadata of the data model structure to be mapped, and perform a quality assessment operation on the metadata of the data model structure to be mapped to generate complete metadata of the data model structure to be mapped. Based on the complete metadata of the data model structure to be mapped, generate the original operational data storage model of the data model structure to be mapped; Based on the pre-built table field mapping big language model, according to the original operation data storage model, a mapping operation is performed on the data model structure to be mapped, mapping the data model structure to be mapped to a preset standard data model structure, and generating the data model mapping relationship of the data model structure to be mapped. Based on the data model mapping relationship, a data model mapping script is generated for the data model structure to be mapped, so as to store the heterogeneous data of the target heterogeneous data source defined by the data model structure to be mapped into the target data detail layer of the target data warehouse.

2. The data model structure mapping method according to claim 1, characterized in that, The methods for performing quality assessment operations on the metadata of the data model structure to be mapped include: Based on preset metadata quality control rules, determine the completeness of the metadata interpretation of the data model structure to be mapped; If the metadata interpretation is incomplete, then supplement the metadata to generate complete metadata for the data model structure to be mapped; If the metadata is fully interpreted, then the metadata is taken as the complete metadata of the data model structure to be mapped.

3. The data model structure mapping method according to claim 2, characterized in that, The methods for supplementing the metadata of the data model structure to be mapped include: Based on a pre-built vendor table structure knowledge base, the target vendor table structure rules of the target heterogeneous data source are obtained, and based on a pre-built organization table structure knowledge base, the target organization table structure rules of the target heterogeneous data source are obtained. Based on the target manufacturer table structure rules and the target organization table structure rules, equal-ratio matching is performed on multiple data tables to be mapped and multiple data fields to be mapped in the data model structure definition to be mapped, to obtain the annotation information of one or more matched data tables and one or more data fields to be mapped, so as to supplement the metadata of the data model structure to be mapped. Based on the pre-built table field annotation large language model, semantic similarity matching is performed on each data table to be mapped and each data field to be mapped according to the target manufacturer table structure rules and the target organization table structure rules. The annotation information of each data table to be mapped and each data field to be mapped is supplemented to further supplement the metadata of the data model structure to be mapped, and the complete metadata of the data model structure to be mapped is obtained.

4. The data model structure mapping method according to claim 3, characterized in that, After supplementing the metadata of the data model structure to be mapped, the target organization table structure rules are updated based on the complete metadata of the data model structure to be mapped, and the organization table structure knowledge base is updated.

5. The data model structure mapping method according to claim 1, characterized in that, The methods for generating the data model mapping relationship of the data model structure to be mapped include: Based on a pre-built standard data model knowledge base, obtain the standard data model structure and its metadata. The metadata of the standard data model structure and the metadata of the data model structure to be mapped are input into the table field mapping language model to perform a mapping operation on the data model structure to be mapped, thereby generating the data model mapping relationship of the data model structure to be mapped.

6. The data model structure mapping method according to claim 5, characterized in that, The methods for performing mapping operations on the data model structure to be mapped include: Based on the metadata of the standard data model structure and the metadata of the data model structure to be mapped, a table mapping operation is performed on multiple data tables to be mapped as defined by the data model structure to be mapped, and a table mapping relationship is established between each data table to be mapped and multiple standard data tables defined by the standard data model structure. Based on the metadata of the standard data model structure, the metadata of the data model structure to be mapped, and the mapping relationships of each table, field mapping operations are performed on multiple data fields to be mapped in each data table defined by the data model structure to be mapped, and field mapping relationships are established between each mapped data field and multiple standard data fields in each standard data table defined by the standard data model structure. Based on the mapping relationships of each table and each field, establish the data model mapping relationship between the data model structure to be mapped and the standard data model structure.

7. The data model structure mapping method according to claim 6, characterized in that, The methods for performing mapping operations on the data model structure to be mapped also include: Obtain instance heterogeneous data of the target heterogeneous data source defined by the data model structure to be mapped, and input the table field mapping language model to understand the data content of the instance heterogeneous data, and update the mapping relationship of each table and the mapping relationship of each field accordingly, and update the data model mapping relationship.

8. The data model structure mapping method according to claim 1, characterized in that, Also includes: Based on the data model mapping relationship, construct a data model mapping relationship knowledge base; Based on the data model mapping script, construct a data model mapping script knowledge base.

9. A data model structure mapping system, characterized in that, include: The metadata synchronization module is used to obtain the data model structure to be mapped from the target heterogeneous data source and the metadata of the mapped data model structure, and to perform a quality assessment operation on the metadata of the data model structure to be mapped to generate complete metadata of the data model structure to be mapped. The model generation module, connected to the metadata synchronization module, is used to generate the original operational data storage model of the data model structure to be mapped based on the complete metadata of the data model structure to be mapped. The model mapping module, connected to the model generation module, is used to map a large language model based on a pre-built table field. According to the original operation data storage model, it performs a mapping operation on the data model structure to be mapped, maps the data model structure to be mapped to a preset standard data model structure, and generates a data model mapping relationship for the data model structure to be mapped. The script generation module, connected to the model mapping module, is used to generate a data model mapping script for the data model structure to be mapped based on the data model mapping relationship, so as to store the heterogeneous data of the target heterogeneous data source defined by the data model structure to be mapped into the target data detail layer of the target data warehouse.

10. A data model structure mapping terminal, characterized in that, include: Memory and processor; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory to enable the terminal to implement the data model structure mapping method as described in any one of claims 1 to 8.

Images