Method and device for uniformly creating data model in ETL process

Abstract

The embodiment of the present invention provides a method and device for unified creation of data models in the ETL process, the method comprising: in response to a detected data model creation instruction, obtaining data model creation parameters corresponding to the data model creation instruction; creating a target data structure model based on the data model creation parameters and a pre-stored data structure instance. A unified one-click creation of data structures is achieved, and a variety of deployment scenarios are uniformly responded to, saving manpower, thereby making the data processing project cycle fast, efficient, and avoiding repeated manual operations and maintenance.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of big data technology, and in particular to a method and device for uniformly creating a data model in an ETL process. Background Art

[0002] With the steady development of medical information construction, medical data centers can provide high-quality data services for clinical, scientific research, drug development and other fields. Among them, medical data centers are inseparable from the creation of ETL (Extract-Transform-Load) data processing systems. ETL (Extract-Transform-Load) is an important part of big data. Big data ETL is responsible for extracting data from scattered and heterogeneous data sources such as relational data and flat data files into the big data platform system, cleaning, converting, integrating, and finally loading into the big data platform, data warehouse or data mart to become data for online analytical processing and data mining to provide decision support.

[0003] Before creating an ETL data processing system, you need to create a corresponding data structure model on the component data engine based on the different selected Hahoop components. When using big data components such as HBASE, HIVE, HDFS, etc. in the big data Hadoop ecosystem, the creation of business models always requires the use of different syntax and creation methods depending on the components used. As more components are used, more data structure layers will appear, and correspondingly, there will be more maintenance of this layer structure. For developers, the existing maintenance method is to manually create and maintain creation scripts, and more traditional file methods are used to store, view, and modify, all of which are based on the modification of the content of text files. Therefore, the existing methods have the defects of low maintenance efficiency, complex modification of data model structures, and waste of human resources. Summary of the invention

[0004] The embodiment of the present invention provides a method and device for uniformly creating a data model in an ETL process, so as to realize uniform creation of a data structure, avoid repeated manual operations and maintenance, and save manpower.

[0005] In a first aspect, an embodiment of the present invention provides a method for uniformly creating a data model in an ETL process, including:

[0006] In response to the detected data model creation instruction, obtaining a data model creation parameter corresponding to the data model creation instruction;

[0007] A target data structure model is created based on the data model creation parameters and the pre-stored data structure instance.

[0008] Optionally, further, obtaining data model creation parameters corresponding to the data model creation instruction includes:

[0009] The target application component identifier corresponding to the data model creation instruction and the model configuration information associated with the target application component identifier are obtained as data model creation parameters.

[0010] Optionally, further, creating a target data structure model based on the data model creation parameters and the pre-stored data structure instance includes:

[0011] Determine the data structure model to be created according to the target application component identifier;

[0012] A target data structure model is created from a pre-stored data structure instance based on the data structure model to be created and the model configuration information.

[0013] Optionally, further, creating a target data structure model from a pre-stored data structure instance based on the data structure model to be created and the model configuration information includes:

[0014] Selecting a target element from a pre-stored data structure instance according to the data structure model to be created;

[0015] Determine definition configuration parameters of the target element according to the data structure model to be created, adjust the definition configuration parameters based on the model configuration information, and obtain target configuration parameters of the target element;

[0016] The target elements are configured based on the target configuration parameters to obtain the target data structure model.

[0017] Optionally, further configure the target element based on the target configuration parameters, including:

[0018] Get the component version ID corresponding to the target application component ID;

[0019] The target configuration mode is determined based on the component version identifier, and the target element is configured based on the target configuration parameter through the target configuration mode.

[0020] Optionally, further, determining a target configuration mode based on the component version identifier, and configuring a target element based on a target configuration parameter through the target configuration mode includes:

[0021] When the component version is identified as a version that does not support the definition of the native table primary key, a note field containing the identification text will be created as the target configuration method;

[0022] Configure a target element by adding identifying text and target configuration parameters in the target element's Comments field.

[0023] Optionally, further, determining a target configuration mode based on the component version identifier, and configuring a target element based on a target configuration parameter through the target configuration mode includes:

[0024] When the component version is identified as a version that supports native table primary key definition, the table definition method is used as the target configuration method;

[0025] Configure target elements based on target configuration parameters through table definition.

[0026] In a second aspect, an embodiment of the present invention further provides a device for uniformly creating a data model in an ETL process, comprising:

[0027] A model creation parameter module, used to obtain data model creation parameters corresponding to the data model creation instruction in response to the detected data model creation instruction;

[0028] The data structure model module is used to create a target data structure model based on data model creation parameters and pre-stored data structure instances.

[0029] In a third aspect, an embodiment of the present invention further provides a computer device, the device comprising:

[0030] one or more processors;

[0031] A storage device for storing one or more programs;

[0032] When one or more programs are executed by one or more processors, the one or more processors implement the unified data model creation method in the ETL process provided by any embodiment of the present invention.

[0033] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a unified method for creating a data model in an ETL process as provided in any embodiment of the present invention.

[0034] The unified creation method of data models in the ETL process provided by the embodiment of the present invention obtains data model creation parameters corresponding to the data model creation instruction in response to the detected data model creation instruction; creates a target data structure model based on the data model creation parameters and the pre-stored data structure instance. It realizes unified one-key creation of data structure, uniformly responds to various deployment scenarios, saves manpower, and thus makes the data processing project cycle fast, efficient, and avoids repeated manual operations and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flowchart of a method for uniformly creating a data model in an ETL process provided by the first embodiment of the present invention;

[0036] Figure 2 It is a flowchart of a method for uniformly creating a data model in an ETL process provided by the second embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the structure of a device for uniformly creating a data model in an ETL process provided by Embodiment 3 of the present invention;

[0038] Figure 4 It is a structural diagram of a computer device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0040] Embodiment 1

[0041] Figure 1 This is a flowchart of a method for uniformly creating a data model in an ETL process provided by the first embodiment of the present invention. This embodiment is applicable to the case of creating a data structure model on a component data engine before creating an ETL data processing system. The method can be executed by a device for uniformly creating a data model in an ETL process. The device for uniformly creating a data model in an ETL process can be implemented in software and / or hardware. For example, the device for uniformly creating a data model in an ETL process can be configured in a computer device. Figure 1 As shown, the method includes:

[0042] S110. In response to the detected data model creation instruction, obtain data model creation parameters corresponding to the data model creation instruction.

[0043] In this embodiment, the data model creation instruction may be an instruction initiated by a user for creating a data structure model corresponding to the Hadoop component in the ETL data processing system. The user may trigger the data model creation instruction through a model creation control in the data processing system. Exemplarily, the user may click on the model creation control to trigger the data model creation instruction. After the user triggers the data model creation instruction, the unified data model creation device in the ETL process may detect the data model creation instruction triggered by the user.

[0044] It is understandable that when using big data components such as HBASE, HIVE, HDFS, etc. under the big data Hadoop ecosystem, the creation of business models always needs to use different syntax and creation methods according to the different components used. In other words, different Hahoop components correspond to different data structure models. Taking the Hbase component as an example, its data structure needs to define column clusters, and when using Hive, specific column names are defined. Based on specific business needs, it is also necessary to add or merge the structures defined in one or more table models. For example, create Hbase large wide tables as needed or use index tables to speed up queries. For the creation of the data model of the source layer, its basic data structure is roughly the same as the business layer model of the source. Therefore, when creating the target data structure model corresponding to the Hadoop component, it is necessary to determine the components, configuration information and other parameters corresponding to the target data structure model. Based on this, when the user triggers the data model creation instruction, it is necessary to carry the creation parameters of the target data structure model at the same time. The data model creation parameters can be the component identifier of the Hadoop component corresponding to the target data structure model. After detecting a data model creation instruction triggered by a user, the unified data model creation device in the ETL process obtains the component identifier carried in the data model creation instruction to create a target data structure model based on the component identifier.

[0045] In one embodiment, obtaining the data model creation parameters corresponding to the data model creation instruction includes: obtaining the target application component identifier corresponding to the data model creation instruction, and the model configuration information associated with the target application component identifier as the data model creation parameter. Among them, the target application component identifier can be understood as the unique identifier of the target application component, such as the name and ID of the target application component. Optionally, the target application component can be any Hadoop component, such as HBASE, HIVE, HDFS and other big data components. Considering that the data structures corresponding to different Hadoop components are different, the user can specify the Hadoop component as the target application component according to the needs, and on the other hand, the configuration information can be added according to the business needs. Optionally, the user triggers the data model creation instruction after inputting the target application component and the model configuration information associated with the target application component. After detecting the data model creation instruction triggered by the user, the unified data model creation device in the ETL process obtains the target application component identifier carried in the data model creation instruction and the model configuration information associated with the target application component identifier as the data model creation parameter.

[0046] It should be noted that the target application component identifier can be one or more, and each target application component can set its corresponding model creation parameters. When the target application component identifier is multiple, the target data structure model corresponding to each target application component identifier can be created in parallel to speed up the model building speed, thereby speeding up the data processing project cycle. When the target application component identifier is multiple, the data structure layer corresponding to the target application component identifier can be created by the method provided in this embodiment, and each layer of the data structure model corresponds to the corresponding target application component identifier. When it is necessary to maintain any layer of the data structure, such as storing, viewing, and modifying, there is no need to manually modify the structure script of the data structure. It is only necessary to modify the data model creation parameters to complete the maintenance of the data structure. Therefore, the data structure model maintenance created by the method for automatically creating a data structure model provided in this embodiment does not require manual operation, saving manpower.

[0047] S120: Create a target data structure model based on the data model creation parameters and the pre-stored data structure instance.

[0048] After obtaining the model creation parameters corresponding to the data model creation instruction, the target data structure model is created based on the data model creation parameters and the pre-stored data structure instance. The pre-stored data structure instance can be any data structure instance. Optionally, the data structure instance of a relational database can be used as the pre-stored data structure instance.

[0049] In general, in this embodiment, element definitions, configurations, etc. in a data structure instance are converted based on the data model construction parameters to obtain a target data structure model corresponding to the data model creation parameters.

[0050] In one embodiment, a target data structure model is created based on data model creation parameters and a pre-stored data structure instance, including: determining the data structure model to be created according to the target application component identifier; creating the target data structure model from the pre-stored data structure instance based on the data structure model to be created and the model configuration information. On the basis that the data model creation parameters include the target application component identifier and the model configuration information, when creating the target data structure model, the data structure model to be created can be first determined according to the target application component identifier, the specific data structure of the data structure model to be created can be obtained, and then the specific data structure can be analyzed. Based on the model configuration information and the data structure model to be created, the pre-stored data structure instance is defined and configured for conversion to obtain the created data structure model. Exemplarily, assuming that the target application component identifier is HBase and Hive, the models of HBase and Hive are the data structure models to be created, and the basic configuration information of the pre-stored data structure instance (such as the corresponding relational database) is specified to be converted to obtain the target data structure model.

[0051] Optionally, a target data structure model is created from a pre-stored data structure instance based on the data structure model to be created and the model configuration information, including: selecting a target element from a pre-stored data structure instance according to the data structure model to be created; determining the definition configuration parameters of the target element according to the data structure model to be created, adjusting the definition configuration parameters based on the model configuration information, and obtaining the target configuration parameters of the target element; configuring the target element based on the target configuration parameters, and obtaining the target data structure model. In this embodiment, the target configuration parameters of the target element are determined in a manner that convention takes precedence over configuration. Specifically, according to the database connection configuration, a connection is made to a pre-stored data structure instance to obtain the metadata structure to be created, and the creation of the target element is pre-set in a manner that convention takes precedence over configuration. In the data structure model of Hbase, it includes the definition of Hbase column clusters, the definition of incremental column names, the definition of delete markers, etc. On this basis, other parameters of the target element are configured in combination with the model configuration information input by the user. Taking Hbase as an example, if partition columns need to be created for large and small tables, the data format of the partition columns and partition column values ​​can be determined through external configuration files. For example, when the date time is used as the partition column, the format can be set to year (yyyy), year-month (yyyy-MM), year-month-day (yyyy-MM-dd). More customized functions can be implemented by reading the model configuration information configured by the user during creation.

[0052] It is understandable that the target data structure model is created for data analysis in the Hadoop component. Therefore, it is necessary to consider whether the Hadoop component for creating the target data structure model is available. Some versions of Hadoop components support native table primary key definitions, while some versions of Hadoop components do not support native table primary key definitions. In order to ensure that the created target data structure model is applicable to different versions of Hadoop components, it is necessary to adopt different data model creation methods based on the versions of Hadoop components. Based on this, the target element is configured based on the target configuration parameters, including: obtaining the component version identifier corresponding to the target application component identifier; determining the target configuration method based on the component version identifier, and configuring the target element based on the target configuration parameters through the target configuration method. Optionally, the version number of the Hadoop component can be automatically obtained as the component version identifier, and whether the Hadoop component supports the native table primary key definition can be determined based on the component version identifier. Different target configuration methods are adopted based on whether the Hadoop component supports the native table primary key definition to achieve the configuration of the target element, thereby achieving the creation of the target data structure model.

[0053] In one embodiment, a target configuration mode is determined based on a component version identifier, and a target element is configured based on a target configuration parameter through the target configuration mode, including: when the component version identifier is a version identifier that does not support native table primary key definition, a remark field containing an identification text is created as a target configuration mode; and the target element is configured by adding the identification text and the target configuration parameter in the remark field of the target element. When it is determined based on the component version identifier that the Hadoop component does not support native table primary key definition, a special identification text "pk" can be added to the remark field of the creation column, and the program specially processes the reading of the primary key column information and adds the reading remark field as the primary key identification source to achieve the definition of the target element. Currently, version 1.0 of the Hadoop component does not support native table primary key definition. Therefore, when the component version identifier is 1.0, the target element is configured by adding the identification text and the target configuration parameter in the remark field of the target element.

[0054] In one embodiment, a target configuration mode is determined based on a component version identifier, and a target element is configured based on a target configuration parameter through the target configuration mode, including: when the component version identifier is a version identifier that supports native table primary key definition, the table definition mode is used as the target configuration mode; and the target element is configured based on the target configuration parameter through the table definition mode. When it is determined based on the component version identifier that the Hadoop component supports native table primary key definition, the DDL syntax of the primary key can be directly used to specify the table definition mode to implement the definition of the target element. Currently, the Hadoop component of version 1.2 does not support native table primary key definition. Therefore, when the component version identifier is 1.2, the target element is configured through the table definition mode.

[0055] Different definition methods are used for different versions of Hadoop components, so that the target data structure model created by definition can adapt to different versions of Hadoop components, ensuring the availability of data model construction.

[0056] The unified creation method of data models in the ETL process provided by the embodiment of the present invention obtains data model creation parameters corresponding to the data model creation instruction in response to the detected data model creation instruction; creates a target data structure model based on the data model creation parameters and the pre-stored data structure instance. It realizes unified one-key creation of data structure, uniformly responds to various deployment scenarios, saves manpower, and thus makes the data processing project cycle fast, efficient, and avoids repeated manual operations and maintenance.

[0057] Embodiment 2

[0058] Figure 2 This is a flow chart of a method for uniformly creating a data model in an ETL process provided by Embodiment 2 of the present invention. This embodiment provides a preferred embodiment based on the above embodiments.

[0059] The embodiment of the present invention provides a unified control to achieve one-click creation of a multi-layer data structure, which distinguishes the data structures of components such as Hbase, Hive, and HDFS. For example, the structure creation of the internal and external tables of Hbase and Hive is based on this.

[0060] HBase is a non-relational database for column cluster storage, used to store structured and unstructured data, and suitable for storing single-table non-relational data. Hive is similar to the structure of traditional relational databases, mainly including databases, files, tables, and views.

[0061] like Figure 2 As shown, the method provided in this embodiment mainly includes the following steps:

[0062] (1) Obtaining specific data structure

[0063] The user specifies the reference model when creating. For example, when creating HBase and Hive models, the basic configuration information of the corresponding relational database (Oracle, SQL Server, MySQL, PostgreSQL, etc.) must be specified, including JDBC-based connection configuration and other custom configurations (whether to delete existing ones, whether to map data field types, and whether to create a temporary table as a transition table for the incremental table).

[0064] (2) Data structure analysis

[0065] Connect to the instance according to the database connection configuration to obtain the metadata structure that needs to be created, and use conventions to take precedence over configuration to pre-set creation elements, such as Hbase column cluster definition, incremental column name definition, and deletion mark definition. For tables that need to create partition columns, external configuration files can be used to determine the data format of the partition columns and partition column values. For example, when the date and time are used as partition columns, the format can be set to year (yyyy), year-month (yyyy-MM), year-month-day (yyyy-MM-dd). More custom functions are provided by reading external configuration files during creation.

[0066] It should be noted that different versions of Hadoop components use different definition methods to configure elements. For example, in Hadoop component version 1.0, native table primary key definition is not supported. Under this premise, by adding a special identification text "pk" to the remark field of the created column, the program specially processes the reading of the primary key column information and adds the reading remark field as the primary key identification source. In Hadoop component version 1.2, native table primary key definition is supported. Under this premise, the primary key DDL syntax is directly used to specify the table definition method.

[0067] At present, the products involved in this embodiment can support related components, including: automatic creation of Hbase, Hive external tables, Hive internal tables and related data extraction tasks based on big data components Hbase and Hive; single table creation based on relational database to Hbase, large wide table creation components. The components of the big data platform include HDFS, Hive, HBase, Solr, YARN, Oozie, Spark, Storm, Sqoop, Pig, Impala, and Zookeeper.

[0068] The embodiment of the present invention adopts a consistent class table structure combined with the storage characteristics of HBase and the relevant rules of table structure definition to form a unified control to realize one-click creation of a multi-layer data structure. Therefore, even in the face of different deployment scenarios 1 and 2 or different ETL requirements, or different HBase and Hive models and relational database field conditions, it is still possible to uniformly respond to various deployment scenarios by adopting a class table structure definition specification (library table column method) through the method of giving priority to configuration by agreement, saving manpower, and fast data processing project cycle, high efficiency, and avoiding repeated manual operations and maintenance.

[0069] Embodiment 3

[0070] Figure 3 1 is a schematic diagram of the structure of a unified data model creation device in an ETL process provided by the third embodiment of the present invention. The unified data model creation device in the ETL process can be implemented in software and / or hardware. For example, the unified data model creation device in the ETL process can be configured in a computer device. Figure 3 As shown, the device includes a model creation parameter module 310 and a data structure model module 320, wherein:

[0071] A model creation parameter module 310 is used to obtain a data model creation parameter corresponding to the data model creation instruction in response to the detected data model creation instruction;

[0072] The data structure model module 320 is used to create a target data structure model based on data model creation parameters and pre-stored data structure instances.

[0073] The embodiment of the present invention obtains data model creation parameters corresponding to the data model creation instruction by responding to the detected data model creation instruction; and creates a target data structure model based on the data model creation parameters and the pre-stored data structure instance. This realizes unified one-click creation of data structures, uniformly responds to various deployment scenarios, saves manpower, and thus makes the data processing project cycle fast, efficient, and avoids repeated manual operations and maintenance.

[0074] Optionally, based on the above solution, the model creation parameter module 310 is specifically used to:

[0075] The target application component identifier corresponding to the data model creation instruction and the model configuration information associated with the target application component identifier are obtained as data model creation parameters.

[0076] Optionally, based on the above solution, the data structure model module 320 includes:

[0077] A model-to-be-created determining unit, used to determine a data structure model to be created according to a target application component identifier;

[0078] The target data model creation unit is used to create a target data structure model from a pre-stored data structure instance based on the data structure model to be created and the model configuration information.

[0079] Optionally, based on the above solution, the target data model creation unit includes:

[0080] A target element subunit is used to select a target element from a pre-stored data structure instance according to a data structure model to be created;

[0081] A target configuration parameter subunit is used to determine the definition configuration parameters of the target element according to the data structure model to be created, adjust the definition configuration parameters based on the model configuration information, and obtain the target configuration parameters of the target element;

[0082] The target element configuration subunit is used to configure the target element based on the target configuration parameters to obtain the target data structure model.

[0083] Optionally, based on the above solution, the target element configuration subunit is specifically used for:

[0084] Get the component version ID corresponding to the target application component ID;

[0085] The target configuration mode is determined based on the component version identifier, and the target element is configured based on the target configuration parameter through the target configuration mode.

[0086] Optionally, based on the above solution, the target element configuration subunit is specifically used for:

[0087] When the component version is identified as a version that does not support the definition of the native table primary key, a note field containing the identification text will be created as the target configuration method;

[0088] Configure a target element by adding identifying text and target configuration parameters in the target element's Comments field.

[0089] Optionally, based on the above solution, the target element configuration subunit is specifically used for:

[0090] When the component version is identified as a version that supports native table primary key definition, the table definition method is used as the target configuration method;

[0091] Configure target elements based on target configuration parameters through table definition.

[0092] The device for uniformly creating data models in the ETL process provided by the embodiment of the present invention can execute the method for uniformly creating data models in the ETL process provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0093] Embodiment 4

[0094] Figure 4 It is a structural diagram of a computer device provided in Embodiment 4 of the present invention. Figure 4 A block diagram of an exemplary computer device 412 suitable for use in implementing embodiments of the present invention is shown. Figure 4 The computer device 412 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0095] like Figure 4 As shown, the computer device 412 is in the form of a general-purpose computing device. The components of the computer device 412 may include, but are not limited to: one or more processors 416, a system memory 428, and a bus 418 that connects different system components (including the system memory 428 and the processor 416).

[0096] Bus 418 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a local bus to processor 416, or a bus structure using any of a variety of bus architectures. These architectures include, by way of example, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0097] The computer device 412 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 412, including volatile and non-volatile media, removable and non-removable media.

[0098] System memory 428 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 430 and / or cache memory 432. Computer device 412 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage device 434 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, usually called a "hard drive"). Although Figure 4 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 418 via one or more data medium interfaces. The memory 428 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0099] A program / utility 440 having a set (at least one) of program modules 442 may be stored, for example, in memory 428, such program modules 442 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 442 generally perform the functions and / or methods of the embodiments described herein.

[0100] The computer device 412 may also communicate with one or more external devices 414 (e.g., keyboard, pointing device, display 424, etc.), may also communicate with one or more devices that enable a user to interact with the computer device 412, and / or communicate with any device that enables the computer device 412 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed via input / output (I / O) interface 422. Furthermore, the computer device 412 may also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) via network adapter 420. As shown, the network adapter 420 communicates with other modules of the computer device 412 via bus 418. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 412, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0101] The processor 416 executes various functional applications and data processing by running the programs stored in the system memory 428, for example, implementing the unified creation method of data model in the ETL process provided by the embodiment of the present invention, including:

[0102] In response to the detected data model creation instruction, obtaining a data model creation parameter corresponding to the data model creation instruction;

[0103] A target data structure model is created based on the data model creation parameters and the pre-stored data structure instance.

[0104] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the unified data model creation method in the ETL process provided by any embodiment of the present invention.

[0105] Embodiment 5

[0106] Embodiment 5 of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for uniformly creating a data model in an ETL process provided by the embodiment of the present invention is implemented. The method includes:

[0107] In response to the detected data model creation instruction, obtaining a data model creation parameter corresponding to the data model creation instruction;

[0108] A target data structure model is created based on the data model creation parameters and the pre-stored data structure instance.

[0109] Of course, the computer-readable storage medium provided by an embodiment of the present invention stores a computer program which is not limited to the above method operations, but can also execute related operations of the unified data model creation method in the ETL process provided by any embodiment of the present invention.

[0110] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0111] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0112] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0113] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0114] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for uniformly creating a data model in an ETL process, characterized in that: include: In response to the detected data model creation instruction, obtaining a target application component identifier corresponding to the data model creation instruction and model configuration information associated with the target application component identifier as the data model creation parameter; Determine a data structure model to be created according to the target application component identifier, and create a target data structure model from a pre-created data structure instance based on the data structure model to be created and the model configuration information; The step of creating the target data structure model from a pre-created data structure instance based on the data structure model to be created and the model configuration information includes: Selecting a target element from a pre-stored data structure instance according to the data structure model to be created; Determine the definition configuration parameters of the target element according to the data structure model to be created, adjust the definition configuration parameters based on the model configuration information, and obtain the target configuration parameters of the target element; The target element is configured based on the target configuration parameters to obtain the target data structure model.

2. The method according to claim 1, characterized in that The configuring the target element based on the target configuration parameter includes: Obtain a component version identifier corresponding to the target application component identifier; A target configuration mode is determined based on the component version identifier, and the target element is configured based on the target configuration parameter through the target configuration mode.

3. The method according to claim 2, characterized in that The determining a target configuration mode based on the component version identifier, and configuring the target element based on the target configuration parameter through the target configuration mode, includes: When the component version identifier is a version identifier that does not support the definition of the native table primary key, a remark field containing an identifier text is created as the target configuration mode; The target element is configured by adding the identification text and the target configuration parameters in the remark field of the target element.

4. The method according to claim 2, characterized in that: The determining a target configuration mode based on the component version identifier, and configuring the target element based on the target configuration parameter through the target configuration mode, includes: When the component version identifier is a version identifier that supports native table primary key definition, the table definition mode is used as the target configuration mode; The target element is configured based on the target configuration parameters in the table definition manner.

5. A device for uniformly creating a data model in an ETL process, characterized in that: include: A model creation parameter module, configured to, in response to a detected data model creation instruction, obtain a target application component identifier corresponding to the data model creation instruction and model configuration information associated with the target application component identifier as the data model creation parameter; A data structure model module, used to determine the data structure model to be created according to the target application component identifier, and create a target data structure model from a pre-created data structure instance based on the data structure model to be created and the model configuration information; Among them, the data structure model module is specifically used for: Selecting a target element from a pre-stored data structure instance according to the data structure model to be created; Determine the definition configuration parameters of the target element according to the data structure model to be created, adjust the definition configuration parameters based on the model configuration information, and obtain the target configuration parameters of the target element; The target element is configured based on the target configuration parameters to obtain the target data structure model.

6. A computer device, characterized in that: The device comprises: one or more processors; A storage device for storing one or more programs; When one or more programs are executed by one or more processors, the one or more processors implement the unified data model creation method in the ETL process as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for uniformly creating a data model in an ETL process as described in any one of claims 1 to 4 is implemented.

Images