Data model development method and device based on low-code platform and medium

Abstract

The invention discloses a data model development method based on a low-code platform. The method comprises the following steps: S1, generating first test data according to a service demand; the first test data comprises a data demand template, a data model, a data mapping relation, an SQL code and a scheduling operation process; step S2, generating a test case according to the first test data and completing unit test, and if the test passes, entering step S3; if the test is not passed, returning to the step S1; and S3, outputting and storing the first test data, and merging the data model and the data mapping relation to a baseline version. The method has the advantages that the development efficiency is high, and the technical threshold is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data development technology, and in particular to a data model development method, device and medium based on a low-code platform. Background Art

[0002] The low-code development technology background in the field of data ETL (Extract-Transform-Load) mainly stems from the changes in enterprises' demand for data development and the development of software development trends in the new era.

[0003] With the development of enterprise businesses and their digital transformation, the demand for data processing is growing. Traditional data development methods often require significant manpower, material resources, and time, and are prone to errors. Therefore, enterprises need more efficient and user-friendly data development methods to achieve rapid iteration and support their business needs. With the rise of low-code development technology, more and more enterprises are adopting low-code platforms for application development and deployment. Low-code platforms provide visual interfaces and predefined components, enabling even non-professional developers to quickly build applications. This development approach significantly lowers the barrier to entry and costs, improving development efficiency. Low-code development technology also has great potential in the data ETL field. By enabling template-based and configuration-based development, enterprises can quickly build data models and implement data extraction, transformation, and loading. Low-code platforms also provide a series of automation tools, such as data modeling, data mapping, and data standardization, to automatically generate and execute data processing and test case code, maximizing development efficiency and quality.

[0004] At present, low-code development technology in the data field lacks complete technical support, and related research and applications are relatively rare; low-code development methods in the data field often cannot adapt to a variety of business scenarios and cannot be widely promoted and applied.

[0005] Therefore, how to propose a data model development method, device and medium based on a low-code platform has become an urgent problem that needs to be solved. Summary of the Invention

[0006] The purpose of the present invention is to provide a data model development method, device and medium based on a low-code platform, which has the advantages of high development efficiency and lower technical barriers.

[0007] To achieve the above-mentioned objectives, the present invention provides a data model development method based on a low-code platform, the method comprising: step S1, generating first test data according to business requirements; the first test data includes a data requirement template, a data model, a data mapping relationship, an SQL code, and a scheduling job process; step S2, generating a test case according to the first test data and completing a unit test, if the test passes, proceeding to step S3; if the test fails, returning to step S1; step S3, outputting and saving the first test data, and merging the data model and data mapping relationship into the baseline version.

[0008] Preferably, the step S1 includes: step S11, configuring a data requirement template according to business requirements; step S12, performing model design according to the data requirement template and outputting a physical data model; step S13, configuring a data mapping relationship according to the data model; step S14, creating SQL code according to the data mapping relationship; step S15, creating a scheduling job process according to the data mapping relationship.

[0009] Preferably, in step S11, the data requirement template includes: data items, data processing rules, data update frequency, data scale, and data life cycle.

[0010] Preferably, step S12 includes: generating a conceptual model design based on the data requirement template, generating a logical model design based on the data requirement and the conceptual model, and then generating a final physical data model; wherein, the conceptual model design refers to configuring the entity name, entity attributes and relationship between entities of the data model in the conceptual model design template based on the data item definition in the data requirement template to generate a conceptual model; the logical model design refers to configuring the logical model design template based on the data processing rules, data update frequency and data scale in the data requirement template in combination with the conceptual model; after the logical model configuration data is determined, the relevant tables and fields are standardized in combination with the standard metadata maintained by the data governance team to obtain a physical model.

[0011] Preferably, the data mapping relationship in step S13 includes the target system, target table, target field constructed based on the model visualization module, and the data source system, data source table, data source field, mapping rules, association conditions, and storage method constructed based on the data model.

[0012] Preferably, step S14 includes reading the target system, target table, target field and data processing rules of the data requirement model in the mapping relationship to automatically generate the table structure modification code of the SQL code part, and automatically creating the data batch processing code of the automatically generated SQL code part according to the data source system, data source table, data source field, mapping rules and association conditions in the mapping relationship. Finally, the system packages the two parts to generate the final SQL code.

[0013] Preferably, step S15 includes: the scheduling operation process is used to control the system's daily data processing process, by reading the data source, data destination, mapping rules, and association rule information in the mapping relationship, selecting the system's preset scheduling operation process template, and generating a scheduling operation process.

[0014] Preferably, the step S2 includes: step S21, matching the test case template of the corresponding scenario according to the first test data; specifically, the test case template is a test case template preset for different scenarios, and the test case template corresponding to the data scenario is determined by reading the mapping rules, association conditions, and storage method information in the data mapping relationship; step S22, reading the data mapping relationship, SQL code, and scheduling job flow, and combining the test case template to generate test cases for checking the SQL code and scheduling job flow respectively, and generating test data by combining the test case and data mapping information; step S23, executing unit testing according to the test case and test data; if the test passes, proceed to step S3, if not, return to step S1.

[0015] A data model development device based on a low-code platform, the device is used to implement any of the methods described above, the device includes: a first development module, used to generate first test data according to business needs; a first test module, which generates test cases based on the first test data and completes unit testing; a first storage module, which saves the first test data.

[0016] A storage medium, comprising stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute a data model development method based on a low-code platform as described in any one of the foregoing items.

[0017] In summary, compared with the existing technology, the data model development method and device based on the low-code platform provided by the present invention have the following beneficial effects:

[0018] First, the present invention can improve development efficiency. It reduces the amount of handwritten code in traditional coding through template and configuration methods, thereby increasing development speed. At the same time, it integrates design, development and testing to improve overall development efficiency.

[0019] Second, the present invention can lower the technical threshold. Since this method saves a lot of handwritten code, non-professional developers can also participate in the development work in a short period of time.

[0020] Third, the present invention can reduce errors and failures because it uses process-based modules and components, thereby reducing errors and non-standard issues in handwritten codes. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a schematic diagram of a data model development method based on a low-code platform proposed in the present invention.

[0022] Figure 2 This is a schematic diagram of a data model development device based on a low-code platform proposed in the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the appended Figure 1 ~Attached Figure 2 , the technical solutions, structural features, objectives achieved and effects in the embodiments of the present invention are described in detail.

[0024] It should be noted that the drawings are in a very simplified form and use non-precise proportions. They are only used to conveniently and clearly assist in explaining the embodiments of the present invention, and are not used to limit the conditions for the implementation of the present invention. Therefore, they have no substantive technical significance. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in the present invention without affecting the efficacy and purpose that can be achieved by the present invention.

[0025] It should be noted that, in the present invention, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only the elements explicitly listed, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0026] This paper proposes a data model development method based on a low-code platform. The method builds a data model based on business needs, enabling rapid and accurate implementation of business needs and solving business problems. The business needs that can be addressed by this method include: adding new data to a business, modifying data types, and deleting data.

[0027] The method comprises:

[0028] Step S1, generating first test data according to business requirements;

[0029] Specifically, the first test data includes but is not limited to: data requirement template, data model, data mapping relationship, SQL (Structured Query Language) code, scheduling job flow, etc.; for specific content, please refer to the following description.

[0030] Step S2, generating a test case based on the first test data and completing the unit test. If the test passes, proceed to step S3; if the test fails, return to step S1;

[0031] Unit testing is a test case written by developers during the development process to test whether the developed functional modules are running normally. In this step, test cases and test data are created based on the first test data, and the test cases are executed to simplify the development process.

[0032] Step S3: output and save the first test data, merge the data model and data mapping relationship into the baseline version;

[0033] Data models and data mapping relationships are managed through baseline control technology. Baseline information refers to a set of configuration items or data states that are officially approved and confirmed at a specific point in the project or data development process. Baseline control technology is a series of management and control methods based on these baselines, used to monitor, record, and manage changes to data or projects. A baseline version is a version at a specific point in time.

[0034] In addition, when configuring a new data model and data mapping relationship, it will be compared and verified with the existing data model and data mapping relationship. When the corresponding data model and data mapping relationship does not exist, it will be entered and the baseline of the specific time point will be marked; when the data model and data mapping relationship exists and there is an update, the new data model and data mapping relationship will be entered and the new baseline of the specific time point will be marked; when the data model and data mapping relationship exists and there is no update, the data model and data mapping relationship will not be entered.

[0035] In a specific embodiment, step S1 includes:

[0036] Step S11, configuring a data requirement template according to business requirements;

[0037] A data requirements template is an information template that directly maps to business requirements. After data requirements analysis, it serves as the input for the first step and forms the foundation for all subsequent steps. Elements of a data requirements template include: data item definitions, data processing rules, data update frequency, data size, and data lifecycle. Each business requirement is designed with a corresponding data requirements template. As previously mentioned, upon passing testing, the configured data requirements template is stored in a storage space, such as a data repository, to form a requirements baseline for tracking and managing data requirements.

[0038] The elements of the data requirement template are defined as follows:

[0039] A data item is the basic unit of data. A data item definition explicitly describes a specific data element, specifying its name, meaning, data type, and value range. It summarizes and defines the essential characteristics of data, ensuring that all participants in the data processing process have a consistent understanding of the data.

[0040] Data processing rules refer to the specific methods and logic for processing and transforming raw data to meet specific business requirements or data quality standards. These rules can include operations such as data cleansing, data transformation, data calculation, and data aggregation.

[0041] The data update frequency refers to the interval between data updates in the system. It reflects the timeliness requirements of the data, and different business scenarios have different requirements for data update frequency.

[0042] Data scale primarily refers to the amount of data, typically measured in bytes, kilobytes, megabytes, gigabytes, or records. Understanding data scale helps select appropriate data storage and processing technologies, as well as assess system performance and resource requirements.

[0043] The data lifecycle describes the entire process of data generation, use, maintenance, and eventual destruction, including stages such as data creation, storage, processing, sharing, archiving, and deletion. Clarifying the data lifecycle helps rationally manage data resources and ensure data security and compliance.

[0044] Step S12: Design a model based on the data requirement template to produce a physical data model;

[0045] Specifically, the physical data model is the blueprint of the entire data warehouse, used to display the data entities, entity attributes, and the connections between entities within the data warehouse. Updating and maintaining the data model is an essential step before the data development process. Model design includes conceptual model design, logical model design, and physical data model design.

[0046] In this step, based on the data requirement template generated in the previous step, a conceptual model is produced, and then a logical model is produced by combining the data requirements and the conceptual model, and then the final physical data model is produced.

[0047] Conceptual model design: Based on the data item definitions in the data requirement template, the conceptual model design template is used to configure the entity names, entity attributes, and relationships of the data model to generate a conceptual model. The conceptual model is rendered and displayed through the model visualization module, allowing users to intuitively view and understand the conceptual model.

[0048] Logical model design: Based on the data processing rules, data update frequency, and data scale in the data requirements template, and in conjunction with the conceptual model, a logical model design template is configured. The table name, field name, field type, field constraints, primary key information, partition information, and field descriptions are configured in the logical model design template to generate the logical model. Once the logical model configuration data is finalized, the relevant tables and fields are standardized using the standard metadata maintained by the data governance team, ultimately resulting in the physical model.

[0049] Step S13: configuring data mapping relationships according to the data model;

[0050] The mapping relationship is a design link in the data development process and is the intermediary between the data model and the SQL code. It is used to display the association relationship between the target entity (entity and entity attributes) and the source entity (entity and entity attributes). The data mapping relationship includes the target system, target table, and target field built based on the model visualization module, and the data source system, data source table, data source field, mapping rules, association conditions, and storage method built based on the data model.

[0051] Step S14, creating SQL (Structured Query Language) code according to the data mapping relationship;

[0052] After the data mapping relationship is configured, select a specific SQL code template, read the data source, data target, mapping rules, association conditions and other information in the mapping relationship, and automatically generate the corresponding SQL code.

[0053] Specifically, the target system, target table, target field and data processing rules of the data requirement model in the mapping relationship are read to automatically generate the table structure modification code of the SQL code part. The data batch processing code of the automatic generation of SQL code part is automatically created according to the data source system, data source table, data source field, mapping rules and association conditions in the mapping relationship. Finally, the system packages the two parts to generate the final SQL code.

[0054] Step S15, creating a scheduling operation process according to the data mapping relationship;

[0055] The scheduling operation process is used to control the system's daily data processing process. By reading the data source, data destination, mapping rules, association rules and other information in the mapping relationship, the system's preset scheduling operation process template is selected to generate the scheduling operation process.

[0056] Specifically, to create a scheduling job process, it is necessary to establish a scheduling job process based on different data mapping rules. When creating a scheduling job process, first read the mapping rules, storage method and other information in the data mapping relationship to determine the data scenario, and then match the scheduling job process template of the corresponding scenario. According to the data information of the data mapping relationship, create the corresponding scheduling jobs in sequence, and configure the scheduling information of the job, including the scheduling time, scheduling type, job dependency, job type, etc. The job types include data integration jobs, data conversion and loading jobs, etc.

[0057] The step S2 comprises:

[0058] Step S21, matching a test case template corresponding to a scenario according to the first test data;

[0059] Specifically, the test case template is a test case template preset for different scenarios, and the test case template corresponding to the data scenario is determined by reading the mapping rules, association conditions, storage method and other information in the data mapping relationship;

[0060] Step S22: Read the data mapping relationship, SQL code, and scheduling workflow, and generate test cases for checking the SQL code and scheduling workflow respectively based on the test case template, and generate test data based on the test case and data mapping information;

[0061] Step S23: Execute unit test according to the test case and test data. If the test passes, proceed to step S3; if not, return to step S1.

[0062] In a specific embodiment, the invention also discloses a data model development device based on a low-code platform, such as Figure 2 As shown, the device includes:

[0063] A first development module, configured to generate first test data according to business requirements;

[0064] A first test module generates test cases based on the first test data and completes unit testing;

[0065] The first storage module stores first test data.

[0066] Although the present invention has been described in detail through the above preferred embodiments, it should be understood that the above description is not intended to limit the present invention. After reading the above description, various modifications and substitutions of the present invention will become apparent to those skilled in the art. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A data model development method based on a low-code platform, characterized in that: The method comprises: Step S1, generating first test data according to business requirements; the first test data includes a data requirement template, a data model, a data mapping relationship, an SQL code, and a scheduling operation process; Step S2, generating a test case based on the first test data and completing the unit test. If the test passes, proceed to step S3; if the test fails, return to step S1; Step S3: output and save the first test data, and merge the data model and data mapping relationship into the baseline version.

2. A data model development method based on a low-code platform according to claim 1, characterized in that: The step S1 comprises: Step S11, configuring a data requirement template according to business requirements; Step S12: Design a model based on the data requirement template to produce a physical data model; Step S13: configuring data mapping relationships according to the data model; Step S14, creating SQL code according to the data mapping relationship; Step S15: Create a scheduling workflow based on the data mapping relationship.

3. A data model development method based on a low-code platform according to claim 2, characterized in that: In step S11, the data requirement template includes: data items, data processing rules, data update frequency, data scale, and data life cycle.

4. A data model development method based on a low-code platform according to claim 3, characterized in that: The step S12 includes: generating a conceptual model design based on the data requirement template, generating a logical model design based on the data requirement and the conceptual model, and then generating a final physical data model; The conceptual model design refers to configuring the entity names, entity attributes and relationships between entities of the data model in the conceptual model design template based on the data item definitions in the data requirement template to generate a conceptual model; Logical model design refers to the configuration of the logical model design template based on the data processing rules, data update frequency, and data scale in the data requirement template, combined with the conceptual model; After the logical model configuration data is determined, the relevant tables and fields are standardized in combination with the standard metadata maintained by the data governance team to obtain the physical model.

5. A data model development method based on a low-code platform according to claim 4, characterized in that: The data mapping relationship in step S13 includes the target system, target table, target field constructed based on the model visualization module, and the data source system, data source table, data source field, mapping rules, association conditions, and storage method constructed based on the data model.

6. A data model development method based on a low-code platform according to claim 5, characterized in that: The step S14 includes reading the target system, target table, target field and data processing rules of the data requirement model in the mapping relationship to automatically generate the table structure modification code of the SQL code part, and automatically creating the data batch processing code of the automatically generated SQL code part according to the data source system, data source table, data source field, mapping rules and association conditions in the mapping relationship. Finally, the system packages the two parts to generate the final SQL code.

7. A data model development method based on a low-code platform according to claim 6, characterized in that: Step S15 includes: the scheduling operation process is used to control the system's daily data processing process, by reading the data source, data destination, mapping rules, and association rule information in the mapping relationship, selecting the system's preset scheduling operation process template, and generating a scheduling operation process.

8. A data model development method based on a low-code platform according to claim 7, characterized in that: The step S2 comprises: Step S21, matching a test case template corresponding to a scenario according to the first test data; specifically, the test case template is a test case template preset for different scenarios, and the test case template corresponding to the data scenario is determined by reading the mapping rules, association conditions, and storage method information in the data mapping relationship; Step S22: Read the data mapping relationship, SQL code, and scheduling workflow, and generate test cases for checking the SQL code and scheduling workflow respectively based on the test case template, and generate test data based on the test case and data mapping information; Step S23, perform unit testing according to the test case and test data; if the test passes, proceed to step S3, if not, return to step S1.

9. A data model development device based on a low-code platform, characterized in that: The device is used to implement the method according to any one of claims 1 to 8, and the device includes: A first development module, configured to generate first test data according to business requirements; A first test module generates test cases based on the first test data and completes unit testing; The first storage module stores first test data.

10. A storage medium, characterized in that: The storage medium includes stored instructions, wherein, when the instructions are executed, the device where the storage medium is located is controlled to execute a data model development method based on a low-code platform as described in any one of claims 1 to 8.

Images