Database model development method and device, electronic equipment and storage medium

Abstract

The invention discloses a database model development method and device, electronic equipment and a storage medium, and relates to the technical field of data processing.The method comprises the steps that in response to selection operation of database model development requirements, developed service scene information and a target service platform are determined, and service data of the target service platform are obtained; determining a target engine in a plurality of preset modeling engines according to the business scene information; and processing the business data through the target engine to form a database model corresponding to the target business platform. The problem that a traditional database model development tool cannot flexibly select and configure a modeling engine for a specific business scene and a specific data platform is solved, and the effect of supporting different OLAP engines to perform database model development under different business scenes is achieved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a database model development method, device, electronic device and storage medium. Background Art

[0002] There are two main types of big data development tools on the market. One type is based on the Hadoop ecosystem, including Spark, Hive, etc. When users choose to develop models, they need to choose the database model corresponding to the Hadoop ecosystem development, so there are many constraints in the choice of models. The other type also includes more diverse self-developed big data products outside the Hadoop ecosystem, such as DataWorks, etc. These self-developed big data storages need to be bound to their own products, such as MaxCompute underlying storage platform or ByteHouse database. In summary, these database model development tools are not focused on the OLAP (Online Analytical Processing) big data analysis field, and cannot support the current mainstream OLAP engines such as ClickHouse / Kylin / Starrocks / Gauss200 database model development in different business scenarios.

[0003] Therefore, how to support different OLAP engines to develop database models in different business scenarios is a problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of this application is to provide a database model development method, device, electronic device and storage medium, aiming to solve the technical problem of how to support different OLAP engines to develop database models in different business scenarios.

[0005] To achieve the above objectives, the present application proposes a database model development method, which includes:

[0006] In response to the selection operation of the database model development requirement, the business scenario information to be developed and the target business platform are determined, and the business data of the target business platform is acquired;

[0007] Determine a target engine among multiple preset modeling engines according to the business scenario information;

[0008] The business data is processed by the target engine to form a database model corresponding to the target business platform.

[0009] In one embodiment, the step of determining a target engine among a plurality of preset modeling engines according to the business scenario information comprises:

[0010] Determine each candidate engine among a plurality of preset modeling engines according to the business scenario information;

[0011] The candidate engines are outputted, and the selected candidate engine is used as the target engine.

[0012] In one embodiment, the step of processing the business data by the target modeling engine to form a database model corresponding to the target business platform includes:

[0013] Determine the to-be-configured items of the database model according to the target cluster in the target modeling engine, and extract target data from the business data according to the to-be-configured items through the target modeling engine;

[0014] The target data is loaded into the item to be configured through the target modeling engine to form the database model.

[0015] In one embodiment, before the step of determining the to-be-configured items of the database model according to the target cluster in the target modeling engine, the step further includes:

[0016] Obtaining indicator data of each cluster associated with the target engine;

[0017] A target cluster among the clusters is determined according to the indicator data, and the target cluster is configured to the target engine.

[0018] In one embodiment, the database model development method further includes:

[0019] After receiving a model adjustment task for adjusting the database model, performing authority verification on the model adjustment task;

[0020] When the authority check of the model adjustment task is passed, the database model is adjusted according to the model adjustment task.

[0021] In one embodiment, the model adjustment task includes a data migration task, a data supplement task and a data modification task, wherein the data migration task is to migrate the model data of the database model to the target database model, the data supplement task is to add new business data to the database model, and the data modification task is to modify the model data of the database model;

[0022] The step of adjusting the database model according to the model adjustment task comprises:

[0023] In the case where the model adjustment task is the data migration task, obtaining a data screening condition configured in the data migration task, and filtering the model data of the database model based on the data screening condition; determining a migration engine from a plurality of preset modeling engines based on the target database model, and processing the filtered model data through the migration model to obtain the target database model;

[0024] In the case where the model adjustment task is the data supplementation task, based on the data supplementation scope configured in the data supplementation task, extracting the business data to be supplemented from the target business platform, and adding the business data to be supplemented to the database model;

[0025] In the case where the model adjustment task is the data modification task, the database model is updated according to the data to be modified configured in the data modification task.

[0026] In one embodiment, before the step of obtaining the service data of the target service platform, the step further includes:

[0027] The source data in the online transaction processing (OLTP) database is converted into a data collection in a preset standard format, and the data collection is transmitted to the target business platform so as to use the data collection as the business data of the target business platform.

[0028] In addition, to achieve the above purpose, the present application also proposes a database model development device, the database model development device comprising:

[0029] A response module, used to respond to the selection operation of the database model development requirement, determine the business scenario information to be developed, and the target business platform, and obtain the business data of the target business platform;

[0030] A determination module, used to determine a target engine among a plurality of preset modeling engines according to the business scenario information;

[0031] A processing module is used to process the business data through the target engine to form a database model corresponding to the target business platform.

[0032] In addition, to achieve the above-mentioned purpose, the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the database model development method as described above.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the database model development method described above are implemented.

[0034] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the database model development method described above.

[0035] One or more technical solutions proposed in this application have at least the following technical effects:

[0036] The present application first responds to the selection operation of the database model development requirements, determines the business scenario information to be developed, and the target business platform, and obtains the business data of the target business platform, thereby realizing the understanding of the business scenarios and model development requirements, and the preliminary collection of business data, providing reliable basic information for subsequent model development, while ensuring that the database model development process is guided by actual business needs, thereby improving the applicability and practicality of the database model; determining the target engine among multiple preset modeling engines according to the business scenario information, realizing the selection of the most suitable modeling engine for a specific business scenario, and ensuring the flexibility and pertinence of database model development; processing the business data through the target engine to form a database model corresponding to the target business platform, realizing the use of the selected modeling engine to process the business data, and finally generating a database model that matches the target business platform, achieving the effect of meeting specific business needs.

[0037] In summary, this application optimizes the process of database model development by automatically selecting the most suitable modeling engine according to the business scenario after the user selects the data source, namely the target business platform, and fills in the business scenario. The engine processes the business data to generate a database model that matches the target business platform. This avoids the problem that traditional database model development tools cannot flexibly select and configure modeling engines for specific business scenarios and specific data platforms, and achieves the effect of supporting different OLAP engines for database model development in different business scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 A flowchart of the first embodiment of the database model development method of the present application is provided;

[0041] Figure 2 A schematic diagram of a business space application process of a database model development method provided in Embodiment 1 of the present application;

[0042] Figure 3 A schematic diagram of a model development process of a database model development method provided in Example 1 of the present application;

[0043] Figure 4 A schematic diagram of a model development background design process for a database model development method provided in Example 1 of the present application;

[0044] Figure 5 A schematic diagram of the exchange platform linkage process of the database model development method provided in Example 1 of the present application;

[0045] Figure 6 A flowchart of the second embodiment of the database model development method of the present application is provided;

[0046] Figure 7 A schematic diagram of a model adjustment module of the database model development method provided in Example 2 of the present application;

[0047] Figure 8 A schematic diagram of the data migration process of the database model development method provided in Example 2 of the present application;

[0048] Fig. 9 A schematic diagram of the model complement process of the database model development method provided in Example 2 of the present application;

[0049] Fig.10 A schematic diagram of the business framework of the database model development method provided in Example 2 of the present application;

[0050] Fig.11 A user usage map diagram of the database model development method provided in Example 2 of the present application;

[0051] Fig.12 A schematic diagram of the physical deployment logic of the database model development method provided in the second embodiment of the present application;

[0052] Fig.13A schematic diagram of the module structure of the database model development device according to an embodiment of the present application;

[0053] Fig.14 A schematic diagram of the device structure of the hardware operating environment involved in the database model development method in the embodiment of the present application.

[0054] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0055] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0056] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0057] The main solution of the embodiment of the present application is: in response to the selection operation of the database model development requirements, determine the business scenario information to be developed, and the target business platform, and obtain the business data of the target business platform; determine the target engine among multiple preset modeling engines according to the business scenario information; process the business data through the target engine to form a database model corresponding to the target business platform.

[0058] Since existing database model development tools do not focus on the field of OLAP big data analysis, they cannot support the database model development of current mainstream OLAP engines such as ClickHouse / Kylin / Starrocks / Gauss200 in different business scenarios. Therefore, how to support database model development for different OLAP engines in different business scenarios is a problem that needs to be solved urgently.

[0059] The present application provides a solution to optimize the process of database model development. After the user selects the data source, namely the target business platform, and fills in the business scenario, the most suitable modeling engine is automatically selected according to the business scenario, and the business data is processed by the engine to generate a database model that matches the target business platform. This avoids the problem that traditional database model development tools cannot flexibly select and configure modeling engines for specific business scenarios and specific data platforms, and achieves the effect of supporting different OLAP engines for database model development in different business scenarios.

[0060] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc. The following takes an electronic device as an example to illustrate this embodiment and the following embodiments.

[0061] Based on this, the embodiment of the present application provides a database model development method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the database model development method of the present application.

[0062] In this embodiment, the database model development method includes steps S10 to S30:

[0063] Step S10, in response to the selection operation of the database model development requirement, determining the business scenario information to be developed and the target business platform, and acquiring the business data of the target business platform;

[0064] It should be noted that the selection operation refers to the operation of users filling in the business scenario on the front-end page; business scenario information refers to various parameters and conditions in a specific business environment, such as data type, data volume, business logic, business needs, etc.; the target business platform refers to a specific business system that requires data analysis and processing, such as Gaussian core warehouse, Hadoop, Kafka, etc.

[0065] It is understandable that since the premise of the correct model development direction is to understand the specific business needs and background, performing step S10 can avoid the problem of model design not meeting actual needs due to lack of understanding of business scenarios, thereby ensuring that the model development is closer to actual business needs and improving the accuracy and practicality of the model.

[0066] Exemplarily, the user selects from multiple preset business scenarios displayed on the front-end page to determine the business scenario information to be input into the OLAP platform. At the same time, the user specifies the source database to be processed as the target business platform in multiple source databases connected to the OLAP platform (such as Gaussian core warehouse, Hadoop, CSV (Comma-Separated Values) standard file, custom delimiter line file, Kafka, Flink, etc.), and obtains the business data in the target business platform. The business data can be static or in batch stream form.

[0067] Step S20, determining a target engine among a plurality of preset modeling engines according to the business scenario information;

[0068] It should be noted that the modeling engine refers to an OLAP engine that analyzes and processes business data, such as open source engines such as ClickHouse, Kylin, and Starrocks; the target engine refers to a modeling engine selected based on the business scenario and suitable for current business needs.

[0069] It can be understood that since the structures of database models developed by different engines are different, performing step S20 can avoid problems such as poor model performance or inability to meet business query requirements due to improper engine selection, ensuring that the selected engine can process business data efficiently and accurately, thereby improving the quality and efficiency of model development.

[0070] In a feasible implementation, step S20 may include steps S21 to S22:

[0071] Step S21, determining each candidate engine among a plurality of preset modeling engines according to the business scenario information;

[0072] It should be noted that the candidate engine refers to a group of modeling engines that are preliminarily screened out based on the business scenario information and may be suitable for processing the business scenario.

[0073] It is understandable that since the same business scenario can be applied to different modeling engines for model development, performing step S21 can avoid directly using an engine that does not meet the actual needs of users for model development, further ensuring that the engine used in the subsequent model development process has a certain degree of applicability and efficiency, and providing a technical path that better meets user needs for business analysis.

[0074] Step S22: output each candidate engine, and use the selected candidate engine as the target engine.

[0075] It is understandable that since the processing and modeling of business data requires a certain engine to execute, performing step S22 can avoid the incorrect use of an engine that may not meet specific business needs due to automatic selection. Through manual intervention and selection, it is ensured that the selected target engine is not only technically feasible, but also can better meet the specific needs of specific business scenarios, thereby improving the accuracy and applicability of model development.

[0076] Exemplarily, when the OLAP platform can support four different OLAP engines, namely modeling engines (Kylin, Starrocks, ClickHouse, Gaussian logical view), two OLAP engines are recommended and output for the user to choose based on the business scenario information filled in by the user. After receiving the engine selection instruction input by the user, the OLAP engine selected by the user among the two OLAP engines is determined as the target engine.

[0077] In this embodiment, by adopting automatic screening and user interaction, it is possible to flexibly match the most suitable modeling engine according to specific business needs, thereby improving data processing efficiency and the accuracy of model development, while also enhancing the flexibility of the system and the user's control over the model development process.

[0078] Step S30: Processing the business data through the target engine to form a database model corresponding to the target business platform.

[0079] It should be noted that the database model refers to the OLAP model obtained through processing by the target engine based on the data characteristics and analysis requirements of the target business platform. This model is a database model that stores multidimensional data, and users can perform data query operations through this model.

[0080] It is understandable that since the actual application value of the original business data is low and it is often difficult to be used directly, performing step S30 can avoid the problem of inaccurate business analysis results or inability to be applied to actual business due to improper model development, realize deep mining and analysis of business data, provide strong support for data query, and thus improve business operation efficiency.

[0081] In a feasible implementation, step S30 may include steps S31-S32:

[0082] Step S31, determining the to-be-configured items of the database model according to the target cluster in the target modeling engine, and extracting target data from the business data according to the to-be-configured items through the target modeling engine;

[0083] It should be noted that the target cluster refers to the cluster selected as the most suitable for performing specific data processing tasks under the target modeling engine; the items to be configured refer to the parameters in the database model that need to be set according to specific business needs and data characteristics; the target data refers to the specific data sets extracted from the original business data by the modeling engine and used to configure the database model. These data sets contain the information required to build the model.

[0084] It is understandable that, since it is necessary to clarify which business data is necessary based on the characteristics of the modeling engine and the requirements of the database model, so as to accurately extract valuable information for model construction from a large amount of business data, step S31 is performed to avoid model configuration errors or incomplete data extraction due to unclear configuration items, thereby affecting the accuracy and performance of the model, ensuring the accuracy of the database model configuration and the targeted data extraction, and laying the foundation for building an efficient and accurate database model.

[0085] For example, please refer to Table 1 below:

[0086]

[0087] Table 1

[0088] Table 1 shows the exclusive configuration items required by different modeling engines (Kylin, Starrocks, ClickHouse). The exclusive configuration items of any modeling engine are combined with the common configuration items common to all modeling engines to obtain the items to be configured of the database model. Based on the content of the items to be configured, the specific content corresponding to each item to be configured, i.e., the target data, is extracted from the business data.

[0089] In a feasible implementation manner, before step S30, steps S301 to S302 may also be included:

[0090] Step S301, obtaining indicator data of each cluster associated with the target engine;

[0091] It should be noted that indicator data refers to data used to measure and evaluate the performance and status of each cluster, including but not limited to the usage and importance of the cluster, the risk protection level of the cluster, etc.

[0092] It is understandable that since a modeling engine includes multiple clusters, and different operating states of different clusters are often not necessarily suitable for processing current business data, performing step S301 can avoid inefficient data processing due to improper cluster selection caused by not understanding the status of cluster-related indicators, or affecting the normal progress of other businesses, thereby causing processing delays or system failures. By selecting the optimal cluster, the execution efficiency of data processing tasks and the resource utilization of the system are improved, thereby ensuring the processing quality of business data.

[0093] Exemplarily, when the user requires to establish a database model with high query efficiency and not affected by other businesses, the risk assurance levels of 12 clusters associated with the ClickHouse engine, i.e., the target engine, are obtained.

[0094] Step S302: determining a target cluster among the clusters according to the indicator data, and configuring the target cluster to the target engine.

[0095] It is understandable that, since the selected target cluster needs to be bound to the target engine so that the engine can use cluster resources to perform data processing work, performing step S302 can avoid the target engine being unable to effectively call cluster resources, thereby affecting the execution of data processing work, ensuring that the target engine can smoothly call the resources of the target cluster, and improving the operating efficiency and data processing capabilities of the entire system.

[0096] Exemplarily, the indicator data of each cluster associated with the target engine includes the risk assurance level of each cluster and the load condition of each cluster. When the indicator data is the risk assurance level, the cluster corresponding to the target risk assurance level higher than the preset level among the risk assurance levels is taken as the target cluster, wherein the risk assurance level refers to the ability level of the cluster to maintain normal operation and service level when facing various potential risks (such as hardware failure, network interruption, software error, etc.); when the indicator data is the load condition, the cluster corresponding to the target load condition lower than the preset load among the load conditions is taken as the target cluster, wherein the load condition refers to the ability and status of the cluster to process the workload at a specific point in time.

[0097] For example, please refer to Figure 2 , users need to agree to the OLAP platform development specification agreement and SLA (Service Level Agreement) guarantee agreement before they can use the OLAP platform development tool to develop OLAP models. When filling in the scenario, the user enters the business scenario information for development, and uses the OLAP engine recommendation and selection, that is, determines the target engine among the candidate engines, and then allocates the engine cluster, that is, determines the target cluster among the clusters, and configures the target cluster to the target engine. For example: when the user requires the database model to be established with high query efficiency and is not affected by other businesses, the indicator data (such as risk assurance level) of the 12 clusters associated with the ClickHouse engine, that is, the target engine, is a cluster with a high or dedicated risk assurance level as the target cluster. Then the approval operation is carried out. If the approval is not passed, the space application fails; if the approval is passed, the space application is successful.

[0098] In this implementation, by adopting a method of dynamic scheduling of cluster resources, the effect of dynamically selecting and configuring optimal resources according to cluster performance and status is achieved, thereby optimizing the data processing process and improving the overall operating efficiency and data processing capability of the system.

[0099] Step S32: Loading the target data into the item to be configured through the target modeling engine to form the database model.

[0100] It is understandable that since the construction of the database model requires the support of specific data content, step S32 is performed. By applying the extracted target data to the configuration of the database model, the problem of missing model data due to improper data loading, which causes poor model performance, is avoided. This ensures that the model can be accurately developed according to the actual data situation, thereby providing strong data support for data query.

[0101] For example, please refer to Figure 3 and Figure 4 , Figure 3 This is a flow chart of the model development module. The user first clicks on the new model in the front-end interface, then selects the OLAP engine to be used, that is, determines the target engine, and selects the data source, that is, the user determines the target business platform among multiple business platforms. Then, the source information configuration, OLAP model information configuration, physical model design, data life cycle configuration, batch time and data offset configuration of the model development are performed to obtain the developed database model. After the database model development is completed, it still needs to go through the production approval process. Figure 4 For the background design part of the model development module, the user first submits the model configuration of the database model to be established, and then calls the model template associated with the OLAP engine after identifying the OLAP engine, i.e. the target engine. In the model template, fill in the table SQL (Structured Query Language) configuration, which contains fields, partitions, engines, and additional information, and then connect to the corresponding OLAP engine to execute the configuration information based on SQL. At the same time, generate a scheduling information configuration, which contains a globally unique job name, execution cycle, trigger conditions, etc., and schedule the upper limit of the configuration information, and then check the online situation. In addition, it is necessary to configure the exchange platform file permissions, and upload the configuration information to the exchange platform, and then check the online situation. After the above three configuration checks are completed, the database model required by the user can be obtained by processing the relevant business information through the OLAP engine.

[0102] In this embodiment, by extracting specific business data and configuring the model, the database model is automatically developed according to specific business needs, thereby improving the efficiency of model development while ensuring the accuracy and practicality of the model.

[0103] This embodiment provides a database model development method, which optimizes the process of database model development. After the user selects the data source, namely the target business platform, and fills in the business scenario, the most suitable modeling engine is automatically selected according to the business scenario, and the business data is processed by the engine to generate a database model that matches the target business platform. This avoids the problem that traditional database model development tools cannot flexibly select and configure modeling engines for specific business scenarios and specific data platforms, and achieves the effect of supporting different OLAP engines to develop database models in different business scenarios.

[0104] In a feasible implementation, step S10 may include step S100:

[0105] Step S100, converting source data in an online transaction processing (OLTP) database into a data collection in a preset standard format, and transmitting the data collection to the target business platform so as to use the data collection as business data of the target business platform.

[0106] It should be noted that source data refers to the original data stored in the OLTP database. This data is usually transactional and records the specific details of business operations; data collection refers to the data set formed after the source data is converted according to a preset standard format. This set contains business-related information that has been sorted and formatted.

[0107] It is understandable that since the OLAP engine has difficulty processing transaction data in different OLTP databases with different formats, step S100 is performed to unify the transaction data in different OLTP databases into a unified format for further analysis and processing in the target business platform, thereby avoiding the problem of data integration difficulties caused by inconsistent data formats, thereby improving the accuracy and efficiency of data analysis and processing on the OLAP platform, and providing reliable data support for business decisions.

[0108] For example, please refer to Figure 5 The OLAP scenario development tool where the database model development method runs can be connected to the exchange platform job development tool through the Open API (Application Programming Interface), so as to obtain the source data in the OLTP database (such as MYSQL, ORACLE, Tidb, etc.), and convert the data format of the source data into a CSV standard file. After the converted CSV standard file is transmitted to the target business platform (which can be Kylin, ClickHouse, Starrocks), the business data in the OLTP database can be successfully connected, thereby realizing the support of the database model development method for the OLTP database, thereby improving the applicability of the database model development method.

[0109] In this implementation, by adopting the method of data extraction, conversion and loading, the source data in the OLTP database is converted into a data collection in a preset standard format, and the data collection is transmitted to the target business platform as business data. This not only ensures the smooth flow and consistency of data between different systems, but also improves the data availability and analysis efficiency, providing an accurate and unified data foundation for business decision-making and operations.

[0110] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 6 , Figure 6 This is a flow chart of the second embodiment of the database model development method of the present application. The database model development method further includes steps S01 to S02:

[0111] Step S01, after receiving a model adjustment task for adjusting the database model, performing authority verification on the model adjustment task;

[0112] It should be noted that model adjustment tasks refer to specific instructions or requests to modify, optimize or update an already constructed database model. These tasks may include changing model parameters, updating data sets, etc.; permission verification refers to verifying whether the subject who initiates the model adjustment task has the authority to perform the task, ensuring that only authorized users or systems can change the database model.

[0113] It is understandable that since arbitrary changes and adjustments are not allowed after the database model is developed, step S01 is performed to avoid risks such as model errors or data leakage caused by unauthorized users making improper modifications to the database model, and to ensure that only users with appropriate permissions can adjust the database model, thereby protecting the security of the model and the privacy of business data.

[0114] Step S02: if the authority check of the model adjustment task is passed, adjusting the database model according to the model adjustment task.

[0115] It is understandable that since the database model after development may not meet new business needs, step S02 is performed. By adjusting the database model in a timely manner, it is possible to avoid the database model from losing accuracy and relevance due to failure to update in a timely manner, thereby ensuring that the model can continue to adapt to business changes, thereby improving the query accuracy and business adaptability of the model.

[0116] For example, please refer to Figure 7 In the model supplement / modification / cluster migration module, adjustment tasks that require modification of the database model are received through the API interface. The adjustment task can be a cluster migration, i.e., a data migration task, a supplement initiation, i.e., a data supplementation task, or a modification initiation, i.e., a data modification task. In addition, permission verification is configured in this module to perform permission verification on the received adjustment tasks. At the same time, during the task execution process, this module supports the functions of scheduled execution, concurrency restriction, manual intervention from the outside world, and failure alarm. Among them, concurrency restriction means that after the task is initiated, the backend application sets a preset cluster synchronization concurrency restriction to limit the synchronization tasks to queue for execution, thereby avoiding the problem of poor cluster resource scheduling leading to model adjustment or other business performance degradation when multiple tasks are initiated at the same time, thereby ensuring the normal operation of related businesses and the efficient implementation of model adjustments.

[0117] In a feasible implementation manner, the model adjustment task includes a data migration task, a data supplement task and a data modification task, wherein the data migration task is to migrate the model data of the database model to the target database model, the data supplement task is to add new business data to the database model, and the data modification task is to modify the model data of the database model;

[0118] The step of adjusting the database model according to the model adjustment task in step S02 may include steps S021 to S023:

[0119] Step S021, when the model adjustment task is the data migration task, obtaining the data screening condition configured in the data migration task, and filtering the model data of the database model based on the data screening condition; determining a migration engine among a plurality of preset modeling engines based on the target database model, and processing the filtered model data through the migration model to obtain the target database model;

[0120] It should be noted that the data migration task refers to migrating the data of a database model from one database model to another database model; the data screening condition refers to the rules determined according to the configuration items of the target database model and used to filter the data content required by the target database model; the model data refers to the data set that constitutes the database model; the migration engine refers to a specific modeling engine used to perform data migration tasks, which can convert data from one database model to another.

[0121] It is understandable that since there are often new business needs in actual business scenarios, and one database model is usually difficult to meet the new business needs, step S021 is performed to migrate business data between different database models, thereby avoiding the current database model from being unable to meet current business needs and improving the applicability of the database model to different business scenarios.

[0122] For example, please refer to Figure 8 In the data migration task, users only need to select the source model information, that is, the database model that needs to be migrated, and the target model information, that is, the target database model after the data migration, and then fill in the filter conditions, that is, the data screening conditions, and choose to perform field mapping, that is, users can choose to add, delete, and modify the fields in the model data, and then submit to initiate the data migration task to complete the migration of data between models.

[0123] Step S022, when the model adjustment task is the data supplementation task, based on the data supplementation scope configured in the data supplementation task, extract the business data to be supplemented from the target business platform, and add the business data to be supplemented to the database model;

[0124] It should be noted that the data supplementation task refers to the process of adding new data to the existing database model to enrich the model's information or improve the accuracy of model queries; the data supplementation scope refers to the scope or conditions for determining which data needs to be supplemented into the database model; the business data to be supplemented refers to the new data that needs to be added to the database model based on the data supplementation scope.

[0125] It is understandable that since the developed database model may have the problem of poor model performance due to incomplete data in the model, performing step S022 can avoid the problem of poor model performance caused by incomplete data and enhance the model performance of the database model by supplementing new data.

[0126] For example, please refer to Fig. 9 In the data supplement task, after the user clicks on the new supplement task in the front-end page, the source table, i.e. the database model that needs to be supplemented with data, and the extraction condition, i.e. the data supplement range, are configured, and then the corresponding OLAP model is selected to execute the supplement task on a scheduled or immediate basis.

[0127] Step S023, when the model adjustment task is the data modification task, the database model is updated according to the data to be modified configured in the data modification task.

[0128] It should be noted that the data modification task refers to the process of modifying or updating the data in the database model, and the data to be modified refers to the data that needs to be modified in the database model according to the data modification task.

[0129] It is understandable that since the model data in the database model may contain errors or need to be updated to reflect the latest business conditions, performing step S023 can avoid model output errors caused by data errors or inaccuracies. By updating the data, the accuracy of the database model is ensured, thereby improving the reliability of the model.

[0130] In this implementation, by performing data migration, data supplementation, and data modification in the developed database model, efficient and accurate data management and adjustment of the database model is achieved, ensuring the accurate migration of model data in data migration tasks, real-time updating and enhancement of the model in data supplementation tasks, and correction of model data in data modification tasks, thereby improving the adaptability and accuracy of the database model and supporting the continuous development of the business and decision-making needs.

[0131] In this embodiment, by adjusting the model after permission verification, strict control and security protection of the database model adjustment operation is achieved, ensuring that only authorized users or systems can perform model adjustment tasks, thereby avoiding the risks brought by unauthorized modifications, protecting the integrity and reliability of the database model, and ensuring the legality and accuracy of the model adjustment.

[0132] For example, to help understand the implementation process of the database model development method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Fig.10 , Fig.10 A business framework diagram of a database model development method is provided, specifically:

[0133] The core of the overall business framework is the OLAP one-stop work platform run by the database model development method, as well as the users, external applications, exchange platforms, scheduling platforms and business platforms connected to the work platform. The business platform can include Hadoop, GAUSS, exchange platform file information and Kafka information, etc. The work platform includes a front-end page connected to the user, an OPEN API connected to the external application, a metadata management module, and an OLAP model management module connected to the exchange platform, the scheduling platform, the business space application module, the model complement module, the inter-cluster data migration module, the model repair module, and the lifecycle management module. The front-end page provides users with a visual scenario-based development tool. The OPEN The API opens functional interfaces such as OLAP engine table information query and user unified addition / modification / deletion. Users only need to provide engine and table information or sources to complete the OLAP engine table creation work, without having to worry about the underlying table creation statements. The OLAP model management module includes a table / model management submodule and a model operation and maintenance monitoring submodule. The table / model management submodule can provide table model management for four OLAP engines (Gauss view, Kylin, ClickHouse, Starrocks), allowing the use of different engine features to build different OLAP query scenarios, such as point query or materialized acceleration, aiming to achieve low-code development by converting them into SQL and instructions for engine operations through an end-to-end development approach. The model operation and maintenance monitoring submodule ensures the update of the model readiness for the day by linking with other external systems, allowing users to locate and change problems in the model. The module also provides the model query function of B / S architecture, allowing users to query model data on the production network segment page. The metadata management module supports obtaining and importing the OLAP engine, cluster and configuration related information used by the model after the user selects the OLAP model. The model complement module allows users to select the complement days from the external multi-business platform scenario and automatically complement the OLAP table required by the user, that is, the database model. The inter-cluster data migration module allows users to complete model switching between the four different OLAP engines currently supported, customizes the automatic use of different engines for user usage scenarios, and also supports users to migrate historical data across clusters of different clusters of the same engine. The model repair module allows for supplementary maintenance of some user data in the OLAP engine. The life cycle management module provides various OLAP engine expired data partition cleanup functions to ensure engine performance.

[0134] For further information, please refer to Fig.11After the user operates in the front-end page, the back-end page module can realize operations such as business space application, model development, model supplementation, model modification, model monitoring and operation, and inter-cluster data migration. At the same time, the back-end page module can support four OLAP query engines including Kylin, Starrocks, Gauss, and ClickHouse.

[0135] For further information, please refer to Fig.12 In the figure, R represents load balancing routing, and A represents execution agent group. The business network and office network initiate service requests to the backend service. The backend service supports the job configuration of the data exchange platform and the scheduling platform, metadata query of ADT (Analytical Data Tier), ADH (Analytical Data Hub) and Kylin, table management of OLAP metadata service, configuration of lifecycle management, initiation of migration tasks to historical data migration execution scheduling, and monitoring data query for operation and maintenance monitoring. The six dotted boxes in the upper part of the figure all represent ACS (Alibaba Cloud Container Compute Service) containers. The lifecycle management performs partition cleaning on OLAP-Router, the historical data migration execution scheduling executes tasks through multiple execution agent groups, and the operation and maintenance monitoring performs the scheduling of Kafka information. The OLAP metadata service, OLAP-Router, and execution agent group are connected to each ClickHouse cluster.

[0136] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the database model development method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0137] This application also provides a database model development device, please refer to Fig.13 , the database model development device comprises:

[0138] The response module 10 is used to respond to the selection operation of the database model development requirement, determine the business scenario information to be developed, and the target business platform, and obtain the business data of the target business platform;

[0139] A determination module 20, configured to determine a target engine among a plurality of preset modeling engines according to the business scenario information;

[0140] The processing module 30 is used to process the business data through the target engine to form a database model corresponding to the target business platform.

[0141] Optionally, the determining module 20 is further configured to:

[0142] Determine each candidate engine among a plurality of preset modeling engines according to the business scenario information;

[0143] The candidate engines are outputted, and the selected candidate engine is used as the target engine.

[0144] Optionally, the determining module 20 is further configured to:

[0145] Determine the to-be-configured items of the database model according to the target cluster in the target modeling engine, and extract target data from the business data according to the to-be-configured items through the target modeling engine;

[0146] The target data is loaded into the item to be configured through the target modeling engine to form the database model.

[0147] Optionally, the determining module 20 is further configured to:

[0148] Obtaining indicator data of each cluster associated with the target engine;

[0149] A target cluster among the clusters is determined according to the indicator data, and the target cluster is configured to the target engine.

[0150] Optionally, the adjustment module 40 in the database model development device is used to:

[0151] After receiving a model adjustment task for adjusting the database model, performing authority verification on the model adjustment task;

[0152] When the authority check of the model adjustment task is passed, the database model is adjusted according to the model adjustment task.

[0153] Optionally, the model adjustment task includes a data migration task, a data supplement task and a data modification task, wherein the data migration task is to migrate the model data of the database model to the target database model, the data supplement task is to add new business data to the database model, and the data modification task is to modify the model data of the database model;

[0154] The adjustment module 40 is used for:

[0155] In the case where the model adjustment task is the data migration task, obtaining a data screening condition configured in the data migration task, and filtering the model data of the database model based on the data screening condition; determining a migration engine from a plurality of preset modeling engines based on the target database model, and processing the filtered model data through the migration model to obtain the target database model;

[0156] In the case where the model adjustment task is the data supplementation task, based on the data supplementation scope configured in the data supplementation task, extracting the business data to be supplemented from the target business platform, and adding the business data to be supplemented to the database model;

[0157] In the case where the model adjustment task is the data modification task, the database model is updated according to the data to be modified configured in the data modification task.

[0158] Optionally, the conversion module 50 in the database model development device is used to:

[0159] The source data in the online transaction processing (OLTP) database is converted into a data collection in a preset standard format, and the data collection is transmitted to the target business platform so as to use the data collection as the business data of the target business platform.

[0160] The database model development device provided by the present application adopts the database model development method in the above embodiment, which can solve the technical problem of how to support different OLAP engines to develop database models in different business scenarios. Compared with the prior art, the beneficial effects of the database model development device provided by the present application are the same as the beneficial effects of the database model development method provided by the above embodiment, and other technical features in the database model development device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0161] The present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the database model development method in the above-mentioned embodiment one.

[0162] Reference below Fig.14 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.14The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0163] like Fig.14 As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0164] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0165] The electronic device provided by the present application adopts the database model development method in the above embodiment, which can solve the technical problem of how to support different OLAP engines to develop database models in different business scenarios. Compared with the prior art, the beneficial effects of the electronic device provided by the present application are the same as the beneficial effects of the database model development method provided by the above embodiment, and the other technical features in the electronic device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0166] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0167] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0168] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the database model development method in the above-mentioned embodiment.

[0169] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0170] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0171] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device: responds to the selection operation of the database model development requirements, determines the business scenario information to be developed, and the target business platform, and obtains the business data of the target business platform; determines the target engine among multiple preset modeling engines according to the business scenario information; processes the business data through the target engine to form a database model corresponding to the target business platform.

[0172] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0173] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0174] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0175] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned database model development method, and can solve the technical problem of how to support different OLAP engines to develop database models in different business scenarios. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the database model development method provided in the above-mentioned embodiment, and will not be repeated here.

[0176] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned database model development method when executed by a processor.

[0177] The computer program product provided by this application can solve the technical problem of how to support different OLAP engines to develop database models in different business scenarios. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the database model development method provided by the above embodiment, which will not be repeated here.

[0178] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A database model development method, characterized in that: The database model development method comprises: In response to the selection operation of the database model development requirement, the business scenario information to be developed and the target business platform are determined, and the business data of the target business platform is acquired; Determine a target engine among multiple preset modeling engines according to the business scenario information; The business data is processed by the target engine to form a database model corresponding to the target business platform.

2. The database model development method according to claim 1, characterized in that: The step of determining a target engine among a plurality of preset modeling engines according to the business scenario information comprises: Determine each candidate engine among a plurality of preset modeling engines according to the business scenario information; The candidate engines are outputted, and the selected candidate engine is used as the target engine.

3. The database model development method according to claim 1, characterized in that: The step of processing the business data by the target modeling engine to form a database model corresponding to the target business platform includes: Determine the to-be-configured items of the database model according to the target cluster in the target modeling engine, and extract target data from the business data according to the to-be-configured items through the target modeling engine; The target data is loaded into the item to be configured through the target modeling engine to form the database model.

4. The database model development method according to claim 3, characterized in that: Before the step of determining the to-be-configured items of the database model according to the target cluster in the target modeling engine, the step further includes: Obtaining indicator data of each cluster associated with the target engine; A target cluster among the clusters is determined according to the indicator data, and the target cluster is configured to the target engine.

5. The database model development method according to claim 1, characterized in that: The database model development method further includes: After receiving a model adjustment task for adjusting the database model, performing authority verification on the model adjustment task; When the authority check of the model adjustment task is passed, the database model is adjusted according to the model adjustment task.

6. The database model development method according to claim 5, characterized in that: The model adjustment task includes a data migration task, a data supplement task and a data modification task, wherein the data migration task is to migrate the model data of the database model to the target database model, the data supplement task is to add new business data to the database model, and the data modification task is to modify the model data of the database model; The step of adjusting the database model according to the model adjustment task comprises: In the case where the model adjustment task is the data migration task, obtaining a data screening condition configured in the data migration task, and filtering the model data of the database model based on the data screening condition; determining a migration engine from a plurality of preset modeling engines based on the target database model, and processing the filtered model data through the migration model to obtain the target database model; In the case where the model adjustment task is the data supplementation task, based on the data supplementation scope configured in the data supplementation task, extracting the business data to be supplemented from the target business platform, and adding the business data to be supplemented to the database model; In the case where the model adjustment task is the data modification task, the database model is updated according to the data to be modified configured in the data modification task.

7. The database model development method according to claim 1, characterized in that: Before the step of acquiring the business data of the target business platform, the following steps are also included: The source data in the online transaction processing (OLTP) database is converted into a data collection in a preset standard format, and the data collection is transmitted to the target business platform so as to use the data collection as the business data of the target business platform.

8. A database model development device, characterized in that: The database model development device comprises: A response module, used to respond to the selection operation of the database model development requirement, determine the business scenario information to be developed, and the target business platform, and obtain the business data of the target business platform; A determination module, used to determine a target engine among a plurality of preset modeling engines according to the business scenario information; A processing module is used to process the business data through the target engine to form a database model corresponding to the target business platform.

9. An electronic device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the database model development method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the database model development method according to any one of claims 1 to 7 are implemented.

Images