Data model creation method, device, equipment and medium
Through simplified modeling tools and automation solutions, the problems of high learning costs and complex operations of database modeling are solved, which improves modeling efficiency and accuracy and reduces errors.
Patent Information
- Application Number
- CN202411017603.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-12-17
- Estimated Expiration
- 2044-07-29
AI Technical Summary
In the prior art, database modeling has high learning costs and complex operations, resulting in low user modeling efficiency and error-prone.
Get the model type through the front-end, set the column types and order of index columns and partitioned columns, generate partition attributes, and divide the model data according to pre-increment, allocate buckets, and splice the required statements to simplify the database modeling process.
It reduces learning costs, simplifies operational processes, improves modeling efficiency and accuracy, and reduces user errors during the modeling process.
Smart Images
Figure CN118550900B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of communications, and in particular to a data model creation method, apparatus, device and medium. Background Art
[0002] Database is the key carrier for storing, managing and retrieving data; using database modeling is the process of designing and creating database structure. Database modeling helps to store, manage and retrieve data efficiently.
[0003] In the prior art, when using database modeling, users are required to have an in-depth understanding of the basic principles of the database, including mastering relevant knowledge such as database table creation, partitioning, models, and syntax. This results in high learning costs and complex operations for database modeling, which is not conducive to users performing database modeling.
[0004] Therefore, the present application proposes a data model creation method that can solve the above problems. Summary of the invention
[0005] The present application provides a data model creation method, device, equipment and medium to solve the problems in the prior art that database modeling has high learning costs and complex operations, resulting in low modeling efficiency and prone to errors for users.
[0006] In a first aspect, the present application provides a data model creation method, which is applied to a backend, comprising:
[0007] Obtaining the model type through the front end, setting the column type and / or column order for the index column and partition column corresponding to the model, and generating partition attributes according to the partition column selected by the front end;
[0008] Obtaining a pre-increment of the model through the front end, and using the pre-increment as a segmentation unit of the model data to obtain the number of buckets of the model data;
[0009] Allocating buckets of the model data among the servers according to the number of buckets and the number of servers in the cluster;
[0010] According to the column type and / or column order, the partition attribute and the bucket, the statements required by the model are concatenated and transmitted to the front end.
[0011] In a possible implementation, obtaining the model type through the front end and setting the column type and / or column order for the index column and partition column corresponding to the model include:
[0012] If the model type is the first type, confirming that the column type is set for the index column and the partition column corresponding to the model;
[0013] If the model type is the second type, confirm that the column type and the column order are set for the index column and the partition column corresponding to the model.
[0014] In a possible implementation manner, if the model type is the first type, the method further includes:
[0015] If the model type is a first subtype of the first type, confirming that the column types of the index column and the partition column corresponding to the model are set to dimension columns;
[0016] If the model type is the second subtype of the first type, confirming that the column types of the index column and the partition column corresponding to the model are set as primary key columns;
[0017] The confirming setting the column type and the column order for the index column and the partition column corresponding to the model includes:
[0018] The index column is set at the front position of a creation statement, and the column types of the index column and the partition column corresponding to the model are set to sequential columns, wherein the creation statement includes the creation of the partition column.
[0019] In a possible implementation, obtaining the number of buckets of the model data includes:
[0020] Acquire a total capacity of the model data, and acquire a first ratio of the total capacity to the segmentation unit;
[0021] If the first ratio is an integer, the first ratio is used as the number of buckets of the model data;
[0022] If the first ratio is a non-integer, the integer digit of the first ratio is increased by 1 as the number of buckets of the model data.
[0023] In a possible implementation, allocating the buckets of the model data among the servers according to the number of buckets and the number of servers in the cluster includes:
[0024] If the number of buckets is greater than or equal to the number of servers, confirm that at least one bucket is allocated to each server, wherein the capacity of each bucket is equal to the pre-increment;
[0025] If the number of buckets is less than the number of servers, confirm and obtain a second ratio of the total capacity of the model data to the number of servers, use the second ratio as the capacity of the bucket, and allocate one bucket to each server.
[0026] In a possible implementation, before obtaining the model type through the front end or generating partition attributes according to the partition column selected by the front end or obtaining the pre-increment of the model through the front end, the method further includes:
[0027] Obtain the task instruction of the front end, and confirm to obtain the model type, column information, the partition column selected by the front end, and the pre-increment of the model according to the task instruction, wherein the column information related to the time type is set as the partition column, and the column information can be set as an index column or a non-index column.
[0028] In a possible implementation, the splicing of statements required by the model according to the column type and / or column order, the partition attribute and the bucketing includes:
[0029] Obtaining a task instruction of the front end, and confirming to obtain basic information of the model according to the task instruction, wherein the basic information includes a grammar database and a hierarchy, and the front end is used to verify and generate the task instruction according to the model type, column information, the partition column selected by the front end, the pre-increment and the basic information of the model;
[0030] According to the basic information, the column type and / or column order, the partition attribute and the bucket, the statements required by the model are concatenated.
[0031] In a second aspect, the present application provides a data model creation device, comprising:
[0032] An acquisition module, used to acquire a model type through a front end, set a column type and / or a column order for an index column and a partition column corresponding to the model, and generate a partition attribute according to the partition column selected by the front end;
[0033] A first processing module, configured to obtain a pre-increment of the model through the front end, and use the pre-increment as a segmentation unit of the model data to obtain the number of buckets of the model data;
[0034] A second processing module, configured to allocate the buckets of the model data among the servers according to the number of buckets and the number of servers in the cluster;
[0035] A splicing module is used to splice the statements required by the model according to the column type and / or column order, the partition attribute and the bucket, and transmit them to the front end.
[0036] In a third aspect, the present application provides a data model creation device, including: at least one processor and a memory;
[0037] The memory stores computer-executable instructions;
[0038] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the data model creation method as described above.
[0039] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data model creation method as described above.
[0040] The present application provides a data model creation method, apparatus, device and medium, which obtains the model type through the front end, sets the column type and / or column order for the index column and partition column corresponding to the model, and generates partition attributes according to the partition column selected by the front end; obtains the pre-increment of the model through the front end, and uses the pre-increment as the segmentation unit of the model data to obtain the number of buckets of the model data; allocates the buckets of the model data in the servers according to the number of buckets and the number of servers in the cluster; and splices the statements required by the model according to the column type and / or column order, the partition attributes and the buckets, and transmits them to the front end.
[0041] In the above method, the front-end page sets options and input box and other controls to obtain the parameters corresponding to the model that the user wants to establish, including model type, partition attributes generated by partition columns, pre-increment, etc.; the back-end can set the corresponding parameters through the parameters corresponding to the model collected by the front-end, and perform statement splicing; when processing the parameters corresponding to the model, according to the model type, set the column type and / or column order for the index column and partition column corresponding to the model, so as to facilitate the created model to query related columns; the front-end selects one from multiple partition columns and generates partition attributes for the partition column, so as to better manage the partition columns; the pre-increment is used as the segmentation unit to obtain the buckets of the model data and the number of buckets, so as to facilitate the subsequent positioning of the buckets of the model data; according to the processed column type and / or column order, partition attributes and buckets, the statements required by the model are spliced, and a model that meets the needs of the front-end is established, and the feedback is given to the front-end, so that the front-end can obtain the model in a simple and efficient way. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0043] Figure 1 A schematic diagram of a scenario for creating a data model provided in an embodiment of the present application;
[0044] Figure 2 A schematic diagram of a data model creation method provided in an embodiment of the present application Figure 1 ;
[0045] Figure 3A schematic diagram of a data model creation method provided in an embodiment of the present application Figure 2 ;
[0046] Figure 4 A schematic diagram of a data model creation method provided in an embodiment of the present application Figure 3 ;
[0047] Figure 5 A data model creation device diagram provided by an embodiment of the present invention;
[0048] Figure 6 A hardware schematic diagram of a data model creation device provided in an embodiment of the present invention.
[0049] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0051] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.
[0052] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0053] In the prior art, when using database modeling, users are required to have an in-depth understanding of the basic principles of the database, including mastering relevant knowledge such as database table creation, partitioning, models, and syntax. This results in high learning costs and complex operations for database modeling, which is not conducive to users performing database modeling.
[0054] For example, when using the Doris database for modeling, the following problems need to be faced:
[0055] Data Definition Language (DDL) is used to define and manage objects in the database. The DDL syntax used when creating tables in the Doris database is different from the DDL syntax used when creating tables in traditional relational databases, and users need to adapt to it again;
[0056] The indexing principle, partitioning method, model selection and column definition of Doris database are different from those of some traditional databases, and users need to understand them again before they can use them correctly. This increases the complexity and learning cost of modeling. The complex modeling method makes it easy for users to make mistakes and perform irregular operations during the modeling process, affecting the efficiency of later data queries.
[0057] Therefore, this application proposes a data model creation method to solve the above problems:
[0058] Reduce learning costs: By providing simplified modeling tools or automated modeling solutions, users can complete modeling without having to deeply learn Doris's unique DDL syntax, indexing strategies, and partitioning methods.
[0059] Simplify the operation process: By providing an intuitive modeling interface or predefined modeling templates, the user's operation process is simplified and the chance of error is reduced.
[0060] Improve modeling efficiency: Through the back-end automatic optimization function, users can quickly assemble mature DDL statements to improve modeling efficiency and accuracy.
[0061] The implementation process of a data model creation method proposed in this application is described in detail below with reference to the accompanying drawings and specific embodiments.
[0062] Figure 1 A schematic diagram of a scenario for creating a data model provided in an embodiment of the present application. Figure 1 As shown, the system includes: a front-end interface; the front-end interface includes different controls, and different controls can be set with different functions;
[0063] In the control for obtaining basic information, click the checkbox of the data source type to display different specific database options, that is, select the syntax database. Click the corresponding option to select the database used for modeling. The database that can be selected by this control can be bound according to the requirements; click the checkbox of the layer to display different specific layer types. Click the corresponding option to select the specific layer type. The layer that can be selected by this control can be bound according to the requirements. The data in the data warehouse is usually stored and processed in layers to improve the manageability, maintainability and availability of the data. Common layer types include the operational data layer (Operational Data Store, ODS), the data mart layer (Data Warehouse Incremental Maintenance, DIM), etc. The main function of ODS is to store and manage these raw data, and the main function of DIM is to realize the incremental maintenance and update of the data warehouse;
[0064] Model type is the type corresponding to the database. Different model types can be displayed as fixed values below the model type on the front-end page. The model type can be selected through the circular selection control. Each field includes a column of data, which can be displayed horizontally below the field on the front-end page. One or more columns of data can form a data table. Each field has a unique field name, and different attributes corresponding to the field name are displayed in the same row as the field name. If some fields are selected as partition columns, they can be displayed below the partition on the front-end page, and there is a circular selection control before each display, which can select a partition column. The front-end page can also set a fill control, which has a pre-increment default value, and the default value is set to allow overwriting.
[0065] The system also includes a backend, which simply collects the parameters needed to create the model on the frontend page, and then processes and models the model based on these parameters:
[0066] The model type is obtained through the front end, and the column type and / or column order are set for the index column and partition column corresponding to the model, and the partition attribute is generated according to the partition column selected by the front end; the pre-increment of the model is obtained through the front end, and the pre-increment is used as the segmentation unit of the model data to obtain the number of buckets of the model data; the buckets of the model data are allocated to the servers according to the number of buckets and the number of servers in the cluster; based on the syntax corresponding to the selected database, the back end has statements that describe the column type and / or column order, partition attributes and buckets. The model creation can be completed by splicing the statements required by the model, and the created model is fed back to the front end to conveniently complete the modeling.
[0067] Figure 2 A schematic diagram of a data model creation method provided in an embodiment of the present application Figure 1 .like Figure 2 As shown, the method includes:
[0068] S201. Obtain a model type through a front end, set a column type and / or a column order for an index column and a partition column corresponding to the model, and generate a partition attribute according to the partition column selected by the front end.
[0069] The system includes a front-end and a back-end. The front-end is provided with a front-end page for interacting with users. At the same time, the display content and controls of the front-end page are arranged according to the needs. Some controls will also be associated with existing database and other information. The implementation logic of the layout page and the associated information is established on the back-end; the layout of the front-end page simplifies the modeling operation, and at the same time can collect modeling-related parameters and generate modeling task instructions to pass to the back-end, and the corresponding modeling is implemented through the logic of the back-end, so as to complete the modeling on the basis of simplified operation;
[0070] The front end collects information and gives instructions to enable the back end to perform subsequent modeling processing:
[0071] For example, a task instruction of the front end is obtained, and it is confirmed that the model type, column information, the partition column selected by the front end, and the pre-increment of the model are obtained according to the task instruction, wherein the column information related to the time type is set as the partition column, and the column information can be set as an index column or a non-index column.
[0072] When the backend receives the task instruction from the frontend, it means that the frontend has collected the information needed for the backend modeling. For example, when the user models the frontend page according to the requirements, he selects and fills in the information to be collected on the frontend page. When the user clicks submit, the collected information is submitted to the backend.
[0073] On the front-end page, you can set the model type, column information, partition columns selected by the front-end, and pre-increment for collection modeling;
[0074] The fields collected on the front-end page are column information. Each column information can be set as an index column or a non-index column. The setting method can be to set a corresponding selection control for each column information. When the selection control is turned on, the column information is selected as the index column.
[0075] Partition columns can be set according to business requirements. For example, column information includes field names and field types. Some field types can be selected for partitioning according to requirements. When the field type corresponding to a field is the field type selected for partitioning, the field is a partition column. In this embodiment, the field type selected for partitioning is related to the time type. For example, the field type of birthday is related to the time type. A field has only one field name, but it may be both an index column and a partition column.
[0076] Fields can be added through the add control on the front-end page or imported from other models. The add control can be bound to some preset fields, and the field type of the field can be modified according to needs; if the added field has a field type selected for partitioning, the field type selected for partitioning can be automatically identified and displayed below the partition, and a single selection can be made to obtain the partition column selected by the front-end;
[0077] The model type is the model type selected according to business needs when creating a model. Different model types have different column type settings, and some model types also have column order settings.
[0078] The model type is selected on the front-end page, and the index column and partition column corresponding to the model are also selected. Therefore, the back-end can set the column type and / or column order for the index column and partition column according to the model type; the partition attribute can also be generated according to the partition column selected on the front-end to facilitate partition management of certain fields;
[0079] The front-end page also collects the estimated increment, which is the estimated data increment for modeling. For example, if the daily data increment of the model after modeling is 5G, the estimated daily increment is 5G, which can be filled in as 5G.
[0080] Confirm the model type based on the selected database. Confirm the column type and column order based on the selected model type:
[0081] For example, if the model type is the first type, confirming that the column type is set for the index column and the partition column corresponding to the model;
[0082] If the model type is the second type, confirm that the column type and the column order are set for the index column and the partition column corresponding to the model.
[0083] For some model types, you only need to set the column type for the index column and partition column corresponding to the model. This type of model type is the first type. For some model types, you can set the column type and column order for the index column and partition column corresponding to the model. This type of model type is the second type. Select the corresponding model type on the front-end page to match the processing of the index column and partition column on the back-end.
[0084] S202: Obtain a pre-increment of the model through the front end, and use the pre-increment as a segmentation unit of the model data to obtain the number of buckets of the model data.
[0085] The front-end page collects pre-increments, and the back-end can use the pre-increments as the unit for bucketing. That is, the capacity of each bucket is the pre-increment. The model data is divided according to the size of each bucket's capacity as the pre-increment to obtain multiple buckets, and the number of buckets is counted. The remaining model data after the division that is less than the pre-increment is also regarded as a bucket. The model establishes a model table, and the model data is table data.
[0086] S203. Allocate the buckets of the model data among the servers according to the number of buckets and the number of servers in the cluster.
[0087] To facilitate data management and query, the storage location of the buckets can be allocated according to the number of servers in the cluster, and the buckets can be stored in different servers in a decentralized manner. This can ensure the work efficiency of each server and facilitate data query and management. The distribution of buckets in the servers can be as balanced as possible. For example, first divide them equally and then distribute the excess buckets to each server. If the model generates new data to form new buckets, the new buckets will be allocated to servers with fewer storage buckets.
[0088] S204: According to the column type and / or column order, the partition attribute and the bucket, the statements required by the model are assembled and transmitted to the front end.
[0089] After the backend sets the column type and / or column order, partition attributes, and bucketing, it can describe them with corresponding statements, assemble the required statements, obtain the created model, and feed it back to the front end; for example, after setting the column order, arrange the statements for creating fields in the corresponding order;
[0090] The front-end collects information including:
[0091] For example, a task instruction of the front end is obtained, and basic information of the model is obtained according to the task instruction, wherein the basic information includes a grammar database and a hierarchy, and the front end is used to verify and generate the task instruction according to the model type, column information, the partition column selected by the front end, the pre-increment and the basic information of the model;
[0092] According to the basic information, the column type and / or column order, the partition attribute and the bucket, the statements required by the model are concatenated.
[0093] The front-end page can also collect basic information required for modeling. After the modeling parameters are filled in and selected according to the requirements of the front-end page and the control is clicked to confirm the completion of the operation, the modeling parameters will be checked. After confirming that the modeling parameters required to be filled in and selected by the front-end page are all correctly set, the task instructions will be generated and the modeling parameters will be given to the back-end; wherein, the modeling parameters include the model type, column information, the partition column selected by the front-end, the pre-increment and basic information of the model;
[0094] The basic information includes information such as syntax database, layering, table name, model description and model explanation. The syntax database is the database type for creating the model. The Doris database is selected in this embodiment; the layering is the table layering method for creating the model; the table name is the name of the table for creating the model; the statements required for the subsequent front-end splicing model come from the syntax in the syntax database in the basic information, and the column type and / or column order, partition attributes and buckets obtained by the back-end processing are spliced with syntax.
[0095] In an embodiment of the present application, the front-end page sets options and input box controls to obtain parameters corresponding to the model that the user wants to establish, including model type, partition column generation partition attributes, pre-increment, etc.; the back-end can perform corresponding parameter settings and statement splicing through the parameters corresponding to the model collected by the front-end; when processing the parameters corresponding to the model, the column type and / or column order are set for the index column and partition column corresponding to the model according to the model type, so as to facilitate the creation of model query related columns; the front-end selects one from multiple partition columns and generates partition attributes for the partition column to better manage the partition columns; the pre-increment is used as the segmentation unit to obtain the buckets of the model data and the number of buckets, so as to facilitate the subsequent positioning of the buckets of the model data; according to the processed column type and / or column order, partition attributes and buckets, the statements required by the model are spliced, and a model that meets the front-end requirements is established, and feedback is given to the front-end, so that the front-end can obtain the model in a simple and efficient manner.
[0096] Figure 3 A schematic diagram of a data model creation method provided in an embodiment of the present application Figure 2 .like Figure 3 As shown, the method includes:
[0097] S301: If the model type is a first subtype of the first type, confirm that the column types of the index column and the partition column corresponding to the model are set to dimension columns.
[0098] After selecting the database, the front end can choose one from different model types for modeling; after receiving the task instruction from the front end, the back end can determine the selected model type; the model type that does not need to be sorted corresponds to the first type; the first type is divided into multiple subtypes according to different column types, including the first subtype and the second subtype; if the back end recognizes that the model type selected by the front end is the first subtype, that is, the Aggregate model, then the back end sets the column type of the index column and the partition column to the dimension column, and the field set to the dimension column can be set to simple calculation processing such as addition.
[0099] S302: If the model type is the second subtype of the first type, confirm that the column types of the index column and the partition column corresponding to the model are set as primary key columns.
[0100] If the backend recognizes that the model type selected by the frontend is the second subtype, that is, the unique model, the backend sets the column type of the index column and the partition column to the primary key column. The field set as the primary key column does not have simple calculation processing such as addition.
[0101] S303: If the model type is the second type, the index column is set at the front position of the creation statement, and the column types of the index column and the partition column corresponding to the model are set to sequential columns, wherein the creation statement includes the creation of the partition column.
[0102] The model type that needs to be sorted corresponds to the second type, that is, the duplicate model. If the backend recognizes that the frontend selects the second type, the index column can be set at the front of the creation statement, that is, when the backend uses the statement to create the index column, the field definition with the index column is written before the field definition without the index column. The column types of the index column and the partition column are both set to sequential columns. The fields defined in the duplicate model are inserted by column when managing data.
[0103] In the embodiment of the present application, the different processing of index columns and partition columns by different models is confirmed through the modeling type selection of the front end, so as to ensure that the corresponding modeling can be implemented based on the simple selection of the front end.
[0104] Figure 4 A schematic diagram of a data model creation method provided in an embodiment of the present application Figure 3 .like Figure 4 As shown, the method includes:
[0105] S401: Obtain a total capacity of the model data, and obtain a first ratio of the total capacity to the segmentation unit.
[0106] When bucketing, it is necessary to confirm the total capacity of the model data, and use the segmentation unit as the unit of bucketing to obtain multiple buckets with the capacity as the segmentation unit; the number of buckets is related to the total capacity and the segmentation unit. By calculating the first ratio of the total capacity and the segmentation unit, the initial number of buckets can be obtained, but the initial number of buckets is not necessarily an integer, and the number of buckets needs to be an integer.
[0107] S402: If the first ratio is an integer, use the first ratio as the number of buckets for the model data.
[0108] If the first ratio is an integer, then the first ratio can directly obtain the current candidate bucket number of the model data.
[0109] S403: If the first ratio is a non-integer, add one to the integer digit of the first ratio as the number of buckets for the model data.
[0110] If the first ratio is a non-integer, the first ratio needs to be rounded to an integer. After rounding, the remaining capacity of the uncreated buckets of the model data is less than the division unit, but a bucket is still needed; that is, when the first ratio is a non-integer, it needs to be rounded up, and the integer value of the first ratio plus one is used as the current candidate bucket number of the model data;
[0111] However, the final bucketing is also related to the relationship between the number of servers and the number of buckets:
[0112] For example, if the number of buckets is greater than or equal to the number of servers, then confirm that at least one bucket is allocated to each server, wherein the capacity of each bucket is equal to the pre-increment;
[0113] If the number of buckets is less than the number of servers, confirm and obtain a second ratio of the total capacity of the model data to the number of servers, use the second ratio as the capacity of the bucket, and allocate one bucket to each server.
[0114] If the number of buckets is greater than or equal to the number of servers, it means that there are too many buckets. You can try to evenly distribute the buckets on each server. In this case, each server is assigned at least one bucket. For example, there are 5 buckets and 3 servers. The buckets of each server are 2, 2, and 1. When the model has a new bucket, the new bucket is stored on the server with 1 bucket to ensure balanced server load and high processing efficiency.
[0115] If the number of buckets is less than the number of servers, it means that there are a large number of servers. Instead of creating buckets according to the pre-increment, you can directly divide the total capacity of the model data evenly and distribute all the data evenly on each server. At this time, the second ratio of the total capacity of the model data to the number of servers is the capacity of each bucket.
[0116] In the embodiment of the present application, by reasonably allocating buckets, it is ensured that data can be efficiently managed and scheduled between servers.
[0117] Figure 5 A data model creation device diagram provided by an embodiment of the present invention, such as Figure 5 As shown, the device includes: an acquisition module 501, a first processing module 502, a second processing module 503, and a splicing module 504;
[0118] The acquisition module 501 is used to acquire the model type through the front end, set the column type and / or column order for the index column and partition column corresponding to the model, and generate partition attributes according to the partition column selected by the front end.
[0119] The acquisition module 501 is further configured to confirm setting the column type for the index column and the partition column corresponding to the model if the model type is the first type;
[0120] If the model type is the second type, confirm that the column type and the column order are set for the index column and the partition column corresponding to the model.
[0121] The acquisition module 501 is further configured to: if the model type is the first type, the method further includes:
[0122] If the model type is a first subtype of the first type, confirming that the column types of the index column and the partition column corresponding to the model are set to dimension columns;
[0123] If the model type is the second subtype of the first type, confirming that the column types of the index column and the partition column corresponding to the model are set as primary key columns;
[0124] The confirming setting the column type and the column order for the index column and the partition column corresponding to the model includes:
[0125] The index column is set at the front position of a creation statement, and the column types of the index column and the partition column corresponding to the model are set to sequential columns, wherein the creation statement includes the creation of the partition column.
[0126] The first processing module 502 is used to obtain the pre-increment of the model through the front end, and use the pre-increment as a segmentation unit of the model data to obtain the number of buckets of the model data.
[0127] The first processing module 502 is further used to obtain the total capacity of the model data and obtain a first ratio of the total capacity to the segmentation unit;
[0128] If the first ratio is an integer, the first ratio is used as the number of buckets of the model data;
[0129] If the first ratio is a non-integer, the integer digit of the first ratio is increased by 1 as the number of buckets of the model data.
[0130] The second processing module 503 is used to distribute the buckets of the model data among the servers according to the number of buckets and the number of servers in the cluster.
[0131] The second processing module 503 is further configured to confirm that at least one bucket is allocated to each server if the number of buckets is greater than or equal to the number of servers, wherein the capacity of each bucket is equal to the pre-increment;
[0132] If the number of buckets is less than the number of servers, confirm and obtain a second ratio of the total capacity of the model data to the number of servers, use the second ratio as the capacity of the bucket, and allocate one bucket to each server.
[0133] The splicing module 504 is used to splice the statements required by the model according to the column type and / or column order, the partition attribute and the bucket, and transmit them to the front end.
[0134] The splicing module 504 is further used to obtain the task instruction of the front end, and confirm to obtain the basic information of the model according to the task instruction, wherein the basic information includes a grammar database and a layer, and the front end is used to verify and generate the task instruction according to the model type, column information, the partition column selected by the front end, the pre-increment and the basic information of the model;
[0135] According to the basic information, the column type and / or column order, the partition attribute and the bucket, the statements required by the model are concatenated.
[0136] The present application also provides a data model creation device, comprising: at least one processor and a memory;
[0137] The memory stores computer-executable instructions;
[0138] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs a data model creation method.
[0139] Figure 6 The hardware diagram of the data model creation device provided in the embodiment of the present invention is shown in FIG. Figure 6As shown, the data model creation device 60 provided in this embodiment includes: at least one processor 601 and a memory 602. The device 60 also includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus 604.
[0140] In the specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above data model creation method.
[0141] The specific implementation process of the processor 601 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.
[0142] In the above Figure 6 In the illustrated embodiment, the processor may be a central processing unit (CPU) or other general-purpose processors. The memory may include a high-speed memory (Random Access Memory, RAM), a non-volatile memory (NVM), etc. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc.
[0143] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method described above is implemented.
[0144] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, and can be any available medium that can be accessed by a general or special-purpose computer. All or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the above-mentioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.
[0145] So far, the technical solution of the present application has been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments, and the above embodiments are only used to illustrate the technical solution of the present application rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A data model creation method, characterized in that: Applied to the backend, including: Obtaining the model type through the front end, setting the column type and / or column order for the index column and partition column corresponding to the model, and generating partition attributes according to the partition column selected by the front end; Obtaining a pre-increment of the model through the front end, and using the pre-increment as a segmentation unit of the model data to obtain the number of buckets of the model data; Allocating buckets of the model data among the servers according to the number of buckets and the number of servers in the cluster; According to the column type and / or column order, the partition attribute and the bucket, the statements required by the model are assembled and transmitted to the front end; The method of obtaining the model type through the front end and setting the column type and / or column order for the index column and partition column corresponding to the model includes: If the model type is the first type, confirming to set the column type for the index column and the partition column corresponding to the model, including, if the model type is the first subtype, confirming to set the column type of the index column and the partition column corresponding to the model as a dimension column; if the model type is the second subtype, confirming to set the column type of the index column and the partition column corresponding to the model as a primary key column, the first type including the first subtype and the second subtype; If the model type is the second type, confirm setting the column type and the column order for the index column and the partition column corresponding to the model, including setting the index column at the front position of the creation statement, and setting the column type of the index column and the partition column corresponding to the model to a sequential column, wherein the creation statement includes the creation of the partition column.
2. The data model creation method according to claim 1, characterized in that: The obtaining the number of buckets of the model data includes: Acquire a total capacity of the model data, and acquire a first ratio of the total capacity to the segmentation unit; If the first ratio is an integer, the first ratio is used as the number of buckets of the model data; If the first ratio is a non-integer, the integer digit of the first ratio is increased by 1 as the number of buckets of the model data.
3. The data model creation method according to claim 1, characterized in that: The allocating the buckets of the model data among the servers according to the number of buckets and the number of servers in the cluster includes: If the number of buckets is greater than or equal to the number of servers, confirm that at least one bucket is allocated to each server, wherein the capacity of each bucket is equal to the pre-increment; If the number of buckets is less than the number of servers, confirm and obtain a second ratio of the total capacity of the model data to the number of servers, use the second ratio as the capacity of the bucket, and allocate one bucket to each server.
4. The data model creation method according to claim 1, characterized in that: Before obtaining the model type through the front end or generating partition attributes according to the partition column selected by the front end or obtaining the pre-increment of the model through the front end, the method further includes: Obtain the task instruction of the front end, and confirm to obtain the model type, column information, the partition column selected by the front end, and the pre-increment of the model according to the task instruction, wherein the column information related to the time type is set as the partition column, and the column information can be set as an index column or a non-index column.
5. The data model creation method according to claim 4, characterized in that: The step of splicing the statements required by the model according to the column type and / or column order, the partition attribute and the bucketing includes: Obtaining a task instruction of the front end, and confirming to obtain basic information of the model according to the task instruction, wherein the basic information includes a grammar database and a hierarchy, and the front end is used to verify and generate the task instruction according to the model type, column information, the partition column selected by the front end, the pre-increment and the basic information of the model; According to the basic information, the column type and / or column order, the partition attribute and the bucket, the statements required by the model are concatenated.
6. A data model creation device, characterized in that: include: An acquisition module, used to acquire a model type through a front end, set a column type and / or a column order for an index column and a partition column corresponding to the model, and generate a partition attribute according to the partition column selected by the front end; A first processing module, configured to obtain a pre-increment of the model through the front end, and use the pre-increment as a segmentation unit of the model data to obtain the number of buckets of the model data; A second processing module, configured to allocate the buckets of the model data among the servers according to the number of buckets and the number of servers in the cluster; A splicing module, used for splicing the statements required by the model according to the column type and / or column order, the partition attribute and the bucket, and transmitting the statements to the front end; The acquisition module is specifically used to confirm setting the column type for the index column and the partition column corresponding to the model if the model type is the first type, including: if the model type is the first subtype, confirming setting the column type of the index column and the partition column corresponding to the model as a dimension column; if the model type is the second subtype, confirming setting the column type of the index column and the partition column corresponding to the model as a primary key column, and the first type includes the first subtype and the second subtype; If the model type is the second type, confirm setting the column type and the column order for the index column and the partition column corresponding to the model, including setting the index column at the front position of the creation statement, and setting the column type of the index column and the partition column corresponding to the model to a sequential column, wherein the creation statement includes the creation of the partition column.
7. A data model creation device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the data model creation method according to any one of claims 1 to 5.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data model creation method as described in any one of claims 1 to 5 are implemented.