A data storage method, device, apparatus and medium

Abstract

The application discloses a data storage method, device, equipment and medium. The method comprises the following steps: obtaining table configuration data from a demand document of a to-be-built data table, wherein the table configuration data is used to represent attribute characteristics of the to-be-built data table; performing feature extraction on the table configuration data to obtain storage feature information of the table configuration data, and classifying the to-be-built data table based on the storage feature information of the table configuration data to determine a storage format corresponding to the to-be-built data table; generating a SQL script corresponding to the to-be-built data table based on the table configuration data, the storage format and a preset SQL script template; and executing the SQL script corresponding to the to-be-built data table to build the to-be-built data table and store the to-be-built data table according to the storage format. The method can improve the accuracy of the determined storage format of the to-be-built data table, improve storage efficiency and storage resource utilization rate, and reduce storage space waste.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a data storage method, apparatus, device, and medium. Background Technology

[0002] With the advent of the big data era, data storage, processing, and analysis have become crucial issues in the field of modern information technology. Traditional database management systems and data storage solutions often face problems such as low efficiency, wasted storage space, and inappropriate data format selection when dealing with complex and massive amounts of data. Especially in the processing and storage of large-scale data, how to automatically and intelligently select the most suitable storage format to improve storage efficiency and query performance has become a pressing technical challenge.

[0003] Currently, existing data storage methods typically rely on manual experience to determine the storage format of the data table to be created, and then store the data table according to the determined storage format. However, this approach suffers from problems such as low accuracy of the determined storage format, low storage efficiency, poor query performance, and potential waste of storage resources. Summary of the Invention

[0004] This application provides a data storage method, apparatus, device, and medium to improve the accuracy of the storage format of a determined data table to be built, and to improve storage efficiency and storage resource utilization, thereby reducing storage space waste.

[0005] In a first aspect, embodiments of this application provide a data storage method, including:

[0006] Obtain table configuration data from the requirements document of the data table to be built, wherein the table configuration data is used to characterize the attribute features of the data table to be built;

[0007] Feature extraction is performed on the table configuration data to obtain the storage feature information of the table configuration data, and the data table to be built is classified based on the storage feature information of the table configuration data to determine the storage format corresponding to the data table to be built;

[0008] Based on the table configuration data, the storage format, and the preset structured query language (SQL) script template, generate the SQL script corresponding to the data table to be created;

[0009] Execute the SQL script corresponding to the data table to be created to create the data table and store it according to the storage format.

[0010] In this embodiment, table configuration data characterizing the attribute features of the data table to be built is obtained from the requirements document of the data table to be built. Feature extraction is performed on the table configuration data to obtain its storage feature information, which comprehensively reflects the inherent characteristics and distribution patterns of the table configuration data. Based on the storage feature information of the table configuration data, the data table to be built is classified, and the corresponding storage format is determined, improving the accuracy of the determined storage format. Furthermore, based on the table configuration data, storage format, and preset SQL script template, this application generates an SQL script corresponding to the data table to be built. Executing the SQL script establishes the data table and stores it according to the storage format, eliminating the need for manual intervention, improving storage efficiency and storage resource utilization, reducing storage space waste, and enhancing data query and processing performance. Therefore, this application can flexibly select the appropriate storage format according to the requirements of different data tables to be built, rather than a single, fixed format, thereby improving the accuracy of the storage format corresponding to the determined data tables to be built, improving the utilization rate of storage resources, reducing storage space waste, and improving the performance of data query and processing. It is especially suitable for big data processing and complex business scenarios, meeting the high requirements of enterprises for storage efficiency and resource optimization.

[0011] In one possible design, the step of extracting features from the table configuration data to obtain storage feature information of the table configuration data, and classifying the data table to be built based on the storage feature information of the table configuration data to determine the storage format corresponding to the data table to be built, includes:

[0012] The table configuration data is input into the feature extraction layer of the trained classification model, and the stored feature information of the table configuration data is extracted through the feature extraction layer.

[0013] The storage feature information of the table configuration data is input into the output layer of the classification model. The output layer classifies the data table to be built based on the storage feature information of the table configuration data to obtain the probability value of each storage format of the data table to be built.

[0014] The storage format corresponding to the maximum probability value shall be used as the storage format.

[0015] In this embodiment, the table configuration data is input into the trained classification model to obtain the probability value of each storage format corresponding to the data table to be built. The storage format corresponding to the highest probability value is used as the storage format. This allows for flexible selection of the appropriate storage format based on the different requirements of the data table to be built, improving the accuracy of the determined storage format for the data table to be built, increasing the utilization rate of storage resources, and reducing storage space waste.

[0016] In one possible design, the storage characteristic information of the table configuration data includes some or all of the data table characteristic information, field characteristic information, and operation characteristic information;

[0017] The data table feature information is used to characterize the basic attribute features of the data table to be built;

[0018] The field feature information is used to characterize the field distribution characteristics of the data table to be built;

[0019] The operational feature information is used to characterize the operational features of the data table to be created.

[0020] In one possible design, after determining the storage format corresponding to the data table to be built, the method further includes:

[0021] The display page shows the storage format corresponding to the data table to be created.

[0022] In response to a user-triggered storage format modification command, the modified storage format is determined, and a target label is determined based on the modified storage format and preset storage formats, wherein the target label includes the actual probability value of the data table to be built corresponding to each storage format;

[0023] Input the table configuration data and the target label into the classification model to obtain the predicted probability value of each storage format of the data table to be built;

[0024] Based on the predicted probability values ​​for each storage format of the data table to be built and the target label, the target loss value is determined, and the parameters of the classification model are updated using the target loss value.

[0025] In this embodiment, the storage format corresponding to the data table to be built is displayed on the display page. In response to the user-triggered storage format modification instruction, the modified storage format is determined. Based on the modified storage format and the preset storage formats, the target label is determined. The classification model is incrementally optimized using the table configuration data and the target label, thereby improving the performance of the classification model, improving the classification accuracy of the classification model, and improving the applicability of the classification model to adapt to the needs of dynamic business scenarios.

[0026] In one possible design, the classification model is trained in the following manner:

[0027] Obtain a training dataset, wherein the training dataset includes multiple training samples, each training sample includes historical table configuration data and labels for a historical data table to be built, and the labels are used to characterize the storage format corresponding to the historical data table to be built;

[0028] Based on the training samples in the training dataset, a preset classification model is trained iteratively. When the preset iteration termination condition is met, the classification model is obtained.

[0029] The following operations are performed during one iteration of training:

[0030] Input any training sample from the training dataset into a preset classification model for processing to obtain the predicted probability value of each storage format for any training sample.

[0031] Based on the predicted probability value of each storage format corresponding to any training sample and the label of any training sample, determine the loss value corresponding to any training sample.

[0032] The parameters of the preset classification model are updated using the loss value corresponding to any one of the training samples.

[0033] In this embodiment of the application, a pre-defined classification model is trained using a training dataset that includes multiple training samples to obtain a classification model, thereby improving the classification accuracy of the classification model.

[0034] Secondly, embodiments of this application provide a data storage device, including:

[0035] The acquisition module is used to acquire table configuration data from the requirements document of the data table to be built, wherein the table configuration data is used to characterize the attribute features of the data table to be built;

[0036] The determination module is used to extract features from the table configuration data to obtain the storage feature information of the table configuration data, and classify the data table to be built based on the storage feature information of the table configuration data to determine the storage format corresponding to the data table to be built.

[0037] The generation module is used to generate the SQL script corresponding to the data table to be built based on the table configuration data, the storage format and the preset structured query language SQL script template.

[0038] The execution module is used to execute the SQL script corresponding to the data table to be created, so as to create the data table to be created and store it according to the storage format.

[0039] In one possible design, the determining module is used for:

[0040] The table configuration data is input into the feature extraction layer of the trained classification model, and the stored feature information of the table configuration data is extracted through the feature extraction layer.

[0041] The storage feature information of the table configuration data is input into the output layer of the classification model. The output layer classifies the data table to be built based on the storage feature information of the table configuration data to obtain the probability value of each storage format of the data table to be built.

[0042] The storage format corresponding to the maximum probability value shall be used as the storage format.

[0043] In one possible design, the storage characteristic information of the table configuration data includes some or all of the data table characteristic information, field characteristic information, and operation characteristic information;

[0044] The data table feature information is used to characterize the basic attribute features of the data table to be built;

[0045] The field feature information is used to characterize the field distribution characteristics of the data table to be built;

[0046] The operational feature information is used to characterize the operational features of the data table to be created.

[0047] In one possible design, after determining the storage format corresponding to the data table to be built, the determining module is further configured to:

[0048] The display page shows the storage format corresponding to the data table to be created.

[0049] In response to a user-triggered storage format modification command, the modified storage format is determined, and a target label is determined based on the modified storage format and preset storage formats, wherein the target label includes the actual probability value of the data table to be built corresponding to each storage format;

[0050] Input the table configuration data and the target label into the classification model to obtain the predicted probability value of each storage format of the data table to be built;

[0051] Based on the predicted probability values ​​for each storage format of the data table to be built and the target label, the target loss value is determined, and the parameters of the classification model are updated using the target loss value.

[0052] In one possible design, the determining module is used to train the classification model in the following manner:

[0053] Obtain a training dataset, wherein the training dataset includes multiple training samples, each training sample includes historical table configuration data and labels for a historical data table to be built, and the labels are used to characterize the storage format corresponding to the historical data table to be built;

[0054] Based on the training samples in the training dataset, a preset classification model is trained iteratively. When the preset iteration termination condition is met, the classification model is obtained.

[0055] The following operations are performed during one iteration of training:

[0056] Input any training sample from the training dataset into a preset classification model for processing to obtain the predicted probability value of each storage format for any training sample.

[0057] Based on the predicted probability value of each storage format corresponding to any training sample and the label of any training sample, determine the loss value corresponding to any training sample.

[0058] The parameters of the preset classification model are updated using the loss value corresponding to any one of the training samples.

[0059] Thirdly, this application provides an electronic device, comprising:

[0060] Memory, used to store program instructions;

[0061] A processor is configured to invoke program instructions stored in the memory and execute the steps of the method described in any one of the first aspects according to the obtained program instructions.

[0062] Fourthly, this application provides a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a computer, cause the computer to perform the method described in any one of the first aspects.

[0063] Fifthly, this application provides a computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform the method described in any one of the first aspects.

[0064] The technical effects of aspects two through five and any one of their designs can be found in the technical effects of the corresponding designs in aspect one, and will not be repeated here. Attached Figure Description

[0065] Figure 1 This is a schematic diagram illustrating an application scenario provided in the embodiments of this application;

[0066] Figure 2 A flowchart illustrating a data storage method provided in an embodiment of this application;

[0067] Figure 3A flowchart illustrating a method for determining the storage format corresponding to a data table to be built, provided in an embodiment of this application;

[0068] Figure 4 A flowchart illustrating a method for incrementally optimizing a classification model, as provided in this application embodiment;

[0069] Figure 5 A flowchart illustrating a classification training method provided in this application embodiment;

[0070] Figure 6 A flowchart illustrating a method for performing one iteration of training on a classification model, as provided in an embodiment of this application;

[0071] Figure 7 A detailed flowchart illustrating a data storage method provided in an embodiment of this application;

[0072] Figure 8 This is a schematic diagram of the structure of a data storage device provided in an embodiment of this application;

[0073] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0075] The terms "first" and "second" in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The term "multiple" in this application can mean at least two, for example, two, three, or more, and the embodiments of this application do not impose limitations.

[0076] The data collection, dissemination, and use in this application all comply with relevant national laws and regulations.

[0077] Before introducing a data storage method provided by the embodiments of this application, the technical background of the embodiments of this application will be described in detail below for ease of understanding.

[0078] With the advent of the big data era, data storage, processing, and analysis have become crucial issues in the field of modern information technology. Traditional database management systems and data storage solutions often face problems such as low efficiency, wasted storage space, and inappropriate data format selection when dealing with complex and massive amounts of data. Especially in the processing and storage of large-scale data, how to automatically and intelligently select the most suitable storage format to improve storage efficiency and query performance has become a pressing technical challenge.

[0079] Currently, existing data storage methods typically rely on manual experience to determine the storage format of the data table to be created, and then store the data table according to the determined storage format. However, this approach suffers from problems such as low accuracy of the determined storage format, low storage efficiency, poor query performance, and potential waste of storage resources.

[0080] To address the aforementioned issues, this application proposes a data storage method, apparatus, device, and medium to improve the accuracy of the storage format of a determined data table to be built, enhance storage efficiency and storage resource utilization, and reduce storage space waste.

[0081] First refer to Figure 1This is a schematic diagram of an application scenario of an embodiment of this application, including a collector 11 and a server 12. The collector 11 and the server 12 communicate with each other through a network, which can be a local area network, a wide area network, etc. The collector 11 can be a portable device (e.g., a mobile phone, tablet, laptop, etc.) or a personal computer (PC). The server 12 can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms. Collector 11 is used to collect table configuration data from the requirements document of the data table to be built and send it to server 12. Server 12 is used to determine the storage format corresponding to the data table to be built based on the table configuration data, generate the SQL script corresponding to the data table to be built based on the table configuration data, storage format and preset Structured Query Language (SQL) script template, and execute the SQL script corresponding to the data table to be built in order to build the data table to be built and store it according to the storage format.

[0082] In this embodiment of the application, as an optional implementation, the server 12 acquires the table configuration data collected by the collector 11 from the requirement document of the data table to be built, wherein the table configuration data is used to characterize the attribute features of the data table to be built; the table configuration data is subjected to feature extraction to obtain the storage feature information of the table configuration data, and the data table to be built is classified based on the storage feature information of the table configuration data to determine the storage format corresponding to the data table to be built; based on the table configuration data, the storage format and the preset SQL script template, the SQL script corresponding to the data table to be built is generated; the SQL script corresponding to the data table to be built is executed to build the data table to be built and store it according to the storage format.

[0083] The following is for reference. Figure 2 The flowchart illustrating a data storage method is used to explain the technical solution provided in the embodiments of this application.

[0084] Step 201: Obtain table configuration data from the requirements document of the data table to be created.

[0085] The requirements document for the data tables to be built includes information such as table size, each field in the table, the field type for each field, query mode, query frequency, and update frequency. Query modes include query engines such as MapReduce (MR), Tez (a data processing framework), and Spark (a computing engine). Update frequency includes some or all of the deletion, modification, and addition frequencies. Table configuration data is used to characterize the attributes of the data tables to be built. This configuration data includes information such as table size, number of fields, and field types.

[0086] For example, the requirements document for the data table to be created includes the following: table name is user_data, table size is a megabytes (MByte, MB), table fields include user_id, user_name, email, and signup_date, field type for user_id is INT (integer), field type for user_name is STRING (string), field type for email is STRING, field type for signup_date is STRING, query mode is Spark, query frequency is b, deletion frequency is c, modification frequency is d, and addition frequency is e.

[0087] The data tables to be built are those that need to be connected to the data warehouse. The data warehouse can be Hive (a data warehouse system), which is built on Hadoop (an intelligent development platform based on a code generator). Hive provides a SQL-like query language (called HiveQL) for managing, querying, and analyzing big data stored in Hadoop HDFS (Hadoop Distributed File System). Hive allows data analysts to manipulate distributed data using SQL query language without needing to master complex MapReduce programming. HDFS is a distributed file system designed specifically for processing big data. It is primarily used to store massive amounts of data and provides high fault tolerance, high throughput, and the ability to adapt to large-scale data processing needs.

[0088] Step 202: Extract features from the table configuration data to obtain the storage feature information of the table configuration data, and classify the data table to be built based on the storage feature information of the table configuration data to determine the storage format corresponding to the data table to be built.

[0089] In this embodiment, table configuration data is input into a trained classification model to extract features from the table configuration data, obtain storage feature information of the table configuration data, and classify the data table to be built based on the storage feature information of the table configuration data, thereby determining the storage format corresponding to the data table to be built.

[0090] The classification model can be LightGBM (Light Gradient Boosting Machine) or other multi-class classification models. LightGBM is an efficient framework based on the gradient boosting algorithm, mainly used for processing large-scale datasets and performing machine learning tasks, especially excelling in classification, regression, and ranking tasks.

[0091] Step 203: Based on the table configuration data, storage format, and preset SQL script template, generate the SQL script corresponding to the data table to be created.

[0092] The preset SQL script templates can be customized according to actual needs. An SQL script is an executable file written in a database query and programming language.

[0093] For example, when the storage format of the data table to be created is Parquet (column-oriented storage format), based on the table configuration data, storage format, and preset SQL script template, the generated SQL script for the data table to be created is as follows:

[0094] CREATE TABLE IF NOT EXISTS user_data( / / Create a table named user_data;

[0095] user_id INT, / / The user_id field in this table is of type INT;

[0096] user_name STRING, / / The field type of the user_name field in this table is STRING;

[0097] email STRING, / / The email field in this table is of type STRING;

[0098] signup_date STRING / / The signup_date field in this table is of type STRING; )

[0100] STORED AS PARQUET / / This data table is stored in Parquet storage format;

[0101] The storage location of this data table is 'hdfs: / / user / hive / warehouse / user_data / '.

[0102] TBLPROPERTIES('parquet.compression' = 'SNAPPY'); / / Compress and store the data table according to the Parquet storage format using SNAPPY (a development package for compression and decompression);

[0103] In this embodiment, the table name, table fields, and field types corresponding to the table fields are directly obtained from the table configuration data. The prefix "hdfs: / / user / hive / warehouse / " of the table storage location is pre-configured by the user, and the storage location " / user_data / " is determined based on the table name. The storage format of the table is predicted by the classification model.

[0104] In this embodiment, a trained classification model is used to process the table configuration data of the input data table to be built, and the storage format corresponding to the data table to be built is output. Based on the table configuration data, storage format and preset SQL script template, the SQL script corresponding to the data table to be built is generated, and automatic configuration of compression format and storage path is also supported.

[0105] Step 204: Execute the SQL script corresponding to the data table to be created, so as to create the data table and store it according to the storage format.

[0106] In this embodiment of the application, after executing the SQL script corresponding to the data table to be created, at least one piece of data is collected using a collector, and each piece of collected data is loaded into the established data table to be created.

[0107] In this embodiment of the application, after executing the SQL script corresponding to the data table to be created, in response to the user-triggered instruction to add new data to the existing data table to be created, the data is loaded into the existing data table to be created.

[0108] In this embodiment, table configuration data characterizing the attribute features of the data table to be built is obtained from the requirements document of the data table to be built. Feature extraction is performed on the table configuration data to obtain its storage feature information, which comprehensively reflects the inherent characteristics and distribution patterns of the table configuration data. Based on the storage feature information of the table configuration data, the data table to be built is classified, and the corresponding storage format is determined, improving the accuracy of the determined storage format. Furthermore, based on the table configuration data, storage format, and preset SQL script template, this application generates an SQL script corresponding to the data table to be built. Executing the SQL script establishes the data table and stores it according to the storage format, eliminating the need for manual intervention, improving storage efficiency and storage resource utilization, reducing storage space waste, and enhancing data query and processing performance. Therefore, this application can flexibly select the appropriate storage format according to the requirements of different data tables to be built, rather than a single, fixed format, thereby improving the accuracy of the storage format corresponding to the determined data tables to be built, improving the utilization rate of storage resources, reducing storage space waste, and improving the performance of data query and processing. It is especially suitable for big data processing and complex business scenarios, meeting the high requirements of enterprises for storage efficiency and resource optimization.

[0109] In this embodiment, the table configuration data is obtained from the requirements document of the data table to be built. This table configuration data needs to be normalized or tagged, and processed to handle missing field information and fill in default values. The table configuration data includes field values ​​corresponding to different fields.

[0110] For example, the table configuration data M = {f1, f2, ..., f n}, where f i Let be the value of the i-th field, and n be the number of fields in the table configuration data. After standardization, M is the processed table configuration data M. ′ ={f1 ′ f2 ′ ,…,f n ′}, where f i ′ This is the processed field value of the i-th field.

[0111] In this embodiment of the application, the field value f of the i-th field... i There are two scenarios for standardization:

[0112] In the first case, the field value f of the i-th field... i If the value is numerical, then it is based on the maximum value of the i-th field, the minimum value of the i-th field, and the field value f of the i-th field. i Determine the processed field value f of the i-th field.i ′ The maximum and minimum values ​​of the i-th field can be set according to the actual situation, or they can be determined based on the field values ​​of the i-th field in multiple historical table configuration data, that is, the maximum and minimum values ​​of the i-th field in multiple historical table configuration data.

[0113] The processed field value f of the i-th field can be determined using the following formula in the embodiments of this application. i ′ :

[0114]

[0115] Among them, f i For the value of the i-th field, min(f) i ) represents the minimum value of the i-th field, max(f i ) represents the maximum value of the i-th field.

[0116] In the second case, the field value f of the i-th field... i If the value is not numeric, then the field value f of the i-th field is encoded using the defined encoding method. i Perform encoding processing to obtain the processed field value f of the i-th field. i ′ The encoding method can be set according to the actual situation.

[0117] The processed field value f of the i-th field can be determined using the following formula in the embodiments of this application. i ′ :

[0118] f i ′ =Encode(f i );

[0119] Where Encode() is the encoding method, f i Let i be the field value of the i-th field.

[0120] In this embodiment of the application, the storage format corresponding to the data table to be built is determined based on the table configuration data. Figure 3 A flowchart illustrating a method for determining the storage format corresponding to a data table to be built, as provided in this application embodiment, is shown below. Figure 3 As shown, step 202 above includes at least the following steps 301-303:

[0121] Step 301: Input the table configuration data into the feature extraction layer of the trained classification model, and extract the stored feature information of the table configuration data through the feature extraction layer.

[0122] In this embodiment of the application, the storage characteristic information of the table configuration data includes part or all of the data table characteristic information, field characteristic information and operation characteristic information;

[0123] Data table feature information is used to characterize the basic attribute features of the data table to be created;

[0124] Field feature information is used to characterize the field distribution characteristics of the data table to be created;

[0125] Operational characteristic information is used to characterize the operational characteristics of the data table to be created.

[0126] The data table characteristics include some or all of the data table size and the number of data table fields; the field characteristics include some or all of the data table field type distribution and nesting; and the operation characteristics include some or all of the query pattern, query frequency, and update frequency. Update frequency includes some or all of the deletion frequency, modification frequency, and addition frequency.

[0127] Specifically, based on the processed table configuration data M ′ It generates some or all of the data table feature information, field feature information, and operation feature information, and performs fusion processing to obtain storage feature information.

[0128] For example, based on the processed table configuration data M ′ This process generates table characteristic information F, field characteristic information D, and operation characteristic information Q. Based on these three information, the storage characteristic information X = F∪D∪Q is obtained, where ∪ represents the union operation. Table characteristic information F includes the table size (file_size) and the number of fields (column_count), therefore, F = {file_size, column_count}. Field characteristic information D includes the field type distribution (type_distribution) and nested structure, therefore, D = {type_distribution, nested_structure}. Operation characteristic information Q includes the query mode (query_mode), query frequency (query_frequency), and update frequency (update_frequency), therefore, Q = {query_mode, query_frequency, update_frequency}.

[0129] Step 302: Input the storage feature information of the table configuration data into the output layer of the classification model. The output layer classifies the data table to be built based on the storage feature information of the table configuration data to obtain the probability value of each storage format of the data table to be built.

[0130] In this embodiment, the preset storage format set can be set according to actual conditions, and the storage format set includes multiple storage formats. The storage format set may include Text (text file), Parquet (column-based storage format), ORC (Optimized Row Columnar), Avro (row-based storage format), RCfile (RecordColumnar File), SequenceFile (binary serialized file), and other storage formats. For example, the storage format set Y includes 6 storage formats, i.e., storage format set Y = {Text, Parquet, ORC, Avro, RCfile, SequenceFile}.

[0131] Step 303: Use the storage format corresponding to the maximum probability value as the storage format.

[0132] In this embodiment of the application, after obtaining the probability value of each storage format corresponding to the data table to be built, the probability values ​​of each storage format can be sorted in descending order, and the storage format corresponding to the highest probability value can be used as the storage format.

[0133] In this embodiment of the application, taking LightGBM as an example, the trained LightGBM multi-class model is loaded, and based on the table configuration data M of the input data table to be built, the probability value of each storage format corresponding to the data table to be built is output, that is, the probability distribution of the storage format of the data table to be built, and the storage format corresponding to the maximum probability value is taken as the storage format corresponding to the data table to be built.

[0134] This application can use the following formula to obtain the probability distribution P(Y|M) of the storage format of the data table to be built:

[0135] P(Y|M) = LightGBM(M);

[0136] Where Y is the set of storage formats, LightGBM() is LightGBM, and M is the table configuration data of the data table to be created.

[0137] In this embodiment of the application, after obtaining the probability distribution of the storage format of the data table to be built, the storage format Y corresponding to the data table to be built can be determined based on the probability distribution of the storage format of the data table to be built using the following formula. recommend :

[0138]

[0139] Where P(Y|M) is the probability distribution of the storage format of the data table to be built, argmax is the input parameter or index position that makes the objective function obtain the maximum value, Y is the set of storage formats, and M is the table configuration data of the data table to be built.

[0140] In this embodiment, after determining the storage format corresponding to the data table to be built, the storage format corresponding to the data table to be built is displayed on the display page so that the user can manually review the storage format of the data table to be built and judge whether the storage format of the data table to be built is reasonable. If the storage format of the data table to be built is reasonable, then the above steps 203-204 are continued. If the storage format of the data table to be built is unreasonable, then the classification model is incrementally optimized based on the modified storage format determined by the user. Figure 4 A flowchart illustrating a method for incrementally optimizing a classification model, as provided in this application embodiment, is shown below. Figure 4 As shown, it includes at least the following steps 401-404:

[0141] Step 401: Display the storage format corresponding to the data table to be created on the display page.

[0142] In this embodiment of the application, in addition to displaying the storage format corresponding to the data table to be created on the display page, the requirement document or table configuration data of the data table to be created can also be displayed, so that the user can judge whether the storage format corresponding to the data table to be created is reasonable based on the requirement document of the data table to be created.

[0143] Step 402: In response to the user-triggered storage format modification command, determine the modified storage format, and determine the target tag based on the modified storage format and the preset storage formats.

[0144] The target label includes the actual probability value for each storage format corresponding to the data table to be built. For example, if the set of storage formats is {storage format 1, storage format 2, storage format 3, storage format 4}, and based on the table configuration data of the data table to be built, the storage format corresponding to the data table to be built is determined to be storage format 2 using a classification model, the display page shows that the storage format corresponding to the data table to be built is storage format 2. In response to a user-triggered storage format modification command, the modified storage format is determined to be storage format 3. Based on the modified storage format and the preset storage formats, the determined target label is {0,0,1,0}.

[0145] Step 403: Input the table configuration data and target labels into the classification model to obtain the predicted probability value for each storage format of the data table to be built.

[0146] Step 404: Based on the predicted probability value and target label for each storage format of the data table to be built, determine the target loss value and use the target loss value to update the parameters of the classification model.

[0147] Optionally, the target loss value is determined based on the predicted probability value and target label for each storage format of the data table to be built, including: determining the target loss value using a set loss function based on the predicted probability value and target label for each storage format of the data table to be built.

[0148] The loss function can be set according to the actual situation. For example, the loss function can be the cross-entropy loss function, the Kullback-Leibler divergence, or other loss functions.

[0149] In this embodiment of the application, taking LightGBM as an example, the storage format Y corresponding to the data table to be built is determined using LightGBM based on the table configuration data M of the data table to be built. recommend In response to a user-triggered storage format modification command, determine the modified storage format Y. manual Based on the table configuration data M and the modified storage format Y manual Incremental training is performed on LightGBM. In this embodiment, LightGBM can be incrementally optimized using the following formula:

[0150] LightGBM new =Train(LightGBM,D feedback );

[0151] Among them, LightGBM new This refers to the incrementally optimized LightGBM, i.e., the LightGBM after updating its parameters. `Train()` is the model training function, and `LightGBM` is the LightGBM before incremental optimization. D feedback For LightGBM input data, D feedback This includes the table configuration data M for the data table to be created and the modified storage format Y. manual D feedback ={(M,Y manual )}.

[0152] In this embodiment, after determining the storage format of the data table to be built, the storage format is displayed on the display page so that the user can manually review it and determine whether it is reasonable. If the storage format is reasonable, an SQL script is generated and executed to create the data table and store it according to the storage format. If the storage format is unreasonable, the classification model is incrementally optimized based on the user-determined modified storage format, thereby improving the performance, accuracy, and applicability of the classification model to meet dynamic business scenario requirements.

[0153] In this embodiment of the application, a classification model is trained based on the acquired training dataset. Figure 5 A flowchart of a classification training method provided in this application embodiment is shown below. Figure 5 As shown, it includes at least the following steps 501-502:

[0154] Step 501: Obtain the training dataset.

[0155] The training dataset includes multiple training samples. Each training sample includes historical table configuration data and labels for historical tables to be built. The labels are used to characterize the storage format corresponding to the historical tables to be built.

[0156] In this embodiment, the label of the historical data table to be built is determined based on the storage format corresponding to the historical data table to be built and each storage format in the set of defined storage formats. Specifically, the label of the historical data table to be built is determined by the following method: the storage format corresponding to the historical data table to be built is labeled as a first label value, and each of the other storage formats in the set of defined storage formats is labeled as a second label value. Based on the label value corresponding to each storage format, the label of the historical data table to be built is obtained. The first label value and the second label value can be set according to the actual situation. For example, the first label value can be 1 and the second label value can be 0.

[0157] For example, if the storage format set is {storage format A, storage format B, storage format C, storage format D}, and the storage format corresponding to the historical data table to be created is storage format B, then the label of the historical data table to be created is {0,1,0,0}.

[0158] Step 502: Based on the training samples in the training dataset, perform iterative training on the preset classification model. When the preset iteration termination condition is met, the classification model is obtained.

[0159] The aforementioned preset iteration termination condition can be set according to the actual situation. For example, the preset iteration termination condition can be that the current iteration number reaches a set iteration number, and the set iteration number can be set according to the actual situation. The preset iteration termination condition can also be that the preset classification model converges.

[0160] The following section will explain in detail the method for performing a single iteration of training on a classification model, such as... Figure 6 As shown:

[0161] Step 601: Input any training sample from the training dataset into the preset classification model for processing to obtain the predicted probability value of any training sample for each storage format.

[0162] In this embodiment of the application, the base model used to train the classification model, i.e. the preset classification model, can be various models with prediction functions, such as convolutional neural network models, neural network models, etc.

[0163] Step 602: Based on the predicted probability value of any training sample for each storage format and the label of any training sample, determine the loss value corresponding to any training sample.

[0164] Step 603: Update the parameters of the preset classification model using the loss value corresponding to any training sample.

[0165] Optionally, based on the predicted probability value of any training sample for each storage format and the label of any training sample, the loss value corresponding to any training sample is determined. This includes: determining the loss value corresponding to any training sample using a defined loss function based on the predicted probability value of any training sample for each storage format and the label of any training sample. The defined loss function can be set according to the actual situation; for example, the defined loss function can be a cross-entropy loss function, KL divergence, or other loss functions.

[0166] The specific steps of the data storage method provided above will be explained in detail below, such as... Figure 7 As shown:

[0167] Step 701: Obtain table configuration data from the requirements document of the data table to be created.

[0168] Step 702: Input the table configuration data into the feature extraction layer of the trained classification model, and extract the stored feature information of the table configuration data through the feature extraction layer.

[0169] Step 703: Input the storage feature information of the table configuration data into the output layer of the classification model. The output layer classifies the data table to be built based on the storage feature information of the table configuration data to obtain the probability value of each storage format of the data table to be built.

[0170] Step 704: Use the storage format corresponding to the maximum probability value as the storage format corresponding to the data table to be built.

[0171] Step 705: Display the storage format corresponding to the data table to be created on the display page.

[0172] Step 706: In response to the user-triggered storage format modification command, determine the modified storage format, and determine the target tag based on the modified storage format and the preset storage formats.

[0173] In this embodiment of the application, after determining the modified storage format, the method further includes: generating an SQL script corresponding to the data table to be created based on the table configuration data, the modified storage format, and the preset SQL script template; executing the SQL script corresponding to the data table to be created to establish the data table to be created and store it according to the modified storage format.

[0174] Step 707: Input the table configuration data and target labels into the classification model to obtain the predicted probability value for each storage format of the data table to be built.

[0175] Step 708: Based on the predicted probability value and target label for each storage format of the data table to be built, determine the target loss value and use the target loss value to update the parameters of the classification model.

[0176] Step 709: In response to the user-triggered storage format determination command, generate the SQL script corresponding to the data table to be created based on the table configuration data, storage format, and preset SQL script template.

[0177] Step 710: Execute the SQL script corresponding to the data table to be created, so as to create the data table and store it according to the storage format.

[0178] Based on the same technical concept, this application provides an exemplary data storage device, such as... Figure 8 As shown, the device includes:

[0179] The acquisition module 801 is used to acquire table configuration data from the requirements document of the data table to be built, wherein the table configuration data is used to characterize the attribute features of the data table to be built;

[0180] The determination module 802 is used to extract features from the table configuration data to obtain storage feature information of the table configuration data, and classify the data table to be built based on the storage feature information of the table configuration data to determine the storage format corresponding to the data table to be built.

[0181] The generation module 803 is used to generate the SQL script corresponding to the data table to be built based on the table configuration data, the storage format and the preset structured query language SQL script template.

[0182] The execution module 804 is used to execute the SQL script corresponding to the data table to be created, so as to create the data table to be created and store it according to the storage format.

[0183] Optionally, the determining module 802 is used to:

[0184] The table configuration data is input into the feature extraction layer of the trained classification model, and the stored feature information of the table configuration data is extracted through the feature extraction layer.

[0185] The storage feature information of the table configuration data is input into the output layer of the classification model. The output layer classifies the data table to be built based on the storage feature information of the table configuration data to obtain the probability value of each storage format of the data table to be built.

[0186] The storage format corresponding to the maximum probability value shall be used as the storage format.

[0187] Optionally, the storage characteristic information of the table configuration data includes part or all of the data table characteristic information, field characteristic information, and operation characteristic information;

[0188] The data table feature information is used to characterize the basic attribute features of the data table to be built;

[0189] The field feature information is used to characterize the field distribution characteristics of the data table to be built;

[0190] The operational feature information is used to characterize the operational features of the data table to be created.

[0191] Optionally, after determining the storage format corresponding to the data table to be built, the determining module 802 is further configured to:

[0192] The display page shows the storage format corresponding to the data table to be created.

[0193] In response to a user-triggered storage format modification command, the modified storage format is determined, and a target label is determined based on the modified storage format and preset storage formats, wherein the target label includes the actual probability value of the data table to be built corresponding to each storage format;

[0194] Input the table configuration data and the target label into the classification model to obtain the predicted probability value of each storage format of the data table to be built;

[0195] Based on the predicted probability values ​​for each storage format of the data table to be built and the target label, the target loss value is determined, and the parameters of the classification model are updated using the target loss value.

[0196] Optionally, the determining module 802 is used to train the classification model in the following manner:

[0197] Obtain a training dataset, wherein the training dataset includes multiple training samples, each training sample includes historical table configuration data and labels for a historical data table to be built, and the labels are used to characterize the storage format corresponding to the historical data table to be built;

[0198] Based on the training samples in the training dataset, a preset classification model is trained iteratively. When the preset iteration termination condition is met, the classification model is obtained.

[0199] The following operations are performed during one iteration of training:

[0200] Input any training sample from the training dataset into a preset classification model for processing to obtain the predicted probability value of each storage format for any training sample.

[0201] Based on the predicted probability value of each storage format corresponding to any training sample and the label of any training sample, determine the loss value corresponding to any training sample.

[0202] The parameters of the preset classification model are updated using the loss value corresponding to any one of the training samples.

[0203] Based on the same inventive concept, this application provides an electronic device that can realize the function of the data storage device described above. Please refer to... Figure 9 The device includes a processor 901 and a memory 902, wherein the memory 902 is used to store program instructions;

[0204] The processor 901 calls the program instructions stored in the memory and executes the program instructions to achieve the following steps:

[0205] Obtain table configuration data from the requirements document of the data table to be built, wherein the table configuration data is used to characterize the attribute features of the data table to be built;

[0206] Feature extraction is performed on the table configuration data to obtain the storage feature information of the table configuration data, and the data table to be built is classified based on the storage feature information of the table configuration data to determine the storage format corresponding to the data table to be built;

[0207] Based on the table configuration data, the storage format, and the preset structured query language (SQL) script template, generate the SQL script corresponding to the data table to be created;

[0208] Execute the SQL script corresponding to the data table to be created to create the data table and store it according to the storage format.

[0209] Optionally, the step of extracting features from the table configuration data to obtain storage feature information of the table configuration data, and classifying the data table to be built based on the storage feature information of the table configuration data to determine the storage format corresponding to the data table to be built, includes:

[0210] The table configuration data is input into the feature extraction layer of the trained classification model, and the stored feature information of the table configuration data is extracted through the feature extraction layer.

[0211] The storage feature information of the table configuration data is input into the output layer of the classification model. The output layer classifies the data table to be built based on the storage feature information of the table configuration data to obtain the probability value of each storage format of the data table to be built.

[0212] The storage format corresponding to the maximum probability value shall be used as the storage format.

[0213] Optionally, the storage characteristic information of the table configuration data includes part or all of the data table characteristic information, field characteristic information, and operation characteristic information;

[0214] The data table feature information is used to characterize the basic attribute features of the data table to be built;

[0215] The field feature information is used to characterize the field distribution characteristics of the data table to be built;

[0216] The operational feature information is used to characterize the operational features of the data table to be created.

[0217] Optionally, after determining the storage format corresponding to the data table to be built, the processor 901 is further configured to execute:

[0218] The display page shows the storage format corresponding to the data table to be created.

[0219] In response to a user-triggered storage format modification command, the modified storage format is determined, and a target label is determined based on the modified storage format and preset storage formats, wherein the target label includes the actual probability value of the data table to be built corresponding to each storage format;

[0220] Input the table configuration data and the target label into the classification model to obtain the predicted probability value of each storage format of the data table to be built;

[0221] Based on the predicted probability values ​​for each storage format of the data table to be built and the target label, a target loss value is determined, and the parameters of the classification model are updated using the target loss value.

[0222] Optionally, the classification model is trained in the following manner:

[0223] Obtain a training dataset, wherein the training dataset includes multiple training samples, each training sample includes historical table configuration data and labels for a historical data table to be built, and the labels are used to characterize the storage format corresponding to the historical data table to be built;

[0224] Based on the training samples in the training dataset, a preset classification model is trained iteratively. When the preset iteration termination condition is met, the classification model is obtained.

[0225] The following operations are performed during one iteration of training:

[0226] Input any training sample from the training dataset into a preset classification model for processing to obtain the predicted probability value of each storage format for any training sample.

[0227] Based on the predicted probability value of each storage format corresponding to any training sample and the label of any training sample, determine the loss value corresponding to any training sample.

[0228] The parameters of the preset classification model are updated using the loss value corresponding to any one of the training samples.

[0229] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium. The computer program product includes computer program code, which, when executed on a computer, causes the computer to perform any of the data storage methods described above. Since the principle by which the computer-readable storage medium solves the problem is similar to that of the data storage method, the implementation of the computer-readable storage medium can be found in the implementation of the method; repeated details will not be elaborated further.

[0230] Based on the same inventive concept, this application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to execute any of the data storage methods discussed above. Since the principle by which the above computer program product solves the problem is similar to that of the data storage method, the implementation of the above computer program product can be referred to the implementation of the method, and repeated details will not be described again.

[0231] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0232] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0233] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0234] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of user-operated steps to be executed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0235] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data storage method, characterized in that, include: Obtain table configuration data from the requirements document of the data table to be built, wherein the table configuration data is used to characterize the attribute features of the data table to be built; Feature extraction is performed on the table configuration data to obtain the storage feature information of the table configuration data, and the data table to be built is classified based on the storage feature information of the table configuration data to determine the storage format corresponding to the data table to be built; Based on the table configuration data, the storage format, and the preset structured query language (SQL) script template, generate the SQL script corresponding to the data table to be created; Execute the SQL script corresponding to the data table to be created to create the data table and store it according to the storage format; The step of extracting features from the table configuration data to obtain storage feature information of the table configuration data, and classifying the data table to be built based on the storage feature information of the table configuration data to determine the storage format corresponding to the data table to be built, includes: The table configuration data is input into the feature extraction layer of the trained classification model, and the stored feature information of the table configuration data is extracted through the feature extraction layer. The storage feature information of the table configuration data is input into the output layer of the classification model. The output layer classifies the data table to be built based on the storage feature information of the table configuration data to obtain the probability value of each storage format of the data table to be built. The storage format corresponding to the maximum probability value shall be used as the storage format.

2. The method as described in claim 1, characterized in that, The storage characteristic information of the table configuration data includes some or all of the data table characteristic information, field characteristic information, and operation characteristic information; The data table feature information is used to characterize the basic attribute features of the data table to be built; The field feature information is used to characterize the field distribution characteristics of the data table to be built; The operational feature information is used to characterize the operational features of the data table to be created.

3. The method as described in claim 1, characterized in that, After determining the storage format corresponding to the data table to be created, the method further includes: The display page shows the storage format corresponding to the data table to be created. In response to a user-triggered storage format modification command, the modified storage format is determined, and a target label is determined based on the modified storage format and preset storage formats, wherein the target label includes the actual probability value of the data table to be built corresponding to each storage format; Input the table configuration data and the target label into the classification model to obtain the predicted probability value of each storage format of the data table to be built; Based on the predicted probability values ​​for each storage format of the data table to be built and the target label, the target loss value is determined, and the parameters of the classification model are updated using the target loss value.

4. The method according to any one of claims 1-3, characterized in that, The classification model is trained in the following way: Obtain a training dataset, wherein the training dataset includes multiple training samples, each training sample includes historical table configuration data and labels for a historical data table to be built, and the labels are used to characterize the storage format corresponding to the historical data table to be built; Based on the training samples in the training dataset, a preset classification model is trained iteratively. When the preset iteration termination condition is met, the classification model is obtained. The following operations are performed during one iteration of training: Input any training sample from the training dataset into a preset classification model for processing to obtain the predicted probability value of each storage format for any training sample. Based on the predicted probability value of each storage format corresponding to any training sample and the label of any training sample, determine the loss value corresponding to any training sample. The parameters of the preset classification model are updated using the loss value corresponding to any one of the training samples.

5. A data storage device, characterized in that, include: The acquisition module is used to acquire table configuration data from the requirements document of the data table to be built, wherein the table configuration data is used to characterize the attribute features of the data table to be built; The determination module is used to extract features from the table configuration data to obtain the storage feature information of the table configuration data, and classify the data table to be built based on the storage feature information of the table configuration data to determine the storage format corresponding to the data table to be built. The generation module is used to generate the SQL script corresponding to the data table to be built based on the table configuration data, the storage format and the preset structured query language SQL script template. The execution module is used to execute the SQL script corresponding to the data table to be created, so as to create the data table to be created and store it according to the storage format; The determining module is used for: The table configuration data is input into the feature extraction layer of the trained classification model, and the stored feature information of the table configuration data is extracted through the feature extraction layer. The storage feature information of the table configuration data is input into the output layer of the classification model. The output layer classifies the data table to be built based on the storage feature information of the table configuration data to obtain the probability value of each storage format of the data table to be built. The storage format corresponding to the maximum probability value shall be used as the storage format.

6. An electronic device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the steps of the method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-4.

8. A computer program product, characterized in that, The computer program product includes: computer program code, which, when run on a computer, causes the computer to perform the method described in any one of claims 1-4.

Images