Data storage method and apparatus, electronic device, computer-readable storage medium, and computer program product

Abstract

The present application provides a data storage method and apparatus, a device, a computer-readable storage medium, and a computer program product. The method comprises: obtaining a plurality of query requests for an application, and parsing each query request to obtain at least one piece of column information included in each query request; on the basis of the at least one piece of column information included in each query request, determining a candidate column combination of tables associated with each query request, such that when table data of the tables associated with the query request is stored according to the candidate column combination, the query overhead for executing the query request is minimized; on the basis of a plurality of candidate column combinations pertaining to a same table, determining a target column combination corresponding to the table, such that when table data of each table is stored according to the respective target column combination, the total query overhead for executing the plurality of query requests is minimized; and storing the table data of each table according to the target column combination.

Description

Data storage method, device, electronic device, computer-readable storage medium, and computer program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202311111908.5 and application date August 31, 2023, and claims the priority of the above Chinese patent application. The entire content of the above Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present application relates to storage technology, and more particularly to a data storage method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0004] Database storage consists of row-based and column-based storage. Row-based storage uses row data as the underlying logical storage unit, making it suitable for random read operations but unsuitable for big data. However, with the widespread application of big data, column-based storage has emerged. Column-based storage stores all data in each column together, while allowing for separate storage for different columns. Column-based storage is widely used in database online analytical processing (OLAP) to improve the processing efficiency of analytical queries.

[0005] Currently, columnar storage stores a column of data in a file. When executing a query request, it is often necessary to access multiple columns, which requires opening multiple files simultaneously for filtered access, resulting in low query efficiency.

[0006] Summary of the Invention

[0007] Embodiments of the present application provide a data storage method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can optimize the storage of table data according to application query requests, thereby improving query efficiency.

[0008] The technical solution of the embodiment of the present application is implemented as follows:

[0009] An embodiment of the present application provides a data storage method, which is applied to an electronic device. The method includes:

[0010] Acquire multiple query requests for an application, parse each query request, and obtain at least one column of information included in each query request;

[0011] determining, based on at least one column information included in each query request, candidate column combinations for each table related to each query request, wherein when table data of the table related to the query request is stored according to the candidate column combinations, query overhead for executing the query request is minimized;

[0012] Determining a target column combination of a corresponding table based on multiple candidate column combinations belonging to the same table, wherein when table data of each table is stored according to the respective target column combinations, a total query overhead of executing the multiple query requests is minimized;

[0013] The table data of each table is stored according to the target column combination.

[0014] An embodiment of the present application provides a data storage device, including: a first acquisition module, configured to acquire multiple query requests for an application, parse each query request, and obtain at least one column information included in each query request; a first determination module, configured to determine, based on the at least one column information included in each query request, candidate column combinations of each table related to each query request, wherein when the table data of the table related to the query request is stored according to the candidate column combination, the query overhead of executing the query request is minimized; a second determination module, configured to determine, based on multiple candidate column combinations belonging to the same table, a target column combination of the corresponding table, wherein when the table data of each table is stored according to the respective target column combinations, the total query overhead of executing the multiple query requests is minimized; and a data storage module, configured to store the table data of each table according to the target column combination.

[0015] An embodiment of the present application provides an electronic device, comprising:

[0016] a memory for storing computer-executable instructions;

[0017] The processor is used to implement the data storage method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0018] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the data storage method provided in the embodiment of the present application when executed by a processor.

[0019] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the data storage method provided in the embodiment of the present application is implemented.

[0020] The embodiments of the present application have the following beneficial effects:

[0021] After obtaining multiple query requests for the application, the query requests are parsed to obtain at least one column information included in the query requests. Then, the candidate column combination of each table that can achieve the minimum query overhead when executing each query request is first determined in the single query request dimension. Then, based on this, the target column combination of each table is determined when executing multiple query requests in the application dimension to achieve the minimum total query overhead, wherein the target column combination is also the column combination that will be frequently accessed simultaneously when executing the query request. For different applications, the target column combination that will be accessed simultaneously when the query request is executed is different. In the embodiment of the present application, based on the multiple query requests for each application and the table data related to each query request, the target column combination that can achieve optimal storage in the application dimension and meets the application data access characteristics is determined, thereby achieving adaptability between the target column combination and the application data access characteristics, and improving the robustness of the data storage method provided in the embodiment of the present application. Finally, the table data of each table is merged and stored according to the corresponding target column combination. In this way, the present application can combine and store column data that are frequently accessed simultaneously in one file when executing a query request, thereby reducing the number of file openings, shortening the query time, and improving data query efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG1A is a schematic diagram showing a column of data stored in a storage unit;

[0023] FIG1B is a schematic diagram of a file storing column data of each column in a table;

[0024] FIG2 is a schematic diagram of the network architecture of the data query system 100 provided in an embodiment of the present application;

[0025] FIG3 is a schematic diagram of the structure of a server 400 provided in an embodiment of the present application;

[0026] FIG4 is a schematic diagram of an implementation flow of a data storage method provided in an embodiment of the present application;

[0027] FIG5 is a schematic diagram of an implementation flow of determining candidate column combinations of various tables related to various query requests provided by an embodiment of the present application;

[0028] FIG6 is a schematic diagram of an implementation flow of determining a basic column combination set of each table related to an i-th query request provided by an embodiment of the present application;

[0029] FIG7 is a schematic diagram of an implementation flow of determining target column combinations for each table according to an embodiment of the present application;

[0030] FIG8 is a schematic diagram of an implementation flow of determining the total query overhead of executing multiple query requests when the table data of each table is stored according to the respective corresponding merged candidate column combinations, provided by an embodiment of the present application;

[0031] FIG9 is a schematic diagram of an implementation flow of storing data using target column combinations of various tables according to an embodiment of the present application;

[0032] FIG10 is a schematic diagram of another implementation flow of the data storage method provided in an embodiment of the present application;

[0033] FIG11A is a schematic diagram of an implementation flow of performing a database query using a query statement according to an embodiment of the present application;

[0034] FIG11B is a schematic diagram of a parse tree obtained by parsing a query statement according to an embodiment of the present application;

[0035] FIG11C is a schematic diagram of the storage format of query records;

[0036] FIG12 is a schematic diagram of determining the basic column combination of T1;

[0037] FIG13 is a schematic diagram of determining read column overhead;

[0038] FIG14 is a schematic diagram of determining the column read overhead when three columns are stored separately and when three columns are stored in a combined manner;

[0039] FIG15 is a schematic diagram of merging three candidate column combinations of T1;

[0040] FIG16A is a schematic diagram of determining the total query overhead of executing multiple query statements according to a merged candidate column combination for table data storage according to an embodiment of the present application;

[0041] FIG16B is a schematic diagram of determining a column combination with the lowest total overhead according to an embodiment of the present application;

[0042] FIG17 is a schematic diagram of determining a sparse column according to an embodiment of the present application;

[0043] FIG18 is a schematic diagram of block storage of sparse optimal column combinations provided by an embodiment of the present application;

[0044] FIG19 is a schematic diagram of a system table provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0046] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0047] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0048] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be fully or partially implemented by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0049] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0050] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0051] 1) Online Analytical Processing (OLAP), a software technology that enables analysts to quickly, consistently, and interactively view information from all aspects to achieve a deep understanding of the data.

[0052] 2) Predicate. In the context of computer languages, a predicate refers to a conditional expression that returns true or false.

[0053] 3) Local predicates are predicates that are only used to access one table.

[0054] 4) Join predicate, which defines the join relationship between tables.

[0055] 5) Materialization: In order to make the column format of column storage correspond to user query habits and expression meaning, data needs to be converted into row format.

[0056] 6) Delayed materialization means delaying materialization until as late as possible in the query lifecycle. Delayed materialization means that during the initial period of query execution, the query execution model is column-based, not relational.

[0057] 7) Path overhead refers to the time it takes to execute a query request along an access path, including the overhead of reading the column data of the table related to the query request along the access path. The overhead of reading column data includes input / output (I / O) overhead and central processing unit (CPU) overhead. I / O overhead includes file opening overhead, random read overhead, and sequential read overhead. CPU overhead includes basic overhead, page read overhead, page scan overhead, and record scan overhead.

[0058] 8) Query overhead refers to the minimum path overhead when executing a query request along different access paths.

[0059] 9) Reference overhead refers to the query overhead when executing a query request when the table data of the table related to the query request is stored according to a basic column combination, wherein the table related to the query request is a table that appears in the query statement corresponding to the query request, the basic column combination is a column combination obtained by merging the columns included in the table related to the query request according to a preset merging condition, and storing the table data of the table related to the query request according to the basic column combination means storing the column data corresponding to the column identifiers included in the basic column combination in the same file.

[0060] 10) Column combination refers to a combination of at least one data column in a data table. For example, data table T1 includes four data columns, namely C1, C2, C3 and C4, then {C1, C2} is a column combination, and {C1, C2, C4} is also a column combination.

[0061] 11) Sparse columns: columns where the ratio of the number of empty data to the total number of data is greater than the sparsity threshold.

[0062] In order to better understand the data storage method provided in the embodiments of the present application, the column storage method in the related art and its shortcomings are first described.

[0063] Currently, column storage stores a column of data in a table in a single file (or storage unit). Figure 1A is a schematic diagram of a column of data in a storage unit. As shown in Figure 1A, in related art, a column is stored in an independent storage unit, a Silo, where a Silo is a compact arrangement of data blocks distributed in a columnar manner. As shown in Figure 1A, a Silo stores Silo header information, a space bitmap, data content, and zero padding for page alignment.

[0064] FIG1B is a schematic diagram of a file storing column data for each column in a table. The "16466" in the first row of FIG1B is the table identifier. The first column in the second to sixteenth rows is the file size, and the second column is the file identifier. 16466_1.0 indicates that the file stores the column data for the first column in table 16466, and the column name of the first column is "o_orderkey." Each file can store up to 1.1 GB of data. When the column data for a column exceeds 1.1 GB, the column data is split and stored in two or more files. For example, the column data for the second column is stored in files 16466_2.0 and 16466_2.1. The data included in each file is as shown in FIG1A , including Silo header information, a space bitmap, data content, and zero padding for page alignment.

[0065] Because certain columns are often accessed simultaneously in queries, multiple files need to be opened simultaneously for filtered access, resulting in low query efficiency. Furthermore, in scenarios like label descriptions, data becomes sparse. For example, when describing an item, labels for different types of items can vary significantly, resulting in very sparse data columns. Each file stores very little data, leading to wasteful access and data storage methods that cannot adapt to the characteristics of the application data.

[0066] Based on this, the embodiments of the present application provide a data storage method, apparatus, device, computer-readable storage medium, and computer program product, which can achieve storage optimization of column storage and improve query efficiency. The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as various types of user terminals such as laptops, tablet computers, desktop computers, set-top boxes, mobile devices (for example, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), smart phones, smart speakers, smart watches, smart TVs, and vehicle-mounted terminals, and can also be implemented as servers. Below, an exemplary application when the device is implemented as a server will be described.

[0067] Refer to Figure 2, which is a schematic diagram of the network architecture of the data query system 100 provided in an embodiment of the present application. As shown in Figure 2, the network architecture includes a terminal 200, a network 300, a server 400 and a database 500, wherein the terminal 200 is connected to the server 400 through the network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two.

[0068] The terminal 200 sends a query request to the server 400, which carries a query statement. After receiving the query request sent by the terminal 200, the server 400 parses the query request, obtains the query statement, and determines the query result corresponding to the query request from the database 500 based on the query statement, and returns the query result to the terminal 200.

[0069] The data in the database 500 is stored after being optimized using the data storage method provided in the embodiment of the present application. When the server 400 performs storage optimization on the data in the database 500, it first obtains multiple query requests for applications received within a preset historical period. Among them, the application can be a shopping application, a video application, a music application, etc. The server 400 parses each query request and obtains at least one column information included in each query request. Then, based on the at least one column information included in each query request, it determines the candidate column combination of each table related to the query request. When the table data of the table related to the query request is stored according to the candidate column combination, the query overhead of executing the query request is minimized. In order to minimize the total overhead of executing the query under various query conditions, the target column combination of the corresponding table is determined based on multiple candidate column combinations belonging to the same table. When the table data of each table is stored according to its respective target column combination, the total query overhead of executing multiple query requests is minimized. At this time, the table data of each table is stored according to the target column combination. Since the table data of each table is stored according to the target column combination, the total overhead of executing multiple query requests is minimized, that is, the database data storage has reached optimal storage. At this time, when the server receives a query request again, it can reduce data access time and improve query efficiency when accessing and querying data based on the current optimized stored table data.

[0070] In some embodiments, the server 400 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal 200 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car terminal, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0071] Referring to Figure 3, Figure 3 is a schematic diagram of the structure of a server 400 provided in an embodiment of the present application. The server 400 shown in Figure 3 includes: at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the server 400 are coupled together via a bus system 440. It can be understood that the bus system 440 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in Figure 3, all various buses are labeled as bus system 440.

[0072] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0073] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0074] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0075] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0076] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0077] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0078] A network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0079] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0080] The input processing module 454 is configured to detect one or more user inputs or interactions from one of the one or more input devices 432 and to translate the detected inputs or interactions.

[0081] In some embodiments, the apparatus provided in the embodiments of the present application can be implemented in software. FIG3 shows a data storage device 455 stored in the memory 450. The data storage device 455 can be software in the form of a program or plug-in, and includes the following software modules: a first acquisition module 4551, a first determination module 4552, a second determination module 4553, and a data storage module 4554. These modules are logical and can be arbitrarily combined or further separated according to the functions implemented. The functions of each module will be described below.

[0082] In other embodiments, the apparatus provided in the embodiments of the present application may be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the data storage method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0083] The data storage method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0084] The following describes the data storage method provided by the embodiment of the present application. As mentioned above, the electronic device that implements the data storage method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution entity of each step will not be repeated below.

[0085] Refer to Figure 4, which is a schematic diagram of an implementation flow of the data storage method provided in an embodiment of the present application. The steps shown in Figure 4 will be explained. The main body of the steps in Figure 4 is the server.

[0086] In step 101, multiple query requests for an application are obtained, each query request is parsed, and at least one column information included in each query request is obtained.

[0087] In some embodiments, the server retrieves multiple query requests for an application received within a preset historical period. For example, the server retrieves multiple query requests for an application received within 10 days. After receiving the multiple query requests, the server parses each query request to obtain at least one query statement, determines at least one column identifier corresponding to each query statement, and determines column attribute information corresponding to each column identifier. The column attribute information includes one of a local predicate column attribute, a join predicate column attribute, and a target selection column attribute. The column identifier and column attribute information constitute the column information.

[0088] In some embodiments, all query requests contain at least one query statement, that is, at least one query request contains a query statement. Multiple query requests may each contain at least one query statement; or all query requests may each contain at least one query statement.

[0089] For example, the query request is parsed and the query statement obtained may be as follows:

[0090] SELECT T1.C3, T2.C2

[0091] FROM T1,T2

[0092] WHERE T1.C1=1

[0093] AND T1.C2=T2.C2

[0094] After obtaining a query statement, the local predicate, join predicate, and target selection statement in the query statement are first determined. A local predicate is a predicate used only to access a single table. In the above query statement, the local predicate is T1.C1=1. A join predicate is a predicate that defines a join relationship between tables. In the above query statement, the join predicate is T1.C2=T2.C2, and the target selection statement is SELECT T1.C3, T2.C2. The column attribute information of the column identifier included in the local predicate is then determined as the local predicate column attribute, the column attribute information of the column identifier included in the join predicate is determined as the join predicate column attribute, and the column attribute information of the column identifier included in the target selection statement is determined as the target selection column attribute.

[0095] However, it should be noted that when determining column attributes, the column attributes of the column identifiers in the local predicate are determined first, followed by the column attributes of the column identifiers in the join predicate, and finally the column attributes of the column identifiers in the target select statement. If a column identifier appears in both a local predicate and a join predicate, the column attributes of the column identifier are considered the local predicate column attributes.

[0096] Taking the query above as an example, if the local predicate is T1.C1=1, then T1.C1 is the local predicate column attribute. If the join predicate is T1.C2=T2.C2, then the column attribute information of T1.C2 and T2.C2 are both join predicate column attributes. The target selection statement is SELECT T1.C3, T2.C2, and the column identifiers included in the target selection statement are T1.C3 and T2.C2. Since the column attributes of T2.C2 have been determined to be join predicate column attributes, only the column attribute information of T1.C3 is the target selection column attribute. For example, T1 represents Table 1, C3 represents the third column, and T1.C3 also represents the third column in Table 1. Similarly, T2.C2 represents the second column in Table 2.

[0097] In step 101, multiple query requests obtained within a preset time period are parsed to determine at least one column information included in each query request. This can provide a comprehensive and accurate data basis for subsequently determining the candidate column combination of each table when achieving the minimum query overhead in a single query request dimension, and determining the target column combination of each table when achieving the minimum total query overhead in the application dimension.

[0098] In step 102, based on at least one column information included in each query request, candidate column combinations of each table related to each query request are determined.

[0099] When the table data of the table related to the query request is stored according to the candidate column combination, the query overhead of executing the query request is minimized. In some embodiments, referring to FIG5 , step 102 can be implemented by the following steps 1021 to 10218 , which are described in detail below.

[0100] In step 1021 , based on the column attribute information corresponding to each column identifier in the i th query request, a basic column combination set of each table related to the i th query request is determined.

[0101] The basic column combination set includes at least one basic column combination, i=1, ...M, and M is the total number of query requests. In some embodiments, referring to FIG6 , step 1021 can be implemented by steps 211 to 2116 , which are described in detail below.

[0102] In step 211 , the column identifier of the column in the j-th table whose column attribute information is the local predicate column attribute is added to the local predicate column combination corresponding to the j-th table.

[0103] Wherein, j=1,…,N, N is the total number of tables related to the i-th query request. Among them, the tables related to the i-th query request refer to the tables that appear in the query statement corresponding to the i-th query request. By parsing the query statement and determining the number of tables that appear in the query statement, the total number of tables related to the i-th query request can be obtained. Since column data with the same column attributes are generally accessed at the same time, in an embodiment of the present application, the column identifiers of the columns whose column attribute information is the local predicate column in each table are used to construct the local predicate column combinations corresponding to each table.

[0104] For example, the tables associated with the i-th query request are T1 and T2, respectively. The column identifiers for columns in T1 whose column attribute information is a local predicate column attribute are T1.C1 and T1.C4, the column identifier for columns in T1 whose column attribute information is a join predicate column attribute is T1.C2, and the column identifiers for columns whose column attribute information is a target selection column attribute are T1.C3 and T1.C5. The column identifier for columns in T2 whose column attribute information is a local predicate column attribute is T2.C1, the column identifier for columns in T2 whose column attribute information is a join predicate column attribute is T2.C2, and the column identifier for columns whose column attribute information is a target selection column attribute is T2.C3.

[0105] Then the local predicate column combination corresponding to T1 is {T1.C1, T1.C4}, and the local predicate column combination corresponding to T2 is {T2.C1}.

[0106] In step 212 , the column identifier of the column in the j-th table whose column attribute information is the join predicate column attribute is added to the join predicate column combination corresponding to the j-th table.

[0107] Continuing with the above example, the join predicate column combination corresponding to T1 is {T1.C2}, and the join predicate column combination corresponding to T2 is {T2.C2}.

[0108] In step 213 , the column identifier of the column whose column attribute information in the j-th table is the target selection column attribute is added to the target selection column combination corresponding to the j-th table.

[0109] Continuing with the above example, the target selection column combination corresponding to T1 is {T1.C3, T1.C5}, and the target selection column combination corresponding to T2 is {T2.C3}.

[0110] In step 214 , it is determined whether the local predicate column combination and the join predicate column combination satisfy the first merge condition.

[0111] The first merge condition is that local materialization is not required, meaning that columns with local predicate column attributes do not need to be processed separately. In some embodiments, the optimizer obtains a first determination result regarding whether local materialization is required. If the first determination result indicates that local materialization is not required, it indicates that no columns with local predicate column attributes require separate processing, meaning that the local predicate column combination and the join predicate column combination meet the first merge condition. If the local predicate column combination and the join predicate column combination meet the first merge condition, the process proceeds to step 215; if the local predicate column combination and the join predicate column combination do not meet the first merge condition, the process proceeds to step 2111.

[0112] In step 215 , the local predicate column combination and the join predicate column combination are merged to obtain a first merged combination.

[0113] In some embodiments, the local predicate column combination and the join predicate column combination may be combined in ascending order according to column identifiers, or they may be combined in random order.

[0114] For example, assuming that the local predicate column combination and the join predicate column combination of T1 meet the first merge condition, the local predicate column combination corresponding to T1 is {T1.C1, T1.C4}, and the join predicate column combination is {T1.C2}. Then, the local predicate column combination and the join predicate column combination are combined in ascending order of column identifiers, resulting in the first merged combination corresponding to T1 being {T1.C1, T1.C2, T1.C4}. The local predicate column combination and the join predicate column combination are combined in random order, resulting in the first merged combination corresponding to T1 being {T1.C1, T1.C4, T1.C2}.

[0115] Assuming that the local predicate column combination of T2 and the join predicate column combination do not satisfy the first merging condition, the local predicate column combination of T2 and the join predicate column combination are not merged.

[0116] In step 216 , it is determined whether the first merging combination and the target selection column combination satisfy a second merging condition.

[0117] The second merge condition may be that delayed materialization is not required, meaning that there are no target selection columns that require separate processing. Similar to step 214, the optimizer obtains a second determination result regarding whether delayed materialization is required. If the second determination result indicates that delayed materialization is not required, the first merge combination and the target selection column combination meet the second merge condition. If the first merge combination and the target selection column combination meet the second merge condition, the process proceeds to step 217; if the first merge combination and the target selection column combination do not meet the second merge condition, the process proceeds to step 219.

[0118] In step 217 , the first merged combination and the target selected column combination are merged to obtain the basic column combination of the j-th table.

[0119] In some embodiments, the first merge combination and the target selected column combination may be merged in ascending order according to column identifiers, or the first merge combination and the target selected column combination may be merged in random order.

[0120] Exemplarily, assuming that the first merge combination of T1 and the target selection column combination meet the second merge condition, the first merge combination of T1 is {T1.C1, T1.C2, T1.C4}, and the target selection column combination is {T1.C3, T1.C5}, then the first merge combination and the target selection column combination are merged in order from small to large according to the column identifier, and the basic column combination of T1 is {T1.C1, T1.C2, T1.C3, T1.C4, T1.C5}; the first merge combination and the target selection column combination are merged in random order, and the basic column combination of T1 is {T1.C1, T1.C2, T1.C4, T1.C3, T1.C5}.

[0121] In step 218 , the base column combination of the j-th table is added to the base column combination set of the j-th table.

[0122] For T1, there is only one basic column combination, so the basic column combination set of T1 is {{T1.C1, T1.C2, T1.C4, T1.C3, T1.C5}}.

[0123] In step 219 , the first merge combination and the target selected column combination are respectively determined as the basic column combination of the j th table.

[0124] Exemplarily, assuming that the first merge combination of T1 and the target selected column combination do not satisfy the second merge condition, the basic column combinations of T1 are {T1.C1, T1.C2, T1.C4}, {T1.C3, T1.C5}.

[0125] In step 2110 , the base column combination of the j th table is added to the base column combination set of the j th table.

[0126] Continuing with the example in step 219 , the basic column combination set of T1 now includes two elements, namely, {{T1.C1, T1.C2, T1.C4}, {T1.C3, T1.C5}}.

[0127] In step 2111 , it is determined whether the join predicate column combination and the target column combination satisfy the second merge condition.

[0128] In some embodiments, when the local predicate column combination and the join predicate column combination do not satisfy the first merge condition, it can be determined whether the join predicate column combination and the target column combination satisfy the second merge condition, and the process proceeds to step 2112. When the join predicate column combination and the target column combination do not satisfy the second merge condition, the process proceeds to step 2115.

[0129] In step 2112 , the join predicate column combination and the target column combination are merged to obtain a second merged combination.

[0130] For example, assuming that the local predicate column combination of T2 and the join predicate column combination do not satisfy the first merge condition, but the join predicate column combination of T2 and the target column combination satisfy the second merge condition, then the join predicate column combination of T2 {T2.C2} and the target column combination {T2.C3} are merged to obtain the second merge combination {T2.C2, T2.C3}.

[0131] In step 2113 , the local predicate column combination and the second merge combination are respectively determined as the base column combination of the j th table.

[0132] Assume that the local predicate column combination of T2 is {T2.C1}, then there are two basic column combinations of T2, namely {T2.C1} and {T2.C2, T2.C3}.

[0133] In step 2114 , the base column combination of the j th table is added to the base column combination set of the j th table.

[0134] Continuing with the example in step 2113, the basic column combination set of T2 is {{T2.C1}, {T2.C2, T2.C3}}.

[0135] In step 2115 , the local predicate column combination, the join predicate column combination, and the target column combination are respectively determined as the base column combination of the j-th table.

[0136] In some embodiments, when the local predicate column combination and the join predicate column combination do not satisfy the first merging condition, and the join predicate column combination and the target column combination do not satisfy the second merging condition, then none of the three column combinations are merged, that is, the local predicate column combination, the join predicate column combination, and the target column combination are respectively determined as the base column combination of the j-th table.

[0137] In step 2116 , the base column combination of the j-th table is added to the base column combination set of the j-th table.

[0138] In some embodiments, if the local predicate column combination, the join predicate column combination, and the target selection column combination of the jth table are all merged, then the base column combination set of the jth table includes three base column combinations.

[0139] Since column data of the same column attribute are generally accessed simultaneously, in the above steps 211 to 2116, the column identifiers of the same column attribute are first used to construct local predicate column combinations, join predicate column combinations, and target column combinations. Then, the local predicate column combinations and join predicate column combinations are merged according to the preset first merge condition and second merge condition. Since the local predicate column combinations or target selection column combinations that do not need to be stored separately are not required when the first merge condition or the second merge condition is met, it is ensured that the column data in the determined basic column combination of each table can be merged and stored, thereby ensuring the accuracy of optimized storage.

[0140] 5 , and proceed to step 1021 for further description.

[0141] In step 1022 , it is determined whether the basic column combination set of the j th table related to the i th query request includes only one basic column combination.

[0142] Among them, when the basic column combination set of the jth table related to the i-th query request includes one basic column combination, enter step 1023; when the basic column combination set of the j-th table related to the i-th query request includes at least two basic column combinations, enter step 1024.

[0143] In step 1023, the basic column combination is determined as the candidate column combination of the j-th table.

[0144] Here, when there is only one basic column combination in the basic column combination set of the j-th table, there is no need to compare the query overheads, and the basic column combination is directly determined as the candidate column combination of the j-th table.

[0145] In step 1024 , at least two basic column combinations of the j th table are merged to obtain at least one merged basic column combination of the j th table.

[0146] In some embodiments, assuming that there are T basic column combinations in the jth table, T is a positive integer greater than or equal to 2, then any two basic column combinations in the jth table are merged to obtain Merge any three basic column combinations in the jth table to get The combined basic column combinations are then deduced by analogy. The T basic column combinations in the jth table are combined to obtain That is, when there are T basic column combinations in the jth table, a total of The combined base columns.

[0147] In step 1025 , it is determined whether the total number N of tables related to the i-th query request is 1.

[0148] When N is 1 and the basic combination set of the j-th table includes at least two basic column combinations, proceed to step 1026; when N is greater than 1 and the basic combination set of the j-th table includes at least two basic column combinations, proceed to step 10212.

[0149] In step 1026 , when the table data of the j th table is stored according to at least two basic column combinations, a first reference cost of executing the i th query request is determined.

[0150] In some embodiments, storing the table data of the j-th table according to the at least two basic column combinations is storing the column data corresponding to the column identifiers included in the basic column combinations in the j-th table into a file. For example, T1 corresponds to two basic column combinations, namely {T1.C1, T1.C2, T1.C4} and {T1.C3, T1.C5}. Then, the column data of C1, C2, and C4 in T1 are stored in a file, and the column data of C3 and C5 are stored in a file. Then, the i-th query request is executed according to different access paths to obtain various candidate query costs, and the minimum candidate query cost is determined as the first reference cost of the i-th query request.

[0151] That is, the first reference cost of the i-th query request is the candidate query cost corresponding to the optimal access path. The process of determining the first reference cost of the i-th query request is also the process of determining the optimal access path. In some embodiments, the optimal access path can be determined using a database optimizer. When executing a query request, the data reading function can be implemented through multiple paths. For example, for queries on the same record, the database can screen the data through indexes, which can be called an index scan. For multi-index data tables, there are multiple access paths to choose from for different indexes. Alternatively, the database can read data through a full table scan instead of using indexes. When using the database optimizer to select the optimal access path, a query path tree is generated based on the query statement according to an internal standard algorithm. The read cost of all access paths in the query path tree that can obtain the required data is then estimated. The estimated costs of all paths are compared, and the access path with the lowest cost is selected as the optimal access path to achieve the purpose of improving access efficiency.

[0152] In some embodiments, the candidate query overhead for executing the i-th query request along a data access path includes the overhead of reading table data of a table associated with the i-th query request along the data access path. The overhead of reading column data includes input / output (I / O) overhead and CPU overhead, where I / O overhead includes file opening overhead, random read overhead, and sequential read overhead; and CPU overhead includes base overhead, page read overhead, page scan overhead, and record scan overhead.

[0153] In step 1027 , when the table data of the j th table is stored according to the combined basic columns, each first query cost of executing the i th query request is determined.

[0154] In some embodiments, when implementing step 1027, first, when the table data of the j-th table is stored according to the k-th merged basic column combination, the corresponding path costs for executing the i-th query request according to multiple different access paths are determined. That is, each access path corresponds to a path cost. When the i-th query request has three different access paths, then the query request corresponds to three path costs. Where k = 1, ..., P, P is the total number of merged basic column combinations of the j-th table; and the minimum path cost among the path costs corresponding to the k-th merged basic column combination is determined as the first query cost for executing the i-th query request when the table data of the j-th table is stored according to the k-th merged basic column combination.

[0155] For example, assuming there are three base column combinations for T1, namely {T1.C1, T1.C4}, {T1.C2}, and {T1.C3, T1.C5}, four merged base column combinations are obtained: {T1.C1, T1.C2, T1.C4}, {T1.C1, T1.C4, T1.C3, T1.C5}, {T1.C2, T1.C3, T1.C5}, and {T1.C1, T1.C4, T1.C2, T1.C3, T1.C5}. The database optimizer then calculates the path costs for each merged base column combination when executing the i-th query request along different paths, and determines the minimum path cost as the first query cost for executing the query request when storage is performed according to the merged base column combination.

[0156] Suppose the merged basic column combination is {T1.C1, T1.C4, T1.C3, T1.C5}. Assuming there are three different data access paths, the database optimizer is used to determine the path costs of executing query requests for the merged basic column combination according to different data access paths. The minimum path cost among the three path costs is determined as the first query cost of the merged basic column combination {T1.C1, T1.C4, T1.C3, T1.C5}.

[0157] In step 1028 , a first minimum query cost among the first query costs is determined.

[0158] In some embodiments, a first minimum query cost among the first query costs may be determined using a preset sorting algorithm.

[0159] Exemplarily, the sorting algorithm may be a bubble sort algorithm, an insertion sort algorithm, etc. In the embodiments of the present application, the sorting algorithm actually used is not limited.

[0160] In step 1029 , it is determined whether the first minimum query cost is less than the first reference cost.

[0161] When the first minimum query cost is less than the first reference cost, the process goes to step 10210 ; when the first minimum query cost is greater than or equal to the first reference cost, the process goes to step 10211 .

[0162] In step 10210, the merged basic column combination corresponding to the first minimum query cost is determined as the candidate column combination of the j-th table.

[0163] In some embodiments, when the first minimum query overhead is less than the first reference overhead, it means that when the table data of the j-th table is stored using the merged basic column combination, a better storage effect and a smaller query overhead can be achieved. At this time, the merged basic column combination corresponding to the first minimum query overhead is determined as the candidate column combination of the j-th table.

[0164] In step 10211, at least two basic column combinations are determined as candidate column combinations for the j-th table.

[0165] In some embodiments, when the first minimum query overhead is greater than or equal to the first reference overhead, it means that when using the merged basic column combination to store the table data of the j-th table, it cannot achieve better storage effect and smaller query overhead before the basic column combination is merged. At this time, at least two basic column combinations are determined as candidate column combinations for the j-th table.

[0166] In step 10212, when there is another table among the N tables that includes at least two basic column combinations, the at least two basic column combinations of the other table are merged to obtain at least one merged basic column combination of the other table.

[0167] In some embodiments, when there are at least two tables related to the i-th query request and the j-th table corresponds to at least two basic column combinations, the at least two basic column combinations corresponding to the j-th table are first merged, and then it is determined whether there are other tables in the N tables that include at least two basic column combinations in addition to the j-th table. If there are other tables that include at least two basic column combinations in addition to the j-th table, the at least two basic column combinations of the other tables are merged to obtain at least one merged basic column combination of the other tables.

[0168] It should be noted that the implementation process of merging at least two basic column combinations of other tables is similar to the implementation process of step 1024 , and reference may be made to the implementation process of step 1204 .

[0169] In some embodiments, when there are no other tables among the N tables except the j-th table that include at least two basic column combinations, that is, all tables except the j-th table among the N tables only include one basic column combination, then there is no need to merge the basic column combinations of these tables, and step 10214 is continued.

[0170] In step 10213 , when the table data of the N tables are stored according to their corresponding basic column combinations, a second reference cost for executing the i-th query request is determined.

[0171] In some embodiments, since the data of N tables related to the i-th query request will be accessed when executing the i-th query request, in this step, it is necessary to determine the path costs of executing the i-th query request according to different access paths when the table data of the N tables are stored according to their respective corresponding basic column combinations, and then determine the minimum path cost as the second reference cost corresponding to the i-th query request.

[0172] In step 10214, when the table data of the table including at least two basic column combinations among the N tables are stored according to their respective corresponding merged basic column combinations, and the table data of the table including one basic column combination are stored according to their respective corresponding basic column combinations, the second query overhead of executing the i-th query request is determined.

[0173] In some embodiments, it is assumed that among N tables, there are A tables including at least two basic column combinations, namely T1 to TA, where A is an integer less than or equal to N, wherein T1 corresponds to B1 combined basic column combinations, T2 corresponds to B2 combined basic column combinations, ..., TA corresponds to B A The table data of the N tables including at least two basic column combinations are stored according to their corresponding combined basic column combinations, and the table data of the tables including one basic column combination are stored according to their corresponding basic column combinations. There are a total of B1*B2*…*B A There are three different execution paths for executing the i-th query request. Each storage method will determine three path costs. The minimum path cost among the three path costs of each storage method is determined as the second query cost for executing the i-th query request according to the storage method. That is, in step 10215, B1*B2*…*B A Second query overhead.

[0174] Exemplarily, there are three tables related to the i-th query request, namely T1, T2 and T3, among which T1 corresponds to three basic column combinations, T2 corresponds to two basic column combinations, and T3 corresponds to one basic column combination. Then T1 corresponds to four merged basic column combinations, T2 corresponds to one merged basic column combination, and T3 corresponds to one basic column combination. Then T1, T2 and T3 have a total of 4*1 storage methods, and 4 second query overheads can be determined in step 10214.

[0175] For example, T1 corresponds to four merged basic column combinations, namely {T1.C1, T1.C2, T1.C4}, {T1.C1, T1.C4, T1.C3, T1.C5}, {T1.C2, T1.C3, T1.C5}, {T1.C1, T1.C4, T1.C2, T1.C3, T1.C5}, T2 corresponds to one merged basic column combination, {T2.C1, T2.C2, T2.C3}, and T3 corresponds to one basic column combination, {T3.C3, T3.C5}. When executing the i-th query request, T1, T2, and T3 have four storage methods:

[0176] The first type: T1 is merged and stored according to {T1.C1, T1.C2, T1.C4}, T2 is merged and stored according to {T2.C1, T2.C2, T2.C3}, and T3 is merged and stored according to {T3.C3, T3.C5};

[0177] The second type: T1 is merged and stored according to {T1.C1, T1.C4, T1.C3, T1.C5}, T2 is merged and stored according to {T2.C1, T2.C2, T2.C3}, and T3 is merged and stored according to {T3.C3, T3.C5};

[0178] The third type: T1 is merged and stored according to {T1.C2, T1.C3, T1.C5}, T2 is merged and stored according to {T2.C1, T2.C2, T2.C3}, and T3 is merged and stored according to {T3.C3, T3.C5};

[0179] The fourth type: T1 is merged and stored according to {T1.C1, T1.C4, T1.C2, T1.C3, T1.C5}, T2 is merged and stored according to {T2.C1, T2.C2, T2.C3}, and T3 is merged and stored according to {T3.C3, T3.C5}.

[0180] Then determine the second query overhead corresponding to the above four storage methods. For example, the second query overhead corresponding to the first storage method is 1 second, the second query overhead corresponding to the second storage method is 0.8 seconds, the second query overhead corresponding to the third storage method is 2 seconds, and the second query overhead corresponding to the fourth storage method is 1.2 seconds.

[0181] In step 10215, the second minimum query cost among the second query costs is determined.

[0182] In some embodiments, the implementation process of step 10215 is similar to the implementation process of step 1028, and can be implemented with reference to the implementation process of step 1028.

[0183] Continuing with the example of step 10214, the second minimum query overhead is 0.8 seconds.

[0184] In step 10216, it is determined whether the second minimum query cost is less than the second reference cost.

[0185] When the second minimum query cost is less than the second reference cost, the process goes to step 10217 ; when the second minimum query cost is greater than or equal to the second reference cost, the process goes to step 10218 .

[0186] In step 10217, the column combination of each table corresponding to the second minimum query cost is determined as the candidate column combination of each table.

[0187] When the second minimum overhead is less than the second reference overhead, it means that when the table data of each table is stored using the column combination of each table corresponding to the second minimum query overhead, a better storage effect and a smaller query overhead can be achieved. At this time, the column combination of each table corresponding to the second minimum query overhead is determined as the candidate column combination of each table.

[0188] For example, assuming the second reference cost is 1 second and the second minimum query cost is 0.8 seconds, and the second minimum query cost is less than the second reference cost, the column combination of each table corresponding to the second minimum query cost is determined as the corresponding candidate column combination. If the second minimum query cost corresponds to the second storage method described above, the candidate column combination for T1 is {T1.C2, T1.C3, T1.C5}, the candidate column combination for T2 is {T2.C1, T2.C2, T2.C3}, and the candidate column combination for T3 is {T3.C3, T3.C5}.

[0189] In step 10218, the basic column combination of each table is determined as the candidate column combination of each table.

[0190] When the second minimum query cost is greater than or equal to the second reference cost, it means that the column combination of each table corresponding to the second minimum query cost cannot achieve a better storage effect and lower query cost than before the merger when storing the table data of each table. Therefore, the basic column combination of each table is directly determined as the candidate column combination of each table.

[0191] In the above steps 1021 to 10218, the candidate column combinations of the jth table are determined respectively when the jth table has only one basic column combination, when the jth table includes at least two basic column combinations and there is only one table related to the i-th query request, and when the jth table includes at least two basic column combinations and there are at least two tables related to the i-th query request. This ensures that in various cases, the candidate column combinations of each table that can achieve the minimum query overhead when executing the i-th query request are accurately determined.

[0192] 4 , the following description continues with step 102 .

[0193] In step 103 , based on multiple candidate column combinations belonging to the same table, a target column combination of the corresponding table is determined.

[0194] In the above step 102, a single query request is used as the granularity to determine the candidate column combination of each table that can minimize the query overhead of each query request. However, the application may receive multiple different query requests, and different query requests may involve the same table. The same table may correspond to different candidate column combinations in different query requests. However, the same table can only be stored in one storage method. At this time, in order to minimize the total query overhead at the entire application level, it is necessary to determine the target column combination of each table. When the table data of each table is stored according to its respective target column combination, the total query overhead of executing the multiple query requests can be minimized.

[0195] In some embodiments, referring to FIG. 7 , step 103 may be implemented through steps 1031 to 1034 , which are described in detail below.

[0196] In step 1031 , at least two candidate column combinations belonging to the same table are merged to obtain at least one merged candidate column combination of the corresponding table.

[0197] In some embodiments, step 102 determines the candidate column combinations of each table related to each query request. Since the same table may be involved in different query requests, the same table will correspond to multiple candidate column combinations. Therefore, in this step, at least two candidate column combinations belonging to the same table are merged to obtain at least one merged candidate column combination of each table.

[0198] The implementation process of merging at least two candidate columns belonging to the same table is similar to the implementation process of step 1205 , and reference may be made to the implementation process of step 1205 .

[0199] To simplify the example, assume that there are three query requests. The tables related to the first query request are T1, T2, and T3. The tables related to the second query request are T2 and T3. The tables related to the third query request are T1 and T3. The candidate column combination of table T1 related to the first query request is {T1.C2, T1.C3, T1.C5}, the candidate column combination of table T2 related to the first query request is {T2.C1, T2.C2, T2.C3}, and the candidate column combination of table T2 related to the first query request is {T2.C1, T2.C2, T2.C3}. The candidate column combination of table T3 related to the first query request is {T3.C3, T3.C5}; the candidate column combination of table T2 related to the second query request is {T2.C1, T2.C4}, and the candidate column combination of table T3 related to the second query request is {T3.C4, T3.C5}; the candidate column combination of table T1 related to the third query request is {T1.C2, T1.C6}, and the candidate column combination of table T3 related to the third query request is {T3.C1, T3.C2}. Therefore, T1 has two candidate column combinations: {T1.C2, T1.C3, T1.C5}, {T1.C2, T1.C6}; T2 has two candidate column combinations: {T2.C1, T2.C2, T2.C3}, {T2.C1, T2.C4}; T3 has three candidate column combinations: {T3.C3, T3.C5}, {T3.C4, T3.C5}, {T3.C1, T3.C2}. Merge the two candidate column combinations of T1 to obtain a merged candidate column combination: {T1.C2, T1.C3, T1.C5, T1.C6}. Merge the two candidate column combinations of T2 to obtain a merged candidate column combination: {T2.C1, T2.C2, T2.C3, T2.C4}. Merge the three candidate column combinations of T3 to obtain four merged candidate column combinations: {T3.C3, T3.C4, T3.C5}, {T3.C1, T3.C2, T3.C4, T3.C5}, {T3.C1, T3.C2, T3.C3, T3.C5}, and {T3.C1, T3.C2, T3.C3, T3.C4, T3.C5}.

[0200] In step 1032 , when the table data of each table is stored according to each corresponding candidate column combination after merging, the total query overhead of executing the multiple query requests is determined.

[0201] In some embodiments, as shown in FIG8 , step 1032 may be implemented through steps 321 to 323 , which are described in detail below.

[0202] In step 321 , when the table data of each table is stored according to the corresponding merged candidate column combinations, each third query cost of executing each query request is determined.

[0203] In some embodiments, it is assumed that there are Q tables related to multiple query requests, where T1 corresponds to D1 candidate column combinations after merging, T2 corresponds to D2 candidate column combinations after merging, ..., TQ corresponds to D Q The combined candidate columns are stored in the table data of each table according to their corresponding combined candidate columns. There are D1*D2*…, *D Q storage modes, and then determining each path cost of executing each query request according to different access paths when each table is stored according to each storage mode, and determining the minimum path cost as the third query cost of the query request.

[0204] When there is D1*D2*…, *D Q In this storage mode, each query request corresponds to D1*D2*…, *D Q A third query overhead.

[0205] In step 322, the execution times of each query request are obtained.

[0206] In some embodiments, the number of executions of the query request may refer to the number of executions of the query request within a preset historical time period.

[0207] In step 323 , a total query cost of executing the multiple query requests is determined based on the number of executions of the query requests and the third query costs corresponding to the query requests.

[0208] In some embodiments, the number of executions of each query request is multiplied by each third query cost corresponding to each query request to obtain each product, and then the products are added together to obtain the total query cost of executing multiple query requests.

[0209] Continuing with the above example, assume that there are M query requests. Obtain the third query cost and execution times corresponding to each of the M query requests under each storage method. Then multiply the third query cost and the corresponding execution times to obtain the total query cost under this storage method. Q storage method, then in this step we get D1*D2*…,*D Q The total query cost.

[0210] For example, assume there are three query requests, four storage methods, and the three query requests are executed 100, 500, and 300 times, respectively. Under the first storage method, the third query overhead of the first query request is 1 second, the third query overhead of the second query request is 0.8 seconds, and the third query overhead of the third query request is 2 seconds. Therefore, under the first storage method, the total query overhead is 1*100 + 0.8*500 + 2*300 = 1100 seconds. Assume that, according to this method, the total query overhead corresponding to the second storage method is 800 seconds, the total query overhead corresponding to the third storage method is 1200 seconds, and the total query overhead corresponding to the third storage method is 1500 seconds.

[0211] In step 1033, the minimum query total cost among the query total costs is determined.

[0212] In some embodiments, the minimum query total cost among the query total costs may be determined according to a preset sorting algorithm. For example, the minimum query total cost is 800 seconds.

[0213] In step 1034 , the candidate column combination of each table corresponding to the minimum total query cost after merging is determined as the target column combination of each table.

[0214] Exemplarily, the minimum total query cost corresponds to the second storage mode, and the candidate column combination after merging the tables in the second storage mode is determined as the target column combination of each table. Assume that in the second storage method, the candidate column combination after merging of T1 is {T1.C2, T1.C3, T1.C5, T1.C6}, the candidate column combination after merging of T2 is {T2.C1, T2.C2, T2.C3, T2.C4}, and the candidate column combination after merging of T3 is {T3.C1, T3.C2, T3.C4, T3.C5}, then the target column combination of T1 is {T1.C2, T1.C3, T1.C5, T1.C6}, the target column combination of T2 is {T2.C1, T2.C2, T2.C3, T2.C4}, and the target column combination of T3 is {T3.C1, T3.C2, T3.C4, T3.C5}.

[0215] Since the application will receive multiple different query requests, different query requests may involve the same table. The same table may correspond to different candidate column combinations in different query requests, but the same table can only be stored in one storage method. At this time, through the above steps 1031 to 1034, the candidate column combinations belonging to the same table are merged to obtain the merged candidate column combinations of each table. Then, when the table data of each table is stored according to the corresponding merged candidate column combinations, the total query costs of multiple query requests are executed, and the merged candidate column combination of each table corresponding to the minimum query total cost is determined as the target column combination of each table. In this way, when the table data of each table is stored according to the respective target column combinations, the total query cost can be minimized at the entire application level, and it can be ensured that the determined target column combination of each table is adaptive to the data characteristics of the application.

[0216] That is, in step 103, the target column combination for each table is determined on an application-by-application basis to minimize the number of queries required when executing multiple query requests. The target column combination is also the combination of columns that are frequently and simultaneously accessed during query execution. Different applications have different target column combinations that are simultaneously accessed during query requests. In this embodiment of the present application, based on the multiple query requests for each application and the table data associated with each query request, a target column combination that enables optimal storage and meets the application data access characteristics is determined on an application-by-application basis, thereby achieving adaptability between the target column combination and the application data access characteristics and improving the robustness of the data storage method provided in this embodiment of the present application.

[0217] 4 , the following description continues with step 103 .

[0218] In step 104, the table data of each table is stored according to the target column combination.

[0219] In some embodiments, assuming that Table 1 (T1) has a total of 10 columns of data, the ten columns of T1 are T1.C1, T1.C2, T1.C3, T1.C4, T1.C5, T1.C6, T1.C7, T1.C8, T1.C9, and T1.C10, the target column combinations of T1 are {T1.C2, T1.C3, T1.C5, T1.C6}, {T1.C1, T1.C4, T1.C7}, and neither target column combination of T1 is a sparse target column combination, then the column data of the four columns T1.C2, T1.C3, T1.C5, and T1.C6 are stored in the same file, the column data of the three columns T1.C1, T1.C4, and T1.C7 are stored in the same file, and T1.C8, T1.C9, and T1.C10 are each stored in a file.

[0220] In some embodiments, referring to FIG. 9 , step 104 may be implemented through steps 1041 to 10411 , which are described in detail below.

[0221] In step 1041 , it is determined whether the target column combinations of the sth table include at least two sparse target column combinations.

[0222] Where s = 1, 2, ..., Q, where Q is the total number of tables to be optimized. In some embodiments, when determining whether the target column combination of the sth table is a sparse target column combination, the first step is to obtain each column identifier in the target column combination, then obtain the number of empty data and the total number of data in the column data corresponding to each column identifier, and determine the ratio of the number of empty data corresponding to each column identifier to the total number of data as the sparsity value corresponding to each column identifier. Columns corresponding to column identifiers having a sparsity value greater than a preset sparsity threshold are determined as sparse columns. When all columns corresponding to each column identifier in a target column combination are sparse columns, the target column combination is determined to be a sparse target column combination. Exemplarily, the target column combination is {T1.C1, T1.C4, T1.C7}. Assuming that the number of empty data corresponding to the column identifier T1.C1 is 3 and the total number of data is 50, then the sparsity value corresponding to the column identifier is 0.06, the number of empty data corresponding to T1.C4 is 5, and the total number of data is 50, then the sparsity value corresponding to the column identifier is 0.1, the number of empty data corresponding to T1.C7 is 6, and the total number of data is 50, then the sparsity value corresponding to the column identifier is 0.12. Assuming that the sparsity threshold is 0.15, then the columns corresponding to T1.C1, T1.C4, and T1.C7 are all sparse columns, that is, the target column combination is a sparse target column combination.

[0223] Determine in turn whether each target column combination in the sth table is a sparse target column combination. When there are at least two sparse target column combinations in the target column combination of the sth table, proceed to step 1042. When the target column combination of the sth table does not include at least two sparse target column combinations, proceed to step 1049.

[0224] In step 1042 , at least two sparse target column combinations are merged to obtain at least one merged sparse target column combination.

[0225] In some embodiments, when merging at least two sparse target column combinations, the occupied space size of each sparse target column combination and the file size of the storage file are first obtained, and then the number of sparse target column combinations that can be stored in each storage file is determined based on the file size, and the sparse target column combinations in the sth table are merged based on the number of sparse target column combinations that can be stored in each storage file.

[0226] For example, the number of sparse target column combinations that can be stored in each storage file is 2, and there are 3 sparse target column combinations in the sth table. Then, two sparse target column combinations are randomly selected from these 3 sparse target column combinations and merged to obtain a merged sparse target column combination. The remaining sparse target column combination that has not been merged is stored as a regular target column combination.

[0227] In step 1043 , new metadata information corresponding to each sparse target column identifier in at least one merged sparse target column combination is determined.

[0228] The new metadata information includes the file identifier, file format, and storage location of the storage file corresponding to the column data.

[0229] In step 1044 , based on the metadata information corresponding to each sparse target column identifier, the column data corresponding to each sparse target column identifier is stored in a corresponding storage file.

[0230] In some embodiments, in order to ensure that a normal response can be achieved when a query request is received during data storage, when the column data corresponding to each sparse target column identifier is stored in the corresponding storage file, the column data corresponding to each sparse target column identifier is copied to the corresponding storage file.

[0231] In step 1045 , the metadata information corresponding to each sparse target column identifier in the system table is updated to the new metadata information corresponding to each sparse target column identifier.

[0232] In some embodiments, after the column data corresponding to each sparse target column identifier is stored in the corresponding storage file, in order to enable the query request to be executed according to the new storage method when a query request is received subsequently, it is necessary to delete the metadata information corresponding to each sparse target column identifier originally stored in the system table, and store the new metadata information corresponding to each sparse target column identifier in the system table.

[0233] In step 1046 , when the sth table further includes other target column combinations in addition to the at least two sparse target column combinations, new metadata information corresponding to each other target column identifier in the other target column combinations is determined.

[0234] In some embodiments, the new metadata information corresponding to each other target column identifier in the other target column combination includes the file identifier, file format, and storage location of the storage file corresponding to the column data of each other target column identifier.

[0235] In step 1047 , based on the new metadata information corresponding to each other target column identifier in the other target column combination, the column data corresponding to each other target column identifier is stored in the corresponding storage file.

[0236] In some embodiments, the implementation process of step 1047 is similar to the implementation process of step 1044 , and the implementation process of step 1044 may be referred to during implementation.

[0237] In step 1048, the metadata information corresponding to each other target column identifier in the system table is updated to the new metadata information corresponding to each other target column identifier.

[0238] In some embodiments, the implementation process of step 1048 is similar to the implementation process of step 1045, and the implementation process of step 1045 can be referred to during implementation.

[0239] In step 1049 , new metadata information corresponding to each target column identifier in the target column combination is determined.

[0240] In step 10410, based on the metadata information corresponding to each target column identifier, the column data corresponding to each target column identifier is stored in the corresponding storage file.

[0241] In step 10411, the metadata information corresponding to each target column identifier in the system table is updated to the new metadata information corresponding to each target column identifier.

[0242] In some embodiments, when there are no at least two sparse target column combinations in the sth table, it means that the target column combinations do not need to be merged again, and the data of the sth table is directly stored through steps 1049 to 10411, wherein the implementation process of steps 1049 to 10411 is similar to the implementation process of steps 1043 to 1045, and the implementation process of the above steps 1043 to 1045 can be referred to.

[0243] In the above steps 1041 to 10411, when there are at least two sparse target column combinations in the target column combination of the table to be optimized, the at least two sparse target column combinations are merged and the merged sparse target column combination is stored in a file. This can improve the storage space utilization of the file. If there are not at least two sparse target column combinations in the target column combination of the table to be optimized, the target column combination determined in step 103 is optimized and stored, that is, the column data corresponding to the column identifiers included in the target column combination is stored in a file, and the metadata information of each table is modified accordingly. In addition, when data is stored, it is stored in segments, that is, one table is optimized and stored each time, thereby reducing the complexity of the optimized storage, and after the optimized storage of a table is completed, the original data of the table before the optimized storage is deleted. After the optimized storage of the table data of each table is completed, when a query request is received, the query process is executed based on the optimized and stored table data.

[0244] In some embodiments, during the process of performing optimized storage, if a query request sent by a client is received, the query request is executed based on the table data before the optimized storage, thereby avoiding erroneous query results due to incomplete optimized storage of table data and ensuring the correctness of the query results.

[0245] In the data storage method provided in the embodiment of the present application, after obtaining multiple query requests for an application, each query request is parsed to obtain at least one column information included in each query request, and then the candidate column combination of each table that can achieve the minimum query overhead when executing each query request in a single query request dimension is first determined. Then, based on this, the target column combination of each table when executing multiple query requests in the application dimension to achieve the minimum total query overhead is determined, so that the table data of each table is merged and stored according to the corresponding target column combination. In this way, when executing the query request, the column data that is frequently accessed at the same time can be merged and stored in one file, thereby reducing the number of file openings, reducing query time, and improving data query efficiency. In addition, when performing data storage, multiple query requests for the application are used for analysis and processing, so that the final optimized storage strategy can be adapted to the characteristics of the application data.

[0246] In some embodiments, as shown in FIG. 10 , during the execution of step 104 , if data update information is received, the following steps 105 to 107 may also be executed, which are described in detail below.

[0247] In step 105 , when data update information is received during the data storage process, the data update information is stored in an update log.

[0248] The data update information may include data deletion information, data modification information or data addition information.

[0249] In step 106 , after the data storage is completed, the table data is updated based on the data update information in the update log to obtain updated table data.

[0250] In some embodiments, completing data storage means completing the optimization of all tables to be optimized and stored in the application. At this time, data update information received during the optimization storage process can be obtained from the update log, and based on the data update information in the update log, the table data is updated to obtain updated table data.

[0251] Exemplarily, the data update information is data deletion information, which carries the location information of the data to be deleted, and then deletes the data to be deleted corresponding to the location information. The location information includes at least a table identifier, and may also include a column identifier and a row identifier.

[0252] In step 107, the data update information is deleted from the update log.

[0253] In some embodiments, after completing the data update based on the data update information, in order to avoid performing the data update again, the data update information is deleted from the update log, which can also reduce the space occupied by the update log.

[0254] In the above steps 105 to 107, in order to avoid omission and uncertainty as to whether the data before optimized storage or the data after optimized storage is updated, when data update information is received during the data storage process, the data will not be updated in real time. Instead, the data update information will be first stored in the update log, and after the data optimized storage is completed, the table data will be updated based on the data update information stored in the update log, so that when the query request is subsequently executed, the updated table data is accessed, thereby ensuring the accuracy of the query results.

[0255] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0256] An embodiment of the present application provides a data storage method that dynamically adjusts a column storage strategy based on multiple query requests received and application data storage characteristics to achieve optimal data storage effect and reading efficiency.

[0257] In an embodiment of the present application, multiple query statements are first analyzed to determine the columns of each table that appear in each query statement, and to determine the candidate column combination of each table when executing each query statement to achieve the minimum query overhead; then, the candidate column combination of each table is combined again to determine the optimal column combination of each table when executing multiple query statements to achieve the minimum total query overhead (corresponding to the target column combination in other embodiments). In order to further improve storage space utilization, sparse column storage optimization is also performed. When a table includes at least two optimal column combinations as diluted column combinations, the at least two optimal column combinations are merged to obtain at least one merged sparse column combination. Finally, data is stored based on the determined optimal column combination for each table.

[0258] The data storage method provided in the embodiment of the present application can be implemented through the following steps, which are described below.

[0259] The first step is to parse the query statement, determine the columns of each table that appear in the query statement, and determine the initial column combination to which the columns that appear in the query statement belong.

[0260] The initial column combination includes: a column combination corresponding to the local predicate, a column combination corresponding to the join predicate, and a target selection column combination.

[0261] Generally speaking, when performing a database query through a query statement, it is implemented through the following steps 1101 to 1106 shown in FIG11A , which will be explained below with reference to FIG11A .

[0262] Step 1101, query and check.

[0263] In this step, the received query request is checked to see if it has already been stored in the system table. If so, it has already been analyzed for optimal storage and no further analysis is required. If not, it has not yet been analyzed for optimal storage and the subsequent steps are performed.

[0264] Step 1102, query analysis.

[0265] In some embodiments, the received query statement is parsed to generate a parse tree, and the column information included in the query request is recorded according to the content of the parse tree.

[0266] Because columns in different positions may be used at different times when the database optimizer calculates the optimal path, the columns required for query statement execution are divided into:

[0267] Columns corresponding to local predicates: The predicate contains only columns from one table (which can be used to establish materialization of this table).

[0268] Columns corresponding to join predicates: Columns in a predicate that contain predicates for more than one table (usually used for join calculations).

[0269] Columns corresponding to target selection: columns that appear in the result set (usually used to build result sets, delay materialization, etc.).

[0270] Figure 11B is a schematic diagram of a parse tree obtained by parsing a query statement provided by an embodiment of the present application. Taking table T1 in Figure 11B as an example, T1.C1 appears in the local predicate T1.C1=10 and is marked as a local predicate column; T1.C2 appears in the join predicate T1.C2=T2.C2 and is marked as a join predicate column; T1.C3 and T1.C4 appear in the target selection statement, and T1.C3 and T1.C4 are marked as target selection columns.

[0271] For example, a query statement is as follows:

[0272] SELECT T1.C3,T2.C2

[0273] FROM T1,T2

[0274] WHERE T1.C1=1

[0275] AND T1.C2=T2.C2

[0276] This query statement involves two tables, T1 and T2. T1.C1 = 1, which is a local predicate. Therefore, T1.C1 is the column used in the local predicate, so T1.C1 is added to the local predicate column combination. T1.C2 = T2.C2 is a join predicate. T1.C2 and T2.C2 are the columns used in the join predicate. Therefore, T1.C2 and T2.C2 are added to the join predicate column combination. T1.C3 appears in the SELECT list. T1.C3 is the target selection column, so T1.C3 is added to the target selection column combination.

[0277] It should be noted that when a column appears in both the local predicate and the join predicate, the column is added to the local predicate column combination; when a column appears in both the local predicate and the target select statement, the column is added to the local predicate column combination; when a column appears in both the join predicate and the target select statement, the column is added to the join predicate columns.

[0278] For example, the query statement is as follows:

[0279] SELECT T1.C1,T2.C4

[0280] FROM T1,T2

[0281] WHERE T1.C1=1

[0282] AND T1.C1=T2.C1

[0283] AND T1.C2=T2.C2

[0284] AND T1.C3=T2.C3;

[0285] For T1, T1.C1 appears in the local predicate. Although T1.C1 also appears in the join predicate and target select statement, it is no longer recorded in the columns corresponding to the join predicate or the target select. This is because during query execution, T1.C1 is first accessed in the column combination corresponding to the local predicate. By the time the result set is generated, T1.C1 has already been accessed and does not need to be accessed again. Therefore, the column combination corresponding to T1's local predicate is {T1.C1}, and the column combination corresponding to the join predicate is {T1.C2, T1.C3}.

[0286] For T2, columns that do not appear in the local predicate, T2.C1, T2.C2, and T2.C3 all appear in the join predicate, and T2.C4 appears in the target selection statement. Therefore, the column combination corresponding to T2's join predicate is {T2.C1, T2.C2, T2.C3}, and the column combination corresponding to the target selection is {T2.C4}.

[0287] Step 1103: query rewriting.

[0288] In this step, the parse tree is rewritten in some ways, such as eliminating subqueries, merging views, etc., to improve query efficiency and obtain a modified parse tree.

[0289] Step 1104: Access path calculation.

[0290] In this step, when the database optimizer is used to calculate the access path, the optimal path is calculated based on the storage information of each column combination, and the minimum query cost corresponding to each column combination is recorded.

[0291] Step 1105: Plan generation.

[0292] In this step, a data structure describing the detailed steps of how to execute the query statement is generated, providing important information and guidance for subsequent query execution.

[0293] Step 1106, query execution.

[0294] During query execution, record the execution information to check whether the column combination used is valid. Record the cost of the previous combination and compare it with the cost of the new combination to check for any improvement.

[0295] The second step is to determine the candidate column combinations for each table when executing each query statement to achieve the minimum query overhead.

[0296] In some embodiments, after receiving multiple query statements, query records can be generated based on the multiple query statements. Figure 11C is a schematic diagram of the storage format of the query record. As shown in Figure 11C, each query record includes a query statement 1111, an access path 1112, table information 1113, the number of executions of the query statement 1114, and column information 1115 of each column contained in the table.

[0297] In some embodiments, to determine candidate column combinations for each table appearing in a query statement, the base column combinations for each table must first be determined. Figure 12 is a schematic diagram of determining the base column combinations for T1. As shown in Figure 12, assuming that in the first step, T1's local predicate column combination 1201 is determined to be {C1, C2, C3}, the join predicate column combination 1202 is determined to be {C4, C5}, and the target selection column combination 1203 is determined to be {C6, C7}, then if T1 does not have local materialization or other situations requiring separate processing of local predicate columns, the local predicate column combination and the join predicate column combination are merged to obtain a first merged combination 1204 of {C1, C2, C3, C4, C5}. If T1 does not have delayed materialization or other situations requiring separate processing of target columns, the target selection column combination 1203 and the first merged combination 1204 are merged to obtain T1's base column combination 1205 of {C1, C2, C3, C4, C5, C6, C7}.

[0298] In some embodiments, if T1 has local materialization or other situations that require separate processing of local predicate columns, the column combination corresponding to the local predicate and the column combination corresponding to the join predicate cannot be merged. In this case, if T1 does not have delayed materialization or other situations that require separate processing of target columns, the column combination corresponding to the join predicate and the target selection column combination can be merged to obtain a second merged combination {C4, C5, C6, C7}. In this case, T1 has two basic column combinations, namely {C1, C2, C3} and {C4, C5, C6, C7}. If T1 has local materialization or other situations that require separate processing of local predicate columns, and has delayed materialization or other situations that require separate processing of target columns, no merging is performed, that is, T1 has three basic column combinations, namely {C1, C2, C3}, {C4, C5}, and {C6, C7}.

[0299] After determining the basic column combinations of each table appearing in the query statement, the query overhead of executing the query statement according to different access paths when each table is merged and stored according to the basic column combinations is determined, and the query overhead corresponding to the optimal path is determined as the basic overhead of executing the query statement (corresponding to the first reference overhead in other embodiments). The basic column combinations of each table are then merged to form at least one merged basic column combination, and the query overhead of executing the query statement according to different access paths when each table is stored according to the merged basic column combination is determined, and the query overhead corresponding to the optimal path is determined as the query overhead corresponding to the merged basic column combination. After determining the query overhead corresponding to each merged basic column combination of each table, if the minimum query overhead is less than the basic overhead corresponding to the basic column combination, it indicates that merging the basic column combinations can achieve better query efficiency, and the merged basic column combination corresponding to the minimum query overhead is determined as a candidate column combination for the table. If the minimum query overhead is greater than or equal to the basic overhead corresponding to the basic column combination, it indicates that merging the basic column combinations cannot achieve better query efficiency, and the basic column combination is determined as a candidate column combination for the table.

[0300] In the embodiment of the present application, query overhead is also the overhead of reading columns. As shown in Figure 13, the overhead of reading columns can include I / O overhead and CPU overhead. The I / O overhead can be the sum of the overhead of opening files, the overhead of random reads, and the overhead of sequential reads; the CPU overhead can be the sum of the basic overhead, the overhead of reading pages, the overhead of scanning pages, and the overhead of record scanning.

[0301] Figure 14 is a schematic diagram of determining the column reading overhead when the three columns are stored separately and merged. As shown in Figure 14, when C1, C2, and C3 are stored in three different files, three files need to be opened when reading C1, C2, and C3. Assuming that the overhead of opening one file is 10 milliseconds (ms), the overhead of opening three files is 30 ms. The overhead of randomly reading one file is 1000 ms, and the overhead of randomly reading three files is 3000 ms. Since reading a large file is more likely to trigger sequential reading than reading a small file, the sequential read overhead of reading three small files (1000 ms) is less than the sequential read overhead of reading a large file (1250 ms).

[0302] Therefore, the I / O overhead of storing C1, C2 and C3 in three different files is 30+3000+1000=4030ms, while the I / O overhead of storing C1, C2 and C3 in the same file is 10+1000+1250=2260ms.

[0303] In terms of CPU overhead, the fewer files opened, the lower the basic overhead. Assuming that the file size remains unchanged when storing files separately and merging them, the overhead of reading and scanning pages is the same. For record scanning, the combined overhead of parsing the number of record columns is one-third of the total overhead of a single column.

[0304] Therefore, the CPU overhead of storing C1, C2 and C3 in three different files is 300+110+55+330=795ms, while the CPU overhead of storing C1, C2 and C3 in the same file is 100+110+55+110=375ms.

[0305] If C1, C2, and C3 are stored in three different files, the query overhead is 4030 + 795 = 4825 ms. If C1, C2, and C3 are stored in the same file, the query overhead is 2260 + 375 = 2365 ms.

[0306] In the third step, multiple candidate column combinations determined by different query statements for each table are combined again to determine the optimal column combination for each table when executing multiple query statements to achieve the minimum total query overhead.

[0307] For an application, multiple different query statements may be obtained. The candidate column combinations corresponding to the same table in different query statements may be different. Therefore, the same table may correspond to multiple different candidate column combinations. In this step, to minimize the total query overhead in the application dimension, the multiple column combinations corresponding to the same table are merged. Generally speaking, there is a certain correlation between the columns of the table, and the scale of the merged column combinations is relatively controllable.

[0308] Figure 15 is a schematic diagram of merging the three candidate column combinations of T1. As shown in Figure 15, T1 has three candidate column combinations, namely {C1, C2, C3}, {C2, C3, C6, C7}, and {C4, C5}. By merging at least two of these three candidate column combinations, four merged candidate column combinations can be obtained: merged candidate column combination 1 {C1, C2, C3, C6, C7}, merged candidate column combination 2 {C2, C3, C6, C7, C4, C5}, merged candidate column combination 3 {C1, C2, C3, C4, C5}, and merged candidate column combination 4 {C1, C2, C3, C4, C5, C6, C7}.

[0309] For other tables that appear in the query statement, each candidate column combination is merged in the above manner to obtain multiple merged candidate column combinations. Then, based on the merged candidate column combinations of each table, the query overhead of executing each query statement is determined. Based on the number of executions of each query statement, the total query overhead of executing multiple query statements is determined.

[0310] Figure 16A is a schematic diagram of determining the total query overhead of executing multiple query statements when storing table data according to a merged candidate column combination, as provided in an embodiment of the present application. As shown in Figure 16A, when the table data is stored according to the merged candidate column combination, the query overhead of executing each query statement is determined, and then the total query overhead corresponding to the merged candidate column combination is determined based on the number of executions of each query statement and the corresponding query overhead.

[0311] FIG16B is a schematic diagram of determining the column combination with the lowest total overhead provided by an embodiment of the present application. As shown in FIG16B , the total overhead of each combination is determined, and the column combination with the lowest total overhead is used as the optimal column combination of the table.

[0312] Step 4: Optimize storage based on the optimal column combination of each table.

[0313] In an embodiment of the present application, a table may correspond to one or more optimal column combinations. For example, there may be two optimal column combinations for T1, namely {C1, C2, C3}, {C4, C5, C6, C7}, that is, storing C1, C2 and C3 in one file, and storing C4, C4, C6 and C7 in one file can achieve the optimal query effect. Therefore, after obtaining the optimal column combination of each table, if a table has only one optimal column combination, then the data is directly stored based on the optimal column combination. If a table corresponds to two or more optimal column combinations, then before performing storage optimization, the sparsity of each optimal column combination of the table is first judged to determine whether there is a sparse optimal column combination.

[0314] When determining the sparsity of each optimal column combination of a table, determine whether each column in the optimal column combination is a sparse column. Figure 17 is a schematic diagram of determining sparse columns provided by an embodiment of the present application. As shown in Figure 17, if the ratio of the number of NULLs in the column data of a column to the total number of records is greater than a preset threshold, then the column can be considered to be a sparse column. If all columns in an optimal column combination are sparse columns, the optimal column combination is determined to be a sparse optimal column combination.

[0315] Since the amount of data contained in the file used to store the sparse optimal column combination is relatively small, it is easy to generate a large number of small files, which affects the efficiency of storage and reading. In an embodiment of the present application, when a table corresponds to two sparse optimal column combinations, the file used to store column data can be divided into blocks to obtain multiple file blocks, each file block stores a sparse optimal column combination, thereby realizing block storage of the sparse column combination. Figure 18 is a schematic diagram of block storage of sparse optimal column combinations provided by an embodiment of the present application. As shown in Figure 18, in a storage file, there are three file blocks, namely file block 1801, file block 1802 and file block 1803, and each of these three file blocks stores column data corresponding to a sparse optimal column combination.

[0316] If a table does not have at least two sparse optimal column combinations, storage is optimized based on the optimal column combination of the table.

[0317] The header information of the sparse optimal column combination stores information about each sparse column in the sparse optimal column combination, such as the column identifier of the sparse column. When reading the column data of a certain column, reading starts from the corresponding file offset according to the metadata information. In addition, if the column data of the sparse columns stored in the file increases, making it impossible to store all the column data in the file, the column data corresponding to the optimal column combination that no longer belongs to the sparse column can be removed from the file and stored in a new file. Storage optimization can also be re-performed to re-divide the storage column combination and modify the metadata information.

[0318] Figure 19 is a schematic diagram of a system table provided in an embodiment of the present application. This system table is used to identify the correspondence between the file (silo) used to store column data and the column. For metadata storage of sparse columns, a sparse system table 1901 is added to record and store relevant information about the sparse column, including the column name, occurrence count, and null value bitmap.

[0319] Before performing storage optimization based on the optimal column combination of a table, the optimization time is first determined based on the historical data of the application. In some embodiments, the historical data of the application can be used to determine a time period when data queries and updates are not frequent, and storage optimization is performed during this time period. When performing storage optimization, the metadata information of the table to be optimized is first determined. The metadata information includes the column identifiers included in the optimal column combination, and the format and location of the storage file of the optimal column combination. Then, according to the optimal column combination of the table, a transformation is performed, that is, the column data corresponding to each column identifier in the optimal column combination is stored in the file corresponding to the optimal column combination. In addition, when performing storage optimization, data transformation is performed in segments and the corresponding metadata information is updated. That is, each time the column data of a table to be optimized is converted, after the column data corresponding to the optimal column combination in the table to be optimized is stored in the corresponding file, the metadata information corresponding to each column identifier included in the optimal column combination in the system table is updated, so that after receiving the query statement again, the new storage format and storage file are used for data query.

[0320] In some embodiments, if data update information is received during the storage optimization process, the data update information is recorded in corresponding log information. After the storage optimization is completed, data changes are made according to the recorded log information.

[0321] After optimizing the column storage data of the database using the data storage method provided in the embodiment of the present application, the query execution time is reduced by reading the columns in combination, the number of read files is reduced and continuous reading is achieved, thereby improving data access efficiency, reducing the number of small files in sparse columns, improving file management efficiency and reducing the impact of fragmented files.

[0322] It is understandable that in the embodiments of the present application, when user information, query requests, table data and other related data are involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent must be obtained, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0323] Next, an exemplary structure of a data storage device 455 provided in an embodiment of the present application implemented as a software module is described. In some embodiments, as shown in FIG3 , the software module stored in the data storage device 455 of the memory 450 may include: a first acquisition module 4551 configured to acquire multiple query requests for an application, parse each query request, and obtain at least one column information included in each query request; a first determination module 4552 configured to determine, based on the at least one column information included in each query request, candidate column combinations for each table associated with each query request, wherein storing the table data of the table associated with the query request according to the candidate column combination minimizes the query overhead of executing the query request; a second determination module 4553 configured to determine, based on multiple candidate column combinations belonging to the same table, a target column combination for the corresponding table, wherein storing the table data of each table according to the respective target column combination minimizes the total query overhead of executing the multiple query requests; and a data storage module 4554 configured to store the table data of each table according to the target column combination.

[0324] In some embodiments, the column information includes column identifiers and column attribute information. The first acquisition module 4551 is further configured to: parse the various query requests to obtain at least one query statement; determine at least one column identifier corresponding to each query statement; determine the column attribute information corresponding to each column identifier, and the column attribute information includes one of a local predicate column attribute, a join predicate column attribute, and a target selection column attribute.

[0325] In some embodiments, the first determination module 4552 is further configured to: determine a basic column combination set of each table related to the i-th query request based on the column attribute information corresponding to each column identifier in the i-th query request, the basic column combination set including at least one basic column combination, i=1,…M, M is the total number of query requests; when the basic column combination set of the j-th table related to the i-th query request includes a basic column combination, determine the basic column combination as a candidate column combination of the j-th table, j=1,…,N, N is the total number of tables related to the i-th query request.

[0326] In some embodiments, the first determination module 4552 is further configured to: when N is 1 and the basic combination set of the j-th table includes at least two basic column combinations, merge the at least two basic column combinations of the j-th table to obtain at least one merged basic column combination of the j-th table; when the table data of the j-th table is stored according to the at least two basic column combinations, determine the first reference overhead of executing the i-th query request; when the table data of the j-th table is stored according to each merged basic column combination, determine each first query overhead of executing the i-th query request; determine the first minimum query overhead among the each first query overhead; when the first minimum query overhead is less than or equal to the first reference overhead, determine the merged basic column combination corresponding to the first minimum query overhead as the candidate column combination of the j-th table; when the first minimum query overhead is greater than the first reference overhead, determine the at least two basic column combinations as the candidate column combination of the j-th table.

[0327] In some embodiments, the first determination module 4552 is further configured to: when N is greater than 1, and the basic combination set of the j-th table includes at least two basic column combinations, merge the at least two basic column combinations of the j-th table to obtain at least one merged basic column combination of the j-th table; when there is another table including at least two basic column combinations among the N tables, merge the at least two basic column combinations of the other tables to obtain at least one merged basic column combination of the other tables; when the table data of the N tables are stored according to their respective corresponding basic column combinations, determine the second reference overhead for executing the i-th query request; when the When the table data of the N tables including at least two basic column combinations are stored according to the corresponding merged basic column combinations, and when the table data of the tables including one basic column combination are stored according to the basic column combination, the second query costs of executing the i-th query request are determined; the second minimum query cost among the second query costs is determined; when the second minimum query cost is less than or equal to the second reference cost, the column combination of each table corresponding to the second minimum query cost is determined as the candidate column combination of each table; when the second minimum query cost is greater than the second reference cost, the basic column combination of each table is determined as the candidate column combination of each table.

[0328] In some embodiments, the first determination module 4552 is further configured to: determine the path costs of executing the i-th query request according to different access paths when the table data of the j-th table is stored according to the k-th merged basic column combination, where k=1,…,P, P is the total number of merged basic column combinations of the j-th table; determine the minimum path cost among the path costs corresponding to the k-th merged basic column combination as the first query cost of executing the i-th query request when the table data of the j-th table is stored according to the k-th merged basic column combination.

[0329] In some embodiments, the first determining module 4552 is further configured to: add the column identifier of the column in the j-th table whose column attribute information is the local predicate column attribute to the local predicate column combination corresponding to the j-th table; add the column identifier of the column in the j-th table whose column attribute information is the join predicate column attribute to the join predicate column combination corresponding to the j-th table; add the column identifier of the column in the j-th table whose column attribute information is the target selection column attribute to the target selection column combination corresponding to the j-th table; when the local predicate column combination and the join predicate column combination meet the first merging condition, merge the local predicate column combination and the join predicate column combination to obtain a first merged combination;

[0330] When the first merge combination and the target selected column combination meet the second merge condition, the first merge combination and the target selected column combination are merged to obtain the basic column combination of the j-th table; and the basic column combination of the j-th table is added to the basic column combination set of the j-th table.

[0331] In some embodiments, the first determination module 4552 is further configured to: when the first merge combination and the target selection column combination do not meet the second merge condition, determine the first merge combination and the target selection column combination as the basic column combination of the j-th table respectively; and add the basic column combination of the j-th table to the basic column combination set of the j-th table.

[0332] In some embodiments, the first determination module 4552 is further configured to: when the local predicate column combination and the join predicate column combination do not satisfy a first merge condition, and the join predicate column combination and the target column combination satisfy a second merge condition, merge the join predicate column combination and the target column combination to obtain a second merge combination; determine the local predicate column combination and the second merge combination as the base column combination of the j-th table respectively; and add the base column combination of the j-th table to the base column combination set of the j-th table.

[0333] In some embodiments, the first determination module 4552 is further configured to: when the local predicate column combination and the join predicate column combination do not satisfy the first merge condition, and the join predicate column combination and the target column combination do not satisfy the second merge condition, determine the local predicate column combination, the join predicate column combination, and the target column combination as the base column combination of the j-th table, respectively; and add the base column combination of the j-th table to the base column combination set of the j-th table.

[0334] In some embodiments, the second determination module 4553 is further configured to: merge at least two candidate column combinations belonging to the same table to obtain at least one merged candidate column combination of the corresponding table; determine the total query costs of executing the multiple query requests when the table data of each table are stored according to their respective corresponding merged candidate column combinations; determine the reference total cost of executing the multiple query requests when the table data of each table are stored according to their respective corresponding candidate column combinations; determine the minimum query total cost among the query total costs; when the minimum query total cost is less than or equal to the reference total cost, determine the merged candidate column combination of each table corresponding to the minimum query total cost as the target column combination of each table.

[0335] In some embodiments, the second determination module 4553 is further configured to: determine the third query overhead of executing each query request when the table data of each table is stored respectively according to their corresponding merged candidate column combinations; obtain the number of executions of each query request; and determine the total query overhead of executing the multiple query requests based on the number of executions of each query request and the third query overhead corresponding to each query request.

[0336] In some embodiments, the data storage module 4554 is further configured to: when there are at least two sparse target column combinations in the target column combination of the s-th table, merge the at least two sparse target column combinations to obtain at least one merged sparse target column combination, where s=1, 2, ..., Q, and Q is the total number of tables to be optimized; determine new metadata information corresponding to each sparse target column identifier in the at least one merged sparse target column combination, the new metadata information including the file identifier, file format, and storage location of the storage file corresponding to the column data; based on the metadata information corresponding to each sparse target column identifier, store the column data corresponding to each sparse target column identifier in the storage file; corresponding storage files; updating the metadata information corresponding to the respective sparse target column identifiers in the system table to the new metadata information corresponding to the respective sparse target column identifiers; when the sth table also includes other target column combinations in addition to the at least two sparse target column combinations, determining the new metadata information corresponding to the respective other target column identifiers in the other target column combinations; based on the new metadata information corresponding to the respective other target column identifiers in the other target column combinations, storing the column data corresponding to the respective other target column identifiers to the corresponding storage files; updating the metadata information corresponding to the respective other target column identifiers in the system table to the new metadata information corresponding to the respective other target column identifiers.

[0337] In some embodiments, the data storage module 4554 is further configured to: when the target column combination of the sth table does not include at least two sparse target column combinations, determine the new metadata information corresponding to each target column identifier in the target column combination; based on the metadata information corresponding to each target column identifier, store the column data corresponding to each target column identifier to the corresponding storage file; and update the metadata information corresponding to each target column identifier in the system table to the new metadata information corresponding to each target column identifier.

[0338] In some embodiments, the device also includes: a second acquisition module, configured to obtain each column identifier in the target column combination, and obtain the number of empty data and the total number of data in the column data corresponding to each column identifier; a third determination module, configured to determine the ratio of the number of empty data corresponding to each column identifier to the total number of data as the sparsity value corresponding to each column identifier; a fourth determination module, configured to determine the column corresponding to the column identifier whose sparsity value is greater than the sparsity threshold as a sparse column; a fifth determination module, configured to determine the target column combination as a sparse target column combination when the columns corresponding to each column identifier in the target column combination are all sparse columns.

[0339] In some embodiments, the device also includes: an information storage module, configured to store data update information in an update log when data update information is received during the data storage process; a data update module, configured to update the table data based on the data update information in the update log after completing the data storage to obtain updated table data; and an information deletion module, configured to delete the data update information from the update log.

[0340] An embodiment of the present application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the data storage method described above in the embodiment of the present application.

[0341] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, which stores computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the data storage method provided by the embodiment of the present application, for example, the data storage method shown in Figures 4 and 10.

[0342] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0343] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0344] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, e.g., in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0345] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0346] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A data storage method, applied to an electronic device, comprising: Acquire multiple query requests for the application, parse each of the query requests, and obtain at least one column of information included in each of the query requests; Based on at least one column information included in each query request, determine a candidate column combination of each table related to each query request, wherein when table data of the table related to the query request is stored according to the candidate column combination, query overhead of executing the query request is minimized; Determining a target column combination of a corresponding table based on multiple candidate column combinations belonging to the same table, wherein when table data of each table is stored according to the respective target column combinations, a total query overhead of executing the multiple query requests is minimized; The table data of each table is stored according to the target column combination.

2. The method according to claim 1, wherein: The column information includes a column identifier and column attribute information, and the parsing of each query request to obtain at least one column information included in each query request includes: Parsing each query request to obtain at least one query statement; Determine at least one column identifier corresponding to each query statement; Column attribute information corresponding to each column identifier is determined, where the column attribute information includes one of a local predicate column attribute, a join predicate column attribute, and a target selection column attribute.

3. The method according to claim 1 or 2, wherein: The determining, based on at least one column information included in each query request, candidate column combinations corresponding to each table related to each query request includes: Based on the column attribute information corresponding to each column identifier in the i-th query request, determine a basic column combination set of each table related to the i-th query request, wherein the basic column combination set includes at least one basic column combination, i=1, ...M, and M is the total number of query requests; When the basic column combination set of the jth table related to the i-th query request includes a basic column combination, the basic column combination is determined as the candidate column combination of the j-th table, j=1,…,N, N is the total number of tables related to the i-th query request.

4. The method according to any one of claims 1 to 3, wherein: The determining, based on at least one column information included in each query request, candidate column combinations corresponding to each table related to each query request includes: When N is 1, and the basic combination set of the j-th table includes at least two basic column combinations, merging the at least two basic column combinations of the j-th table to obtain at least one merged basic column combination of the j-th table; In a case where the table data of the j-th table is stored according to the combination of the at least two basic columns, determining a first reference cost for executing the i-th query request; Under the condition that the table data of the j-th table is stored according to each combined basic column, determining each first query cost of executing the i-th query request; Determining a first minimum query cost among the first query costs; When the first minimum query cost is less than or equal to the first reference cost, determining the merged basic column combination corresponding to the first minimum query cost as the candidate column combination of the j-th table; When the first minimum query cost is greater than the first reference cost, the at least two basic column combinations are determined as candidate column combinations of the j-th table.

5. The method according to any one of claims 1 to 4, wherein: The determining, based on at least one column information included in each query request, candidate column combinations corresponding to each table related to each query request includes: When N is greater than 1, and the basic combination set of the j-th table includes at least two basic column combinations, merging the at least two basic column combinations of the j-th table to obtain at least one merged basic column combination of the j-th table; When there is another table including at least two basic column combinations among the N tables, merging the at least two basic column combinations of the other table to obtain at least one merged basic column combination of the other table; In a case where the table data of the N tables are stored according to respective corresponding basic column combinations, determining a second reference cost for executing the i-th query request; In the case where table data of a table including at least two basic column combinations among the N tables are stored according to the corresponding merged basic column combinations, and table data of a table including one basic column combination are stored according to the basic column combinations, determining each second query cost of executing the i-th query request; Determining a second minimum query cost among the second query costs; When the second minimum query cost is less than or equal to the second reference cost, each table corresponding to the second minimum query cost Determine the column combination as the candidate column combination of each table; When the second minimum query cost is greater than the second reference cost, the basic column combinations of the respective tables are determined as candidate column combinations of the respective tables.

6. The method according to any one of claims 1 to 5, wherein: The determining of each first query cost of executing the i-th query request when the table data of the j-th table is stored according to each combined basic column combination includes: When the table data of the j-th table is stored according to the k-th combined basic column combination, determining the path costs corresponding to executing the i-th query request respectively according to a plurality of different access paths, where k=1, ..., P, and P is the total number of combined basic columns of the j-th table; The minimum path cost among the path costs corresponding to the kth merged basic column combination is determined as the first query cost of executing the i-th query request when the table data of the j-th table is stored according to the kth merged basic column combination.

7. The method according to any one of claims 1 to 6, wherein: Based on the column attribute information corresponding to each column identifier in the i-th query request, determining a basic column combination set of each table related to the i-th query request includes: Adding the column identifier of the column whose column attribute information in the j-th table is the local predicate column attribute to the local predicate column combination corresponding to the j-th table; Adding the column identifier of the column whose column attribute information in the j-th table is the join predicate column attribute to the join predicate column combination corresponding to the j-th table; Add the column identifier of the column whose column attribute information in the j-th table is the target selection column attribute to the target selection column combination corresponding to the j-th table; When the local predicate column combination and the join predicate column combination meet a first merging condition, merging the local predicate column combination and the join predicate column combination to obtain a first merged combination; When the first merging combination and the target selected column combination meet a second merging condition, merging the first merging combination and the target selected column combination to obtain a basic column combination of the j-th table; The basic column combination of the j-th table is added to the basic column combination set of the j-th table.

8. The method according to any one of claims 1 to 7, wherein: Based on the column attribute information corresponding to each column identifier in the i-th query request, determining a basic column combination set of each table related to the i-th query request includes: When the first merging combination and the target selected column combination do not satisfy the second merging condition, the first merging combination and the target selected column combination are respectively determined as the basic column combination of the j-th table; The basic column combination of the j-th table is added to the basic column combination set of the j-th table.

9. The method according to any one of claims 1 to 8, wherein: The determining, based on the column attribute information corresponding to each column identifier in the i-th query request, a basic column combination set of each table related to the i-th query request includes: When the local predicate column combination and the join predicate column combination do not satisfy a first merging condition, and the join predicate column combination and the target column combination satisfy a second merging condition, merging the join predicate column combination and the target column combination to obtain a second merged combination; Determine the local predicate column combination and the second merged combination as the basic column combination of the j-th table respectively; The basic column combination of the j-th table is added to the basic column combination set of the j-th table.

10. The method according to any one of claims 1 to 9, wherein: The determining, based on the column attribute information corresponding to each column identifier in the i-th query request, a basic column combination set of each table related to the i-th query request includes: When the local predicate column combination and the join predicate column combination do not satisfy the first merging condition, and the join predicate column combination and the target column combination do not satisfy the second merging condition, the local predicate column combination, the join predicate column combination and the target column combination are respectively determined as the basic column combination of the j-th table; The basic column combination of the j-th table is added to the basic column combination set of the j-th table.

11. The method according to any one of claims 1 to 10, wherein: The determining the target column combination of the corresponding table based on multiple candidate column combinations belonging to the same table includes: Merging at least two candidate column combinations belonging to the same table to obtain at least one merged candidate column combination of the corresponding table; Determine the total query costs of executing the multiple query requests when the table data of each table is stored according to the respective corresponding merged candidate column combinations; Determining a reference total cost of executing the plurality of query requests when the table data of the respective tables are stored according to the respective corresponding candidate column combinations; Determine a minimum total query cost among the total query costs; When the minimum query total cost is less than or equal to the reference total cost, the merged candidate column combination of each table corresponding to the minimum query total cost is determined as the target column combination of each table.

12. The method according to any one of claims 1 to 11, wherein: The step of determining the total query costs of executing the plurality of query requests when the table data of each table is stored according to the respective corresponding merged candidate column combinations includes: Determine each third query cost of executing each query request when the table data of each table is stored according to the corresponding merged candidate column combinations; Obtaining the execution times of each query request; Based on the number of executions of each query request and each third query cost corresponding to each query request, a total query cost of executing the plurality of query requests is determined.

13. The method according to any one of claims 1 to 13, wherein: The storing the table data of each table according to the target column combination includes: When there are at least two sparse target column combinations in the target column combination of the sth table, the at least two sparse target column combinations are merged to obtain at least one merged sparse target column combination, where s=1, 2, ..., Q, where Q is the total number of tables to be optimized; Determine new metadata information corresponding to each sparse target column identifier in the at least one merged sparse target column combination, wherein the new metadata information includes a file identifier, a file format, and a storage location of a storage file corresponding to the column data; Based on the metadata information corresponding to each sparse target column identifier, storing the column data corresponding to each sparse target column identifier in a corresponding storage file; Update the metadata information corresponding to each sparse target column identifier in the system table to new metadata information corresponding to each sparse target column identifier; When the s-th table also includes other target column combinations except the at least two sparse target column combinations, determining new metadata information corresponding to each other target column identifier in the other target column combinations; Based on the new metadata information corresponding to each other target column identifier in the other target column combination, storing the column data corresponding to each other target column identifier in the corresponding storage file; The metadata information corresponding to each other target column identifier in the system table is updated to new metadata information corresponding to each other target column identifier.

14. The method according to any one of claims 1 to 13, wherein: The storing the table data of each table according to the target column combination includes: When the target column combination of the s-th table does not include at least two sparse target column combinations, determining new metadata information corresponding to each target column identifier in the target column combination; Based on the metadata information corresponding to each target column identifier, storing the column data corresponding to each target column identifier in a corresponding storage file; The metadata information corresponding to each target column identifier in the system table is updated to new metadata information corresponding to each target column identifier.

15. The method according to any one of claims 1 to 14, wherein: The method further comprises: Obtaining each column identifier in the target column combination, and obtaining the number of empty data and the total number of data in the column data corresponding to each column identifier; Determine the ratio of the number of empty data corresponding to each column identifier to the total number of data as the sparsity value corresponding to each column identifier; Determine the column corresponding to the column identifier whose sparsity value is greater than the sparsity threshold as a sparse column; When the columns corresponding to the column identifiers in the target column combination are all sparse columns, the target column combination is determined to be a sparse target column combination.

16. The method according to any one of claims 1 to 15, wherein: The method further comprises: When data update information is received during the data storage process, the data update information is stored in the update log; After the data storage is completed, the table data is updated based on the data update information in the update log to obtain updated table data; The data update information is deleted from the update log.

17. A data storage device, the device comprising: A first acquisition module is configured to acquire multiple query requests for an application, parse each query request, and obtain at least one column information included in each query request; a first determining module configured to determine, based on at least one column information included in each query request, candidate column combinations of each table related to each query request, wherein when table data of the table related to the query request is stored according to the candidate column combination, query overhead of executing the query request is minimized; A second determination module is configured to determine a target column combination of a corresponding table based on a plurality of candidate column combinations belonging to the same table, wherein when the table data of each table is stored according to the respective target column combinations, a total query overhead of executing the plurality of query requests is minimized; The data storage module is configured to store the table data of each table according to the target column combination.

18. An electronic device, comprising: A memory for storing computer executable instructions; A processor, used to implement the data storage method according to any one of claims 1 to 16 when executing the computer executable instructions stored in the memory.

19. A computer-readable storage medium storing computer-executable instructions or a computer program, wherein the computer-executable instructions or the computer program, when executed by a processor, implements the data storage method according to any one of claims 1 to 16.

20. A computer program product, comprising computer executable instructions or a computer program, wherein the computer executable instructions or the computer program, when executed by a processor, can implement the data storage method according to any one of claims 1 to 16.

Images