A data storage method, device, electronic equipment and storage medium

Abstract

The application discloses a data storage method and device, electronic equipment and storage medium. The method comprises the following steps: obtaining a to-be-stored data table; determining the data amount of the to-be-stored data table and a service type matched with the to-be-stored data table; wherein the service type comprises online and / or batch; determining the storage mode of the to-be-stored data table according to the data amount of the to-be-stored data table and the service type; the technical scheme quantifies the data amount of the to-be-stored data table and matches the service type of the to-be-stored data table, reduces the emergence of redundancy during data export, reduces unnecessary export and import pressure, and reduces the resource occupancy rate of the equipment. The to-be-stored data table is matched with a reasonable storage mode, and the complexity of the storage architecture is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data storage method and device, electronic equipment and a storage medium. BACKGROUND

[0002] There are two major categories of information processing methods: online and batch. Online, also known as online transaction processing, is suitable for single event transactions and can quickly feedback processing results with strong real-time interactivity. Online generally uses traditional relational databases such as Oracle, DB2, etc. Such databases use row storage, and data is logically stored in rows as a storage unit. The data in a row is stored in a continuous form in the storage medium. Batch, also known as online analytical processing, is suitable for a large number of work tasks and has relatively long processing time and poor real-time performance, but is more efficient and saves resource consumption. Batch generally uses analytical column storage databases such as GBase, Hbase, ClickHouse, etc. Such databases use column storage, and data is logically stored in columns as a storage unit. The data in a column is stored in a continuous form in the storage medium.

[0003] In order to meet the business needs of online and batch at the same time, a system usually needs to deploy an online database for online transactions and a batch database for batch processing. This setup requires considering two sets of databases during design, development, operation, and maintenance, which has high technical threshold and cost, and frequently exporting and importing data from the database consumes a lot of central processing unit, memory, and other resources. The prior art provides a TiDB database that simultaneously performs row storage and column storage on all data tables, and uses a data synchronization mechanism between the two sets of data. The row storage data runs online, and the column storage data runs batch. However, this method has high data synchronization pressure and data redundancy, which wastes storage resources. SUMMARY

[0004] The present application provides a data storage method, device, electronic equipment and storage medium to realize mixed storage of row and column, reduce the complexity of the database architecture and the data import and export pressure, and reduce the data synchronization pressure and data redundancy.

[0005] In a first aspect, an embodiment of the present application provides a data storage method, which comprises:

[0006] obtaining a data table to be stored;

[0007] determining the data volume of the data table to be stored and the business type matched with the data table to be stored;

[0008] The service types include online and / or batch processing;

[0009] Determine the storage method for the data table to be stored based on the data volume and business type of the data table to be stored.

[0010] Secondly, embodiments of the present invention also provide a data storage device, the device comprising:

[0011] The data table acquisition module is used to acquire the data table to be stored.

[0012] The attribute determination module is used to determine the amount of data in the data table to be stored, and the business type that matches the data table to be stored;

[0013] The service types include online and / or batch processing;

[0014] The storage method determination module is used to determine the storage method of the data table to be stored based on the data volume and business type of the data table to be stored.

[0015] The storage method includes row-based storage, column-based storage, or a hybrid row-column storage.

[0016] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data storage method as described in any of the embodiments of the present invention.

[0017] Fourthly, embodiments of the present invention also provide a storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform any of the data storage methods described in the embodiments of the present invention.

[0018] The technical solution of this invention involves acquiring a data table to be stored, determining the data volume of the data table using a data volume calculation method, and matching the business type to the data table according to the user's actual operational needs. Based on the data volume and business type of the data table, the storage method for the data table is determined. This embodiment of the application determines the storage method for the data table by quantifying its data volume and matching it with its business type. By rationally matching the storage method to the data table, the pressure of data export and import is reduced, and the complexity of the storage architecture is decreased.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a data storage method provided in Embodiment 1 of the present invention;

[0022] Figure 2 This is a flowchart of a data storage method provided in Embodiment 2 of the present invention;

[0023] Figure 3 This is a schematic diagram of the structure of a data storage device provided in Embodiment 3 of the present invention;

[0024] Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] Example 1

[0028] Figure 1The flowchart of a data storage method provided in Embodiment 1 of the present invention is applicable to situations where the data storage method is determined. The method can be executed by a data storage device, which can be implemented in hardware and / or software and can be configured in any device with storage and computing capabilities.

[0029] like Figure 1 As shown, the method includes:

[0030] S110. Obtain the data table to be stored;

[0031] The data table to be stored can be a data table that will be stored in the database.

[0032] For example, if the data table to be stored is in a relational database, it is necessary to find out which database the data table should be in, retrieve and obtain the data table to be stored. If the current data is stored in text form, it is necessary to write a program to generate the data table to be stored, and convert each piece of data in the text into the form specified by the data table to be stored.

[0033] S120. Determine the data volume of the data table to be stored, and the service type that matches the data table to be stored; wherein, the service type includes online and / or batch processing;

[0034] Here, data volume can refer to the total number of data rows in a data table stored on storage media. Online processing can be a process where a terminal that generates data inputs data into the system, the system processes the data, and then directly transmits the data to the device that needs it. Batch processing can be an online analytical processing operation that handles a large number of tasks.

[0035] Optionally, determining the amount of data in the data table to be stored may include: determining the number of fields in the data table to be stored, the field data length corresponding to each field, and the number of data rows; and determining the amount of data in the data table to be stored based on the number of fields, the field data length corresponding to each field, and the number of data rows.

[0036] The number of fields can be the total number of attribute columns in the stored data table. The field data length can be the length of bytes allocated to this field by the computer. The number of data rows can be the total number of rows in the stored data table.

[0037] Specifically, the number of attribute columns in the data table to be stored is calculated as the number of fields. The data length of each field is determined based on its data type. The number of rows in the table is then determined by counting each record. Optionally, weights can be assigned to the number of fields, field data length, and number of data rows. A value is assigned to the number of fields matching the number of fields, a value to the field data length matching the field's data length, and a value to the number of data rows matching the number of data rows. The sum of the products of each value and its weight is used as the overall data volume evaluation value of the data table to be stored.

[0038] For example, the data volume of the data table to be stored is determined according to the above method. Then, based on the customer needs that the data table needs to support, such as users needing to quickly obtain feedback results for real-time interactive operations, if the online service type can meet the user's needs, the service type of the data table to be stored is determined to be online. Alternatively, if users need to perform online analysis and processing of a large number of tasks based on the data table to be stored without real-time interaction, if the batch service type can meet the user's needs, the service type of the data table to be stored is determined to be batch.

[0039] S130. Determine the storage method of the data table to be stored based on the data volume and business type of the data table to be stored; wherein, the storage method includes row storage, column storage, or a hybrid row-column storage.

[0040] Storage methods refer to how data is stored in storage space, differentiated by the physical address distribution of the data within that space. Row-based storage is built upon the operating system's file system, storing data row-wise as the basic logical storage unit. Databases using row-based storage include Oracle, DB2, MySQL, and SQL Server. Column-based storage, in contrast to row-based storage, stores data column-wise as the basic logical storage unit, with each column stored contiguously. Databases using column-based storage include distributed databases such as HBase. Hybrid row- and column-based storage involves using both row-based and column-based storage for the data table. This method distributes the data table across both row-based and column-based databases, resulting in some data redundancy. Furthermore, data modification, addition, and deletion require synchronization between the row-based and column-based storage tables.

[0041] Specifically, the above steps determine the amount of data in the table to be stored, and the business type of the table to be stored is determined according to customer needs. It is also determined whether the table to be stored is stored in column form or row form.

[0042] For example, when writing data in rows, row storage stores a row at a time, while column storage stores each field value of a row sequentially. Therefore, column storage is more time-consuming than row storage. When modifying data, row storage modifies the data at a specified location at a time, while the address access complexity of column storage is several times that of row storage. When reading data, row storage is prone to redundant columns when reading several columns from a stored data table, while column storage reads a portion or all of the data set, eliminating redundancy issues. Since online operations are suitable for reading one or a few rows, row storage is appropriate. Batch processing, suitable for handling many rows of data from a single column, is suitable for column storage.

[0043] The technical solution of this invention involves acquiring a data table to be stored, determining the data volume of the data table using a data volume calculation method, and matching the business type to the data table according to the user's actual operational needs. Based on the data volume and business type of the data table, the storage method for the data table is determined. This embodiment of the application determines the storage method for the data table by quantifying its data volume and matching it with its business type. By rationally matching the storage method to the data table, the pressure of data export and import is reduced, and the complexity of the storage architecture is decreased.

[0044] Example 2

[0045] Figure 2 This is a flowchart of a data storage method provided in Embodiment 2 of the present invention. Based on the above embodiments, the present invention further specifies the method for selecting the storage method of the data table to be stored.

[0046] like Figure 2 As shown, the method includes:

[0047] S210. Obtain the data table to be stored;

[0048] Specifically, based on the current storage address of the data table to be stored, the data table to be stored is retrieved and exported, or the data table to be stored is generated by transforming the data structure of the current data to be stored.

[0049] S220. Determine the data volume of the data table to be stored, and the service type that matches the data table to be stored; wherein, the service type includes online and / or batch processing;

[0050] Specifically, the data volume of the data table to be stored is determined based on the method for determining the data volume of the data table to be stored. The business type matching the data table to be stored is determined based on the type of customer needs that the data table to be stored needs to respond to. The methods for determining the data volume and business type have been described in the above embodiments and will not be repeated here.

[0051] S230. Determine whether the amount of data in the data table to be stored is less than a preset first threshold. If yes, execute S240; otherwise, execute S250.

[0052] The preset first threshold refers to a value that is pre-set to be the same as the data type of the data table to be stored. It can be set based on historical data or in response to user input data.

[0053] Specifically, the data volume of the data table to be stored is calculated according to the data volume calculation method. The data volume of the data table to be stored is then compared with a preset first threshold.

[0054] S240. Determine that the storage method of the data table to be stored is row-based storage;

[0055] Specifically, if the amount of data in the data table to be stored is less than a preset first threshold, then the storage method for the data table to be stored is determined to be row storage.

[0056] S250. Determine whether the amount of data in the data table to be stored is greater than or equal to a preset second threshold. If yes, execute S260; otherwise, execute S270.

[0057] In this embodiment, the determination of data size is based on the example where the first threshold and the second threshold are the same. It should be noted that the first threshold and the second threshold can be different. When the first threshold and the second threshold are different, if the data volume of the table to be stored is greater than or equal to the first threshold and less than the second threshold, the storage method can be flexibly determined based on the current system's processing performance, memory size, etc., or based on the business type of the table to be stored.

[0058] The preset second threshold refers to a value that is consistent with the data volume and data type of the data table to be stored, distinguishing it from the preset first threshold. It can be set based on historical data distribution patterns or in response to user input. The preset second threshold can be equal to or different from the preset first threshold.

[0059] Specifically, compare the size relationship between the data volume of the data table to be stored and the preset second threshold. S260. Determine the storage method of the data table to be stored based on its business type;

[0060] As an optional but not limited implementation method, the storage method of the data table to be stored is determined according to the business type of the data table to be stored, and may include, but is not limited to, the following five cases A1-A5:

[0061] Step A1: If the business type of the data table to be stored is determined to be online, then the storage method of the data table to be stored is determined to be row-based storage;

[0062] For example, when a user needs to quickly obtain feedback results for real-time interactive operations, and the online service type can meet the user's needs, then the service type of the data table to be stored is determined to be online. Since online is suitable for operations that read one or a few rows, row-based storage is appropriate.

[0063] Step A2: If the business type of the data table to be stored is determined to be batch, then the storage method of the data table to be stored is determined to be columnar storage.

[0064] For example, when a user needs to process a large number of tasks for online analytical processing (OLAP), the batch business type can meet the user's needs, so the business type of the data table to be stored is determined to be batch. A large number of tasks consists of multiple rows of data in a single column, and since batch processing is suitable for processing many rows of data in a single column, columnar storage is appropriate.

[0065] Step A3: If the business type of the data table to be stored is determined to be online and batch, and the difference between the number of online operations and the number of batch operations of the data table to be stored is greater than or equal to the preset third threshold, then the storage method of the data table to be stored is determined to be row-based storage.

[0066] The preset third threshold can be a preset data type that is consistent with the difference between the number of online operations and the number of batch operations of the data table to be stored. It can be set according to the distribution of historical data or in response to user settings.

[0067] Specifically, the number of online operations and the number of batch operations on the data table to be stored can be determined. When the difference between the number of online operations and the number of batch operations on the data table to be stored is greater than or equal to a preset third threshold, that is, when the number of read operations performed by the user on one or more rows is greater than or equal to the number of read operations performed by the user on multiple rows of column data, the data table to be stored is determined to be stored in row-based storage.

[0068] Step A4: If the business type of the data table to be stored is determined to be online and batch, and the difference between the number of batch operations and the number of online operations of the data table to be stored is greater than or equal to the preset fourth threshold, then the storage method of the data table to be stored is determined to be columnar storage; wherein, the third threshold is the same as or different from the fourth threshold.

[0069] The preset fourth threshold can be a preset data type that is consistent with the difference between the number of online operations and the number of batch operations of the data table to be stored. It can be set according to the distribution of historical data or in response to user settings. The preset fourth threshold can be equal to or different from the preset third threshold.

[0070] Specifically, when the difference between the number of batch operations and the number of online operations on the data table to be stored is greater than or equal to the preset fourth threshold, that is, when the number of column data read operations performed by the user is greater than or equal to the number of read operations performed by the user on one or more rows, the data table to be stored is determined to be stored in row-based storage.

[0071] Step A5: If the business type of the data table to be stored is determined to be online and batch, and the absolute value of the difference between the number of online operations and the number of batch operations of the data table to be stored is less than or equal to the fifth threshold, then the storage method of the data table to be stored is determined to be a hybrid row-column storage.

[0072] The fifth threshold can be data of the same type as the absolute value of the difference between the number of online operations and the number of batch operations of the data table to be stored.

[0073] For example, when the absolute value of the difference between the number of online operations and the number of batch operations of the data table to be stored is less than or equal to the fifth threshold, it indicates that the number of online operations and the number of batch operations of the data table to be stored are similar. That is, when the difference between the number of read operations performed by a user on one or more rows and the number of read operations performed by a user on multiple rows of column data is small, the storage method of the data table to be stored is determined to be a hybrid row-column storage.

[0074] S270, End.

[0075] The technical solution of this invention involves determining the data volume of the data table to be stored using a data volume calculation method, and matching the business type to the data table according to the user's actual operational needs. Based on the data volume and business type of the data table to be stored, the storage method of the data table is determined. This embodiment refines the operation of matching the storage method of the data table to be stored based on its data volume and business type, matching a reasonable storage method to the data table to be stored, reducing data redundancy when importing multi-row, single-column data, reducing unnecessary export and import pressure, and minimizing device resource usage.

[0076] Example 3

[0077] Figure 3 This is a schematic diagram of a data storage device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes:

[0078] Data table acquisition module 310 is used to acquire the data table to be stored;

[0079] The attribute determination module 320 is used to determine the amount of data in the data table to be stored, and the business type that matches the data table to be stored;

[0080] The service types include online and / or batch processing;

[0081] The storage method determination module 330 is used to determine the storage method of the data table to be stored based on the data volume and business type of the data table to be stored.

[0082] The storage method includes row-based storage, column-based storage, or a hybrid row-column storage.

[0083] In this embodiment of the application, the attribute determination module 320 includes:

[0084] The data table attribute determination unit is used to determine the number of fields in the data table to be stored, the field data length corresponding to each field, and the number of data rows;

[0085] The data volume determination unit is used to determine the data volume of the data table to be stored based on the number of fields, the field data length corresponding to each field, and the number of data rows.

[0086] In this embodiment of the application, the device further includes:

[0087] The first determining module is used to determine that the storage method of the data table to be stored is row-based storage if the amount of data in the data table to be stored is less than a preset first threshold.

[0088] In this embodiment of the application, the storage method determination module 330 includes:

[0089] The second determining unit is used to determine the storage method of the data table to be stored based on the business type of the data table to be stored if the amount of data in the data table to be stored is greater than or equal to a preset second threshold.

[0090] Wherein, the first threshold may be the same as or different from the second threshold.

[0091] In this embodiment of the application, the second determining unit is specifically used for:

[0092] If the business type of the data table to be stored is determined to be online, then the storage method of the data table to be stored is determined to be row-based storage;

[0093] If the business type of the data table to be stored is determined to be batch, then the storage method of the data table to be stored is determined to be columnar storage.

[0094] If the business type of the data table to be stored is determined to be online and batch, and the difference between the number of online operations and the number of batch operations of the data table to be stored is greater than or equal to the preset third threshold, then the storage method of the data table to be stored is determined to be row-based storage.

[0095] If the business type of the data table to be stored is determined to be online and batch, and the difference between the number of batch operations and the number of online operations of the data table to be stored is greater than or equal to the preset fourth threshold, then the storage method of the data table to be stored is determined to be columnar storage.

[0096] The third threshold may be the same as or different from the fourth threshold.

[0097] If the business type of the data table to be stored is determined to be online and batch, and the absolute value of the difference between the number of online operations and the number of batch operations of the data table to be stored is less than or equal to the fifth threshold, then the storage method of the data table to be stored is determined to be a hybrid row-column storage.

[0098] The data storage device provided in the embodiments of the present invention can execute the data storage method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0099] Example 4

[0100] Figure 4 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0101] like Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0102] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0103] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data storage methods.

[0104] In some embodiments, the data storage method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data storage method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the data storage method by any other suitable means (e.g., by means of firmware).

[0105] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0106] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0107] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0108] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0109] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0110] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0111] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0112] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data storage method, characterized by, The method comprises: obtaining a to-be-stored data table; determining the data amount of the to-be-stored data table and the service type matched with the to-be-stored data table; wherein the service type comprises online and / or batch; determining the storage mode of the to-be-stored data table according to the data amount of the to-be-stored data table and the service type; wherein the storage mode comprises row storage, column storage or row-column hybrid storage, the row-column hybrid storage is to store the to-be-stored data table in row storage and column storage at the same time, and the to-be-stored data table is distributed in the row storage database and the column storage database; wherein after determining the data amount of the to-be-stored data table, the method further comprises: if it is determined that the data amount of the to-be-stored data table is less than a preset first threshold, determining that the storage mode of the to-be-stored data table is row storage; wherein the determination of the storage mode of the to-be-stored data table according to the data amount of the to-be-stored data table and the service type comprises: if it is determined that the data amount of the to-be-stored data table is greater than or equal to a preset second threshold, determining the storage mode of the to-be-stored data table according to the service type of the to-be-stored data table; wherein the first threshold and the second threshold are the same or different; wherein the determination of the storage mode of the to-be-stored data table according to the service type of the to-be-stored data table comprises: if it is determined that the service type of the to-be-stored data table is online and batch, and the difference between the online operation amount and the batch operation amount of the to-be-stored data table is greater than or equal to a preset third threshold, determining that the storage mode of the to-be-stored data table is row storage; if it is determined that the service type of the to-be-stored data table is online and batch, and the difference between the batch operation amount and the online operation amount of the to-be-stored data table is greater than or equal to a preset fourth threshold, determining that the storage mode of the to-be-stored data table is column storage; wherein the third threshold and the fourth threshold are the same or different; if it is determined that the service type of the to-be-stored data table is online and batch, and the absolute value of the difference between the online operation amount and the batch operation amount of the to-be-stored data table is less than or equal to a fifth threshold, determining that the storage mode of the to-be-stored data table is row-column hybrid storage.

2. The method of claim 1, wherein, The determination of the data amount of the to-be-stored data table comprises: determining the field quantity of the to-be-stored data table, the field data length corresponding to each field, and the data quantity; determining the data amount of the to-be-stored data table according to the field quantity, the field data length corresponding to each field, and the data quantity.

3. The method of claim 1, wherein, The determination of the storage mode of the to-be-stored data table according to the service type of the to-be-stored data table comprises: if it is determined that the service type of the to-be-stored data table is online, determining that the storage mode of the to-be-stored data table is row storage; if it is determined that the service type of the to-be-stored data table is batch, determining that the storage mode of the to-be-stored data table is column storage.

4. A data storage device, characterized by The method comprises: a data table obtaining module for obtaining a to-be-stored data table; an attribute determining module for determining the data amount of the to-be-stored data table and the service type matched with the to-be-stored data table; wherein the service type comprises online and / or batch; The storage mode determination module is configured to determine the storage mode of the to-be-stored data table according to the data volume of the to-be-stored data table and the service type of the to-be-stored data table. The storage mode includes row storage, column storage, or row-column hybrid storage, the row-column hybrid storage is to store the to-be-stored data table in the row storage and the column storage, and the to-be-stored data table is distributed in the row storage database and the column storage database. The device further includes: The first determination module is configured to determine the storage mode of the to-be-stored data table as the row storage if the data volume of the to-be-stored data table is less than a preset first threshold. The storage mode determination module includes: The second determination unit is configured to determine the storage mode of the to-be-stored data table according to the service type of the to-be-stored data table if the data volume of the to-be-stored data table is greater than or equal to a preset second threshold. The first threshold and the second threshold are the same or different. The second determination unit is specifically configured to: determine the storage mode of the to-be-stored data table as the row storage if the service type of the to-be-stored data table is online and batch, and the difference between the online operation quantity and the batch operation quantity of the to-be-stored data table is greater than or equal to a preset third threshold; determine the storage mode of the to-be-stored data table as the column storage if the service type of the to-be-stored data table is online and batch, and the difference between the batch operation quantity and the online operation quantity of the to-be-stored data table is greater than or equal to a preset fourth threshold; The third threshold and the fourth threshold are the same or different. determine the storage mode of the to-be-stored data table as the row-column hybrid storage if the service type of the to-be-stored data table is online and batch, and the absolute value of the difference between the online operation quantity and the batch operation quantity of the to-be-stored data table is less than or equal to a fifth threshold.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the data storage method of any one of claims 1-3.

6. A storage medium storing computer-executable instructions, wherein: The computer executable instructions are executed by the computer processor to implement the data storage method of any one of claims 1-3.

Images