Data storage method, apparatus, medium, and computing device

Abstract

The embodiment of the present disclosure provides a data storage method, device, medium and computing equipment, the method comprises: acquiring various enumeration values of a service, the enumeration values comprising a data query mode, a query response time length and a data volume of a data table where query data is located; determining a target storage engine matched with the various enumeration values, and acquiring a service level corresponding to the service; determining a target database corresponding to the service level in the target storage engine, and creating a data table in the target database. In the present disclosure, the target storage engine for storing data is determined according to the data query mode of the service, the query response time length and the data volume of the data table where the query data is located, so that the data table is created in the target database corresponding to the service level in the target storage engine to store data, which does not rely on manual selection of the storage engine and the database in the storage engine, shortens the selection time of the storage engine and the database, and improves the data storage efficiency.

Description

Technical Field

[0001] The embodiments of this disclosure relate to the field of big data technology, and more specifically, the embodiments of this disclosure relate to data storage methods, apparatus, media and computing devices. Background Technology

[0002] This section is intended to provide background or context for embodiments of this disclosure. The description herein is not intended to imply that it is prior art simply because it is included in this section.

[0003] Before performing data queries, data needs to be stored in a database. When storing data, a storage engine needs to be selected, and the data to be stored is then stored in the database of that storage engine.

[0004] In the exemplary technology, the selection of the storage engine and the database within the storage engine are done manually by technical personnel. That is, the selection of the storage engine and the database depends on the experience of the technical personnel. If the technical personnel are not experienced enough, it will take a long time to select the storage engine and the database, resulting in a long data storage time and low data storage efficiency. Summary of the Invention

[0005] This disclosure provides a data storage method, apparatus, medium, and computing device to address the problem of low data storage efficiency.

[0006] In a first aspect of this disclosure, a data storage method is provided, comprising: obtaining various enumeration values ​​of a service, the enumeration values ​​including a data query method, a query response time, and the amount of data in the data table where the query data is located; determining a target storage engine matching each of the enumeration values, and obtaining a service level corresponding to the service; determining a target database corresponding to the service level in the target storage engine, and creating a data table in the target database, the data table being used to store data generated by the service.

[0007] In one embodiment of this disclosure, determining the target storage engine that matches each of the enumeration values ​​includes: constructing a tuple parameter based on each of the enumeration values; and determining, among the storage engines, the storage engine that matches the tuple parameter as the target storage engine.

[0008] In another embodiment of this disclosure, determining the target database corresponding to the service level in the target storage engine includes: determining a plurality of databases to be determined corresponding to the service level in the target storage engine; obtaining a load balancing strategy corresponding to the target storage engine; and determining, among the databases to be determined, the database that matches the load balancing strategy as the target database.

[0009] In another embodiment of this disclosure, after creating a data table in the target database, the method further includes: parsing the data stored in the data table based on the query statement of the business to generate a business node; constructing a data synchronization task corresponding to the data output task of the business node; and sending the data synchronization task to a data scheduling device, wherein the data synchronization task is used to synchronize the data produced by the data output task to the data table.

[0010] In another embodiment of this disclosure, after sending the data synchronization task to the data scheduling device, the method further includes: obtaining the first data synchronized by the data synchronization task, and obtaining the data synchronization method of the data table; and storing the first data in the data table based on the data synchronization method.

[0011] In another embodiment of this disclosure, storing the first data in the data table based on the data synchronization method includes: in response to the data synchronization method being full synchronization, copying the data table to obtain a temporary table; storing the first data in the temporary table to obtain a synchronized temporary table; changing the identifier of the synchronized temporary table to the identifier of the data table, and deleting the data table.

[0012] In another embodiment of this disclosure, storing the first data in the data table based on the data synchronization method includes: in response to the data synchronization method being incremental synchronization, comparing the first data with the second data in the data table to determine sub-data in the first data, wherein the sub-data is incremental data not stored in the data table; and storing the sub-data in the data table.

[0013] In another embodiment of this disclosure, the method further includes: in response to the deletion of the data table or the suspension of the query service corresponding to the data table, sending a deletion request to the data scheduling device, the deletion request being used to request the data scheduling device to delete the data synchronization task.

[0014] In another embodiment of this disclosure, after creating a data table in the target database, the method further includes: obtaining the query frequency of each data in the data table; and deleting the data corresponding to the query frequencies that are lower than a preset frequency.

[0015] In a second aspect of this disclosure, a data storage device is also provided, comprising: a first acquisition module, configured to acquire various enumeration values ​​of a service, the enumeration values ​​including a data query method, a query response time, and the amount of data in the data table where the query data is located; a first determination module, configured to determine a target storage engine matching each of the enumeration values ​​and acquire the service level corresponding to the service; and a second determination module, configured to determine a target database corresponding to the service level in the target storage engine and create a data table in the target database, the data table being used to store the data generated by the service.

[0016] In one embodiment of this disclosure, the first determining module includes: a construction unit, configured to construct a tuple parameter based on each of the enumerated values; and a first determining unit, configured to determine, among the various storage engines, a storage engine that matches the tuple parameter as the target storage engine.

[0017] In another embodiment of this disclosure, the second determining module includes: a second determining unit, configured to determine a plurality of databases to be determined corresponding to the service level in the target storage engine; an obtaining unit, configured to obtain a load balancing strategy corresponding to the target storage engine; and a third determining unit, configured to determine, among the databases to be determined, a database matching the load balancing strategy as the target database.

[0018] In another embodiment of this disclosure, the data storage device further includes:

[0019] The parsing module is used to parse the data stored in the data table and generate business nodes based on the query statement of the business;

[0020] The construction module is used to construct the data synchronization task corresponding to the data output task of the business node;

[0021] The first sending module is used to send the data synchronization task to the data scheduling device, wherein the data synchronization task is used to synchronize the data produced by the data output task to the data table.

[0022] In another embodiment of this disclosure, the data storage device further includes: a second acquisition module, configured to acquire the first data synchronized by the data synchronization task and acquire the data synchronization method of the data table; and a storage module, configured to store the first data into the data table based on the data synchronization method.

[0023] In another embodiment of this disclosure, the storage module includes: a copying unit, configured to copy the data table to obtain a temporary table in response to the data synchronization mode being full synchronization; a first storage unit, configured to store the first data into the temporary table to obtain a synchronized temporary table; and a processing unit, configured to change the identifier of the synchronized temporary table to the identifier of the data table and delete the data table.

[0024] In another embodiment of this disclosure, the storage module includes: a comparison unit, configured to compare the first data with the second data of the data table in response to the data synchronization mode being incremental synchronization, to determine sub-data in the first data, wherein the sub-data is incremental data not stored in the data table; and a second storage unit, configured to store the sub-data in the data table.

[0025] In another embodiment of this disclosure, the data storage device further includes: a second sending module, configured to send a deletion request to the data scheduling device in response to the deletion of the data table or the suspension of the query service corresponding to the data table, wherein the deletion request is used to request the data scheduling device to delete the data synchronization task.

[0026] In another embodiment of this disclosure, the data storage device further includes: a third acquisition module, configured to acquire the query frequency of each data in the data table; and a deletion module, configured to delete the data corresponding to the query frequencies that are lower than a preset frequency.

[0027] In a third aspect of this disclosure, a medium is provided, comprising: computer execution instructions, which, when executed by a processor, are used to implement the method described above.

[0028] In a fourth aspect of this disclosure, a computing device is provided, comprising:

[0029] Memory and processor;

[0030] The memory stores computer-executed instructions;

[0031] The processor executes computer execution instructions stored in the memory, causing the processor to perform the method described above.

[0032] In this embodiment, the target storage engine for storing data is determined by the data query method of the business, the query response time, and the amount of data in the data table where the query data is located. Then, a data table is created in the target database corresponding to the service level of the business in the target storage engine for data storage. This eliminates the need for manual selection of the storage engine and the database in the storage engine, shortens the selection time of the storage engine and the database, and improves the data storage efficiency. Attached Figure Description

[0033] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0034] Figure 1 A schematic diagram illustrating an application scenario of the data storage method according to embodiments of the present disclosure is shown.

[0035] Figure 2 A schematic flowchart according to an embodiment of the present disclosure is shown;

[0036] Figure 3 A schematic flowchart according to another embodiment of the present disclosure is shown;

[0037] Figure 4 A schematic flowchart according to yet another embodiment of the present disclosure is shown;

[0038] Figure 5 A schematic flowchart according to another embodiment of the present disclosure is shown;

[0039] Figure 6 A schematic diagram of a program product provided according to an embodiment of the present disclosure is shown.

[0040] Figure 7 A schematic diagram of the structure of a data storage device provided according to an embodiment of the present disclosure is shown.

[0041] Figure 8 A schematic diagram of the structure of a computing device provided according to an embodiment of the present disclosure is shown.

[0042] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0043] The principles and spirit of this disclosure will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0044] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0045] According to embodiments of this disclosure, a data storage method, apparatus, medium, and computing device are proposed.

[0046] Furthermore, the number of any elements in the accompanying drawings is for illustrative purposes only and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0047] In addition, the data involved in this disclosure may be data authorized by the user or fully authorized by all parties. The collection, dissemination and use of the data shall comply with the requirements of relevant national laws and regulations. The implementation methods / executives of this disclosure may be combined with each other.

[0048] The principles and spirit of this disclosure will be explained in detail below with reference to several representative embodiments. Invention Overview

[0050] Before performing data queries, data needs to be stored in a database. When storing data, a storage engine needs to be selected, and the data to be stored is then stored in the database of that storage engine.

[0051] The inventors of this disclosure have discovered that the selection of storage engines and databases within storage engines is done manually by technical personnel. In other words, the selection of storage engines and databases depends on the experience of technical personnel. If the technical personnel lack experience, it will take a long time to select storage engines and databases, resulting in longer data storage time and lower data storage efficiency.

[0052] The inventors of this disclosure therefore conceived of determining the target storage engine for storing data based on the business's data query method, query response time, and the amount of data in the data table containing the queried data. This allows for the creation of a data table within the target database corresponding to the business's service level in the target storage engine for data storage. This eliminates the need for manual selection of the storage engine and the database within it, shortening the selection time for the storage engine and database and improving data storage efficiency.

[0053] Application Scenarios Overview

[0054] First refer to Figure 1 , Figure 1This is a schematic diagram illustrating an application scenario of the data storage method according to an embodiment of the present disclosure. The data storage device 100 can be a terminal device with data storage capabilities, a data processing system, or a cloud server. The data storage device 100 is associated with a service and obtains various enumeration values ​​of the service. These enumeration values ​​include data query methods, query response times, and the amount of data in the data table containing the queried data. The data storage device 100 includes different types of storage engines, such as storage engine A, storage engine B, and storage engine C. The data storage device 100 determines a target storage engine among multiple storage engines based on the various enumeration values. For example, storage engine A is the target storage engine. The data storage device 100 determines a target database within each database of the target storage engine. The target database is determined by the service level corresponding to the service. For example, the database includes database a, database b, and database c, with database c being the target database. The data storage device 100 creates a data table 200 in the target database to store data generated by the service.

[0055] Exemplary methods

[0056] The following is combined with Figure 1 Application scenarios, refer to Figures 2-5 This document describes a data storage method according to exemplary embodiments of the present disclosure. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in any way. Rather, the embodiments of the present disclosure can be applied to any applicable scenario.

[0057] Reference Figure 2 , Figure 2 An exemplary flowchart of an embodiment of a data storage method provided according to this disclosure is shown. The data storage method includes:

[0058] Step S201: Obtain the enumeration values ​​of the business, including the data query method, query response time, and the amount of data in the data table where the queried data is located.

[0059] In this embodiment, the executing entity is a data storage device. For ease of description, the term "device" will be used to refer to the data storage device below. A device is a terminal device, cloud server, or data processing system with data storage capabilities.

[0060] Each business has a corresponding query scenario, which is represented by a combination of enumeration values ​​for that business. The device obtains the various enumeration values ​​corresponding to the business. For example, a business may be associated with multiple enumeration values, and the device obtains these values ​​through the business's identifier. Enumeration values ​​include the data query method, query response time, and the amount of data in the table containing the queried data. For example, the data query method includes single-table queries and multi-table join queries; the query response time refers to the range of response times for a query request, such as less than 100 milliseconds, 100 to 500 milliseconds, 500 to 1000 milliseconds, 1 to 5 seconds, and 5 to 30 seconds; the data volume refers to the range of data volume in the table, including less than 100,000, 100,000 to 1 million, 1 million to 5 million, 5 million to 10 million, and more than 10 million, with the unit of data volume being KB. The above query response time and data volume are merely illustrative examples and are not limited to these ranges; they can be set according to requirements.

[0061] Step S202: Determine the target storage engine that matches each enumeration value and obtain the service level corresponding to the business.

[0062] The device is equipped with different types of storage engines, such as MySQL (relational database management system), Doris (analytic database), and HBase (distributed open source database).

[0063] Different storage engines have different performance characteristics, and the enumeration values ​​represent the performance requirements of the business for each storage engine. The device then determines the target storage engine based on these enumeration values.

[0064] In one example, the device determines the target storage engine that matches each enumeration value. For instance, the storage engine has tags such as supporting multi-table joins, having more than ten million data entries in the table, and a query response time of less than one hundred milliseconds. The device determines whether each enumeration value has a matching tag in the storage engine. If each enumeration value has a matching tag in the storage engine, then that storage engine can be identified as the target storage engine matching the enumeration value. For example, if the enumeration value is a single-table query, and the storage engine's tag is supporting single-table queries, then the enumeration value matches that tag.

[0065] In another example, the device constructs tuple parameters based on each enumeration value and determines the target storage engine from among the various storage engines that matches the tuple parameters. For example, the enumeration values ​​constitute triple parameters, such as (single table query, less than 100 milliseconds, less than 100,000). Each storage engine is associated with a corresponding tuple parameter. The device compares the constructed tuple parameters with the tuple parameters associated with the storage engine. If they match, the storage engine is selected as the target storage engine. For example, if the constructed tuple parameters are the same as the tuple parameters associated with the storage engine, then the two tuple parameters are matched. It should be noted that if the constructed tuple parameters do not have a matching storage engine, the tuple parameters formed by the enumeration values ​​do not conform to the specifications, and a prompt message needs to be output to change the enumeration values ​​for the business logic.

[0066] Step S203: Determine the target database corresponding to the service level in the target storage engine, and create a data table in the target database. The data table is used to store the data generated by the business.

[0067] The storage engine is configured with multiple databases, each used by different tenants within the business, and each tenant may correspond to multiple parallel databases. After determining the target storage engine, it is necessary to find a suitable database among these multiple databases to store the business data.

[0068] Services have different levels, which refer to the service level defined in the application management system. This level is defined as the service level. The service level is associated with a service identifier, and the device obtains the service level corresponding to the service through the service identifier. Each service level corresponds to a different database. For example, service level 1 corresponds to databases 1 to 30; service level 2 corresponds to databases 31 to 50; and service level 3 corresponds to databases 51 to 59. The device determines the target database based on the service level.

[0069] In one example, one database is selected from the databases corresponding to the service level as the target database.

[0070] In another example, multiple databases corresponding to the service level are identified within the target storage engine; these databases are defined as databases to be determined. The device then obtains the load balancing strategy corresponding to the target storage engine and selects the database matched by the load balancing strategy among the databases to be determined as the target database. For example, the load balancing strategy may maintain a small difference in the amount of data among the databases; the device then selects the database with the smallest amount of data among the databases to be determined as the target database. Alternatively, the load balancing strategy could be round-robin, where the databases are ordered. The device obtains the first sequence number of the database that last created the data table and selects the database corresponding to the second sequence number below the first sequence number as the target database.

[0071] After identifying the target database, the device creates a data table within the target database to store data generated by business operations. For example, the device obtains the structure of a source table in the database and creates a data table based on that structure. For instance, if the source table is a HIVE type table, after obtaining the structure of the HIVE source table, the device converts some of the fields included in the structure to complete the creation of the data table. For example, it converts fields representing different data types, such as converting fields representing MySQL or Doris to string->varchar.

[0072] In this embodiment, the target storage engine for storing data is determined by the data query method of the business, the query response time, and the amount of data in the data table where the query data is located. Then, a data table is created in the target database corresponding to the service level of the business in the target storage engine for data storage. This eliminates the need for manual selection of the storage engine and the database in the storage engine, shortens the selection time of the storage engine and the database, and improves the data storage efficiency.

[0073] Reference Figure 3 , Figure 3 An exemplary flowchart of another embodiment of the data storage method provided according to the present disclosure is shown. Based on the above embodiment, after step S203, the method further includes:

[0074] Step S301: Based on the business query statement, parse the data stored in the data table to generate business nodes.

[0075] In this embodiment, the offline tasks of the database include tasks and nodes. A task typically consists of multiple nodes with runtime dependencies, where a runtime dependency refers to the relationship between the execution of one node and the execution result of another node. The data stored in the database is generated by these nodes. Nodes are defined as business nodes, and each node corresponds to a data output task; that is, the data generated by the data output task needs to be stored in the data tables already created in the database. Therefore, the device needs to determine the lineage relationship between the business nodes and the data tables.

[0076] In response, the device obtains the query statement of the business, such as an SQL (Structured Query Language) statement, so as to obtain the relationship between the source table and the business node in the database. The data table is created based on the source table, so the lineage relationship between the business node and the data table can be found. That is, parsing the query statement can determine the business node generated by the data stored in the data table.

[0077] Step S302: Construct the data synchronization task corresponding to the data output task of the business node.

[0078] Business nodes have data output tasks. The device pulls the metadata of the tasks. Based on the metadata, the task relationship corresponding to the business node can be determined. From the task relationship, the data output task corresponding to the business node can be determined. The device can construct the lineage relationship between the data output task and the data table, so that the data produced by the data output task is stored in the data table based on the lineage relationship.

[0079] In the exemplary technology, after the data output task is completed, the data needs to be transmitted to the data table through the data link and data transmission cycle set by the business node. The data transmission link is relatively long and the data transmission cycle is also relatively long, for example, the data transmission cycle is every other day. In order to schedule the output data in a timely manner, the device constructs a data synchronization task corresponding to the data output task.

[0080] Step S303: The data synchronization task is sent to the data scheduling device. The data synchronization task is used to synchronize the data produced by the data output task to the data table.

[0081] The device sends the data synchronization task to the data scheduling device, which is the physical device of the task scheduling platform. This device monitors whether the data output task is completed and acquires the data produced by the task. Once the data output task is completed, the data synchronization task promptly synchronizes the data generated by the task to the data table, thus ensuring timely data storage and avoiding excessively long data storage periods, achieving timely data scheduling.

[0082] In addition, to prevent changes in the upstream task scheduling cycle or time, the device sets up a monitoring thread to periodically monitor the scheduling configuration of the data output task. If the scheduling configuration of the data output task changes, the data output task will also change. The device will then update the data synchronization task based on the changed scheduling configuration and send the updated data synchronization task to the data scheduling device, so that the updated data synchronization task can obtain the data generated by the data output task after the scheduling configuration has changed.

[0083] In this embodiment, the device parses the business nodes generated by the data stored in the data table based on the business query statement, constructs the data synchronization task corresponding to the data output task of the business node, and then quickly obtains the data produced by the data output task based on the data synchronization task to store it in the data table for timely data scheduling.

[0084] Reference Figure 4 , Figure 4 An exemplary flowchart of yet another embodiment of the data storage method provided according to embodiments of the present disclosure is shown, based on... Figure 3 In the embodiment shown, after step 303, the method further includes:

[0085] Step S401: Obtain the first data synchronized by the data synchronization task, and obtain the data synchronization method of the data table.

[0086] In this embodiment, in order to ensure data consistency and shield the synchronization details from business operations, and to always provide a complete and consistent snapshot, it is necessary to perform pre-processing and post-processing of the data synchronization process based on different data synchronization methods during the data synchronization task scheduling. That is, the data synchronized by the data synchronization task is stored according to the data synchronization method.

[0087] To address this, the device acquires the data synchronized by the data synchronization task, which is defined as the first data. The device then acquires the data synchronization method for the data table. Data synchronization methods include full synchronization and incremental synchronization. The data synchronization method is associated with the storage engine; the device acquires the data synchronization method based on the target storage engine.

[0088] Step S402: Store the first data into the data table based on the data synchronization method.

[0089] After determining the data synchronization method, the device stores the first data in the data table based on the data synchronization method.

[0090] In one example, when the data synchronization method is full synchronization, a temporary table is used for data storage. The device first copies the data table as a temporary table, and uses this temporary table as the data table during the data synchronization process. That is, the device stores the first data in the temporary table to obtain the synchronization temporary table. The device then changes the identifier of the synchronization temporary table to the identifier of the data table, and deletes the data table to complete the data synchronization. By changing the representation first and then deleting the data table, the query business is unaware of this, meaning that it will not affect the query task.

[0091] In another example, when the data synchronization method is incremental synchronization, the device compares the first data with the second data in the data table to determine the sub-data in the first data. The sub-data is incremental data, that is, the sub-data is incremental data that is not stored in the data table. The device stores the sub-data in the data table, and the data storage is completed.

[0092] In this embodiment, the device obtains the first data synchronized by the data synchronization task and the data synchronization method of the data table, and then stores the first data into the data table according to the data synchronization method.

[0093] Reference Figure 5 , Figure 5 An exemplary flowchart of another embodiment of the data storage method provided according to embodiments of the present disclosure is shown, based on... Figure 3 or Figure 4 In the embodiment shown, after step 303, the method further includes:

[0094] In step S501, in response to the deletion of a data table or the suspension of the query service corresponding to the data table, a deletion request is sent to the data scheduling device. The deletion request is used to request the data scheduling device to delete the data synchronization task.

[0095] In this embodiment, after data synchronization is combined with data query, the device has standardized constraints and lifecycle consistent with the data service, and can perform automated operation and maintenance of data synchronization.

[0096] In one example, the device uses a scheduled task to perform routine checks on data synchronization tasks according to specifications, such as checking the number of single database tables and single database tables to generate monitoring alarms.

[0097] In another example, for event-driven scenarios, the lifecycle of data synchronization tasks is managed based on the data's lifecycle. For instance, when a data table deletion or the corresponding query service is paused, the scheduling of the data synchronization task can be cancelled, i.e., the data synchronization task is deleted. The device then sends a deletion request to the data scheduling device, requesting the data scheduling device to delete the data synchronization task.

[0098] In another example, the device can identify "cold data," meaning it identifies and deletes data with low query frequency, freeing up storage space in the database. To do this, after creating a data table in the target database, the device obtains the query frequency of each piece of data in the table and deletes data with query frequencies below a preset threshold. Furthermore, it can generate and output alarm information based on data with query frequencies below the preset threshold, allowing relevant personnel to locate the data and determine whether it can be deleted.

[0099] In this embodiment, the device manages the data synchronization task to terminate the storage of data in the data table, avoids the query service from being suspended, and the data table corresponding to the query service continues to store data, thus avoiding the waste of storage resources in the database.

[0100] Exemplary media

[0101] After introducing the methods of exemplary embodiments of this disclosure, the following references are made. Figure 6 The storage medium of the exemplary embodiments of this disclosure will be described.

[0102] refer to Figure 6 As shown, the storage medium 60 stores a program product for implementing the above-described method according to an embodiment of the present disclosure. This program product may be a portable compact disc read-only memory (CD-ROM) and includes computer-executable instructions for causing a computing device to execute the data storage method provided in this disclosure. However, the program product of this disclosure is not limited thereto.

[0103] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0104] A readable signal medium may include data signals propagated in baseband or as part of a carrier wave, carrying computer-executed instructions. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium.

[0105] Computer-executable instructions for performing the operations disclosed herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The computer-executable instructions can be executed entirely on the user's computing device, partially on the user's computing device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN).

[0106] Exemplary device

[0107] Having introduced the medium of exemplary embodiments of this disclosure, the following references are made to... Figure 7 The terminal device described is provided as an example of an exemplary embodiment. The terminal device is used to implement the method in any of the above data storage method embodiments, and its implementation principle and technical effect are similar.

[0108] refer to Figure 7 , Figure 7 A schematic diagram of the structure of a data storage device provided according to an embodiment of the present disclosure is shown.

[0109] like Figure 7 As shown, the data storage device includes: a first acquisition module 710, used to acquire various enumeration values ​​of the business, the enumeration values ​​including the data query method, query response time, and the amount of data in the data table where the query data is located; a first determination module 720, used to determine the target storage engine that matches each enumeration value and acquire the service level corresponding to the business; and a second determination module 730, used to determine the target database corresponding to the service level in the target storage engine and create a data table in the target database, the data table being used to store the data generated by the business.

[0110] In one embodiment, the first determining module 720 includes: a construction unit for constructing tuple parameters based on each enumeration value; and a first determining unit for determining, among each storage engine, a storage engine that matches the tuple parameters as a target storage engine.

[0111] In one embodiment, the second determining module 730 includes: a second determining unit, configured to determine multiple databases to be determined corresponding to service levels in the target storage engine; an obtaining unit, configured to obtain a load balancing strategy corresponding to the target storage engine; and a third determining unit, configured to determine, among the databases to be determined, the database matching the load balancing strategy as the target database.

[0112] In one embodiment, the data storage device further includes: a parsing module, used to parse the business nodes generated by the data stored in the data table based on the business query statement; a construction module, used to construct the data synchronization task corresponding to the data output task of the business node; and a first sending module, used to send the data synchronization task to the data scheduling device, wherein the data synchronization task is used to synchronize the data produced by the data output task to the data table.

[0113] In one embodiment, the data storage device further includes: a second acquisition module, configured to acquire the first data synchronized by the data synchronization task and acquire the data synchronization method of the data table; and a storage module, configured to store the first data into the data table based on the data synchronization method.

[0114] In one embodiment, the storage module includes: a copying unit, configured to copy a data table to obtain a temporary table in response to a full data synchronization mode; a first storage unit, configured to store first data into the temporary table to obtain a synchronized temporary table; and a processing unit, configured to change the identifier of the synchronized temporary table to the identifier of the data table and delete the data table.

[0115] In one embodiment, the storage module includes: a comparison unit, configured to compare first data with second data in a data table in response to incremental synchronization, to determine sub-data in the first data, wherein the sub-data is incremental data not stored in the data table; and a second storage unit, configured to store the sub-data in the data table.

[0116] In one embodiment, the data storage device further includes: a second sending module, configured to send a deletion request to the data scheduling device in response to data table deletion or the suspension of the query service corresponding to the data table, wherein the deletion request is used to request the data scheduling device to delete the data synchronization task.

[0117] In one embodiment, the data storage device further includes: a third acquisition module for acquiring the query frequency of each data in the data table; and a deletion module for deleting data corresponding to query frequencies lower than a preset frequency.

[0118] Exemplary computing device

[0119] Having described the methods, media, and apparatus of exemplary embodiments of this disclosure, the following references... Figure 8 A computing device according to an exemplary embodiment of the present disclosure will be described.

[0120] Figure 8 The computing device 80 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein. Figure 8As shown, the computing device 80 is presented in the form of a general-purpose computing device. The components of the computing device 80 may include, but are not limited to: at least one processing unit 801, at least one storage unit 802, and a bus 803 connecting different system components (including the processing unit 801 and the storage unit 802). The at least one storage unit 802 stores computer-executable instructions; the at least one processing unit 801 includes a processor that executes the computer-executable instructions to implement the methods described above.

[0121] The 803 bus includes a data bus, a control bus, and an address bus.

[0122] Storage unit 802 may include readable media in the form of volatile memory, such as random access memory (RAM) 8021 and / or cache memory 8022, and may further include readable media in the form of non-volatile memory, such as read-only memory (ROM) 8023.

[0123] Storage unit 802 may also include a program / utility 8025 having a set (at least one) program module 8024, such program module 8024 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0124] The computing device 80 can also communicate with one or more external devices 804 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 805. Furthermore, the computing device 80 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 806. Figure 8 As shown, network adapter 806 communicates with other modules of computing device 80 via bus 803. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with computing device 80, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0125] It should be noted that although several units / modules or sub-units / modules of the terminal device / server are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0126] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0127] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A data storage method, characterized in that, include: Obtain the enumeration values ​​of the business, which include the data query method, query response time, and the amount of data in the data table where the queried data is located; the enumeration values ​​are used to represent the performance requirements of the storage engine for the business. Determine the target storage engine that matches each of the enumerated values, and obtain the service level corresponding to the service. In the target storage engine, determine the target database corresponding to the service level, and create a data table in the target database. The data table is used to store the data generated by the service. Based on the query statement of the business, a business node is generated by parsing and storing the data in the data table; Construct the data synchronization task corresponding to the data output task of the business node; The data synchronization task is sent to the data scheduling device, and the data synchronization task is used to synchronize the data produced by the data output task to the data table.

2. The data storage method according to claim 1, characterized in that, The determination of the target storage engine that matches each of the enumerated values ​​includes: Construct a tuple parameter based on each of the enumerated values; Among the various storage engines, the storage engine that matches the tuple parameters is determined as the target storage engine.

3. The data storage method according to claim 1, characterized in that, Determining the target database corresponding to the service level in the target storage engine includes: In the target storage engine, identify multiple databases corresponding to the service level to be determined; Obtain the load balancing strategy corresponding to the target storage engine; Among the databases to be determined, the database that matches the load balancing strategy is selected as the target database.

4. The data storage method according to claim 1, characterized in that, After sending the data synchronization task to the data scheduling device, the method further includes: Obtain the first data synchronized by the data synchronization task, and obtain the data synchronization method of the data table; The first data is stored in the data table based on the data synchronization method described above.

5. The data storage method according to claim 4, characterized in that, The step of storing the first data in the data table based on the data synchronization method includes: In response to the data synchronization method being full synchronization, the data table is copied to obtain a temporary table; The first data is stored in the temporary table to obtain a synchronized temporary table; Change the identifier of the synchronous temporary table to the identifier of the data table, and then delete the data table.

6. The data storage method according to claim 4, characterized in that, The step of storing the first data in the data table based on the data synchronization method includes: In response to the data synchronization method being incremental synchronization, the first data is compared with the second data in the data table to determine the sub-data in the first data, wherein the sub-data is incremental data not stored in the data table; The sub-data is stored in the data table.

7. The data storage method according to any one of claims 1-6, characterized in that, Also includes: In response to the deletion of the data table or the suspension of the query service corresponding to the data table, a deletion request is sent to the data scheduling device, the deletion request being used to request the data scheduling device to delete the data synchronization task.

8. The data storage method according to any one of claims 1-6, characterized in that, After creating the data table in the target database, the process also includes: Obtain the query frequency of each data item in the data table; Data corresponding to query frequencies below the preset frequency will be deleted.

9. A data storage device, characterized in that, include: The first acquisition module is used to acquire various enumeration values ​​of the business, including the data query method, query response time, and the amount of data in the data table where the queried data is located; the enumeration values ​​are used to represent the performance requirements of the business for the storage engine; The first determining module is used to determine the target storage engine that matches each of the enumerated values ​​and to obtain the service level corresponding to the service. The second determining module is used to determine the target database corresponding to the service level in the target storage engine, and to create a data table in the target database, the data table being used to store the data generated by the service. The parsing module is used to parse the query statement of the business and generate business nodes that store data in the data table; The construction module is used to construct the data synchronization task corresponding to the data output task of the business node; The first sending module is used to send the data synchronization task to the data scheduling device, wherein the data synchronization task is used to synchronize the data produced by the data output task to the data table.

10. The data storage device according to claim 9, characterized in that, The first determining module includes: A construction unit is used to construct tuple parameters based on each of the enumerated values; The first determining unit is used to determine, among the various storage engines, the storage engine that matches the tuple parameters, as the target storage engine.

11. The data storage device according to claim 9, characterized in that, The second determining module includes: The second determining unit is used to determine multiple databases to be determined in the target storage engine corresponding to the service level; The acquisition unit is used to acquire the load balancing strategy corresponding to the target storage engine. The third determining unit is used to determine, among the various databases to be determined, the database that matches the load balancing strategy, as the target database.

12. The data storage device according to claim 9, characterized in that, The data storage device further includes: The second acquisition module is used to acquire the first data synchronized by the data synchronization task and to acquire the data synchronization method of the data table. A storage module is used to store the first data into the data table based on the data synchronization method.

13. The data storage device according to claim 12, characterized in that, The storage module includes: The replication unit is used to replicate the data table to obtain a temporary table in response to the data synchronization mode being full synchronization. The first storage unit is used to store the first data into the temporary table to obtain a synchronized temporary table; The processing unit is configured to change the identifier of the synchronous temporary table to the identifier of the data table and delete the data table.

14. The data storage device according to claim 12, characterized in that, The storage module includes: The comparison unit is used to compare the first data with the second data of the data table in response to the data synchronization method being incremental synchronization, so as to determine the sub-data in the first data, wherein the sub-data is incremental data not stored in the data table; The second storage unit is used to store the sub-data in the data table.

15. The data storage device according to any one of claims 9-14, characterized in that, The data storage device further includes: The second sending module is used to send a deletion request to the data scheduling device in response to the deletion of the data table or the suspension of the query service corresponding to the data table. The deletion request is used to request the data scheduling device to delete the data synchronization task.

16. The data storage device according to any one of claims 9-14, characterized in that, The data storage device further includes: The third acquisition module is used to acquire the query frequency of each data in the data table; The deletion module is used to delete data corresponding to query frequencies that are lower than a preset frequency.

17. A storage medium, characterized in that, include: Computer execution instructions, when executed by a processor, are used to implement the method as described in any one of claims 1 to 8.

18. A computing device, characterized in that, include: Memory and processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1 to 8.

Images