Data storage method, device, system and storage medium

Abstract

The present invention provides a data storage method, device, system and storage medium, which are applied to the field of data storage technology. The method includes: using real-time Flink CDC to establish a connection with a target database to obtain business data in the target database; performing data conversion processing on the business data according to the target data format supported by the remote storage device, and determining the data to be stored in the target data format; dynamically generating a target object storage file corresponding to the business data in the remote storage device, and storing the data to be stored in the target object storage file according to the object storage protocol corresponding to the target data format; the access method of the file stored in the remote storage device is the same as the access method of the local database, the storage capacity of the remote storage device is greater than the storage capacity of the target database, and the storage cost of the remote storage device is lower than the storage cost of the local storage device. The technical solution of the present invention can reduce the storage cost of business data.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to a data storage method, device, system and storage medium. Background Art

[0002] Multi-tenant SaaS (Software as a Service) application systems generate a large amount of business data every day. As time goes by or the number of tenants gradually increases, the storage capacity of the generated large amount of business data also grows exponentially. Therefore, how to store business data becomes particularly important.

[0003] In the related art, more storage space or instances are usually added to store the increasing business data by dividing the database and table.

[0004] However, the above technology will increase the storage cost of business data. Summary of the invention

[0005] The present invention provides a data storage method, device, system and storage medium, which are used to solve the defect in the prior art that storing more business data by adding more storage space or instances will increase the storage cost of business data, and achieve the purpose of obtaining business data in a database through a real-time data integration tool Flink CDC and performing data conversion processing on the business data and then storing it in a remote storage device with lower cost to reduce the storage cost of business data.

[0006] The present invention provides a data storage method, which is applied to a data storage system. The data storage system is built based on a data integration tool Flink CDC, including: Use real-time Flink CDC to establish a connection with the target database and obtain business data from the target database; Obtaining a target data format supported by a remote storage device, and performing data conversion processing on business data according to the target data format to determine data to be stored in the target data format; Dynamically generate a target object storage file corresponding to the business data in a remote storage device, and store the data to be stored in the target object storage file according to an object storage protocol corresponding to the target data format; Among them, the access method of the files stored in the above-mentioned remote storage device is the same as the access method of the files stored in the local database, the storage capacity of the remote storage device is larger than the storage capacity of the target database, and the storage cost of the remote storage device is lower than the storage cost of the local storage device.

[0007] According to a data storage method provided by the present invention, the data conversion processing is performed on the business data according to the target data format to determine the data to be stored in the target data format, including: Acquiring data operation information of the user on the target database; the data operation information includes at least one target data table that needs to be operated on the target database, and the materialized view modeling operation required to be performed on the target data table; Acquire a target data table in a target database according to the data operation information, and perform a materialized view modeling operation on the target data table to obtain a target materialized view corresponding to the target data table; The target materialized view is format converted according to the target data format to determine the data to be stored in the target data format.

[0008] According to a data storage method provided by the present invention, the business data includes an identifier of a target data generating object that generates the business data, and the data to be stored is stored in a target object storage file according to an object storage protocol corresponding to the target data format, including: According to the identifier of the target data generating object, determining the target classifier corresponding to the identifier of the target data generating object in a preset first corresponding relationship; the above-mentioned first corresponding relationship includes the identifiers of different data generating objects and the classifier corresponding to the identifier of each data generating object; According to the mapping relationship between different classifiers and different resource groups, the target resource group corresponding to the target classifier is determined; the above different resource groups are isolated from each other; According to the object storage protocol corresponding to the target data format, use the target resource group to store the data to be stored in the target object storage file.

[0009] According to a data storage method provided by the present invention, the business data includes a target session variable corresponding to the business data, and the data to be stored is stored in a target object storage file according to an object storage protocol corresponding to the target data format, including: According to the target session variable, a target resource group corresponding to the target session variable is determined in a preset second corresponding relationship; the second corresponding relationship includes different session variables and resource groups corresponding to each session variable, and different resource groups are isolated from each other; According to the object storage protocol corresponding to the target data format, use the target resource group to store the data to be stored in the target object storage file.

[0010] According to a data storage method provided by the present invention, the method further includes: Obtaining a data query request; the data query request includes an identifier of the data to be queried; Determine whether the data to be queried is hot data or cold data according to the identifier of the data to be queried; hot data is data with a query frequency higher than a threshold, and cold data is data with a query frequency lower than a threshold; If the data to be queried is hot data, a data query is performed in a local storage device according to the identifier of the data to be queried to obtain the data to be queried; If the data to be queried is cold data, the data to be queried is obtained from a remote storage device according to an identifier of the data to be queried, and the data to be queried is loaded into a local storage device for caching.

[0011] According to a data storage method provided by the present invention, the method further includes: Get data query request; Parsing the data query request to determine whether the data query request includes annotation information; the annotation information is used to characterize routing information of the data query request; If the data query request includes annotation information, the data query request is routed to the slave node, so that the data query is performed according to the data query request through the slave node; The data storage system comprises a master node and slave nodes, and the master node is used for performing data conversion and storage processing on business data.

[0012] According to a data storage method provided by the present invention, the business data includes real-time change data in a target database, and the business data in the target database is obtained, including: Use Flink CDC to parse the binary log data of the target database in real time and build a streaming pipeline to obtain real-time change data in the target database; According to the primary key of each data table and / or the row size of each data table in the target database, the real-time change data is divided into multiple snapshot blocks; Multiple threads are used to read multiple snapshot blocks in parallel; wherein each thread reads one snapshot block correspondingly.

[0013] The present invention also provides a data storage device, which is applied to a data storage system. The data storage system is built based on the data integration tool Flink CDC and includes the following modules: The data acquisition module is used to establish a connection with the target database using real-time Flink CDC and obtain business data from the target database; The data conversion module is used to obtain the target data format supported by the remote storage device, and perform data conversion processing on the business data according to the target data format to determine the data to be stored in the target data format; A data storage module is used to dynamically generate a target object storage file corresponding to the business data in a remote storage device, and store the data to be stored in the target object storage file according to the object storage protocol corresponding to the target data format; wherein the access method of the files stored in the above-mentioned remote storage device is the same as the access method of the files stored in the local database, the storage capacity of the remote storage device is greater than the storage capacity of the target database, and the storage cost of the remote storage device is lower than the storage cost of the local storage device.

[0014] The present invention also provides a data storage system, which is built based on the data integration tool Flink CDC, and includes: The data source connection layer is used to establish a connection with the target database using real-time Flink CDC and obtain business data from the target database; A data construction synchronization layer is used to obtain a target data format supported by a remote storage device, and to perform data conversion processing on business data according to the target data format to determine the data to be stored in the target data format; a target object storage file corresponding to the business data is dynamically generated in the remote storage device, and the data to be stored is stored in the target object storage file according to the object storage protocol corresponding to the target data format; wherein the access method of the files stored in the remote storage device is the same as the access method of the files stored in the local database, the storage capacity of the remote storage device is greater than the storage capacity of the target database, and the storage cost of the remote storage device is lower than the storage cost of the local storage device; The data query operation layer is used to obtain data query requests and perform data queries based on the data query requests.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described data storage methods when executing the computer program.

[0016] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the data storage method described above is implemented.

[0017] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the data storage methods described above.

[0018] The present invention provides a data storage method, device, system and storage medium. The data storage method is applied to a data storage system. The data storage system is built based on a data integration tool Flink CDC. The real-time Flink CDC tool is used to establish a connection with a target database and obtain business data in the target database. At the same time, a target data format supported by a remote storage device is obtained, and data conversion processing is performed on the business data according to the target data format to determine the data to be stored in the target data format; a target object storage file corresponding to the business data is dynamically generated in the remote storage device, and the data to be stored is stored in the target object storage file according to the object storage protocol corresponding to the target data format; wherein the access method of the file stored in the remote storage device is the same as the access method of the file stored in the local database, the storage capacity of the remote storage device is greater than the storage capacity of the target database, and the storage cost of the remote storage device is lower than the storage cost of the local storage device. In this method, since the business data in the database can be obtained through the real-time data integration tool Flink CDC and the business data is converted and stored in a remote storage device with lower cost, this can reduce the storage cost of the business data while ensuring accurate data storage; at the same time, the remote storage device has a larger storage capacity and lower storage cost, and the remote storage device can be used / accessed like a local database when data needs to be accessed, thereby supporting low-cost storage and use of multi-tenant massive data and meeting the actual storage needs of the ever-increasing business data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a schematic diagram of the data storage system framework provided by the present invention.

[0021] Figure 2 This is one of the flow charts of the data storage method provided by the present invention.

[0022] Figure 3 It is a schematic diagram of the data storage architecture provided by the present invention.

[0023] Figure 4 This is the second flow chart of the data storage method provided by the present invention.

[0024] Figure 5 It is a structural schematic diagram of the data storage device provided by the present invention.

[0025] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] At present, with the accumulation of time and the increase in the number of tenants, the amount of business data stored in the SaaS application system has increased exponentially. At the same time, the frequency of use of historical data will usually become lower and lower over time. At this time, how to store and use these business data at low cost has become a technical problem that needs to be solved urgently. The usual solution is to add more instances and storage space, and improve performance by sharding libraries and tables. However, as time goes by, the following problems exist: 1. The storage cost is uncontrollable. The storage cost is proportional to the amount of data. The amount of data grows exponentially over time, which also leads to an exponential increase in the storage cost of the database over time. 2. The computing performance decreases. The computing performance of the database will drop rapidly after the single instance storage space exceeds the TB (Terabyte, trillion bytes) level of data. 3. The development and operation costs increase. The problem of increased data volume and decreased computing performance is solved by sharding libraries and tables, which increases the complexity of database maintenance and use, and adds huge costs to operation and maintenance and development. Based on this, the embodiments of the present invention provide a data storage method, device, system and storage medium, which can solve the above technical problems.

[0028] The following first describes the architecture of the data storage system according to the embodiment of the present invention.

[0029] See also Figure 1The data storage system framework schematic diagram shown in the figure shows that the data storage method of the embodiment of the present invention can be applied to the data storage system. The data storage system is built based on the data integration tool Flink CDC. The data storage system may include a data source link layer, a data construction synchronization layer, and a data query operation layer, wherein the data source link layer is used to establish a connection with the target database using real-time Flink CDC and obtain business data in the target database; the data construction synchronization layer is used to obtain the target data format supported by the remote storage device, and perform data conversion processing on the business data according to the target data format to determine the data to be stored in the target data format; dynamically generate a target object storage file corresponding to the business data in the remote storage device, and store the data to be stored in the target object storage file according to the object storage protocol corresponding to the target data format; wherein the access method of the files stored in the remote storage device is the same as the access method of the files stored in the local database, the storage capacity of the remote storage device is greater than the storage capacity of the target database, and the storage cost of the remote storage device is lower than the storage cost of the local storage device; the data query operation layer is used to obtain a data query request and perform a data query according to the data query request.

[0030] Among them, the data source link layer can establish a connection with the target database, and at the same time, it can parse the change log of the binlog (binary log) of the target database to obtain the business log, that is, the business data; at the same time, data can be buried, such as burying the data at the currently updated data, so that it is convenient to quickly know the location of the last updated data next time and realize rapid data update.

[0031] The data storage system may further include a remote storage device for storing data. The remote storage device may be a remote storage device based on the S3 object storage protocol, such as a cloud, a remote server, etc. The storage capacity of these remote storage devices is generally very large and the storage cost is relatively low.

[0032] The above-mentioned data construction synchronization layer can be connected to the data source link layer and can be connected to the remote storage device at the same time. The data source link layer can transmit the business data of the target database it obtains to the data construction synchronization layer, so that the data construction synchronization layer can obtain the business data of the target database. The data construction synchronization layer includes multiple synchronization storage nodes, such as synchronization storage node 1, synchronization storage node 2, etc., where the synchronization storage node can be a computer device (such as a server). The main function of the data construction synchronization layer is to obtain the target data format supported by the remote storage device, and perform data conversion processing on the business data according to the target data format to determine the data to be stored in the target data format; dynamically generate the target object storage file corresponding to the business data in the remote storage device, and store the data to be stored in the target object storage file according to the object storage protocol corresponding to the target data format. In addition, the data construction synchronization layer can also realize functions such as metadata query or writing, client connection management, and replay rollback.

[0033] The above-mentioned data query operation layer can also be connected to a remote storage device. The data query operation layer can query the data after the data construction synchronization layer stores the business data in the remote storage device. The data query operation layer may include multiple operation nodes (also called query computing nodes), such as operation node 1, operation node 2, etc. The operation node here can be a computer device (such as a server). When querying data, the data query operation layer can query data in S3 object storage / remote storage device or local storage device (such as data cache), and at the same time, the specific node for query can be realized through routing election during query to improve query efficiency.

[0034] In addition, the above data storage system may also include a data application consumption layer, which is mainly used for querying historical data, generating dynamic reports, and monitoring data in real time.

[0035] Combine the following Figure 2-Figure 4 The data storage method according to the embodiment of the present invention is described.

[0036] It should be noted that the execution subject of the embodiments of the present invention can be a data storage system, a data storage device, or an electronic device including a data storage device, etc., which is not specifically limited here. The following embodiments will be explained using the data storage system as the execution subject.

[0037] Figure 2 It is one of the flow charts of the data storage method provided by the present invention, such as Figure 2 As shown, the method comprises the following steps: S202: Use real-time Flink CDC to establish a connection with a target database and obtain business data in the target database.

[0038] Among them, the data storage system in this embodiment is built using the real-time data integration tool Flink CDC. The data source link layer in the built data storage system can establish a connection with the target database. After the connection is established, the data source link layer can perform real-time detection on the target database to obtain the business data in the target database in real time. In addition, the data storage system can use MPP (Massively Parallel Processing) database and distributed architecture for data caching, querying, and writing functions, so as to achieve efficient parallel computing and data processing, with high scalability, fast query performance, and high fault tolerance.

[0039] The target database here may be a database that needs to store data in a remote storage device, such as a SQL (Structured Query Language) database, a MySql database, an Oracle database, etc.

[0040] The target database stores business data, and the business data in the target database obtained above may be all business data in the target database, or may be part of the business data (such as updated business data), etc. In addition, the business data may be historical data in the target database.

[0041] It should be noted that in this embodiment, only one target database is used as an example for explanation. When actually storing data, all databases on the business side can be used as target databases for data migration and storage. At the same time, the sub-libraries and tables, schema changes, etc. on the business side can be stored synchronously.

[0042] S204, obtaining a target data format supported by the remote storage device, and performing data conversion processing on the business data according to the target data format to determine the data to be stored in the target data format.

[0043] In this step, when the data in the target database needs to be stored in the remote storage device, the data construction synchronization layer can first obtain the data format supported by the remote storage device, which is recorded as the target data format. Optionally, the remote storage device here gives priority to the remote storage device of the S3 object storage protocol, which can improve the efficiency and accuracy of data storage. Accordingly, the target data format here can be, for example, a data format based on the S3 protocol.

[0044] After determining the target data format, the data construction synchronization layer can directly perform format conversion on the acquired business data to convert the business data into data in the target data format and record it as data to be stored. Alternatively, the acquired business data can be first subjected to data analysis and other processing, and then the analyzed data can be format converted after the data analysis to convert it into data to be stored in the target data format, or both the analyzed data and the business data can be format converted to convert both into data to be stored in the target data format. Alternatively, other data conversion processing methods or processes can be used, in short, the data to be stored in the target data format can be obtained in the end.

[0045] The following further describes the data construction synchronization layer in the data transmission system. The data construction synchronization layer may include synchronization storage nodes (such as computer devices). The synchronization storage nodes can be divided into master nodes and slave nodes. The slave nodes are responsible for obtaining metadata from the data source link layer and transmitting it to the master node. The master node is responsible for converting the obtained metadata and writing or updating it to the remote storage device. At the same time, the master node can use the BDE JE (Borland DatabaseEngine JE, open source database engine) tool to synchronize the metadata changes / data conversion processed data or change operations to the slave nodes. Generally, after the data changes are synchronized to more than the general slave nodes, the data synchronization write is considered successful. It should be noted that the master node and slave nodes of the synchronization storage node here are essentially slave nodes. The master node is generally generated by election from the slave nodes. When electing the master node, more than the general slave nodes in the synchronization storage node cluster are usually required to be active. When the master node fails, a new master node can be re-elected to perform corresponding operations.

[0046] S206, dynamically generate a target object storage file corresponding to the business data in the remote storage device, and store the data to be stored in the target object storage file according to the object storage protocol corresponding to the target data format; wherein the access method of the files stored in the above remote storage device is the same as the access method of the files stored in the local database, the storage capacity of the remote storage device is greater than the storage capacity of the target database, and the storage cost of the remote storage device is lower than the storage cost of the local storage device.

[0047] In this step, when storing the above-mentioned data to be stored in a remote storage device, the data construction synchronization layer can first dynamically generate an object storage file corresponding to the data to be stored / business data in the remote storage device, such as a Segment file, wherein different business data have different object storage files established, and different object storage files can be distinguished by identification information such as the index of each object storage file, which makes it easier to quickly find data through the file index during subsequent queries.

[0048] After the object storage file corresponding to the business data is established, the data construction synchronization layer can store the data to be stored after the business data is converted into the corresponding object storage file in the remote storage device according to the object storage protocol corresponding to the target data format (such as the S3 protocol).

[0049] It is understandable that in the scenario where multiple tenants use the SaaS application system, each tenant will generate its own business data, and the business data of different tenants can be stored in the same database or in different databases. When establishing an object storage file in a remote storage device, it is also possible to obtain the identifiers of different tenants (such as tenant IDs) and business data of different tenants by parsing the acquired business data, and then establish a corresponding object storage file for each tenant to store its own data to be stored / business data, which can facilitate subsequent rapid data query.

[0050] It should be noted that the storage capacity of the remote storage device in this embodiment is relatively large, generally much larger than the storage capacity of the database and the storage capacity of the local storage device, so it can meet the continuous storage of massive business data. At the same time, the storage cost of the remote storage device in this embodiment is lower than the storage cost of the local storage device and the database, so it can significantly reduce the storage cost of the increasing business data.

[0051] In addition, the above-mentioned remote storage device can be some currently existing remote storage devices. In this embodiment, the business data in the target database can be directly moved / migrated to the remote storage device for storage without building or using other external components to store data. This can reduce the complexity of system deployment and maintenance, and can be dynamically expanded horizontally to improve the scalability of services and the reliability of data storage. At the same time, the access method of the files stored in the remote storage device in this embodiment is the same as the access method of the files stored in the local database, that is, when it is necessary to access the files stored in the remote storage device, the remote storage device can be accessed as the local database, so that low-cost and efficient storage and use of multi-tenant massive data can be achieved.

[0052] For example, Figure 3 This is a schematic diagram of the data storage architecture provided by the present invention, see Figure 3As shown, taking the target database as a MySql database as an example, it may include multiple shared storages, such as shared1, shared2, shared3, etc., and each shared storage may store one or more data tables. The data storage system of the present invention may use the Flink CDC tool to read the Binlog binary log data in the target database, and perform data conversion processing on the read data and store / update it to a remote storage device (such as an S3 object storage device).

[0053] In this embodiment, the data storage method is applied to a data storage system, and the data storage system is built based on the data integration tool Flink CDC. The real-time Flink CDC tool is used to establish a connection with the target database and obtain the business data in the target database. At the same time, the target data format supported by the remote storage device is obtained, and the business data is converted according to the target data format to determine the data to be stored in the target data format; a target object storage file corresponding to the business data is dynamically generated in the remote storage device, and the data to be stored is stored in the target object storage file according to the object storage protocol corresponding to the target data format; wherein the access method of the stored files is the same as the access method of the files stored in the local database, the storage capacity of the remote storage device is greater than the storage capacity of the target database, and the storage cost of the remote storage device is lower than the storage cost of the local storage device. In this method, since the business data in the database can be obtained through the real-time data integration tool Flink CDC and the business data is converted and stored in a remote storage device with lower cost, this can reduce the storage cost of the business data while ensuring accurate data storage; at the same time, the remote storage device has a larger storage capacity and lower storage cost, and the remote storage device can be used / accessed like a local database when data needs to be accessed, thereby supporting low-cost storage and use of multi-tenant massive data and meeting the actual storage needs of the ever-increasing business data.

[0054] The following embodiment describes a possible implementation of the data conversion process of business data.

[0055] In some embodiments, the "performing data conversion processing on the business data according to the target data format to determine the data to be stored in the target data format" in the above S204 may include the following steps: Step A1, obtaining data operation information of a user on a target database; the data operation information includes at least one target data table that needs to be operated on in the target database, and a materialized view modeling operation that needs to be performed on the target data table.

[0056] In the process of moving and storing the data in the target database to the remote storage device, in order to facilitate the subsequent query and analysis of the stored data, an analysis process of the business data can be added here according to user needs.

[0057] Specifically, when the user needs to analyze the business data, he can input the data operation information for the target database on the display interface of the data transmission system. Generally, the target database may include multiple data tables, each of which may store different business data, such as order data tables, payment data tables, etc. The data operation information may include which data tables in the target database need to be operated, and what operations need to be performed on these data tables. The data table to be operated here can be recorded as the target data table, and the operation to be performed may include, for example, materialized view modeling operation on the target data table, that is, after operating the target data table, the materialized view corresponding to the target data table can be obtained. For example, the total payment amount of each order can be calculated for the order data table and the payment data table.

[0058] Step A2: acquiring a target data table in a target database according to the data operation information, and performing a materialized view modeling operation on the target data table to obtain a target materialized view corresponding to the target data table.

[0059] In this step, after obtaining the data operation information input by the user, one or more target data tables that specifically need to be subjected to materialized view modeling operations can be obtained from the data operation information, and then the materialized view modeling operations can be performed on the metadata in these target data tables to obtain the materialized views corresponding to these target data tables, which are recorded as target materialized views. It can be understood that the above-mentioned materialized view modeling operations can be materialized view modeling operations performed on one target data table, or can be materialized view modeling operations performed on multiple target data tables.

[0060] After materialized view modeling is performed on the business data of the target data table in the target database in this step, the target materialized view corresponding to the business data can be obtained. It can be understood that the target materialized view obtained here can be one or more, and each target materialized view can reflect the target data table from which it comes.

[0061] In addition, the above-mentioned materialized view modeling operation on the target data table is actually to analyze the business data in the target data table and obtain corresponding analysis results, which can facilitate the subsequent rapid acquisition of required analysis results during data query and improve query efficiency.

[0062] Step A3: Perform format conversion processing on the target materialized view according to the target data format to determine the data to be stored in the target data format.

[0063] In this step, after obtaining the target materialized view corresponding to the business data, the target materialized view may be format converted according to the target data format to convert the relevant data of the target materialized view into data to be stored in the target data format.

[0064] In this embodiment, the business data in the target database is first converted to materialized view modeling based on the data operation information input by the user, and then converted to the target data format and stored in a remote storage device. This makes it easier to quickly obtain the required analysis results during subsequent data queries, improves query efficiency, and improves the computing performance of business data.

[0065] The following embodiment describes an implementation method for determining a resource group used when storing data when the business data includes an identifier of a target data generating object that generates the business data or the business data includes a target session variable corresponding to the business data.

[0066] In one embodiment, the business data includes an identifier of a target data generating object that generates the business data. The step of "storing the data to be stored in the target object storage file according to the object storage protocol corresponding to the target data format" in S206 may include the following steps: Step B11, according to the identifier of the target data generating object, determine the target classifier corresponding to the identifier of the target data generating object in a preset first corresponding relationship; the above first corresponding relationship includes the identifiers of different data generating objects and the classifier corresponding to each identifier of the data generating object.

[0067] Step B12: determining the target resource group corresponding to the target classifier according to the mapping relationship between different classifiers and different resource groups; the above different resource groups are isolated from each other.

[0068] Step B13: storing the data to be stored in the target object storage file using the target resource group according to the object storage protocol corresponding to the target data format.

[0069] Among them, the business data obtained above may include the identifier of the target data generating object that generates the business data. The target data generating object may be, for example, a tenant, an application (APP, Application), a service, etc. The corresponding identifier of the target data generating object may be the tenant's ID, the name of the application, the name of the service, etc.

[0070] In order to facilitate the resource isolation of different data generation objects, the identifiers of different data generation objects can be collected in advance, and the resources can be grouped to obtain multiple isolated resource groups (that is, the resources are divided into multiple groups, where the resources can be physical resources or virtual resources, etc.), and then a corresponding classifier is set for each resource group, and different classifiers have different identifiers. After that, the identifiers corresponding to different data generation can be bound one by one with the identifiers of different classifiers to obtain a first corresponding relationship. Then, after obtaining the identifier of the target data generation object included in the business data, the classifier corresponding to the identifier of the corresponding data generation object can be matched in the first corresponding relationship, and then the resource group associated / belonging to the classifier can be obtained, which is recorded as the target resource group. After that, the target resource group can be used to quickly and successfully store the data to be stored in the target object storage file according to the object storage protocol corresponding to the target data format.

[0071] In another implementation manner, the business data includes a target session variable corresponding to the business data, and the step of "storing the data to be stored in the target object storage file according to the object storage protocol corresponding to the target data format" in S206 may further include the following steps: Step B21, according to the target session variable, determining the target resource group corresponding to the target session variable in a preset second corresponding relationship; the second corresponding relationship includes different session variables and resource groups corresponding to each session variable, and different resource groups are isolated from each other; Step B22: storing the data to be stored in the target object storage file using the target resource group according to the object storage protocol corresponding to the target data format.

[0072] The service data obtained above may include a session variable that carries the service data, which is recorded as a target session variable.

[0073] At the same time, in this embodiment, different session variables can also be collected in advance, and resources can be grouped to obtain multiple mutually isolated resource groups (that is, resources are divided into multiple groups, where resources can be physical resources or virtual resources, etc.), and then different session variables can be bound to different resource groups one by one to obtain a second corresponding relationship. Then, after obtaining the target session variable, the corresponding session variable can be matched in the second corresponding relationship, and then the resource group corresponding to the matched session variable can be obtained, which is recorded as the target resource group. Afterwards, the target resource group can be used to quickly and successfully store the data to be stored in the target object storage file according to the object storage protocol corresponding to the target data format.

[0074] Accordingly, when querying data later, data can also be queried through mutually isolated resource groups, thereby improving the efficiency and accuracy of data query.

[0075] In this embodiment, the corresponding isolated resource group is obtained through the identifier of the data generating object included in the business data or the session variable that carries the business data, and the data to be stored is stored in the remote storage device using a matching resource group. In this way, resource isolation of the business data of different data generating objects when writing to the remote storage device can be achieved, thereby achieving accurate writing and accurate query of the business data.

[0076] The following embodiments illustrate different query processes for hot data and cold data in a data query process.

[0077] Figure 4 This is the second flow chart of the data storage method provided by the present invention, see Figure 4 As shown, the above method may further include the following steps: S302, obtaining a data query request; the data query request includes an identifier of the data to be queried.

[0078] In this step, when data needs to be queried, the user can input a data query request in the data query operation layer, and the data query request can include the identifier of the data to be queried, which is recorded as the identifier of the data to be queried. Here, the identifier of the data to be queried can be, for example, the ID of the tenant, the name of the application, the name of the service, etc.

[0079] Of course, the above data query request may also include other content, such as data query scope (how long the data is to be queried), data query frequency, etc., which are not specifically limited here.

[0080] The following further explains the data query operation layer in the data storage system. The data query operation layer may include one or more independent query computing nodes, which are mainly responsible for data routing selection, computing query tasks, and caching hot data, etc. The query computing nodes can be dynamically deleted or added according to the number of query tasks. That is, the architecture of the data query operation layer of this embodiment is highly elastic and scalable, which can greatly reduce data storage costs and query costs while ensuring better resource isolation.

[0081] S304, determining whether the data to be queried is hot data or cold data according to the identifier of the data to be queried; hot data is data with a query frequency higher than a threshold, and cold data is data with a query frequency lower than the threshold.

[0082] Among them, hot data refers to data with high query frequency and short data generation time (for example, data generation time is close to the current time), and cold data refers to data with low query frequency and long data generation time (for example, data generation time is far from the current time). The high and low data query frequency here can be distinguished by a threshold. A query frequency higher than the threshold can be considered to be a high query frequency, and a query frequency lower than or equal to the threshold can be considered to be a low query frequency. The threshold size here can be set according to actual conditions.

[0083] The above can count the query frequency and data generation time of the data stored in the remote storage device within a period of time to distinguish the hot data and cold data in the stored data, and set the hot and cold data labels for the corresponding data identification.

[0084] Specifically, after obtaining the identifier of the data to be queried in the data query request, the identifier of the data to be queried can be matched with the pre-set labels of hot and cold data to determine whether the data to be queried is hot data or cold data.

[0085] S306, if the data to be queried is hot data, data query is performed in the local storage device according to the identifier of the data to be queried to obtain the data to be queried; if the data to be queried is cold data, the data to be queried is obtained in the remote storage device according to the identifier of the data to be queried, and the data to be queried is loaded into the local storage device for caching.

[0086] In this step, if the data to be queried is hot data, then generally the data to be queried may be stored in a local storage device, and then the query can be performed in the local storage device first. Specifically, the local cache in the local storage device can be queried first, and when there is no data in the local cache, the local disk can be queried to obtain the corresponding query data; when there is no corresponding data to be queried in the local storage device, the query can be continued in the remote storage device. When the data to be queried is hot data, the corresponding query data is stored locally and queried locally first, and the efficiency of local query is higher, so the query efficiency of hot data can be improved.

[0087] If the data to be queried is cold data, then generally such data is not stored locally. You can directly query it in the remote storage device, and after finding the corresponding query data, you can load this part of the data into the local storage device (for example, you can load it into the local cache first) for caching, so that the next time you query the data, you can quickly find the data in the local storage device, thereby improving query efficiency.

[0088] In this embodiment, by distinguishing between cold data and hot data for query, and prioritizing the hot data in the local storage device after the distinction, the query efficiency and query performance of the hot data can be improved, high computing performance and cost-effective storage can be achieved, and user experience can be guaranteed; at the same time, the cold data can be queried in the remote storage device to ensure the query success rate, and after the data is queried, the queried data is loaded into the local cache, which can facilitate the next time the data is queried to quickly query the data in the local storage device, thereby improving the query efficiency and query performance.

[0089] The following embodiment describes the process of querying data by routing to a slave node.

[0090] In some embodiments, the above method may further include the following steps: Step C1, obtaining a data query request.

[0091] In this step, when data needs to be queried, the user can input a data query request in the data query operation layer, which may include the identification of the data to be queried, and may also include annotation information of the data to be queried or may not include annotation information of the data to be queried.

[0092] Step C2, parsing the data query request to determine whether the data query request includes annotation information; the annotation information is used to characterize the routing information of the data query request.

[0093] In this step, after obtaining the data query request, the data query request may be parsed to obtain the content included therein, and it is determined whether the content obtained by the parsing includes preset annotation information, where the annotation information may be, for example, @QueryHint (query hint).

[0094] The annotation information here can reflect the specific routing information of the data query request, that is, the routing information of whether the data query request is specifically executed by the slave node or the master node.

[0095] Step C3, if the data query request includes annotation information, the data query request is routed to the slave node, so that the data query is performed through the slave node according to the data query request; wherein the above-mentioned data storage system includes a master node and a slave node, and the above-mentioned master node is used to perform data conversion processing and storage processing on the business data.

[0096] Among them, the data query operation layer of the data transmission system may include a master node and a slave node. The master node is used to perform data conversion processing and storage processing on business data, and can also perform query processing on data. The slave node is used to perform query processing on data.

[0097] In this step, if the content obtained by parsing the data query request does not include the preset annotation information, the data query request can be routed to the master node, and the master node executes the data query request, that is, the data query is performed. If the content obtained by parsing the data query request includes the preset annotation information, the data query request is routed to the slave node, and the slave node executes the data query request, that is, the data query is performed.

[0098] From the above description, it can be seen that the master node performs more functions, so the query efficiency is low, while the slave node performs fewer functions, so the query efficiency is high. Therefore, here, by adding annotation information to the query data during the data query process to route the data query request to the slave node for execution, the data query efficiency can be effectively improved.

[0099] In this embodiment, by adding annotation information to the query data during the data query process to route the data query request to the slave node for execution, the data query efficiency can be effectively improved and the user query experience can be improved.

[0100] The following example illustrates an implementation method of obtaining the business data.

[0101] In some embodiments, the business data includes real-time change data in the target database, and the "obtaining business data in the target database" in S202 may include the following steps: Flink CDC is used to parse the binary log data of the target database in real time, and a streaming pipeline is built to obtain the real-time change data in the target database. The real-time change data is divided into multiple snapshot blocks according to the primary key of each data table and / or the row size of each data table in the target database. Multiple threads are used to read multiple snapshot blocks in parallel, where each thread reads one snapshot block.

[0102] Among them, the Flink CDC tool used in the data source link layer can monitor the binary log binlog in the target database in real time, obtain the changed binary log data, and parse the changed binary log data. At the same time, it builds a streaming ELT (Extract Load Transform) pipeline to extract, load and transform the parsed data to obtain real-time change data in the target database.

[0103] Afterwards, the real-time change data can be cut into snapshot blocks according to the primary key of each data table and / or the row size of each data table in the target database to obtain multiple snapshot blocks. The specific number of snapshot blocks can be determined according to the primary key of the data table or the row size of the data table or the number of threads in the data source connection layer. After obtaining multiple snapshot blocks, each snapshot block can be read separately by each thread (or reader) of the data source connection layer, where each thread reads one snapshot block, so that the real-time change data in the target database can be read in parallel, improving data acquisition efficiency.

[0104] In this embodiment, the real-time changed data in the target database is divided into multiple snapshot blocks, and each snapshot block is read in parallel by multiple threads, so that the data reading / acquisition efficiency can be improved.

[0105] Through the description of the above multiple embodiments, it can be known that in the embodiments of the present invention, a data storage system is built based on the real-time Flink CDC tool, and the binlog binary log in the database is parsed through the built data storage system. At the same time, the master node elected by multiple synchronization storage nodes synchronizes the data to the remote storage device in seconds in parallel, and uses resource grouping, classifiers and session variables to achieve data isolation of multiple data generation objects, solving the problems of high historical data storage cost, high maintenance cost and complex maintenance of multiple data generation objects. Further, through multiple independent query computing nodes in the data storage system, dynamic routing to different nodes for query and dynamic routing to different data sources (such as local storage devices or remote storage devices using the S3 protocol) for data query, for historical data stored in remote storage devices using the S3 protocol, local cache, preloading strategy, object index and other methods can be used to improve query performance, and query computing nodes can be dynamically expanded to improve query performance and query experience of historical data.

[0106] In addition, the data storage system in this embodiment divides data storage and data query into two different layers. Data storage is implemented through the data construction synchronization layer, and data query is implemented through the data query operation layer. This can decouple data storage and query calculation, thereby improving the resource utilization of related nodes for data storage or data query and reducing maintenance costs.

[0107] The data storage device provided by the present invention is described below. The data storage device described below and the data storage method described above can be referenced to each other.

[0108] Figure 5 is a schematic diagram of the structure of the data storage device provided by the present invention, which is applied to the data storage system. The data storage system is built based on the data integration tool Flink CDC. Figure 5 As shown, the device may include: The data acquisition module 410 is used to establish a connection with a target database using real-time Flink CDC and acquire business data in the target database; The data conversion module 420 is used to obtain the target data format supported by the remote storage device, and perform data conversion processing on the business data according to the target data format to determine the data to be stored in the target data format; The data storage module 430 is used to dynamically generate a target object storage file corresponding to the business data in a remote storage device, and store the data to be stored in the target object storage file according to the object storage protocol corresponding to the target data format; wherein the access method of the files stored in the above-mentioned remote storage device is the same as the access method of the files stored in the local database, the storage capacity of the remote storage device is greater than the storage capacity of the target database, and the storage cost of the remote storage device is lower than the storage cost of the local storage device.

[0109] In some embodiments, the data conversion module 420 includes: An operation information acquisition unit, used to acquire data operation information of a user on a target database; the data operation information includes at least one target data table in the target database that needs to be operated, and a materialized view modeling operation that needs to be performed on the target data table; A modeling unit, used for acquiring a target data table in a target database according to the data operation information, and performing a materialized view modeling operation on the target data table to obtain a target materialized view corresponding to the target data table; The conversion unit is used to perform format conversion processing on the target materialized view according to the target data format, and determine the data to be stored in the target data format.

[0110] In some embodiments, the above-mentioned business data includes the identifier of the target data generating object that generates the business data, and the above-mentioned data storage module 430 is specifically used to determine the target classifier corresponding to the identifier of the target data generating object in a preset first correspondence relationship according to the identifier of the target data generating object; the above-mentioned first correspondence relationship includes the identifiers of different data generating objects and the classifiers corresponding to the identifiers of each data generating object; according to the mapping relationship between different classifiers and different resource groups, the target resource group corresponding to the target classifier is determined; the above-mentioned different resource groups are isolated from each other; according to the object storage protocol corresponding to the target data format, the target resource group is used to store the data to be stored in the target object storage file.

[0111] In some embodiments, the business data includes a target session variable corresponding to the business data. The data storage module 430 is specifically used to determine, according to the target session variable, a target resource group corresponding to the target session variable in a preset second correspondence relationship; the second correspondence relationship includes different session variables and resource groups corresponding to each session variable, and different resource groups are isolated from each other; and the target resource group is used to store the data to be stored in the target object storage file according to the object storage protocol corresponding to the target data format.

[0112] In some embodiments, the above apparatus further comprises: A first query request acquisition module is used to acquire a data query request; the data query request includes an identifier of the data to be queried; A judgment module, used to determine whether the data to be queried is hot data or cold data according to the identifier of the data to be queried; hot data is data with a query frequency higher than a threshold, and cold data is data with a query frequency lower than the threshold; The query module is used to query the data in the local storage device according to the identifier of the data to be queried if the data to be queried is hot data, and obtain the data to be queried; if the data to be queried is cold data, obtain the data to be queried in the remote storage device according to the identifier of the data to be queried, and load the data to be queried into the local storage device for caching.

[0113] In some embodiments, the above apparatus further comprises: A second query request acquisition module, used to acquire a data query request; A parsing module, used to parse the data query request and determine whether the data query request includes annotation information; the annotation information is used to characterize the routing information of the data query request; A routing module is used to route the data query request to a slave node if the data query request includes annotation information, so as to perform data query according to the data query request through the slave node; wherein the above-mentioned data storage system includes a master node and a slave node, and the above-mentioned master node is used to perform data conversion processing and storage processing on the business data.

[0114] In some embodiments, the data acquisition module 410 is specifically used to use Flink CDC to parse the binary log data of the target database in real time, and build a streaming pipeline to obtain the real-time change data in the target database; according to the primary key of each data table in the target database and / or the row size of each data table, the real-time change data is divided into multiple snapshot blocks; multiple threads are used to read multiple snapshot blocks in parallel; wherein each thread reads one snapshot block correspondingly.

[0115] It should be noted here that the above-mentioned device provided in the embodiment of the present invention can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effect. The parts and beneficial effects that are the same as the method embodiment in this embodiment will not be described in detail here.

[0116] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the data storage method, which includes: using real-time Flink CDC to establish a connection with a target database and obtain business data in the target database; obtaining a target data format supported by a remote storage device, and performing data conversion processing on the business data according to the target data format to determine the data to be stored in the target data format; dynamically generating a target object storage file corresponding to the business data in the remote storage device, and storing the data to be stored in the target object storage file according to the object storage protocol corresponding to the target data format; wherein the storage capacity of the above remote storage device is greater than the storage capacity of the target database, and the storage cost of the remote storage device is lower than the storage cost of the local storage device.

[0117] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0118] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data storage method provided by the above methods, which includes: using real-time Flink CDC to establish a connection with a target database and obtain business data in the target database; obtaining a target data format supported by a remote storage device, and performing data conversion processing on the business data according to the target data format to determine the data to be stored in the target data format; dynamically generating a target object storage file corresponding to the business data in the remote storage device, and storing the data to be stored in the target object storage file according to the object storage protocol corresponding to the target data format; wherein the storage capacity of the above-mentioned remote storage device is greater than the storage capacity of the target database, and the storage cost of the remote storage device is lower than the storage cost of the local storage device.

[0119] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the data storage method provided by the above methods, the method comprising: using real-time Flink CDC to establish a connection with a target database and obtain business data in the target database; obtaining a target data format supported by a remote storage device, and performing data conversion processing on the business data according to the target data format, and determining the data to be stored in the target data format; dynamically generating a target object storage file corresponding to the business data in the remote storage device, and storing the data to be stored in the target object storage file according to the object storage protocol corresponding to the target data format; wherein the storage capacity of the above remote storage device is greater than the storage capacity of the target database, and the storage cost of the remote storage device is lower than the storage cost of the local storage device.

[0120] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0121] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data storage method, characterized in that: Applied to a data storage system, the data storage system is built based on the data integration tool Flink CDC, the method includes: Use real-time Flink CDC to establish a connection with the target database and obtain business data in the target database; Acquire a target data format supported by a remote storage device, and perform data conversion processing on the business data according to the target data format to determine data to be stored in the target data format; Dynamically generate a target object storage file corresponding to the business data in the remote storage device, and store the data to be stored in the target object storage file according to an object storage protocol corresponding to the target data format; Among them, the access method of the files stored in the remote storage device is the same as the access method of the files stored in the local database, the storage capacity of the remote storage device is greater than the storage capacity of the target database, and the storage cost of the remote storage device is lower than the storage cost of the local storage device.

2. The data storage method according to claim 1, characterized in that: The performing data conversion processing on the business data according to the target data format to determine the data to be stored in the target data format includes: Acquiring data operation information of the user on the target database; the data operation information includes at least one target data table in the target database that needs to be operated, and a materialized view modeling operation required to be performed on the target data table; Acquire a target data table in the target database according to the data operation information, and perform a materialized view modeling operation on the target data table to obtain a target materialized view corresponding to the target data table; The target materialized view is format converted according to the target data format to determine the data to be stored in the target data format.

3. The data storage method according to claim 1, characterized in that: The business data includes an identifier of a target data generating object that generates the business data, and storing the data to be stored in the target object storage file according to an object storage protocol corresponding to the target data format includes: According to the identifier of the target data generating object, determining the target classifier corresponding to the identifier of the target data generating object in a preset first corresponding relationship; the first corresponding relationship includes identifiers of different data generating objects and classifiers corresponding to the identifiers of each data generating object; Determine the target resource group corresponding to the target classifier according to the mapping relationship between different classifiers and different resource groups; the different resource groups are isolated from each other; The target resource group is used to store the data to be stored in the target object storage file according to the object storage protocol corresponding to the target data format.

4. The data storage method according to claim 1, characterized in that: The business data includes a target session variable corresponding to the business data, and storing the data to be stored in the target object storage file according to the object storage protocol corresponding to the target data format includes: According to the target session variable, determining a target resource group corresponding to the target session variable in a preset second corresponding relationship; the second corresponding relationship includes different session variables and resource groups corresponding to each session variable, and different resource groups are isolated from each other; The target resource group is used to store the data to be stored in the target object storage file according to the object storage protocol corresponding to the target data format.

5. The data storage method according to any one of claims 1 to 4, characterized in that: The method further comprises: Obtaining a data query request; the data query request includes an identifier of the data to be queried; Determine whether the data to be queried is hot data or cold data according to the identifier of the data to be queried; the hot data is data with a query frequency higher than a threshold, and the cold data is data with a query frequency not higher than the threshold; If the data to be queried is hot data, performing data query in the local storage device according to the identifier of the data to be queried to obtain the data to be queried; If the data to be queried is cold data, the data to be queried is obtained in the remote storage device according to the identifier of the data to be queried, and the data to be queried is loaded into the local storage device for caching.

6. The data storage method according to any one of claims 1 to 4, characterized in that: The method further comprises: Get data query request; Parsing the data query request to determine whether the data query request includes annotation information; the annotation information is used to represent routing information of the data query request; If the data query request includes annotation information, routing the data query request to a slave node, so as to perform data query according to the data query request through the slave node; The data storage system includes a master node and a slave node, and the master node is used to perform data conversion processing and storage processing on the business data.

7. The data storage method according to any one of claims 1 to 4, characterized in that: The business data includes real-time change data in the target database, and obtaining the business data in the target database includes: The Flink CDC is used to parse the binary log data of the target database in real time, and a streaming pipeline is constructed to obtain real-time change data in the target database; According to the primary key of each data table and / or the row size of each data table in the target database, the real-time change data is divided into a plurality of snapshot blocks; A plurality of threads are used to read the plurality of snapshot blocks in parallel; wherein each thread reads one snapshot block correspondingly.

8. A data storage device, characterized in that: Applied to a data storage system, the device comprises: A data acquisition module is used to establish a connection with a target database using the real-time data integration tool Flink CDC and acquire business data from the target database; A data conversion module, used to obtain a target data format supported by a remote storage device, and perform data conversion processing on the business data according to the target data format to determine the data to be stored in the target data format; A data storage module is used to dynamically generate a target object storage file corresponding to the business data in the remote storage device, and store the data to be stored in the target object storage file according to the object storage protocol corresponding to the target data format; wherein the access method of the files stored in the remote storage device is the same as the access method of the files stored in the local database, the storage capacity of the remote storage device is greater than the storage capacity of the target database, and the storage cost of the remote storage device is lower than the storage cost of the local storage device.

9. A data storage system, characterized in that: The data storage system is built based on the data integration tool Flink CDC, and the system includes: The data source link layer is used to establish a connection with the target database using real-time Flink CDC and obtain business data from the target database; A data construction synchronization layer is used to obtain a target data format supported by a remote storage device, and perform data conversion processing on the business data according to the target data format to determine the data to be stored in the target data format; dynamically generate a target object storage file corresponding to the business data in the remote storage device, and store the data to be stored in the target object storage file according to the object storage protocol corresponding to the target data format; wherein the access method of the files stored in the remote storage device is the same as the access method of the files stored in the local database, the storage capacity of the remote storage device is greater than the storage capacity of the target database, and the storage cost of the remote storage device is lower than the storage cost of the local storage device; The data query operation layer is used to obtain a data query request and perform a data query according to the data query request.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data storage method according to any one of claims 1 to 7 is implemented.

Images