Data processing method and device, and electronic device

Abstract

Embodiments of the present application provide a data processing method and device, and electronic equipment. The data processing method comprises: reading first data in a time series database, wherein the first data is stored in the time series database in a first storage format; dumping the first data into second data in a preset storage format, and uploading the second data to an open data storage platform. Through the scheme provided by the embodiments, the query efficiency can be improved, and the problem of time series database exception can be avoided as much as possible.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of database, and in particular, to a data processing method and device, and an electronic device. BACKGROUND

[0002] Generally, in order to save a specific type of data and achieve a specific purpose, a special database is used, for example, a time series database is used to store time series data. Through the time series database, a large amount of data can be inserted into the database according to the corresponding time sequence relationship, and can be clustered or filtered based on time.

[0003] However, in order to store a specific type of data or achieve a specific purpose, the data storage method or data format stored in the database may change, that is, the data is stored in the database through a specific storage format of the database, for example, the time series database generally uses a log-structured merge tree (LSM tree) to store data.

[0004] However, for this storage method, when the amount of data queried is large, the query efficiency is low, and the time series database is prone to abnormality. SUMMARY

[0005] Therefore, one of the technical problems solved by embodiments of the present application is to provide a data processing method and device, and an electronic device, to solve or alleviate the above problems.

[0006] In a first aspect of the embodiments of the present application, a data processing method is provided, comprising: reading first data in a time series database, wherein the first data is stored in the time series database in a first storage format; dumping the first data into second data in a preset storage format, and uploading the second data to an open data storage platform.

[0007] In a second aspect of the embodiments of the present application, a data processing method is provided, comprising: reading first data in a time series database, wherein the first data is stored in the time series database in a first storage format; dumping the first data into second data in a preset storage format, and uploading the second data to an open data storage platform based on cloud-native object storage, to provide a batch query service for data through the open data storage platform based on cloud-native object storage.

[0008] In a third aspect of the embodiments of the present application, a data processing apparatus is provided, comprising: a reading module configured to read first data in a time series database, wherein the first data is stored in the time series database in a first storage format; a dumping module configured to dump the first data into second data in a preset storage format; and an uploading module configured to upload the second data to an open data storage platform.

[0009] In a fourth aspect of the embodiments of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus; the memory is configured to store at least one executable instruction, the executable instruction causes the processor to perform the operation corresponding to the data processing method as described above.

[0010] In a fifth aspect of the embodiments of the present application, a computer storage medium is provided, which stores a computer program, the program is executed by a processor to implement the data processing method as described above.

[0011] The data processing scheme provided by the embodiments of the present application, by reading the first data in the time series database, wherein the first data is stored in the time series database in a first storage format; the first data is dumped into second data in a preset storage format, and the second data is uploaded to an open data storage platform. The open data storage platform can conveniently and directly open the data to the user for user query, and the second data stored in the open data storage platform is obtained by dumping in a preset storage format, and the compatibility of the preset data storage format is good, so that the user can directly query the second data through the open data storage platform, conveniently obtain the query result, improve the query efficiency, and can avoid the problem of abnormal time series database caused by large amount of query data. BRIEF DESCRIPTION OF DRAWINGS

[0012] Some specific embodiments of the embodiments of the present application will be described in detail hereinafter with reference to the accompanying drawings in an exemplary and non-limiting manner. The same reference signs in the drawings denote the same or similar components or parts. Those skilled in the art should understand that the drawings are not necessarily drawn to scale. In the drawings:

[0013] Figure 1 A schematic diagram of the principle of querying data provided by the present application;

[0014] Figure 2A A flowchart of the data processing method in the first embodiment of the present application;

[0015] Figure 2B A schematic diagram of the use scenario of one of the embodiments of the present application;

[0016] Figure 3 Figure 1 is a flowchart of a data processing method according to an embodiment of the present application;

[0017] Figure 4A Figure 2 is a flowchart of a data processing method according to another embodiment of the present application;

[0018] Figure 4B Figure 3 is a structural diagram of a database shard according to the embodiment of the present application;

[0019] Figure 4B Figure 4 is a flowchart of reading first data according to the embodiment of the present application;

[0020] Figure 4B Figure 5 is a flowchart of obtaining second data according to the embodiment of the present application;

[0021] Figure 4B Figure 6 is a flowchart of a data processing method according to another embodiment of the present application;

[0022] Figure 5 Figure 7 is a structural diagram of a data processing device according to another embodiment of the present application;

[0023] Figure 6 Figure 8 is a structural diagram of some electronic devices for implementing the data processing method according to the embodiments of the present application. DETAILED DESCRIPTION

[0024] It is not necessary for all the advantages of the present application to be achieved simultaneously, as each advantage can be achieved by a single embodiment of the present application.

[0025] In order to enable persons skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art should belong to the scope of protection of the present application.

[0026] The specific implementation of the embodiments of the present application will be further described below with reference to the drawings in the embodiments of the present application.

[0027] In order to more conveniently understand the schemes provided by the embodiments of the present application, the commonly used database will be described first.

[0028] Taking time series databases as an example, they are primarily used to store large amounts of real-time time series data, such as real-time monitoring metrics for database instances, machine performance metrics, and network latency statistics for data transmission. When storing files, time series databases not only store the latest data but also historical data, adding timestamps to historical data to establish temporal relationships. Furthermore, time series databases can also perform pre-aggregation on large amounts of time series data.

[0029] To ensure the storage of time series data, time series databases generally use a log-structured merge-tree (LSM tree) to store data and can add specific timestamps or tags to the data, which is significantly different from the preset data. Therefore, in order to facilitate the time series database to operate on the data, the data is generally stored in the first storage format proprietary to the time series database.

[0030] However, if Figure 7 As shown in the figure, when performing a query, the user's client generally sends a query request to the time series database and calls the query interface of the time series database's computing engine. The time series database itself queries the data stored in it, converts it into a format that the user can parse, and then returns the data to the user as response data.

[0031] Therefore, when the amount of data being queried is large, it will put a lot of pressure on the recommendation engine, thereby affecting the operation of the time series database. It is very likely that due to the large amount of data requested by a certain query request, the processing resources of other query requests will be occupied, resulting in a longer processing time for other query requests, and may even cause the time series database to crash.

[0032] Furthermore, since the data in the time series database is stored in the first storage format, the user cannot parse it and cannot locate the data that the user wants to query, so the user cannot query by directly reading the files in the database.

[0033] To solve the above problems, the embodiment of the present application provides a data processing method, comprising the following steps: reading first data in a time series database, wherein the first data is stored in the time series database in a first storage format; dumping the first data into second data in a preset storage format, and uploading the second data to an open data storage platform. The open data storage platform can conveniently and directly open the data to the user for querying. The open data storage platform stores the second data in the preset storage format obtained by dumping, and the preset data storage format has good compatibility, so that the user can directly query the second data through the open data storage platform, conveniently obtain the query result, improve the query efficiency, and can avoid the problem of time series database exception caused by large amount of query data.

[0034] The scheme provided by the present application will be described in detail below through a specific method flow.

[0035] Figure 1 The schematic diagram of the data processing method in the embodiment of the present application is shown in FIG. 1, which comprises the following steps: Figure 2A

[0036] S101, reading first data in a time series database.

[0037] The first data is stored in the time series database in a first storage format.

[0038] The first storage format is a storage format special for the time series database, which can be generally parsed by the time series database itself or by a special parsing software.

[0039] In the embodiment, the above steps can be executed by the time series database, that is, the first data is read by the time series database itself and parsed.

[0040] In addition, the above steps can also be executed by an electronic device installed with the parsing software, that is, the electronic device reads the first data in the time series database through the installed parsing software.

[0041] S102, dumping the first data into second data in a preset storage format, and uploading the second data to an open data storage platform.

[0042] In the embodiment, the preset storage format can be a storage format compatible with a general open data storage platform, or a storage format that can be directly parsed by most users. For example, the preset storage format can be an ORC file (Optimized Record Columnar File), parquet, etc., which is not limited in the embodiment.

[0043] ​In this embodiment, the first data is converted from the first storage format to the preset storage format, and the second data in the preset storage format is obtained and uploaded to the open data storage platform. When a user performs a query, especially a query with a large amount of data, the user can directly perform the query based on the second data in the preset storage format stored in the open data storage platform, conveniently obtains the query result, improves the query efficiency, and can avoid the problem of abnormality of the time series database caused by a large amount of query data.

[0044] The above process is exemplarily described below through a specific scenario.

[0045] As shown in Figure 2A , the data processing system can include a time series database and an open data storage platform.

[0046] The time series database is embedded with a dump module.

[0047] The time series database can be used to store time series data, and specifically can include a plurality of database shards, etc.

[0048] The dump module is used for format conversion. Specifically, the dump module can read the first data stored in the storage module of the time series database, and dump the first data into second data in a preset storage format, and then upload the second data in the preset storage format to the open data storage platform. The dump module can perform real-time conversion on the first data.

[0049] When a user performs a query, especially a query with a large amount of data, for example, queries traffic data in Beijing for a month, the user can directly perform the query based on the second data stored in the open data storage platform. The query process can not call the query interface of the computing engine of the time series database, that is, part of the query service of the time series database can be provided by the open data storage platform, so as to avoid the problem of abnormality of the time series database caused by a large amount of query data.

[0050] The scheme provided by the embodiment comprises the following steps: reading first data in a time series database, wherein the first data is stored in the time series database in a first storage format; dumping the first data into second data in a preset storage format, and uploading the second data to an open data storage platform. The open data storage platform can conveniently and directly open the data to the user for the user to query. The open data storage platform stores the second data in the preset storage format obtained by dumping, and the preset data storage format has good compatibility and can be directly parsed by the user or other databases. Thus, the user can directly query the second data through the open data storage platform, conveniently obtain the query result, improve the query efficiency, and can avoid the problem of time series database exception caused by large query data volume.

[0051] The data processing method of the embodiment can be executed by any appropriate electronic device with data processing capability, including but not limited to a mobile terminal, a PC, a node in a time series server, and the like.

[0052] Figure 2B FIG. 2 is a schematic diagram of the data processing method in the second embodiment of the present application; as shown in the figure, the method comprises the following steps: Figure 3

[0053] S201, reading first data in a time series database, wherein the first data is stored in the time series database in a first storage format.

[0054] The specific implementation of the step can refer to the above step S102.

[0055] In addition, the step S201 can be executed in real time or at a fixed time.

[0056] Optionally, if the step S201 is executed in real time, the first data is data written in the time series database in real time.

[0057] In the embodiment, by taking the data written in the time series database in real time as the first data, a part of the first data can be converted into the second data in the preset storage format every time the first data is written, which can guarantee the consistency of the second data with the data in the time series database and improve the real-time performance of the second data.

[0058] In another implementation, if the step S201 is executed at a fixed time, the step S201 can be executed after the newly written data in the time series database reaches a certain data volume, or the step S201 can be executed at a certain time interval, which is not limited in the embodiment.

[0059] ​Optionally, in the embodiment, the first data stored in the time series database in the first storage format can be read by calling a read function of the time series database. Thus, the first data stored in the first storage format can be read from the time series database by means of the data reading method of the time series database itself, without the need to additionally install software or the like for parsing the first data.

[0060] S202, a file in a preset storage format corresponding to the data table where the first data is located is created.

[0061] A data table is the basis and core of a database at a logical level, and data is generally stored in a data table. At a physical level, data is stored in data blocks, and data in a data table can be stored in multiple data blocks, and data in a data block belongs to the same data table.

[0062] When writing, writing is generally based on a data table, such as writing data to a certain row of a certain data table. Thus, in order to ensure the logical accuracy between files for storing second data, a file in a preset storage format corresponding to a data table can be created.

[0063] Optionally, in the embodiment, step S202 can include:

[0064] According to the organization structure information of the data table, organization structure information of the file corresponding to the data table is generated, and according to the organization structure information of the file, the file in the preset storage format with the same name as the data table is created.

[0065] Generally, organization structure information (schema information) can be used to define the type of a field and the type of the value of the field. Generally, the organization structure information of data in a table is the same, and after the data in the table is converted into second data in a preset storage format, the data still needs to be stored according to the organization structure information.

[0066] Therefore, in the embodiment, the organization structure information of the file corresponding to the data table can be generated according to the organization structure information of the data table, that is, the fields of the file and the values of the fields can be defined, and then according to the organization structure information of the file, the file in the preset storage format with the same name as the data table is created, so that the fields of the created file are consistent with the field definitions of the data table.

[0067] S203, according to the index information of the data table, data of the data table is written into the created file.

[0068] The second data is data written into the file.

[0069] Since the file is a file in the preset storage format, the data written in the file is data in the preset storage format, and the process of writing data in the file is a dumping process.

[0070] The index information of the data table can be used to indicate the order of each data in the data table. According to the index information, the data in the data table can be written into the corresponding file through multiple write operations, and the reading order of the data written in the file can be ensured, that is, the accuracy of the data in a file at the logical level is ensured.

[0071] In this application, when dumping, a file in the preset storage format corresponding to the data table can be created through step S202, and the data in the data table can be written into the corresponding file according to the index information of the data table through step S203, so as to ensure the accuracy of the file for storing the second data at the logical level, that is, the accuracy of the second data.

[0072] Optionally, step S203 can include: determining the data table corresponding to the data block read into the memory according to the index information, wherein the data of the data table is stored in at least one data block; and writing the data read into the memory into the file corresponding to the data table.

[0073] Generally, the data in a data table can be stored in multiple data blocks. In step S201, the data blocks can be read into the memory, and in this step, the data table where the read data blocks are located can be determined according to the index information, and the data can be written into the file corresponding to the data table by operating the memory. Thus, only one or more data blocks can be read each time, without the need to read all data blocks corresponding to the data table, thereby reducing the requirement of the dumping process on the memory.

[0074] S204, uploading the file after writing data into the open data storage platform, and deleting the file from the local storage of the time series database.

[0075] In this embodiment, the file can be uploaded to the open data storage platform and deleted from the local storage of the time series database after the data in each data table is written into the file. Alternatively, the file can be uploaded to the open data storage platform and deleted from the local storage of the time series database after the data amount of the file reaches a certain size after writing is completed, which is not limited in this embodiment.

[0076] After the file is uploaded to the open data storage platform, the file is deleted from the local storage, which can reduce the occupation of the local storage.

[0077] Optionally, in the embodiment, the file can be stored under a multi-level storage directory of the open data storage platform, the time series database is a database with a multi-level storage structure, and the multi-level storage directory corresponds to a storage location of a data table corresponding to the file in the multi-level storage structure.

[0078] When performing a query, the file can be directly queried according to the multi-level storage directory, and the multi-level storage directory of the open data storage platform can follow the multi-level storage structure of the time series database, so that the file can be directly obtained from the corresponding multi-level file directory according to the multi-level storage structure of the time series database, and the query is more convenient.

[0079] For example, the multi-level storage structure can include region-host-database shard, and the corresponding multi-level storage directory can be region / host / database shard / file name.

[0080] Further, in the embodiment, since the time series database is used to store time series data, the multi-level storage directory also corresponds to the creation time of the file. Therefore, the time series of the data stored in the file can be directly determined according to the creation time of the file, so that the query is more convenient. For example, the multi-level storage directory can be region / host / database shard / yyyymmdd / file name, where yyyymmdd is the year-month-day of creating the file.

[0081] The scheme provided in the embodiment can read first data in the time series database, the first data is stored in the time series database in a first storage format, convert the first data into second data in a preset storage format, and upload the second data to an open data storage platform. The open data storage platform can conveniently and directly open the data to the user for querying, and the open data storage platform stores the second data in the preset storage format obtained by conversion. The preset data storage format has good compatibility and can be directly analyzed by the user or other databases. Therefore, the user can directly query the second data through the open data storage platform, conveniently obtain the query result, improve the query efficiency, and can avoid the problem of abnormal time series database caused by large amount of query data.

[0082] The data processing method of the embodiment can be executed by any appropriate electronic device with data processing capability, including but not limited to: mobile terminal, PC, node in time series server, etc.

[0083] Figure 3 The schematic diagram of the data processing method in Embodiment Three of the present application; as shown in the figure, it includes the following steps:

[0084] S301, reading data in a database shard stored on a disk of the time series database through a database shard reading function Shard Reader of the time series database.

[0085] In this embodiment, referring to Figure 4A , a method for reading data in a database shard is shown.

[0086] In this embodiment, for each database shard, the method shown can be performed once Figure 4B until the dumping is completed.

[0087] As shown in Figure 4B , it includes:

[0088] S3011, generating a corresponding shard reader for the current database shard of the time series database.

[0089] As shown in Figure 4B , the data structure of a database shard is shown. As described in the figure, the database shard includes a data header Header, index information Index of the database shard, storage space information Bucket of the database, and data blocks Blk1, Blk2, etc.

[0090] S3012, reading the index information index in the database shard into the memory through the shard reader.

[0091] In this embodiment, since the index information index is a sortable data structure in the database management system, it is used to assist fast query and update of data in the database table, and is not used to store specific data. Therefore, the data amount of the index information index is small, and the occupied memory space is also small.

[0092] S3013, traversing the index information index to obtain the storage space information (Bucket information) in the database shard, and obtaining the storage location of the data blocks blocks according to the Bucket information.

[0093] S3014, writing the data in the blocks into the memory according to the storage location of the data blocks blocks.

[0094] In this embodiment, since the data amount in a database shard shard is large, the data in the database shard is loaded into the memory in batches. Each time, only the data in one database block can be read into the memory, and the subsequent step S302 is performed for dumping. After the dumping is completed, the information in the next block is read, and the traversal of the blocks in the current database shard is completed according to the index information index.

[0095] In this embodiment, by reading the index information first, and then reading the data blocks blocks according to the index information, the memory occupation of the database in the entire dumping process can be reduced.

[0096] Of course, the above is only an example, and in other implementations, the data in the database shards can also be read into the memory according to other conditions, and the present embodiment does not limit this.

[0097] S302, dump the data in the read database shard to generate an ORC file.

[0098] The ORC file format is a columnar storage format that can reduce data storage space and speed up query speed.

[0099] The ORC file is currently also supported by some open data storage platforms. In addition, the ORC file has an index and can be read by SDKs in multiple languages, which can improve the efficiency of file reading, facilitate user use, and also improve the compatibility of the second data stored in the file.

[0100] In addition, when writing data into the ORC file, a compression method can be selected, so that by selecting an appropriate compression method, the storage space of the ORC file can be saved.

[0101] Of course, in other implementations of the present embodiment, instead of generating an ORC file, a file in other preset storage formats can be generated, and the present embodiment does not limit this.

[0102] In this embodiment, as shown in Figure 4C , step S302 can specifically include:

[0103] S3021, according to the parameters of the data table in the database shard, determine the parameters of the ORC file corresponding to the data table, and create a file corresponding to the data table.

[0104] For example, the name of the data table table can be used as the file name of the ORC file; according to the organization structure information (schema information) of the data stored in the data table, the schema information of the ORC file is generated; according to the storage path of the data table, the storage path path of the ORC file is determined; according to the data amount of the data table, the batch size of the ORC file is set, that is, the update data amount of the ORC file is set.

[0105] Optionally, in this embodiment, each bucket of the database shard can correspond to a data table, and of course, one data table can correspond to multiple buckets, or one bucket can correspond to multiple data tables, and the present embodiment does not limit this.

[0106] S3022, create an ORC writer corresponding to the data table, for writing data into an ORC file corresponding to the data table.

[0107] In this embodiment, the data written can be the data of the memory in step S3014.

[0108] S3023, write the data stored in the memory belonging to the data table into the ORC file through the created ORC writer.

[0109] In this embodiment, in the above steps, the schema information of the ORC file is generated according to the schema information of the data, and then in this step, the data can be written into the column of the ORC file with corresponding schema information according to the schema information of the data.

[0110] Of course, the above-mentioned ORC file is only for illustration, and in other implementation manners of the present application, it can also be stored in other preset storage formats, such as Parquet, etc., and this embodiment does not limit this.

[0111] S3024, after the writing is completed, write the generated ORC file into the hard disk.

[0112] In this embodiment, the writing can be performed according to the batch size of the ORC file, and when one batch size is satisfied, the ORC file is flushed to the hard disk of the database.

[0113] After all the data is written, the ORC file can be closed. The ORC file will automatically generate description information.

[0114] S303, upload the generated ORC file to an open data storage platform based on cloud-native object storage, as open data of a time series database, and provide a batch query service for data through the open data storage platform based on cloud-native object storage.

[0115] In this embodiment, after the generated ORC file is uploaded, the batch query service for data can be provided through the open data storage platform based on cloud-native object storage. Especially when a user obtains a large amount of non-real-time time series data, the user does not need to call the query interface of the computing engine of the time series database, but can directly query based on the open data storage platform based on cloud-native object storage, so as to obtain batch non-real-time data, and meet the user's demand for obtaining a large amount of non-real-time data.

[0116] In the embodiment, since the storage space of the electronic device performing the above steps S301-S303 is limited, the ORC file can be uploaded to the open data storage platform based on the cloud-native object storage, and the local ORC file can be deleted to save storage space.

[0117] Of course, in the embodiment, only the open data storage platform based on the cloud-native object storage is exemplarily described, and in other implementation manners, the ORC file can also be uploaded to other open data storage platforms, such as a cloud disk, a big data platform, and the like, which are also within the protection scope of the present application.

[0118] In addition, in order to facilitate users to obtain the required data, in the embodiment, the ORC file stored in the cloud-native object storage platform is divided into storage directories.

[0119] In the embodiment, the storage directory can include multiple levels.

[0120] Specifically, the storage directory can include seven levels. The first level storage directory can correspond to the region information of the database, for example, the database in the Shanghai region corresponds to a first level storage directory, and the database in the Beijing region corresponds to another first level storage directory. The second level storage directory can correspond to the host of the database, which can be directly named according to the name of the host, for example, the name of the second level storage directory is name1, which can correspond to host1, and the name of the second level storage directory is name2, which can correspond to host2. The third level storage directory can correspond to the data table table stored in the host. The fourth level storage directory can correspond to the time of dumping the data table. The fifth level directory can correspond to the schema information of the ORC file. The data with the same schema information in the data table can be stored under the same fifth level directory. The sixth level storage directory can correspond to the database partition information (partition information) and step information (step information) of the time series database. The seventh level storage directory corresponds to the shard.

[0121] In the embodiment, since the storage directory corresponds to the multi-level storage structure in the database and the ORC file creation time, when querying, the query request can be directly queried according to the storage directory, so that the data query is more convenient.

[0122] Of course, the above is only for example description, and does not limit the present application. In other implementation manners of the present application, the multi-level storage directory can also be set in other manners, which is also within the protection scope of the present application.

[0123] The scheme provided by the embodiment comprises the following steps: reading first data in a time series database, wherein the first data is stored in the time series database in a special first storage format; dumping the first data into second data in a general preset storage format, and uploading the second data to an open data storage platform. The open data storage platform can conveniently and directly open the data to the user for the user to query. The open data storage platform stores the second data in the general preset storage format obtained by dumping, and the general preset data storage format has good compatibility and can be directly parsed by the user or other databases. Thus, the user can directly query the second data through the open data storage platform, conveniently obtain the query result, improve the query efficiency, and can avoid the problem of time series database exception caused by large query data volume.

[0124] The data processing method of the embodiment can be executed by any appropriate electronic device with data processing capability, including but not limited to: a mobile terminal, a PC, a node in a time series server, etc.

[0125] Figure 4D FIG. 4 is a schematic diagram of the data processing method in the fourth embodiment of the present application; as shown in the figure, it comprises the following steps: Figure 5

[0126] S401, reading first data in a time series database, wherein the first data is stored in the time series database in a first storage format.

[0127] The first storage format is a storage format special for the time series database, which can be parsed by the time series database itself or by special parsing software.

[0128] In the embodiment, the above steps can be executed by the time series database, i.e. the first data is read and parsed by the time series database itself.

[0129] In addition, the above steps can also be executed by an electronic device installed with parsing software, i.e. the first data in the time series database is read by the electronic device through the installed parsing software.

[0130] S402, dumping the first data into second data in a preset storage format, and uploading the second data to an open data storage platform based on cloud-native object storage, to provide batch query service of data through the open data storage platform based on cloud-native object storage.

[0131] ​In this embodiment, after uploading the generated second data, a batch query service of the data can be provided through an open data storage platform (OSS) based on cloud-native object storage. In particular, when a user obtains a large amount of non-real-time time series data, the user does not need to call the query interface of the computing engine of the time series database again, but can directly query based on the open data storage platform of cloud-native object storage, so as to obtain batch non-real-time data and meet the user's demand for obtaining a large amount of non-real-time data.

[0132] Of course, in this embodiment, only the open data storage platform based on cloud-native object storage is exemplarily described, and in other implementation manners, the second data can also be uploaded to other open data storage platforms, such as a cloud disk, a big data platform, and the like, which are also within the protection scope of the present application.

[0133] In addition, in order to facilitate the user to obtain the required data, in this embodiment, the second data stored in the cloud-native object storage platform is divided into a storage directory.

[0134] The specific division method of the directory can refer to the above-mentioned embodiments, which will not be described here again.

[0135] In this embodiment, since the format of the first data is converted from the first storage format to the preset storage format, the second data in the preset storage format is obtained, and the second data is uploaded to the open data storage platform. When the user queries, especially when the data volume is large, the second data in the preset storage format stored in the open data storage platform can be directly queried, so that the query result is obtained conveniently, the query efficiency is improved, and the pressure on the computing engine of the time series database caused by the query service can be avoided as much as possible. In addition, the problem of time series database exception caused by large data volume of query can also be avoided.

[0136] The scheme provided by the embodiment comprises the following steps: reading first data in a time series database, wherein the first data is stored in the time series database in a first storage format; dumping the first data into second data in a preset storage format, and uploading the second data to an open data storage platform based on cloud-native object storage, so as to provide a batch query service for data by the open data storage platform based on cloud-native object storage. The open data storage platform based on cloud-native object storage can conveniently and directly open the data to the user for querying, especially for batch querying of non-real-time data. The second data stored in the open data storage platform based on cloud-native object storage is in the preset storage format obtained by dumping, and the preset data storage format has good compatibility and can be directly parsed by the user or other databases. Therefore, the user can directly query the second data through the open data storage platform, conveniently obtain the query result, improve the query efficiency, and can avoid the problem of abnormal time series database caused by large amount of query data.

[0137] The data processing method of the embodiment can be executed by any appropriate electronic device with data processing capability, including but not limited to: a mobile terminal, a PC, a node in a time series server, etc.

[0138] Figure 5 FIG. 5 is a structural schematic diagram of a data processing apparatus according to an embodiment of the present application. Figure 6 As shown in the figure, the data processing apparatus comprises a reading module 501, a dumping module 502, and an uploading module 503.

[0139] The reading module 501 is configured to read first data in a time series database, wherein the first data is stored in the time series database in a first storage format.

[0140] The dumping module 502 is configured to dump the first data into second data in a preset storage format.

[0141] The uploading module 503 is configured to upload the second data to an open data storage platform.

[0142] Optionally, the reading of the first data in the time series database comprises: reading the first data stored in the time series database in the first storage format by calling a reading function of the time series database.

[0143] Optionally, the dumping of the first data into second data in a preset storage format comprises: creating a file in the preset storage format corresponding to a data table where the first data is located; and writing data of the data table into the created file according to index information of the data table, so as to dump the first data into second data in the preset storage format.

[0144] Optionally, the storing the obtained second data into the open data storage platform comprises: uploading the file after writing data into the open data storage platform, and deleting the file from the local storage of the time series database.

[0145] Optionally, the creating the file in the preset storage format corresponding to the data table in which the first data is located comprises: generating organization structure information of the file corresponding to the data table according to organization structure information of the data table; and creating the file in the preset storage format and having the same name as the data table according to the organization structure information of the file.

[0146] Optionally, the writing the data of the data table into the created file according to the index information of the data table to dump the first data as second data in the preset storage format comprises: determining the data table corresponding to the data block read into the memory according to the index information, wherein the data of the data table is stored into at least one data block; and writing the data read into the memory into the file corresponding to the data table.

[0147] Optionally, the file is stored under a multi-level storage directory of the open data storage platform, the time series database is a database in a multi-level storage structure, and the multi-level storage directory corresponds to a storage position of the data table corresponding to the file in the multi-level storage structure.

[0148] Optionally, the multi-level storage directory also corresponds to a creation time of the file.

[0149] Optionally, the first data is real-time written data in the time series database.

[0150] The scheme provided by the embodiment comprises: reading first data in a time series database, wherein the first data is stored in the time series database in a first storage format; dumping the first data as second data in a preset storage format; and uploading the second data into an open data storage platform. The open data storage platform can conveniently and directly open the data to a user for querying. The open data storage platform stores the second data in the preset storage format obtained by dumping, and the preset data storage format has good compatibility, so that the user can directly query the second data through the open data storage platform, conveniently obtain the query result, improve the query efficiency, and can avoid the problem of time series database exception caused by large query data volume.

[0151] Figure 6 A hardware structure schematic diagram of some electronic devices for performing the data processing method of the present application. According to the schematic diagram shown in the figure, the device comprises: Figure 7 ​

[0152] one or more processors 602 and a memory 604, Figure 7 The one processor 602 is taken as an example.

[0153] The device for performing the information acquisition method can further include a communications interface 606 and a communications bus 608.

[0154] The communications interface 606 is configured to communicate with other electronic devices or servers.

[0155] The processor 602, the memory 604, and the communications interface 606 can communicate with each other through the communications bus 608.

[0156] The memory 604, as a non-volatile computer readable storage medium, can be used to store at least one executable instruction, which can specifically include a program 610, and the program 610 can specifically include a non-volatile software program, a non-volatile computer executable program, and a module, such as program instructions / modules corresponding to the information acquisition method in the embodiments of the present application. The processor 602 implements the information acquisition method in the above method embodiments by running the program 610 stored in the memory 604.

[0157] The memory 604 can include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required by a function. The data storage area can store data created according to the use of the information acquisition device, and the like. In addition, the memory 604 can include a high-speed random access memory 604, and can also include a non-volatile memory 604, such as at least one magnetic disk memory 604, a flash memory device, or other non-volatile solid-state memory 604. In some embodiments, the memory 604 can optionally include a memory 604 remotely arranged relative to the processor 602, and these remote memories 604 can be connected to the information acquisition device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0158] The one or more modules are stored in the memory 604, and when executed by the one or more processors 602, perform the information acquisition method in any of the above method embodiments.

[0159] The above product can execute the method provided in the embodiments of the present application, and has the corresponding function modules and beneficial effects of executing the method. Technical details not described in detail in the embodiments can be referred to the method provided in the embodiments of the present application.

[0160] The electronic device of the embodiments of the present application exists in various forms, including but not limited to:

[0161] (1) Mobile communication device: This kind of device is characterized by having mobile communication function and providing voice and data communication as the main target. This kind of terminal includes: smart phone (such as iPhone), multimedia phone, functional phone, and low-end phone, etc.

[0162] (2) Ultra-mobile personal computer device: This kind of device belongs to the category of personal computer, has computing and processing function, and generally has mobile internet feature. This kind of terminal includes: PDA, MID and UMPC device, such as iPad.

[0163] (3) Portable entertainment device: This kind of device can display and play multimedia content. This kind of device includes: audio and video player (such as iPod), hand-held game machine, electronic book, and smart toy and portable car navigation device.

[0164] (4) Server: Device providing computing service, the server is composed of processor 602, hard disk, memory, system bus, etc., the server is similar to the general computer architecture, but due to the need to provide high reliable service, therefore in the processing capacity, stability, reliability, security, scalability, manageability, etc. Higher requirements.

[0165] (5) Other electronic devices with data interaction function.

[0166] So far, specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0167] In the 1990s, it was quite obvious to distinguish whether an improvement in a technology was in hardware (e.g., improvement in circuit structures of diodes, transistors, switches, etc.) or in software (improvement in method flow). However, as technology has evolved, many improvements in method flow today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flow into hardware circuits. Therefore, it cannot be said that an improvement in a method flow cannot be implemented by hardware entity modules. For example, a programmable logic device (PLD) such as a field programmable gate array (FPGA) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a piece of PLD by the designer programming it by himself, without having to ask a chip manufacturer to design and manufacture a special integrated circuit chip. Moreover, instead of manually fabricating an integrated circuit chip, this programming is now mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing a program, and the original code before compilation also has to be written in a specific programming language, which is called a hardware description language (HDL), and there are many types of HDL, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should be aware that, as long as the method flow is logically programmed in the above-mentioned hardware description languages and programmed into an integrated circuit, a hardware circuit that implements the logical method flow can be easily obtained.

[0168] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to being implemented in pure computer readable program code, the controller can equally well be implemented to perform the same functions using logic gates, switches, an application specific integrated circuit, a programmable logic controller and an embedded microcontroller, etc. by means of a logical programming of the method steps. The controller can thus be considered as a hardware component, and the means comprised therein for performing the various functions can be considered as structures within the hardware component. Alternatively, the means for performing the various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0169] The systems, apparatuses, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0170] For the sake of description, the above apparatuses are described in various units by functions respectively. Of course, the functions of the units can be implemented in one or more software and / or hardware in the implementation of the present application.

[0171] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage etc.) containing computer-usable program code.

[0172] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 7 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0173] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0174] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 Figure 1 Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0175] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0176] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory. The memory is an example of computer-readable media.

[0177] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0178] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0179] Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system or computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code.

[0180] The present application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular transactions or implement particular abstract data types. The present application can also be practiced in a distributed computing environment where transactions are performed by remote processing devices that are connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including storage devices.

[0181] The various embodiments in the specification are described in progressive manner, and the same or similar parts among the various embodiments can be mutually referred to, and each embodiment focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0182] The above only describes the embodiments of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of the claims of the present application.

Claims

1. A data processing method, comprising: Reading first data from a time series database into a memory, wherein the first data is stored in the time series database in a first storage format, and the time series database includes multiple database shards; the first data is data written in real time into the time series database; Dumping the first data in the memory into second data in a preset storage format, and uploading the second data to an open data storage platform, including: generating organizational structure information of a file corresponding to the data table based on organizational structure information of the data table; creating the file with the same name as the data table and in a preset storage format based on the organizational structure information of the file; writing the data of the data table into the created file based on index information of the data table to dump the first data into the second data in the preset storage format; wherein the file is stored in a multi-level storage directory of the open data storage platform, the time series database is a database with a multi-level storage structure, and the multi-level storage directory corresponds to the storage location of the data table corresponding to the file in the multi-level storage structure; The reading the first data from the time series database into the memory includes: reading the first data stored in the time series database in a first storage format into the memory by calling a read function of the time series database.

2. The method according to claim 1, wherein The uploading of the second data to the open data storage platform further includes: The file after data is written is uploaded to the open data storage platform, and the file is deleted from the local storage of the time series database.

3. The method according to claim 1, wherein Writing the data of the data table into the created file according to the index information of the data table to dump the first data into second data in a preset storage format includes: Determining, based on the index information, a data table corresponding to the data block read into the memory, wherein the data in the data table is stored in at least one data block; The data read into the memory is written into the file corresponding to the data table.

4. The method according to claim 1, wherein The multi-level storage directory also corresponds to the creation time of the file.

5. A data processing method, comprising: Reading first data from a time series database, comprising: reading the first data stored in the time series database in a first storage format into a memory by calling a read function of the time series database, wherein the first data is stored in the time series database in the first storage format, the time series database includes a plurality of database shards, and the first data is data written in real time into the time series database; Dumping the first data in the memory into second data in a preset storage format, and uploading the second data to an open data storage platform based on cloud native object storage, so as to provide a batch query service for data through the open data storage platform based on cloud native object storage; The dumping of the first data in the memory into second data in a preset storage format and uploading the second data to an open data storage platform includes: generating organizational structure information of a file corresponding to the data table based on organizational structure information of the data table; creating the file with the same name as the data table and in a preset storage format based on the organizational structure information of the file; writing the data of the data table into the created file based on index information of the data table to dump the first data into the second data in a preset storage format; wherein the file is stored in a multi-level storage directory of the open data storage platform, the time series database is a database with a multi-level storage structure, and the multi-level storage directory corresponds to the storage position of the data table corresponding to the file in the multi-level storage structure.

6. A data processing device comprising: A reading module, configured to read first data from a time series database, comprising: reading the first data stored in the time series database in a first storage format into a memory by calling a read function of the time series database, wherein the first data is stored in the time series database in the first storage format, the time series database includes multiple database shards; and the first data is data written in real time into the time series database; A dump module, configured to dump the first data in the memory into second data in a preset storage format, comprising: generating organizational structure information of a file corresponding to the data table based on organizational structure information of the data table; creating the file having the same name as the data table and in a preset storage format based on the organizational structure information of the file; and writing the data of the data table into the created file based on index information of the data table to dump the first data into the second data in the preset storage format; wherein the file is stored in a multi-level storage directory of an open data storage platform, the time series database is a database with a multi-level storage structure, and the multi-level storage directory corresponds to the storage location of the data table corresponding to the file in the multi-level storage structure; An uploading module is used to upload the second data to the open data storage platform.

7. An electronic device comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the data processing method according to any one of claims 1 to 4.

8. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the data processing method according to any one of claims 1 to 4 is implemented.

Images