Data processing method and device
By dividing data partitions and subpartitions based on data attribute information in a timing database, the problem that the timing database cannot store multiple data types is solved, and the secure and efficient storage of data and rapid query are achieved.
Patent Information
- Application Number
- CN202110182725.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-02-10
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2041-02-10
AI Technical Summary
Due to the single storage data type, existing timing databases lead to data loss and low query efficiency, and cannot effectively store and manage timing data of multiple data types.
By receiving data storage requests, data labels are determined based on the attribute information of the data, data partitions are created and divided, molecular partitions are divided according to the data type, and data is stored in the corresponding sub-partitions, realizing data storage by type.
It avoids data loss, improves the security and efficiency of data storage, supports the rapid storage of massive time-series data, and improves data query efficiency.
Smart Images
Figure CN113297151B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this specification relate to the field of computer technology, and more particularly to a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, and a computer-readable storage medium. Background Art
[0002] As the Internet or the Internet of Things is becoming more and more widely used, the detection systems or Internet of Things devices in the Internet generate more and more time series-based data, which are called time series data. Although time series data comes from different sources, due to reasons such as the version upgrade of the acquisition device or the system upgrade, the data types of time series data are diverse. With the increase in data types, the time series database cannot store data due to the single storage data type set, resulting in data loss, low data security, and inconvenience for subsequent data queries, which reduces data query efficiency. Summary of the invention
[0003] In view of this, the present specification provides a data processing method. One or more embodiments of the present specification also relate to a data processing device, a computing device, and a computer-readable storage medium to solve the technical defects existing in the prior art.
[0004] According to a first aspect of an embodiment of this specification, a data processing method is provided, including:
[0005] Receive a data storage request, wherein the data storage request carries data to be stored, attribute information of the data to be stored, and a data type of the data to be stored;
[0006] Determining a data tag of the data to be stored based on the attribute information of the data to be stored;
[0007] Creating a data partition based on the data tag, and determining the data type of the data to be stored corresponding to the data tag;
[0008] Based on the data type of the to-be-stored data corresponding to the data tag, dividing the data partition corresponding to the data tag into sub-partitions corresponding to the data type;
[0009] Based on the data type of the to-be-stored data corresponding to the data tag, the to-be-stored data corresponding to the data tag is stored in the sub-partition corresponding to the data type.
[0010] According to a second aspect of an embodiment of this specification, there is provided a data processing device, including:
[0011] A receiving module is configured to receive a data storage request, wherein the data storage request carries data to be stored, attribute information of the data to be stored, and a data type of the data to be stored;
[0012] A determination module, configured to determine a data tag of the data to be stored based on the attribute information of the data to be stored;
[0013] A creation module, configured to create a data partition based on the data tag and determine the data type of the to-be-stored data corresponding to the data tag;
[0014] a partitioning module configured to, based on the data type of the to-be-stored data corresponding to the data tag, partition the data partition corresponding to the data tag into sub-partitions corresponding to the data type;
[0015] The storage module is configured to store the data to be stored corresponding to the data tag in the sub-partition corresponding to the data type based on the data type of the data to be stored corresponding to the data tag.
[0016] According to a third aspect of an embodiment of this specification, a computing device is provided, including:
[0017] Memory and processor;
[0018] The memory is used to store computer-executable instructions, and the processor is used to implement the steps of the data processing method when executing the computer-executable instructions.
[0019] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of any one of the data processing methods are implemented.
[0020] In one embodiment of the present specification, a data label of the data to be stored is determined through attribute information of the data to be stored, a data partition is created according to the data label of the data to be stored, and the data type of the data to be stored corresponding to the data label is determined, and then the data partition is divided into corresponding sub-partitions according to the data type of the data to be stored, and the data to be stored is stored in the sub-partition corresponding to the data type, thereby realizing the storage of the data to be stored according to the data type, avoiding data loss caused by the inability to store the data to be stored in the database due to the data type of the data to be stored, not only improving the security of data storage, but also meeting the needs of fast storage of massive time series data, improving storage efficiency, and subsequently improving data query efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 It is a schematic diagram of a time series database model of a data processing method provided by an embodiment of this specification;
[0022] Figure 2 is a flow chart of a data processing method provided by an embodiment of this specification;
[0023] Figure 3 It is a structural schematic diagram of a data processing method provided by an embodiment of this specification applied to recording temperature values;
[0024] Figure 4 is a processing flow chart of a data processing method provided by an embodiment of this specification;
[0025] Figure 5 is a structural schematic diagram of a data processing device provided by an embodiment of this specification;
[0026] Figure 6 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION
[0027] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.
[0028] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.
[0029] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0030] First, the terms involved in one or more embodiments of this specification are explained.
[0031] Metric: represents a collection of similar time series data.
[0032] Tag: describes the characteristics of a data source and usually does not change over time.
[0033] Field: A measurement indicator that describes a data source, which usually changes over time.
[0034] Time Series: A time series formed by the change of a certain indicator of a data source over time. For example, Metric+Tags+Field is a time series.
[0035] Data Point: A measurement indicator value (Field Value) generated by a data source at a certain time.
[0036] Timestamp: The time when the data is generated.
[0037] Dynamic data type: The data type of the data point indicator value that supports writing is dynamic.
[0038] Time series databases are used to describe the changes of things over time and are widely used in industry scenarios such as application performance testing, IoT device testing systems, and industrial Internet. Since the data model of time series databases is different from that of general databases, they need to support the writing of data points in seconds, and at the same time, they need to have functions such as high compression ratio and low-cost storage, pre-downsampling, interpolation, and multi-dimensional aggregation calculations to meet the storage and processing of massive time series data. It is not easy to efficiently manage time series data.
[0039] like Figure 1 As shown, Figure 1 A schematic diagram of a time series database model of a data processing method provided in one embodiment of the present specification is shown.
[0040] Figure 1 It is the industry time series model view, and also refers to the time series model view that stores time series data in a time series database. Figure 1 Part A is a timestamp sequence, which is used to record the time when the time series data is stored, such as 2020-10-24 10:01, 2020-10-24 10:02, 2020-10-24 10:03, etc. It should be noted that the timestamp data can be set with continuous timestamps at a certain time interval according to the actual application; Figure 1 Part B is a tag sequence, which may include information such as a device number and a region. It should be noted that a tag may represent information such as a device that records the time series data and a region where the device is located. Different tag information may be set according to actual applications, and is not limited in any way in the embodiments of this specification. Figure 1Part C is a field sequence, which may include information such as a temperature field and a description field. It should be noted that the field data sequence may be a sequence for storing actual data.
[0041] In practical applications, the storage method of time series data can be found in the above Figure 1 A time series data storage model, in which there can be multiple field sequences of time series data, and the type of the value written in each field sequence is fixed, such as Figure 1 In the example, the data written to the temperature field is floating point data, and the data written to the description field is string data. Based on this, when the data type written to the database is integer data, it cannot be written to the database. In other words, since the data type of the temperature field in the time series database is floating point data, when the temperature field data written is integer data, the corresponding time series database does not support data writing.
[0042] In order to solve the problem that the above-mentioned time series database cannot support the writing of data after data type conversion, the data processing method provided in the embodiment of this specification sets partitions of different data types in the same field sequence to support dynamic type changes of field data, and also provides great help for the flexibility of adapting to more project scenarios.
[0043] It should be noted that time series data has the characteristic of flexible data types. Not only are the data point types of different time lines inconsistent, but the data types of the same time line often change dynamically. Based on this, the writing of multiple data types of time series data on the same time line can be applied to any scenario that stores multiple types of time series data, and this manual does not impose any limitations on this.
[0044] In this specification, a data processing method is provided. This specification also relates to a data processing apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.
[0045] Figure 2 A flow chart of a data processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.
[0046] Step 202: Receive a data storage request, wherein the data storage request carries data to be stored, attribute information of the data to be stored, and a data type of the data to be stored.
[0047] The attribute information of the data to be stored may be understood as a storage timestamp of the data to be stored, measurement information of the data to be stored, and the like.
[0048] The data type of the data to be stored can be understood as the data form or type of the data to be stored, such as integer data, floating point data, numerical string data, and the like.
[0049] Specifically, the server receives a data storage request from the user, wherein the data storage request carries the data to be stored, attribute information of the data to be stored, and the data type of the data to be stored. For example, the server receives a data storage request for temperature data at time t, wherein the data to be stored carried in the data storage request may be the actual temperature data of 15 degrees, the attribute information of the data to be stored may be the time t and the temperature label, and the data type of the data to be stored may be an integer.
[0050] It should be noted that the embodiments of this specification are described in detail by taking the recording of meteorological data of Beijing as an example, wherein the meteorological data includes temperature data, humidity data, air pressure data, etc.
[0051] Step 204: Determine a data tag of the data to be stored based on the attribute information of the data to be stored.
[0052] The data label of the data to be stored can be understood as a measurement label of the data to be stored. For example, the data label can be any label information that can represent data attributes, such as temperature, humidity, air quality, etc.
[0053] It should be noted that the data tags provided in the embodiments of this specification are described in detail using temperature or humidity as an example, but no limitation is imposed on this.
[0054] Specifically, the data label of the data to be stored is determined according to the attribute information of the data to be stored. For example, according to the attribute of the data to be stored being the time t and the temperature label, it can be determined that the data label of the data to be stored is temperature, and then the field of the data to be stored in the time series database can be represented as temperature.
[0055] Step 206: Create a data partition based on the data tag, and determine the data type of the to-be-stored data corresponding to the data tag.
[0056] Among them, data partition can be understood as a data partition for storing time series data. Different data partitions can be divided according to the difference of each field. The storage location of the temperature field is a data partition.
[0057] Specifically, the database creates a data partition for storing the data to be stored based on the data tag of the data to be stored, one data tag corresponds to one data partition, and the database determines the data type of the data to be stored.
[0058] For example, a time series database needs to store the meteorological data of a city within 7 days. The fields stored in the database can be divided into temperature, humidity, air pressure, etc. Then, it can be determined that the data labels of the data to be stored can be temperature, humidity, air pressure, etc. Based on each data label, a corresponding data partition is created in the time series database, and the data stored in temperature is determined to be integer data, the data stored in humidity is floating-point data, and the data stored in air pressure is string data.
[0059] Step 208: Based on the data type of the to-be-stored data corresponding to the data tag, divide the data partition corresponding to the data tag into sub-partitions corresponding to the data type.
[0060] The sub-partition may be understood as a sub-partition divided in the data partition to store data of different data types to be stored.
[0061] Specifically, according to the data type of the data to be stored corresponding to the data tag, the data partition corresponding to the data tag can be divided into sub-partitions corresponding to the data type. Continuing with the above example, when it is determined that the data type of the data to be stored for the storage temperature is integer data, the data partition for storing the temperature can be divided into sub-partitions corresponding to the integer data, and the time series data of the storage temperature in the sub-partition is integer data.
[0062] In practical applications, due to the upgrade of data acquisition equipment or system update, the data type of the data to be stored may be different. In this case, different sub-partitions can be divided according to different data types for data storage in the time series database. Specifically, based on the data type of the data to be stored corresponding to the data tag, the data partition corresponding to the data tag is divided into sub-partitions corresponding to the data type, including:
[0063] Based on at least two data types of the to-be-stored data corresponding to the data tag, the data partition corresponding to the data tag is divided into at least two sub-partitions corresponding to the at least two data types.
[0064] Specifically, the data to be stored may include at least two data types, and the data partition corresponding to the data tag of the data to be stored is divided into at least two sub-partitions, wherein each sub-partition corresponds to a data type of the data to be stored.
[0065] For example, to record time series data of humidity, in the data partition corresponding to the humidity data, integer data can be divided into one sub-partition, floating-point data can be divided into one sub-partition, and string data can be divided into one sub-partition.
[0066] In practical applications, when a large amount of time series data is stored, the data to be stored corresponding to each data label can be at least two data types. At least two types of data to be stored can be determined in a large amount of time series data by classification, and at least two sub-partitions can be divided in the data partition based on the types of multiple data to be stored.
[0067] The embodiments of this specification divide the data partition corresponding to each data tag into at least two sub-partitions, so that the data to be stored corresponding to each data tag is stored in sub-partitions corresponding to different data types, thereby storing the data to be stored in different partitions according to the data type. This not only avoids the inability to write into the time series database due to different data types, but also improves the security of data storage.
[0068] Step 210: Based on the data type of the to-be-stored data corresponding to the data tag, store the to-be-stored data corresponding to the data tag into the sub-partition corresponding to the data type.
[0069] Specifically, after determining the sub-partition corresponding to the data type of the to-be-stored data, the to-be-stored data corresponding to the received data tag is stored in the sub-partition corresponding to the data type.
[0070] In order to determine that the data to be stored of a certain timestamp is quickly stored in the corresponding position of the time series database, the storage position can be determined in the subpartition corresponding to the data type of the data to be stored; specifically, based on the data type of the data to be stored corresponding to the data tag, storing the data to be stored corresponding to the data tag in the subpartition corresponding to the data type includes:
[0071] Determining a target storage file for the data to be stored based on the data to be stored corresponding to the data tag;
[0072] Determine a storage timestamp of the data to be stored in the target storage file, and determine a storage location of the data to be stored in the subpartition corresponding to the data type based on the storage timestamp and the subpartition corresponding to the data type of the data to be stored;
[0073] The data to be stored is stored in a to-be-stored location of the sub-partition corresponding to the data type, wherein the data to be stored includes a storage timestamp and a storage data value.
[0074] The target storage file may be understood as a storage file storing time series data to be stored.
[0075] Specifically, after determining the data tag of the data to be stored, the target storage file of the data to be stored can be determined according to the data to be stored corresponding to the data tag, and the storage timestamp corresponding to the data to be stored can be determined in the target storage file, and then based on the storage timestamp and the sub-partition corresponding to the data type of the data to be stored, the storage position of the data to be stored in the sub-partition corresponding to the data type can be determined, and finally the data to be stored is stored in the sub-partition corresponding to the data type of the data to be stored, wherein the data to be stored includes the storage timestamp and the storage data value.
[0076] In actual applications, the target storage file of the data to be stored can be determined based on the index data of the data to be stored corresponding to the data tag. The target storage file stores the data to be stored, and the target storage file includes a timeline of the data to be stored. According to the timeline of the data to be stored, the data tag is determined according to the timeline of the data to be stored, and sub-partitions are divided according to the data type of the data to be stored in the data partition determined by the data tag, and the storage location of the data to be stored is determined according to the sub-partition and the storage timestamp of the data to be stored, and then the data to be stored is stored.
[0077] For example, when the time series database receives the data to be stored, which is the outdoor temperature data of Beijing at 0 degrees at 2021-1-1 00:00, it can determine that the index data of the data to be stored is the region, and then the target storage file can be determined to be a meteorological data file storing the Beijing area. According to the time carried by the data to be stored, the timeline in the target storage file can be determined, and the storage timestamp can be determined to be 2021-1-1 00:00. According to the temperature data of 0 degrees, it can be determined that the data type of the data to be stored is an integer type. According to the storage timestamp and the data type, the storage location of the data to be stored in the data partition is determined to be A, and the data to be stored at 0 degrees is stored in the storage location A.
[0078] The data processing method provided in the embodiments of the present specification determines the storage location of the data to be stored in the corresponding sub-partition of the data partition, thereby realizing rapid storage of the data to be stored in the data storage file.
[0079] See also Figure 3 , Figure 3 A structural schematic diagram showing the application of a data processing method provided in an embodiment of the present specification to recording temperature values.
[0080] With the above Figure 1 Similarly, the time series data model includes timestamps, labels, and fields. Figure 3 Part A is the timestamp sequence. Figure 3 Part B is the tag sequence. Figure 3 Part C is a sequence of fields, where: Figure 3 The sequence of fields in Figure 1 The field sequences in the data partition are clearly different. For example, the data partition with the temperature field can be divided into multiple data types. Figure 3 The temperature field in is divided into floating-point partitions, integer partitions, and string partitions. The data corresponding to each timestamp corresponds to only one data type. For example, the data type of the data to be stored with a timestamp of 2020-10-24 10:01 is floating-point data, that is, 12.1 degrees. Then the data of 12.1 degrees is stored in the data partition with the data label as temperature, and stored in the floating-point data type sub-partition of the data partition. Then, no data is written in the sub-partitions of other data types, which can be set to empty, or different data can be written according to preset conditions. The embodiments of this specification do not impose any limitations on this.
[0081] Furthermore, after storing the to-be-stored data corresponding to the data tag in the sub-partition corresponding to the data type, the method further includes:
[0082] Determine the data type of the data to be stored corresponding to the data tag as the target data type;
[0083] Determine the data type to be filled in the data partition corresponding to the data tag based on the target data type, and determine the position to be filled for the data to be stored based on the data type to be filled;
[0084] The position to be filled is filled with data based on a preset filling rule.
[0085] The preset filling rule may be understood as a rule for the data content to be filled in the position to be filled, for example, "0" or "null" may be filled in the position to be filled.
[0086] Specifically, the data type of the data to be stored corresponding to the data tag is determined as the target data type. In the data partition corresponding to the data tag, other data types except the target data type can be determined as the data type to be filled, and then the to-be-filled position of the timeline data for the data to be stored is determined, and the data is filled in the to-be-filled position according to the preset filling rules.
[0087] In actual applications, in a list storing time series data, in each timeline, when the data type of the data to be stored is determined to be A, and the data partition also includes data type B and data type C, then data type B and data type C are the data types to be filled, then in the sub-partition corresponding to the data type to be filled, the corresponding filling data can be filled.
[0088] It should be noted that the data filled in the to-be-filled positions in certain timelines do not occupy storage space.
[0089] The data processing method provided in the embodiments of the present specification can store data in an orderly manner in a storage file of time series data by determining a position to be filled in each timeline and filling the position to be filled with empty data, thereby improving storage efficiency.
[0090] After storing the data to be stored, storing the data of the same type, and compressing the data of the same type together, so that the compression ratio is higher, so as to reduce the storage cost; specifically, after storing the data to be stored corresponding to the data tag in the sub-partition corresponding to the data type, it also includes:
[0091] Based on the data type of the to-be-stored data corresponding to the data tag, the to-be-stored data stored in the sub-partition corresponding to each data type is compressed and stored.
[0092] Specifically, after the data to be stored is stored in the sub-partitions corresponding to the corresponding different types, the sub-partitions corresponding to each data type may store a large amount of time series data, and the data to be stored in the sub-partitions corresponding to each data type are compressed and stored.
[0093] In actual applications, the data type corresponding to each data tag is compressed and stored together when stored at the bottom layer, so that data of the same type can be stored together and compressed together to achieve a higher compression ratio, thereby reducing storage costs; further, in subsequent data queries, data of the same type can be traversed continuously when browsing, so that the query disk efficiency is more efficient, and according to different types, the corresponding data tag can be directly queried directly through the index data. The sub-partition corresponding to the data tag.
[0094] The data processing method provided in the embodiments of this specification compresses and stores data of the same type together to improve the compression ratio and reduce data storage costs. This not only enables rapid storage of massive time series data and improves storage efficiency, but also provides convenience for subsequent data queries.
[0095] In addition, the data processing method provided in the embodiment of this specification supports users to write data of different data types on the same data tag by dividing the data partition of the same data tag into sub-partitions of multiple data types. In the subsequent query data, the user also includes:
[0096] Receive a data query request, wherein the data query request carries time attribute information of the data to be queried and a data tag of the data to be queried;
[0097] Determine the target storage file corresponding to the data to be queried based on the time attribute information of the data to be queried;
[0098] Determining the data type of the data to be queried in the target storage file based on the data tag of the data to be queried;
[0099] The sub-partition corresponding to the data type of the data to be queried is determined, and the data to be queried is queried from the sub-partition corresponding to the data type of the data to be queried.
[0100] The time attribute information of the data to be queried can be understood as the time information of the data that the user needs to query.
[0101] Specifically, the time series database receives a data query request, wherein the data query request carries time attribute information of the data to be queried and a data tag of the data to be queried, determines a target storage file corresponding to the data to be queried according to the time attribute information of the data to be queried, and then determines the data type of the data to be queried in the target storage file according to the data tag of the data to be queried, and then determines the sub-partition corresponding to the data type of the data to be queried, and queries the data to be queried from the sub-partition.
[0102] In actual applications, the user's data query request can be a query request for a single piece of data. The storage location of the data to be queried can be determined based on the time attribute information and data tags carried in the single data query request, and the data to be queried can be queried from the sub-partition corresponding to the data type of the data to be queried.
[0103] For example, the user's data query request is to query the temperature data at 2021-1-1 00:01, then the time attribute information of the data to be queried is determined to be 2021-1-1 00:01, and the data tag of the data to be queried is temperature data. Then the target storage file of the data to be queried is determined according to the timestamp, wherein the target storage file has been stored in the order of the timeline, and the target storage file can be determined by the index file of the timeline. In the target storage file, it is determined that the data type of the temperature data stored at 2021-1-1 00:01 is integer, then continue to query specific integer data from the integer sub-partition, and finally the temperature data queried for 2021-1-1 00:01 is 15 degrees.
[0104] It should be noted that, in actual applications, if the queried data is a numerical string data, the data abnormality at that moment can be determined through the string data, and the embodiments of this specification do not impose any limitation on the abnormal data.
[0105] Furthermore, after the time series database receives at least two data query requests, the data to be queried is queried in the sub-partitions corresponding to the data type of each data to be queried. Specifically, the data processing method further includes:
[0106] Receive at least two data query requests, wherein the data query requests carry time attribute information of at least two data to be queried and data tags of at least two data to be queried;
[0107] Determine a target storage file corresponding to each data to be queried based on the time attribute information of the at least two data to be queried;
[0108] Determine the data type of each to-be-queried data in the target storage file corresponding to each to-be-queried data based on the data label of each to-be-queried data;
[0109] The sub-partition corresponding to the data type of each data to be queried is determined, and the data to be queried is queried from the sub-partition corresponding to the data type of each data to be queried.
[0110] Specifically, the time series database receives at least two data query requests, wherein each data query request carries the time attribute information of each data to be queried and the data tag of each data to be queried, determines the target storage file corresponding to each data to be queried according to the time attribute information of the data to be queried, and then determines the data type of each data to be queried in the corresponding target storage file according to the data tag of each data to be queried, and then queries the data to be queried from each subpartition by determining the subpartition corresponding to the data type of each data to be queried.
[0111] For example, the user's data query request is to query the humidity data within 5 days from 2021-1-1 to 2021-1-5. The target storage file of the data to be queried is determined according to the timestamp of the query data, and the temperature data in each timeline is determined in the target storage file, where the queried temperature data on 2021-1-1 is 5 degrees, the temperature data on 2021-1-2 is 4 degrees, the temperature data on 2021-1-3 is 2.1 degrees, the temperature data on 2021-1-4 is 1 degree, and the temperature data on 2021-1-5 is 3.2 degrees.
[0112] The data processing method provided in the embodiments of the present specification queries the data to be queried in the sub-partitions corresponding to the different data types of the data to be queried respectively, so as to enable the user to perform effective data query by querying the specified data type, thereby improving the data query efficiency.
[0113] After multiple pieces of data are queried, operations such as data aggregation or downsampling may be performed based on the multiple pieces of data to implement data query; specifically, after querying the data to be queried based on the sub-partition corresponding to the data type of each piece of data to be queried, the method further includes:
[0114] receiving a data aggregation request for the data to be queried, and determining a data type of each data to be queried based on the data aggregation request;
[0115] Determining data to be processed based on the data type of each data to be queried, and processing the data to be processed to obtain at least two processed data to be queried, wherein the data types of the at least two data to be queried are the same;
[0116] Aggregate data is calculated based on the processed at least two pieces of data to be queried.
[0117] The data aggregation request may be understood as a processing request for performing an aggregation operation on the data, for example, performing a sum operation on the data to be queried, or performing an average operation on the data to be queried, to obtain the data to be queried after the operation.
[0118] Specifically, after receiving a data aggregation request for data to be queried, the time series database can determine the data type of each data to be queried, and then determine the data to be processed according to different data types, process the data to be processed, and obtain at least two processed data to be queried, wherein the data types of the processed data to be queried are the same, and calculate aggregated data for the at least two processed data to be queried.
[0119] In actual applications, due to different devices for recording data or different system levels, the received data types are different. When subsequently processing at least two data to be queried, data processing cannot be achieved due to different data types. Therefore, it is necessary to perform format conversion on different types of data to be queried to ensure that the data types of all data to be queried are the same, so as to facilitate the subsequent calculation of aggregated data based on the same type of data to be queried.
[0120] Continuing with the above example, among the data to be queried, only the temperature data of 2021-1-3 is 2.1 degrees and the temperature data of 2021-1-5 is 3.2 degrees which are floating point data. Then these two floating point data can be converted into integer data. After the format conversion, the temperature data of 2021-1-3 is 2 degrees, and the temperature data of 2021-1-5 is 3 degrees. It is realized that the humidity data within 5 days from 2021-1-1 to 2021-1-5 are all integer data. Then the integer data of 5 days can be summed or averaged.
[0121] It should be noted that format conversion can also be performed on string data, wherein the specific format conversion method can be set according to actual applications, and the embodiments of this specification do not impose any limitation on this.
[0122] In summary, the data processing method provided in the embodiments of the present specification determines the data label of the data to be stored through the attribute information of the data to be stored, creates a data partition according to the data label of the data to be stored, and determines the data type of the data to be stored corresponding to the data label, and then divides the data partition into corresponding sub-partitions according to the data type of the data to be stored, and stores the data to be stored in the sub-partition corresponding to the data type, thereby realizing the storage of the data to be stored according to the data type, avoiding the data loss caused by the inability to store the data to be stored in the database due to the data type of the data to be stored, not only improving the security of data storage, but also meeting the needs of rapid storage of massive time series data, improving storage efficiency, and further improving the efficiency of data query.
[0123] See also Figure 4 , Figure 4 A schematic diagram of the temperature storage process of a data processing method provided in an embodiment of this specification is shown.
[0124] Figure 4 Part A is the data storage file format. The bottom of part A is index data. The storage file mentioned in part A can be found through the index data. The storage location of the data to be stored is determined to be after storage file 3 through the index data. Storage file 3 includes Figure 4 The timeline index data and timeline data in part B, each timeline data includes a data label, such as Figure 4 The timeline data 2 in part B includes multiple data tags in part C, where the data tag 1 in part C in the figure includes three types, namely floating point type, integer type, and string type. The data to be stored is stored in the partition of each data type, such as Figure 4 As shown in part D, the data to be stored is as follows Figure 4 The E part is the format of the data to be stored, including the timestamp and data value.
[0125] In practical applications, the storage location of the data to be stored is determined by the offset in the index data, so as to achieve data storage of a large amount of time series data through data classification. Based on this, by creating dynamic data types, data of different data types are stored separately, which greatly improves the convenience of users using time series databases.
[0126] In summary, the data label of the data to be stored is determined through the attribute information of the data to be stored, the data partition is created according to the data label of the data to be stored, and the data type of the data to be stored corresponding to the data label is determined, and then the data partition is divided into corresponding sub-partitions according to the data type of the data to be stored, and the data to be stored is stored in the sub-partition corresponding to the data type. This realizes the storage of the data to be stored according to the data type, avoids the data loss caused by the inability to store the data in the database due to the data type of the data to be stored, not only improves the security of data storage, but also meets the rapid storage of massive time series data, improves storage efficiency, and then can also improve data query efficiency later.
[0127] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 5 FIG. 1 is a schematic diagram showing the structure of a data processing device provided by an embodiment of the present specification. Figure 5 As shown, the device comprises:
[0128] The receiving module 502 is configured to receive a data storage request, wherein the data storage request carries data to be stored, attribute information of the data to be stored, and a data type of the data to be stored;
[0129] A determination module 504, configured to determine a data tag of the data to be stored based on the attribute information of the data to be stored;
[0130] A creation module 506 is configured to create a data partition based on the data tag and determine a data type of the data to be stored corresponding to the data tag;
[0131] A partitioning module 508 is configured to partition the data partition corresponding to the data tag into sub-partitions corresponding to the data type based on the data type of the to-be-stored data corresponding to the data tag;
[0132] The storage module 510 is configured to store the data to be stored corresponding to the data tag in the sub-partition corresponding to the data type based on the data type of the data to be stored corresponding to the data tag.
[0133] Optionally, the division module 508 is further configured to:
[0134] Based on at least two data types of the to-be-stored data corresponding to the data tag, the data partition corresponding to the data tag is divided into at least two sub-partitions corresponding to the at least two data types.
[0135] Optionally, the storage module 510 is further configured to:
[0136] Determining a target storage file for the data to be stored based on the data to be stored corresponding to the data tag;
[0137] Determine a storage timestamp of the data to be stored in the target storage file, and determine a storage location of the data to be stored in the subpartition corresponding to the data type based on the storage timestamp and the subpartition corresponding to the data type of the data to be stored;
[0138] The data to be stored is stored in a to-be-stored location of the sub-partition corresponding to the data type, wherein the data to be stored includes a storage timestamp and a storage data value.
[0139] Optionally, the data processing device further includes:
[0140] Based on the data type of the to-be-stored data corresponding to the data tag, the to-be-stored data stored in the sub-partition corresponding to each data type is compressed and stored.
[0141] Optionally, the data processing device further includes:
[0142] storing the to-be-stored data corresponding to the data tag into the data type and determining it as the target data type;
[0143] Determine the data type to be filled in the data partition corresponding to the data tag based on the target data type, and determine the position to be filled for the data to be stored based on the data type to be filled;
[0144] The position to be filled is filled with data based on a preset filling rule.
[0145] Optionally, the data processing device further includes:
[0146] Receive a data query request, wherein the data query request carries time attribute information of the data to be queried and a data tag of the data to be queried;
[0147] Determine the target storage file corresponding to the data to be queried based on the time attribute information of the data to be queried;
[0148] Determining the data type of the data to be queried in the target storage file based on the data tag of the data to be queried;
[0149] The sub-partition corresponding to the data type of the data to be queried is determined, and the data to be queried is queried from the sub-partition corresponding to the data type of the data to be queried.
[0150] Optionally, the data processing device further includes:
[0151] Receive at least two data query requests, wherein the data query requests carry time attribute information of at least two data to be queried and data tags of at least two data to be queried;
[0152] Determine a target storage file corresponding to each data to be queried based on the time attribute information of the at least two data to be queried;
[0153] Determine the data type of each to-be-queried data in the target storage file corresponding to each to-be-queried data based on the data label of each to-be-queried data;
[0154] The sub-partition corresponding to the data type of each data to be queried is determined, and the data to be queried is queried from the sub-partition corresponding to the data type of each data to be queried.
[0155] Optionally, the data processing device further includes:
[0156] receiving a data aggregation request for the data to be queried, and determining a data type of each data to be queried based on the data aggregation request;
[0157] Determining data to be processed based on the data type of each data to be queried, and processing the data to be processed to obtain at least two processed data to be queried, wherein the data types of the at least two data to be queried are the same;
[0158] Aggregate data is calculated based on the processed at least two pieces of data to be queried.
[0159] The data processing device provided in the embodiments of the present specification determines the data tag of the data to be stored through the attribute information of the data to be stored, creates a data partition according to the data tag of the data to be stored, and determines the data type of the data to be stored corresponding to the data tag, and then divides the data partition into corresponding sub-partitions according to the data type of the data to be stored, and stores the data to be stored in the sub-partition corresponding to the data type, thereby realizing the storage of the data to be stored according to the data type, avoiding the data loss caused by the inability to store the data to be stored in the database due to the data type of the data to be stored, not only improving the security of data storage, but also meeting the rapid storage of massive time series data, improving storage efficiency, and further improving the query efficiency of the data.
[0160] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.
[0161] Figure 6The block diagram of a computing device 600 according to an embodiment of the present specification is shown. The components of the computing device 600 include but are not limited to a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and the database 650 is used to store data.
[0162] The computing device 600 also includes an access device 640 that enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.
[0163] In one embodiment of the present specification, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 6 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0164] The computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 600 may also be a mobile or stationary server.
[0165] The processor 620 is used to implement the steps of the data processing method when executing the following computer executable instructions.
[0166] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.
[0167] An embodiment of the present specification also provides a computer-readable storage medium storing computer instructions, which are used to implement the steps of the data processing method as described above when executed by a processor.
[0168] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.
[0169] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0170] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0171] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.
[0172] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0173] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.
Claims
1. A data processing method, comprising: Receive a data storage request, wherein the data storage request carries data to be stored, attribute information of the data to be stored, and a data type of the data to be stored; the data type is a data form or type of the data to be stored, including integer data, floating point data, and numerical string data; Determining a data tag of the data to be stored based on the attribute information of the data to be stored, wherein the data tag represents tag information of the data attribute of the data to be stored; Creating a data partition based on the data tag, and determining the data type of the data to be stored corresponding to the data tag; Based on the data type of the data to be stored corresponding to the data tag, the data partition corresponding to the data tag is divided into sub-partitions corresponding to the data type, wherein based on the data type of the data to be stored corresponding to the data tag, the data partition corresponding to the data tag is divided into sub-partitions corresponding to the data type, including: based on at least two data types of the data to be stored corresponding to the data tag, the data partition corresponding to the data tag is divided into at least two sub-partitions corresponding to the at least two data types; Based on the data type of the to-be-stored data corresponding to the data tag, the to-be-stored data corresponding to the data tag is stored in the sub-partition corresponding to the data type.
2. The data processing method according to claim 1, storing the data to be stored corresponding to the data tag in the sub-partition corresponding to the data type based on the data type of the data to be stored corresponding to the data tag, comprises: Determining a target storage file for the data to be stored based on the data to be stored corresponding to the data tag; Determine a storage timestamp of the data to be stored in the target storage file, and determine a storage location of the data to be stored in the subpartition corresponding to the data type based on the storage timestamp and the subpartition corresponding to the data type of the data to be stored; The data to be stored is stored in a to-be-stored location of the sub-partition corresponding to the data type, wherein the data to be stored includes a storage timestamp and a storage data value.
3. The data processing method according to claim 2, after storing the to-be-stored data corresponding to the data tag in the sub-partition corresponding to the data type, further comprising: Based on the data type of the to-be-stored data corresponding to the data tag, the to-be-stored data stored in the sub-partition corresponding to each data type is compressed and stored.
4. The data processing method according to claim 1 or 3, after storing the to-be-stored data corresponding to the data tag in the sub-partition corresponding to the data type, further comprising: Determine the data type of the data to be stored corresponding to the data tag as the target data type; Determine the data type to be filled in the data partition corresponding to the data tag based on the target data type, and determine the position to be filled for the data to be stored based on the data type to be filled; The position to be filled is filled with data based on a preset filling rule.
5. The data processing method according to claim 1, further comprising: Receive a data query request, wherein the data query request carries time attribute information of the data to be queried and a data tag of the data to be queried; Determine the target storage file corresponding to the data to be queried based on the time attribute information of the data to be queried; Determining the data type of the data to be queried in the target storage file based on the data tag of the data to be queried; The sub-partition corresponding to the data type of the data to be queried is determined, and the data to be queried is queried from the sub-partition corresponding to the data type of the data to be queried.
6. The data processing method according to claim 5, further comprising: Receive at least two data query requests, wherein the data query requests carry time attribute information of at least two data to be queried and data tags of at least two data to be queried; Determine a target storage file corresponding to each data to be queried based on the time attribute information of the at least two data to be queried; Determine the data type of each to-be-queried data in the target storage file corresponding to each to-be-queried data based on the data label of each to-be-queried data; The sub-partition corresponding to the data type of each data to be queried is determined, and the data to be queried is queried from the sub-partition corresponding to the data type of each data to be queried.
7. The data processing method according to claim 6, after querying the data to be queried based on the sub-partition corresponding to the data type of each data to be queried, further comprising: receiving a data aggregation request for the data to be queried, and determining a data type of each data to be queried based on the data aggregation request; Determining data to be processed based on the data type of each data to be queried, and processing the data to be processed to obtain at least two processed data to be queried, wherein the data types of the at least two data to be queried are the same; Aggregate data is calculated based on the processed at least two pieces of data to be queried.
8. A data processing device, comprising: The receiving module is configured to receive a data storage request, wherein the data storage request carries data to be stored, attribute information of the data to be stored, and a data type of the data to be stored; the data type is a data form or type of the data to be stored, including integer data, floating point data, and numerical string data; a determination module, configured to determine a data tag of the data to be stored based on the attribute information of the data to be stored, wherein the data tag represents tag information of the data attribute of the data to be stored; A creation module, configured to create a data partition based on the data tag and determine the data type of the data to be stored corresponding to the data tag; A partitioning module, configured to partition the data partition corresponding to the data tag into sub-partitions corresponding to the data type based on the data type of the to-be-stored data corresponding to the data tag; The partitioning module is further configured to, based on at least two data types of the to-be-stored data corresponding to the data tag, partition the data partition corresponding to the data tag into at least two sub-partitions corresponding to the at least two data types; The storage module is configured to store the data to be stored corresponding to the data tag in the sub-partition corresponding to the data type based on the data type of the data to be stored corresponding to the data tag.
9. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, wherein the processor implements the steps of the data processing method according to any one of claims 1 to 7 when executing the computer-executable instructions.
10. A computer-readable storage medium storing computer instructions, wherein the computer instructions, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 7.