Data processing method and device

Abstract

Embodiments of the present specification provide a data processing method and device, wherein the data processing method comprises: receiving a data storage request for data to be stored for a target object, wherein the data storage request carries attribute information of the data to be stored and an object tag of the target object; determining a creation timestamp and a latest timestamp of the data to be stored based on the attribute information of the data to be stored, and determining a data partitioning strategy based on the creation timestamp and the latest timestamp, and creating data partitions according to the data partitioning strategy; determining the data to be stored of the target object in each data partition, and determining index data corresponding to the data to be stored; constructing a first index table based on the data partition corresponding to the index data, the target object corresponding to the data partition, and the object tag of the target object, and storing the data to be stored of each data partition based on the first index table.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and more particularly to a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, and a computer-readable storage medium. Background Art

[0002] As the Internet or the Internet of Things is increasingly used, detection systems or IoT devices on the Internet generate more and more time series-based data, which are called time series data. Although time series data comes from different sources, new timelines are always created and old timelines are gradually eliminated due to the upgrade of the acquisition device version, system upgrade, or the disappearance of the acquisition object for various reasons. As the amount of data continues to increase, the storage efficiency of time series data is low, which brings inconvenience to subsequent queries and reduces query efficiency. Summary of the invention

[0003] In view of this, the present specification provides a data processing method. One or more embodiments of the present specification also relate to a data processing device, a computing device, and a computer-readable storage medium to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a data processing method is provided, including:

[0005] Receiving a data storage request for data to be stored of a target object, wherein the data storage request carries attribute information of the data to be stored and an object tag of the target object;

[0006] Determine a creation timestamp and a latest timestamp of the data to be stored based on attribute information of the data to be stored, determine a data partitioning strategy based on the creation timestamp and the latest timestamp, and create a data partition according to the data partitioning strategy;

[0007] Determine the data to be stored of the target object in each data partition, and determine the index data corresponding to the data to be stored;

[0008] A first index table is constructed based on the data partition corresponding to the index data, the target object corresponding to the data partition and the object label of the target object, and the data to be stored of each data partition is stored based on the first index table.

[0009] According to a second aspect of an embodiment of this specification, there is provided a data processing device, including:

[0010] A receiving module is configured to receive a data storage request for data to be stored of a target object, wherein the data storage request carries attribute information of the data to be stored and an object tag of the target object;

[0011] a creation module configured to determine a creation timestamp and a latest timestamp of the data to be stored based on attribute information of the data to be stored, determine a data partitioning strategy based on the creation timestamp and the latest timestamp, and create a data partition according to the data partitioning strategy;

[0012] a determination module, configured to determine the data to be stored of the target object in each data partition, and determine index data corresponding to the data to be stored;

[0013] The storage module is configured to construct a first index table based on the data partition corresponding to the index data, the target object corresponding to the data partition and the object tag of the target object, and store the data to be stored in each data partition based on the first index table.

[0014] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0015] Memory and processor;

[0016] The memory is used to store computer executable instructions, and the processor is used to implement the steps of the data processing method when executing the computer executable instructions.

[0017] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of any one of the data processing methods are implemented.

[0018] An embodiment of the present specification determines the creation timestamp and the latest timestamp of the data to be stored through the attribute information of the data to be stored, and then determines the data partitioning strategy and creates the data partitions. Based on the data partitions, a first index table with a target object and an object label is constructed, so that the index adds a time attribute, so that the time dimension can be supported for subsequent filtering of data that meets the query conditions, and the data to be stored is stored based on the first index table. In the scenario where massive timelines are constantly created and destroyed, the index data is partitioned according to the time dimension, thereby avoiding the loss of index data and improving the security of data storage. The method provides an effective indexing strategy, thereby improving the storage efficiency of the data to be stored, and can also improve the query efficiency of the data subsequently. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1It is a time series diagram of time series data of a data processing method provided by an embodiment of this specification;

[0020] Figure 2 is a flow chart of a data processing method provided by an embodiment of this specification;

[0021] Figure 3 It is a data partition diagram of a data processing method provided by an embodiment of this specification;

[0022] Figure 4 This is a data merging and storing schematic diagram of a data processing method provided by an embodiment of this specification;

[0023] Figure 5 It is a flowchart of a data processing method provided by an embodiment of this specification applied to a processing process of querying data;

[0024] Figure 6 is a structural schematic diagram of a data processing device provided by an embodiment of this specification;

[0025] Figure 7 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0026] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0027] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0028] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0029] First, the terms involved in one or more embodiments of this specification are explained.

[0030] Time Series Database (TSDB): The full name of TSDB is Time Series Database, which is a data management system that provides efficient storage and statistical analysis of time series data.

[0031] Time Series Data: A series of indicator monitoring data generated continuously at a stable frequency. For example, when monitoring the air quality of a city, a series of data is generated by collecting a sulfur dioxide concentration value every second.

[0032] Metric: Although the indicator item to be monitored is specified, it does not specify the object for which the indicator item is to be monitored. The indicator of the monitoring data, such as wind speed and temperature.

[0033] Tag: It is used to indicate the specific object that the indicator item is monitoring and belongs to the data subcategory under the specified measurement.

[0034] Timestamp: The time point when the data (measurement value) is generated.

[0035] Timeline: Equivalent to the concept of time series.

[0036] Time Partitioned Index: Indexes the timeline by time period to record the life cycle of the timeline.

[0037] Write-Ahead Log (WAL): An efficient way to record data.

[0038] Bitmap: A data structure that represents a dense set in a finite field, where each element appears at least once and no other data is associated with the element.

[0039] Bloom filter: A binary vector data structure that is space- and time-efficient and is used to detect whether an element is a member of a set.

[0040] A tag consists of a tag key and a corresponding tag value. For example, "City (TagKey) = Hangzhou (TagValue)" is a tag. More tag examples: Computer room = A, IP = 172.220.110.1.

[0041] It should be noted that a tag is considered the same only when both the tag key and the tag value are the same; if the tag key is the same but the tag value is different, then they are not the same tag. When monitoring data, if the specified metric is "temperature" and the tag is "city = Hangzhou", then the temperature of Hangzhou is monitored.

[0042] Tag key (TagKey, Tagk): monitors the specified object type for the metric (there will be a corresponding tag value to locate the specific object under the object type), such as country, province, city, computer room, IP, etc.

[0043] Tag value (TagValue, Tagv): The value corresponding to the tag key (TagKey). For example, when the tag key (TagKey) is "country", the tag value (TagValue) can be specified as "China".

[0044] Value: The value corresponding to the measurement, such as 15 (wind force) and 20℃ (temperature).

[0045] Data Point: Each metric value collected at a specific time interval (continuous timestamps) for a certain indicator of the monitored object (defined by metrics and tags) is a data point. "One metric + N tags (N>=1) + one timestamp + one value" defines a data point.

[0046] Time Series: A description of a certain indicator (defined by metrics and labels) of a certain monitoring object. "One metric + N label KV combinations (N>=1)" is defined as a time series. The increase in the data value generated in a time series will not lead to an increase in the time series.

[0047] See also Figure 1 , Figure 1 A time series diagram of time series data of a data processing method provided in an embodiment of this specification is shown, Figure 1 There are 5 data points in the time series, one timestamp and one value correspond to one time series. Figure 1Taking temperature as an example, the label is: floor 33, conference room number 3302, equipment ID 7649501. In the order of timestamps, the temperature values ​​are 26 degrees, 25.8 degrees, 26.1 degrees, 26.3 degrees, and 26.5 degrees. For example, if the timestamp is 1492158910, the temperature value is 26.

[0048] In actual applications, general search engines can provide tag retrieval, but they do not have a time dimension. Tag retrieval is a basic function of a time series database. The typical time series processing flow is based on the core component TSI. TSI gives the time series database the ability to index the timeline, mainly including timeline management and cascade analysis. For example, you can query tags and metric names based on prefix matching, and query timeline metadata information based on filtering conditions, including complete tag and metric information.

[0049] Data query can query the timeline ID set according to the user's filtering conditions, further query the time series data corresponding to the timeline, and query the cascade relationship of Tags and metadata distribution according to the conditions.

[0050] Time series data is mainly used for continuous data updates and IoT application scenarios. When collecting data, new timelines are constantly created and old timelines are gradually eliminated due to reasons such as version upgrades of collection devices, system upgrades, and disappearance of collection objects due to various reasons. The data processing method provided in the embodiments of this specification is mainly used to solve the problems of instantaneous data storage in the above scenarios and the subsequent query efficiency of time series data.

[0051] In this specification, a data processing method is provided. This specification also relates to a data processing apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0052] Figure 2 A flow chart of a data processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0053] Step 202: Receive a data storage request for data to be stored of a target object, wherein the data storage request carries attribute information of the data to be stored and an object tag of the target object.

[0054] The data storage request may be understood as a storage request for data to be stored of the target object. For example, if the data to be stored is temperature data, the data storage request may be a storage request for storing the acquired temperature data.

[0055] Among them, the attribute information of the data to be stored can be understood as the measurement of the data to be stored and the recording time of the data to be stored; the object label of the target object can be understood as the object label corresponding to the target object targeted by the data to be stored. For example, if the data to be stored is 5 degrees, then the attribute information of the data to be stored is temperature, and the recording time of the data to be stored is 10:01 am.

[0056] Specifically, the server receives a data storage request for data to be stored for a target object, wherein the data storage request carries attribute information of the data to be stored and an object tag of the target object. For example, the server receives a data storage request for meteorological data of a conference room, wherein the data storage request carries measurement information of temperature and humidity of the meteorological data, and identification information for the conference room, such as Conference Room 1 on the 1st floor.

[0057] Step 204: Determine the creation timestamp and the latest timestamp of the data to be stored based on the attribute information of the data to be stored, determine a data partitioning strategy based on the creation timestamp and the latest timestamp, and create a data partition according to the data partitioning strategy.

[0058] The data partitioning strategy can be understood as a strategy for partitioning the timeline lifecycle according to certain time intervals, and the data partitioning can be understood as time partitioning the timeline lifecycle according to certain time intervals.

[0059] Specifically, after receiving the attribute information of the data to be stored, the creation timestamp and the latest timestamp of the data to be stored can be determined based on the attribute information, and the corresponding data partitioning strategy is determined by writing the creation timestamp and the latest timestamp of the stored data, and the data partition is created according to the data partitioning strategy.

[0060] In actual applications, when data is written to storage, the new timeline is marked with the creation timestamp and the latest access timestamp in the memory. Each time a timeline is written, the memory timestamp is updated. The total time of writing the target object can be obtained by subtracting the creation timestamp from the latest access timestamp. The time partitioning strategy is determined based on the total time of writing data, and data partitions are created based on the time partitioning strategy.

[0061] For example, the server records the weather data of the conference room in Building 1. When writing the weather data, the creation timestamp will be marked as 00:00 on January 1, 2020. With each writing of the time series data, the timestamp when the data is written is updated. When the time series data is recorded for one month, it is determined to be sharded for one month according to actual needs. The data partitioning strategy is to use ten days as one shard. Then, the first time shard for recording the weather data of the conference room in Building 1 is 00:00 on January 1, 2020-24:00 on January 10, 2020, the second time shard is 00:00 on January 11, 2020-24:00 on January 20, 2020, and the third time shard is 00:00 on January 21, 2020-24:00 on January 30, 2020.

[0062] Further, the determining of the creation timestamp and the latest timestamp of the data to be stored based on the attribute information of the data to be stored includes:

[0063] A creation timestamp and a latest timestamp of the data to be stored are obtained based on the attribute information of the data to be stored, and the creation timestamp and the latest timestamp are written into a log file for storage.

[0064] Specifically, after receiving the attribute information of the data to be stored for the target object, the creation timestamp and the latest write timestamp of the data to be stored are recorded according to the attribute information of the data to be stored, and the marked creation timestamp and the latest timestamp are written into the log file for storage.

[0065] In practical applications, the timestamp of when the data to be stored is written can be recorded by means of a pre-write log, so that the reliability of data writing can be ensured by subsequently searching for the write record of the data to be stored.

[0066] The data processing method provided in the embodiments of the present specification not only ensures the security of the data by writing the creation timestamp and the latest write timestamp of the data to be stored into the log file, but also facilitates the subsequent determination of the data partition strategy according to the timestamp data for data storage.

[0067] See also Figure 3 , Figure 3 A data partition diagram of a data processing method provided according to an embodiment of the present specification is shown.

[0068] Figure 3 The horizontal axis in the middle represents the timeline life cycle, and the vertical axis represents multiple timelines. Figure 3The timeline life cycle is divided into four time slices, namely time slice 1, time slice 2, time slice 3, and time slice 4. The vertical axis is divided into six time lines, namely time line 1, time line 2, time line 3, time line 4, time line 5, and time line 6. It should be noted that a time line can represent a target object, such as a meeting room; Figure 3 In the example, the duration of the time series data recorded in each time line is different, that is, the creation timestamp and the latest timestamp of the records are different. In each time slice, the corresponding time line that falls in the time slice can be found, which means that the time slice has storage data corresponding to the time line. For example, time slice 1 has time series data corresponding to time line 1, time line 2 and time line 3, time slice 2 has time series data corresponding to time line 2, time line 3, time line 4 and time line 5, and so on.

[0069] In actual applications, the time slices are determined according to the creation timestamp and the latest access timestamp of the stored time series data and are sliced ​​according to the preset time intervals, which is in line with the application scenarios where massive time lines are constantly created and destroyed. Using the above example, each time line is a conference room. When recording the meteorological data of the conference room, the meteorological data of Conference Room 1, Conference Room 2, and Conference Room 3 on the first floor are recorded in time slice 1.

[0070] The data processing method provided in the embodiments of this specification, in a scenario where massive timelines are constantly created and destroyed, partitions the index data by slicing it according to the time dimension, so that the index data can be subsequently filtered by the time dimension to screen out index data that meets the required conditions, and the data to be stored can be further queried more quickly through the index data, thereby improving the efficiency of querying data.

[0071] Step 206: Determine the data to be stored of the target object in each data partition, and determine the index data corresponding to the data to be stored.

[0072] The index data may be understood as data with data labels in the data to be stored.

[0073] Specifically, when writing data, the data to be stored for the target object in each data partition is determined, and the index data corresponding to the data to be stored is determined according to the data to be stored of the target object.

[0074] In actual applications, index data is stored to facilitate subsequent searches for stored data. The data processing method provided in this embodiment, when storing data, shards the index data according to the time dimension, adds time attributes to the index, and supports time dimension filtering. The index data strategy can facilitate subsequent searches for stored data.

[0075] Continuing with the above example, in time slice 1 (data partition 1), determine to record the meteorological data of conference room 1, conference room 2 and conference room 3, and determine that the index data of the meteorological data may include the label of the target object, the measurement information of the meteorological data, or the storage location of the meteorological data, etc. This specification does not impose any limitations on this.

[0076] Step 208: construct a first index table based on the data partition corresponding to the index data, the target object corresponding to the data partition and the object tag of the target object, and store the data to be stored in each data partition based on the first index table.

[0077] The first index table may be understood as an inverted index table with data partitions, target objects, and object labels of the target objects.

[0078] Specifically, after determining the index data of the data to be stored, the corresponding data partition is determined based on the index data. In the data partition, the target object corresponding to the time slice and the object label corresponding to the target object can be determined, and then a first index table is constructed. Based on the constructed first index table, the data to be stored in each data partition is stored.

[0079] Continuing with the above example, in time slice 1 (data partition 1), there is meteorological data for Conference Room 1, Conference Room 2, and Conference Room 3, that is, the first index table constructed has target objects, data partitions, and object tags corresponding to the target objects, where the target objects are Conference Room 1, Conference Room 2, and Conference Room 3, the data partition is time slice 1, and the object tags corresponding to the target objects are Conference Room 1 on the 1st floor, Conference Room 2 on the 1st floor, and Conference Room 3 on the 2nd floor.

[0080] After the first index table is created, a second index table needs to be created to further obtain the storage offset of the data to be stored, and the data to be stored is stored based on the storage offset; specifically, storing the data to be stored of each data partition based on the first index table includes:

[0081] constructing a second index table based on the target object corresponding to the data partition in the first index table, the object label of the target object, and the attribute information of the data to be stored of the target object;

[0082] Acquire the storage offset of the data to be stored in each data partition based on the first index table and the second index table;

[0083] The data to be stored is stored according to the storage offset bit.

[0084] The second index table may be understood as a positive index table containing target objects, target object labels, and attribute information of data to be stored corresponding to the data partitions in the first index table.

[0085] Specifically, when writing data on each timeline, each data partition can determine the timeline that falls on the data partition time, determine the target object corresponding to the timeline, the object tag of the target object, and the attribute information of the target object's data to be stored, construct a second index table, and based on the first index table and the second index table, determine the storage offset bit of the data to be stored for each target object, and store each data to be stored according to the determined storage offset bit.

[0086] It should be noted that the records in the second index table are part of generating the storage offset bits, wherein there are multiple ways to determine the storage offset bits, such as a combination method, a self-increment identification method, a file block mapping, and the like. Taking the combination method to determine the storage offset bits as an example, the first index table and the second index table are spliced ​​and combined to generate the storage offset bits. The method for determining the storage offset bits provided in the embodiments of this specification is not limited in detail here.

[0087] Continuing with the above example, each column in the second index table in time slice 1 includes data on the target object, label, and attribute information of the data to be stored, where the labels include layer 1, layer 1, and layer 2; the target objects include conference room 1, conference room 2, and conference room 3; the attribute information includes temperature, humidity, temperature, and humidity; in the second index table, the horizontal arrangement is layer 1-conference room 1-temperature, layer 1-conference room 2-humidity, and layer 2-conference room 3-temperature and humidity. It should be noted that each timeline in the table has a storage offset bit for storing the corresponding data to be stored, and the data to be stored is stored.

[0088] The data processing method provided in the embodiments of the present specification constructs a second index table based on the information in the first index table so as to obtain the storage offset of the data to be stored in each data partition, thereby realizing rapid storage of the data to be stored.

[0089] Further, after storing the data to be stored according to the storage offset bit, the method further includes:

[0090] Merging the first index tables according to each of the data partitions to obtain a merged inverted index file, and storing the merged inverted index file; and

[0091] The second index tables are merged according to each of the data partitions to obtain a merged forward index file, and the merged forward index file is stored.

[0092] Specifically, after the time dimension is sliced, the index data of each time slice can be merged into a storage file for storage, and the first index table is merged according to each data partition to obtain a merged inverted index file, and the merged index file is stored; and in the created second index table, the data are also divided and merged according to each data partition to obtain a merged forward index file, and the merged index file is stored.

[0093] It should be noted that when merging index data files, the time attributes of the merged files are updated and the old files are removed from the file manager to save storage space. The query logic after the file merge is the same as the query logic before the merge, except that the timelines created in a small range are merged into a larger time period based on the time window.

[0094] The data processing method provided in the embodiments of this specification merges and stores multiple index shards so that the index data can be queried by querying the merged index files later, which not only saves storage space but also improves the efficiency of long-term historical queries.

[0095] See also Figure 4 , Figure 4 A schematic diagram of data merging and storing according to a data processing method provided according to an embodiment of the present specification is shown.

[0096] Figure 4 In the process of recording data writing by writing logs, after the data is written, the index data of the data to be stored will be stored in the timeline index table, and the data to be stored will be stored in the timeline table. It should be noted that the timeline index table does not store real data, only index data, while the timeline table stores full data, which has a large amount of real storage data. In comparison, the storage space occupied by the timeline table is much larger than that occupied by the timeline index table. Each time slice corresponds to a timeline index file. The merge manager merges and stores multiple timeline files. The timeline index file format uses hash indexes to build inverted indexes to support efficient point queries and range queries. At the same time, the timeline files are also merged and stored. Management, where the timeline files use data structures such as Bitmap and BloomFilter to achieve high-performance queries, perform statistical analysis on the timeline, and achieve adaptive query optimization.

[0097] The data processing method provided in the embodiment of the present specification merges the index files of the same data partition through a merging module to facilitate subsequent rapid storage and search.

[0098] In addition, as time series data is continuously updated and written, data partitions can also be continuously created, and the data to be stored is stored in new data partitions. Specifically, storing the data to be stored in each data partition based on the first index table includes:

[0099] Creating a new data partition based on the latest timestamp of the data to be stored and the data partition strategy;

[0100] A first index table corresponding to the new data partition is updated according to the new data partition, and the data to be stored of the new data partition is stored based on the first index table.

[0101] Specifically, during the continuous data writing process, new data partitions can continue to be created based on the latest timestamp of the data to be stored and the data partitioning strategy, and the first index table can be updated based on the index data of the target object to be stored recorded in the new data partition, and the updated first index table can store the data to be stored in the new data partition.

[0102] In practical applications, time series data is continuously written and stored. When the timestamp of the newly written data cannot be divided in the data partition, a new data partition can be created according to the data partition strategy, and the index data of the data to be stored is recorded in the first index table in the new data partition, and the data to be stored is stored based on the newly recorded first index table.

[0103] In the embodiments of this specification, data partitions are created to partition newly written data to be stored, and the data to be stored is stored so that the data to be stored can be quickly queried later through index data in the new data partitions.

[0104] As time series data is continuously written, when the time of the data partition is relatively old compared to the real-time time dimension, the data to be stored stored in the initially created data partition can be deleted; specifically, after storing the data to be stored of each data partition based on the first index table, the method further includes:

[0105] The data to be stored in the data partition is deleted based on preset requirements.

[0106] Among them, the preset requirements can be understood as different user requirements based on actual applications. For example, when the timeline period is long, the time period of the initial data partition is one year ago compared to the real-time period. In actual applications, the data efficiency is poor when stored in the data partition with a long historical time, and the actual application rate is not high. The data to be stored in the data partition with a low application rate can be deleted.

[0107] The data processing method provided in the embodiments of this specification deletes the data to be stored in the data partition according to preset requirements, thereby deleting the data with poor timeliness and saving storage space.

[0108] Furthermore, in the scenario where timelines are frequently created and aged, it is easy to cause index expansion, thereby causing query amplification problems caused by invalid timelines, which seriously affects data query efficiency. Therefore, the data processing method provided in the embodiment of this specification also includes:

[0109] Receiving a data query request, wherein the data query request carries attribute information of the data to be queried;

[0110] Determine a data partition of the data to be queried in the first index table based on the attribute information of the data to be queried, and determine a target object corresponding to the data partition and an object label of the target object in the first index table based on the data partition;

[0111] A data query is performed based on the target object and the object tag of the target object.

[0112] Specifically, after receiving a data query request, the data query request is parsed to obtain attribute information of the data to be queried, and the data partition of the data to be queried is determined in the first index table based on the attribute information of the data to be queried, and the target object corresponding to the data partition and the object label corresponding to the target object are determined in the first index table according to the data partition, and then the data to be queried is queried according to the target object and the object label of the target object.

[0113] Continuing with the above example, the meteorological data at time t is received and the partition in the first index table determines in which data partition the time t falls. It is determined that the time t is within the range of data partition 1. The index data of the meteorological data at time t can be filtered out and stored in the file of data partition 1. The data with data partition 1 is determined in the first index table, and then it is determined that the target objects in data partition 1 are conference room 1, conference room 2, and conference room 3, and the object labels of the target objects are conference room 1-1 floor, conference room 2-1 floor, and conference room 3-2 floor, respectively. The meteorological data at time t is queried based on the determined target objects and the object labels of the target objects.

[0114] The data processing method provided in the embodiments of this specification determines the corresponding data partition in the first index table through the attribute information in the data query request, and then determines the corresponding target object and the object identifier of the target object through the first index table. Then, the storage location of the required query data in the query data can be quickly located through the index table to query the query data, thereby improving the query efficiency.

[0115] After determining the corresponding target object and the object identifier of the target object in the query data, it can be determined in the second index table whether the query request meets the data query condition; specifically, performing data query based on the target object and the object tag of the target object includes:

[0116] Based on the target object and the object tag of the target object, determining whether there is attribute information matching the attribute information of the data to be queried in the second index table;

[0117] If so, the storage offset of the to-be-queried data is determined based on the target object, and the to-be-queried data is queried based on the storage offset.

[0118] Specifically, after obtaining the target object corresponding to the query data request and the object tag of the target object, it is determined whether there is attribute information in the second index table that matches the attribute information of the data to be queried. If it is determined that there is, the storage offset of the data to be queried is determined based on the target object, and data query is performed.

[0119] In actual applications, there may be invalid query requests in the data query request. For example, a user writes a query for the humidity data of the conference room on the first floor at time 1t. If it is determined in the second index table that the conference room 1 on the first floor does not store humidity data, then the written data query request is an effective data query request. The data query fails and no data information can be returned to the user. If the user queries the temperature data of the conference room on the first floor at time 1t, it is determined in the second index table that the conference room 1 on the first floor has stored temperature data. Then the written data query request is a valid data query request. Based on the target object determined in the second index table, the storage offset of the data to be queried can be obtained to perform data query.

[0120] The data processing method provided in the embodiment of this specification can effectively exclude expired time lines by determining whether the data query request written in the second index table is reasonable, and then filtering the time lines according to the time range, thereby realizing the screening of index data and improving data query efficiency.

[0121] In summary, the embodiments of the present specification determine the creation timestamp and the latest timestamp of the data to be stored through the attribute information of the data to be stored, and then determine the data partitioning strategy and create data partitions, and construct a first index table with a target object and an object label based on the data partition, so that the index adds a time attribute, so that it can support the time dimension to filter data that meets the query conditions in the subsequent time dimension, and store the data to be stored based on the first index table, so that in the scenario where massive timelines are constantly created and destroyed, the index data is partitioned according to the time dimension, which is an effective indexing strategy, thereby improving the storage efficiency of the data to be stored, and can also improve the query efficiency of the data in the subsequent period.

[0122] See also Figure 5 , Figure 5 A data processing method provided in an embodiment of this specification is applied to query data as an example for detailed description. Figure 5 It is a flowchart of a data processing method provided by an embodiment of the present specification and applied to a processing process of querying data.

[0123] It should be noted that the start timestamp and the latest timestamp of the data to be stored are determined based on the attribute information of the data to be stored, and then the data partition strategy is determined. The data partition strategy determined in this embodiment can use one month as the time period of a data partition. For example, the determined data partitions are January 1, 2020-February 1, 2020, February 1, 2020-March 1, 2020, and March 1, 2020-April 1, 2020, respectively, and then the meteorological data of the conference room in each data partition and the index data corresponding to the meteorological data to be stored are determined. Based on this, the query steps for the meteorological data of the conference room are as follows:

[0124] Step 502: Receive a query request for the temperature data of the conference room on January 10, 2020.

[0125] Step 504: Determine, based on the time in the query request, that the time of the temperature data to be queried falls in data partition 1, then determine in the first index table in data partition 1 that Conference Room 1, Conference Room 2, and Conference Room 3 have meteorological data for January 10, 2020.

[0126] Step 506: Determine in the first index table that the object identifier corresponding to conference room 1 is 1st floor-conference room 1, the object identifier corresponding to conference room 2 is 1st floor-conference room 2, and the object identifier corresponding to conference room 3 is 2nd floor-conference room 1.

[0127] Step 508: Determine in the second index table based on the acquired target object and target object identifier that Floor 1 - Conference Room 1 has a humidity tag storing temperature data, Floor 1 - Conference Room 2 has a temperature tag storing temperature data, and Floor 2 - Conference Room 1 has a humidity tag storing humidity data.

[0128] Step 510: by searching the attribute information of the temperature data in the data query request, it is determined that only the data of the 1st floor - conference room 2 meets the query condition.

[0129] Step 512: Obtain the storage offset of the data to be queried for Floor 1 - Conference Room 2 in the second index table, and search for the temperature data of Floor 1 - Conference Room 2 stored on January 10, 2020 based on the storage offset.

[0130] The data processing method provided in the embodiments of this specification utilizes an indexing scheme created by time partitioning, performs screening through a first index table, determines query conditions in a second index table, and then accurately searches a data table storing the full amount of data for the actual data storage location to perform data query, thereby improving data query efficiency.

[0131] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 6 FIG. 1 is a schematic diagram showing the structure of a data processing device provided by an embodiment of the present specification. Figure 6 As shown, the device comprises:

[0132] The receiving module 602 is configured to receive a data storage request for data to be stored of a target object, wherein the data storage request carries attribute information of the data to be stored and an object tag of the target object;

[0133] A creation module 604 is configured to determine a creation timestamp and a latest timestamp of the data to be stored based on the attribute information of the data to be stored, determine a data partitioning strategy based on the creation timestamp and the latest timestamp, and create a data partition according to the data partitioning strategy;

[0134] A determination module 606 is configured to determine the data to be stored of the target object in each data partition, and determine index data corresponding to the data to be stored;

[0135] The storage module 608 is configured to construct a first index table based on the data partition corresponding to the index data, the target object corresponding to the data partition and the object tag of the target object, and store the data to be stored in each data partition based on the first index table.

[0136] Optionally, the storage module 608 is further configured to:

[0137] constructing a second index table based on the target object corresponding to the data partition in the first index table, the object label of the target object, and the attribute information of the data to be stored of the target object;

[0138] Acquire the storage offset of the data to be stored in each data partition based on the second index table;

[0139] The data to be stored is stored according to the storage offset bit.

[0140] Optionally, the data processing device further includes:

[0141] Receiving a data query request, wherein the data query request carries attribute information of the data to be queried;

[0142] Determine a data partition of the data to be queried in the first index table based on the attribute information of the data to be queried, and determine a target object corresponding to the data partition and an object label of the target object in the first index table based on the data partition;

[0143] A data query is performed based on the target object and the object tag of the target object.

[0144] Optionally, the data processing device further includes:

[0145] Based on the target object and the object tag of the target object, determining whether there is attribute information matching the attribute information of the data to be queried in the second index table;

[0146] If so, the storage offset of the to-be-queried data is determined based on the target object, and the to-be-queried data is queried based on the storage offset.

[0147] Optionally, the data processing device further includes:

[0148] Merging the first index tables according to each of the data partitions to obtain a merged inverted index file, and storing the merged inverted index file; and

[0149] The second index tables are merged according to each of the data partitions to obtain a merged forward index file, and the merged forward index file is stored.

[0150] Optionally, the creation module 604 is further configured to:

[0151] A creation timestamp and a latest timestamp of the data to be stored are obtained based on the attribute information of the data to be stored, and the creation timestamp and the latest timestamp are written into a log file for storage.

[0152] Optionally, the storage module 608 is further configured to:

[0153] Creating a new data partition based on the latest timestamp of the data to be stored and the data partition strategy;

[0154] A first index table corresponding to the new data partition is updated according to the new data partition, and the data to be stored of the new data partition is stored based on the first index table.

[0155] Optionally, the data processing device further includes:

[0156] The data to be stored in the data partition is deleted based on preset requirements.

[0157] The data processing device provided in the embodiments of the present specification determines the creation timestamp and the latest timestamp of the data to be stored through the attribute information of the data to be stored, and then determines the data partitioning strategy and creates the data partition, and constructs the first index table with the target object and the object label based on the data partition, so that the index adds the time attribute, so that the data that meets the query condition can be supported in the time dimension in the subsequent filtering, and the data to be stored is stored based on the first index table, so that in the scenario where massive time lines are constantly created and disappeared, the index data is partitioned according to the time dimension, thereby avoiding the loss of index data and improving the security of data storage. The method provides an effective indexing strategy, thereby improving the storage efficiency of the data to be stored, and can also improve the query efficiency of the data in the subsequent period.

[0158] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.

[0159] Figure 7 The block diagram of a computing device 700 according to an embodiment of the present specification is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0160] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0161] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0162] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 700 may also be a mobile or stationary server.

[0163] The processor 720 is used to implement the steps of the data processing method when executing the computer executable instructions.

[0164] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.

[0165] An embodiment of the present specification also provides a computer-readable storage medium storing computer instructions, which are used to implement the steps of the data processing method as described above when executed by a processor.

[0166] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.

[0167] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0168] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0169] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0170] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0171] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: Receiving a data storage request for data to be stored of a target object, wherein the data storage request carries attribute information of the data to be stored and an object tag of the target object; Determine the creation timestamp and the latest timestamp of the data to be stored based on the attribute information of the data to be stored, determine the data partition strategy based on the creation timestamp and the latest timestamp, and create the data partition according to the data partition strategy, wherein the determination of the data partition strategy based on the creation timestamp and the latest timestamp includes: subtracting the creation timestamp from the latest timestamp to obtain the total time of writing the target object, and determine the data partition strategy according to the total time of writing the data; Determine the data to be stored of the target object in each data partition, and determine the index data corresponding to the data to be stored; A first index table is constructed based on the data partition corresponding to the index data, the target object corresponding to the data partition and the object label of the target object, and the data to be stored of each data partition is stored based on the first index table.

2. The data processing method according to claim 1, storing the data to be stored in each data partition based on the first index table, comprises: constructing a second index table based on the target object corresponding to the data partition in the first index table, the object label of the target object, and the attribute information of the data to be stored of the target object; Acquire the storage offset of the data to be stored in each data partition based on the first index table and the second index table; The data to be stored is stored according to the storage offset bit.

3. The data processing method according to claim 2, further comprising: Receiving a data query request, wherein the data query request carries attribute information of the data to be queried; Determine a data partition of the data to be queried in the first index table based on the attribute information of the data to be queried, and determine a target object corresponding to the data partition and an object label of the target object in the first index table based on the data partition; A data query is performed based on the target object and the object tag of the target object.

4. The data processing method according to claim 3, wherein the data query based on the target object and the object tag of the target object comprises: Based on the target object and the object tag of the target object, determining whether there is attribute information matching the attribute information of the data to be queried in the second index table; If so, the storage offset of the to-be-queried data is determined based on the target object, and the to-be-queried data is queried based on the storage offset.

5. The data processing method according to claim 2, after storing the data to be stored according to the storage offset bit, further comprising: Merging the first index tables according to each of the data partitions to obtain a merged inverted index file, and storing the merged inverted index file; as well as The second index tables are merged according to each of the data partitions to obtain a merged forward index file, and the merged forward index file is stored.

6. The data processing method according to claim 1 or 5, wherein determining the creation timestamp and the latest timestamp of the data to be stored based on the attribute information of the data to be stored comprises: A creation timestamp and a latest timestamp of the data to be stored are obtained based on the attribute information of the data to be stored, and the creation timestamp and the latest timestamp are written into a log file for storage.

7. The data processing method according to claim 6, wherein storing the data to be stored in each data partition based on the first index table comprises: Creating a new data partition based on the latest timestamp of the data to be stored and the data partition strategy; The first index table corresponding to the new data partition is updated according to the new data partition, and the data to be stored of the new data partition is stored based on the first index table.

8. The data processing method according to claim 2, after storing the data to be stored in each data partition based on the first index table, further comprising: The data to be stored in the data partition is deleted based on preset requirements.

9. A data processing device, comprising: A receiving module is configured to receive a data storage request for data to be stored of a target object, wherein the data storage request carries attribute information of the data to be stored and an object tag of the target object; A creation module, configured to determine a creation timestamp and a latest timestamp of the data to be stored based on the attribute information of the data to be stored, determine a data partitioning strategy based on the creation timestamp and the latest timestamp, and create a data partition according to the data partitioning strategy, wherein the creation module is further configured to: obtain a total time of writing the target object by subtracting the creation timestamp from the latest timestamp, and determine the data partitioning strategy according to the total time of writing the data; A determination module is configured to determine the data to be stored of the target object in each data partition, and determine index data corresponding to the data to be stored; The storage module is configured to construct a first index table based on the data partition corresponding to the index data, the target object corresponding to the data partition and the object tag of the target object, and store the data to be stored in each data partition based on the first index table.

10. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to implement the steps of the data processing method according to any one of claims 1 to 8 when executing the computer-executable instructions.

11. A computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 8.

Images