Data processing method and device, storage medium and electronic equipment
By splitting the time-series data collected by IoT devices into sequential and out-of-order data and storing them in different databases, the problem of low data storage efficiency of IoT devices is solved, and efficient data management and querying are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
- Filing Date
- 2022-09-23
- Publication Date
- 2026-08-04
AI Technical Summary
Existing IoT device data storage systems struggle to effectively handle out-of-order data, resulting in low data storage efficiency.
The time-series data collected by IoT devices is split into sequential data and out-of-order data, and written into different databases: a high-performance first database and a low-performance second database that supports out-of-order data.
It enables the categorized storage of data, improves data storage efficiency and query accuracy, and ensures fast writing of sequential data and effective management of out-of-order data.
Smart Images
Figure CN115510281B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of electronic technology, and in particular relates to a data processing method, apparatus, storage medium and electronic device. Background Technology
[0002] With the development of IoT technology, IoT-based application service systems are becoming increasingly common. These systems typically require various data collected by IoT devices to perform their functions. Therefore, it is necessary to store the data collected by IoT devices for later use. Summary of the Invention
[0003] This application provides a data processing method, apparatus, storage medium, and electronic device that can achieve classified storage of data.
[0004] In a first aspect, embodiments of this application provide a data processing method, including:
[0005] Obtain the timing data to be written;
[0006] If the timing data to be written includes out-of-order data, then the timing data to be written is split into out-of-order data to be written and sequential data to be written.
[0007] Write the sequential data to be written into the first database;
[0008] The out-of-order data to be written is written to the second database.
[0009] Secondly, embodiments of this application provide a data processing apparatus, including:
[0010] The data acquisition module is used to acquire the timing data to be written.
[0011] The data splitting module is used to split the time-series data to be written into out-of-order data to be written and sequential data to be written if the time-series data to be written includes out-of-order data.
[0012] A sequential data writing module is used to write the sequential data to be written into a first database;
[0013] The out-of-order data writing module is used to write the out-of-order data to be written into the second database.
[0014] Thirdly, embodiments of this application provide a storage medium storing a computer program thereon, which, when executed on a computer, causes the computer to perform the data processing method provided in embodiments of this application.
[0015] Fourthly, embodiments of this application also provide an electronic device, including a memory and a processor, wherein the processor executes the data processing method provided in embodiments of this application by calling a computer program stored in the memory.
[0016] In this embodiment of the application, if the time-series data to be written includes out-of-order data, the time-series data to be written is split into out-of-order data to be written and sequential data to be written; the sequential data to be written is written to the first database; and the out-of-order data to be written is written to the second database, thereby realizing the classified storage of data. Attached Figure Description
[0017] The technical solution and its beneficial effects will become apparent from the following detailed description of specific embodiments of this application, in conjunction with the accompanying drawings.
[0018] Figure 1 This is a flowchart illustrating the data processing method provided in the embodiments of this application.
[0019] Figure 2 This is a schematic diagram of the first scenario of the data processing method provided in the embodiments of this application.
[0020] Figure 3 This is a schematic diagram of a second scenario of the data processing method provided in the embodiments of this application.
[0021] Figure 4 This is a schematic diagram of the structure of the data processing device provided in the embodiments of this application.
[0022] Figure 5 This is a schematic diagram of the first structure of the electronic device provided in the embodiments of this application.
[0023] Figure 6 This is a schematic diagram of a second structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0024] It should be noted that the terms "first" and "second," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the listed steps or modules, but some embodiments also include steps or modules not listed, or some embodiments also include other steps or modules inherent to these processes, methods, products, or devices.
[0025] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0026] This application provides a data processing method, a data processing apparatus, a storage medium, and an electronic device. The entity executing the data processing method can be the data processing apparatus provided in this application, or an electronic device integrating the data processing apparatus. The data processing apparatus can be implemented in hardware or software. The electronic device can be a smartphone, tablet computer, PDA, laptop computer, or other device equipped with a processor and possessing data processing capabilities.
[0027] Please see Figure 1 , Figure 1 This is a flowchart illustrating the data processing method provided in an embodiment of this application. The process may include:
[0028] In step 101, retrieve the timing data to be written.
[0029] The timing data to be written is data carrying the data acquisition time. For example, the data to be written could be temperature data reported by an IoT device, such as a temperature sensor, with the acquisition time being 19:56:15 on June 1, 2022. Specifically, this temperature data could be:
[0030] temperature{homeid=1,deviceid=2}value=28,timestamp=2022-06-0119:56:15s.
[0031] For example, temperature sensors C1, C2, and C3 can send their collected temperature data to the electronic device at preset intervals, such as 15 seconds, and humidity sensors H1, H2, and H3 can also send their collected humidity data to the electronic device at preset intervals, such as 15 seconds, so that the electronic device continuously acquires the timing data to be written. The preset intervals can be set by the user or determined by the electronic device based on certain rules.
[0032] like Figure 2As shown, taking the electronic device receiving temperature data sent by temperature sensor C2 in a household as an example, due to network or message queue processing delays, at 19:56:30 on June 1, 2022, the electronic device successively received temperature data collected by temperature sensor C2 at 19:56:15, 19:56:00, and 19:55:45 on June 1, 2022, respectively: temperature data D1, temperature data D2, and temperature data D3. Temperature data D1, D2, and D3 can be used as timing data to be written.
[0033] In 102, if the timing data to be written includes out-of-order data, then the timing data to be written is split into out-of-order data to be written and sequential data to be written.
[0034] For example, suppose the electronic device acquires temperature data D1, temperature data D2, and temperature data D3 as the timing data to be written. We know that temperature data D1, D2, and D3 are sorted by acquisition time as: temperature data D3, D2, and D1. Temperature data D1, D2, and D3 are sorted by reception time as: temperature data D1, D2, and D3. Therefore, temperature data D2 and D3 belong to the out-of-order data to be written, while temperature data D1 belongs to the sequential data to be written. That is to say, the timing data to be written includes out-of-order data. So, the data to be written can be split into out-of-order data to be written and sequential data to be written, that is, split into temperature data D2 and D3 (out-of-order data to be written) and temperature data D1 (sequential data to be written).
[0035] In step 103, the sequential data to be written is written to the first database.
[0036] The first database is a high-performance database that supports writing sequential data, but does not support writing out-of-order data, including but not limited to the Prometheus database.
[0037] For example, such as Figure 2 As shown, since the first database supports writing sequential data, after splitting out the sequential data D1 to be written, the electronic device writes the sequential data D1 to be written into the first database.
[0038] In step 104, the out-of-order data to be written is written to the second database.
[0039] The second database is a database that supports writing out-of-order data but has low performance, including but not limited to InfluxDB, in-memory databases, etc. Therefore, in this embodiment, only out-of-order data is written to the second database.
[0040] For example, such as Figure 2As shown, after separating the out-of-order data D2 and D3 to be written, the electronic device writes the out-of-order data D2 and D3 to the second database.
[0041] In this embodiment, if the time-series data to be written includes out-of-order data, the time-series data to be written is split into out-of-order data to be written and sequential data to be written; the sequential data to be written is written to the first database; and the out-of-order data to be written is written to the second database, thereby achieving classified storage of data.
[0042] In an optional embodiment, writing the out-of-order data to be written to the second database may include:
[0043] (1) Determine the data formats supported by the second database;
[0044] (2) Convert the data format of the out-of-order data to be written into the data format supported by the second database to obtain the converted out-of-order data;
[0045] (3) Write the converted out-of-order data into the second database.
[0046] In this embodiment, the data received by the electronic device from the IoT device is generally sequential data, and therefore is usually written to the first database. The IoT device typically collects data in a data format supported by the first database, allowing the electronic device to quickly write the received data into the database. That is, the data format of the data collected by the IoT device is consistent with the data format supported by the first database. However, different databases support different data formats. Therefore, after separating the out-of-order data to be written, the electronic device also needs to determine the data format supported by the second database and convert the out-of-order data to the format supported by the second database, obtaining the converted out-of-order data. After obtaining the converted out-of-order data, the electronic device writes the converted out-of-order data into the second database.
[0047] For example, suppose there is some out-of-order data to be written, such as temperature data D2:
[0048] temperature{homeid=1, deviceid=C2}value=28.5, timestamp=2022-06-0119:56:0s.
[0049] The data format of temperature data D2 (the data format supported by the first database) is as follows:
[0050] The data tag structure is used to specifically distinguish which household's IoT device reported the data:
[0051] type Label struct{
[0052] Name string
[0053] Value string
[0054] }; that is, Name = homeid, Value = 1; Name = deviceid, Value = C2; Name = "__name__", Value = temperature. Note that the indicator name temperature is an element in the Label.
[0055] A data value structure used to specifically represent the value reported by the IoT device at a certain moment:
[0056] type Sample struct{
[0057] Value float64
[0058] Timestamp int64
[0059] }; that is, the Timestamp here is 2022-06-01 19:56:15s, and the Value is 28.
[0060] A structure that links data labels and data values together represents a sampling action.
[0061] type Timeseries struct{
[0062] Labels[]Label
[0063] Samples[]Sample
[0064] };
[0065] The structures supported by Time Series Database 2 are:
[0066] The data label set is used to specifically distinguish which IoT device reported the data:
[0067] type LabelSet map[string] string, where map[homeid] = 1 and map[deviceid] = C2.
[0068] The sampled data structure SSamples is used to represent the numerical value and time point of a single sample:
[0069] type SSamples struct{
[0070] Metric LabelSet
[0071] Value float64
[0072] Timestamp int64
[0073] };
[0074] Understandably, after determining the data format supported by the second database, the electronic device can convert the data format of the out-of-order data to be written from the data format supported by the first database to the data format supported by the second database. That is, the converted out-of-order data is:
[0075] map[homeid]=1, map[deviceid]=C2, map[__name__]=temperature, Value=28.5, Timestamp=2022-06-01 19:56:0s.
[0076] In an optional embodiment, writing the sequential data to be written to the first database may include:
[0077] (1) Determine the target data to be written from the data to be written in sequence. The target data to be written in sequence is the data whose difference between the acquisition time and the current time is less than a preset difference.
[0078] (2) Write the target data to be written into the first database.
[0079] The preset difference can be set by the user or determined by the electronic device based on certain rules.
[0080] For example, taking the temperature sensor C2 in a home sending temperature data to an electronic device as an example, assuming the preset difference is 5 hours, at 19:56:30 on June 1, 2022, the electronic device receives temperature data D4 and D5 collected by the temperature sensor C2 at 19:56:00 and 19:56:15 on June 1, 2022, respectively. The electronic device can determine that temperature data D4 and D5 are both data to be written in sequence. Furthermore, since the difference between the collection time of temperature data D4 and D5 and the current time is less than the preset difference, the electronic device can determine that temperature data D4 and D5 are both target data to be written in sequence. Therefore, the electronic device can write temperature data D4 and D5 into the first database.
[0081] In an optional embodiment, the data processing method may further include:
[0082] Write the remaining sequential data (excluding the target sequential data) to the third database.
[0083] The third database supports writing timeout data. The aforementioned second database also typically supports writing timeout data; therefore, the third and second databases can refer to the same database. Timeout data is data where the difference between the acquisition time and the current time is greater than or equal to a preset difference. This preset difference can be set by the user or determined by the electronic device based on certain rules.
[0084] For example, taking the temperature data sent from temperature sensors C1 and C2 in a home to an electronic device as an example, assuming the preset difference is 5 hours, at 19:56:30 on June 1, 2022, the electronic device receives temperature data D6 and D7 collected by temperature sensor C1 at 13:56:00 on June 1, 2022, and by temperature sensor C2 at 19:56:15 on June 1, 2022. The electronic device can determine that temperature data D6 and D7 are both data to be written in sequence. Furthermore, since the difference between the acquisition time of temperature data D6 and the current time is greater than the preset difference, the electronic device can determine that temperature data D6 is other data to be written in sequence. Therefore, the electronic device can write temperature data D6 into a third database.
[0085] In an optional embodiment, the electronic device can directly write the out-of-order data to the second database after acquiring it, without waiting for the number of received out-of-order data to reach a preset quantity before writing the received out-of-order data to the second database in batches. The preset quantity can be set by the user or determined by the electronic device based on certain rules.
[0086] In an optional embodiment, writing the out-of-order data to be written to the second database may include:
[0087] If the number of out-of-order data to be written and the number of historical out-of-order data to be written reach a preset number, the out-of-order data to be written will be written to the second database.
[0088] In this embodiment, the electronic device can write out-of-order data into the second database in batches.
[0089] For example, assuming the preset quantity is 5, at 19:56:15 on June 1, 2022, the electronic device acquires out-of-order data D8, D9, and D10 to be written. At 19:56:30 on June 1, 2022, the electronic device acquires out-of-order data D11 and D12 to be written. The out-of-order data D8, D9, and D10 become the historical out-of-order data to be written, and the out-of-order data D11 and D12 become the out-of-order data to be written. The electronic device can determine that the number of out-of-order data to be written and the historical out-of-order data to be written has reached the preset quantity. Therefore, the electronic device can write the data D8 and D9 to be written into the second database.
[0090] In an optional embodiment, the data processing method may further include:
[0091] Write the historical data to be written out of order into the second database.
[0092] For example, assuming the preset quantity is 5, at 19:56:15 on June 1, 2022, the electronic device acquires out-of-order data D8, D9, and D10 to be written, and at 19:56:30 on June 1, 2022, the electronic device acquires out-of-order data D11 and D12 to be written. The out-of-order data D8, D9, and D10 become the historical out-of-order data to be written, and the out-of-order data D11 and D12 become the out-of-order data to be written. The electronic device can determine that the quantity of out-of-order data to be written and the historical out-of-order data to be written have reached the preset quantity. Therefore, the electronic device can batch write the historical out-of-order data D8, D9, and D10, as well as the out-of-order data D11 and D12 to be written, into the second database.
[0093] In an optional embodiment, after acquiring the historical out-of-order data D8, D9, and D10 to be written, the electronic device can place the historical out-of-order data D8, D9, and D10 into a sending queue that currently has no data, and determine whether the number of data in the sending queue has reached a preset number. Assuming the preset number is 5, the electronic device determines that the number of data in the sending queue has not reached the preset number. Subsequently, the electronic device acquires the out-of-order data D11 and D12 to be written, and can place the out-of-order data D11 and D12 into the sending queue, and determine whether the number of data in the sending queue has reached the preset number. Since the sending queue contains the historical out-of-order data D8, D9, and D10, as well as the out-of-order data D11 and D12, the electronic device can determine that the number of data in the sending queue has reached the preset number. Therefore, the electronic device can batch-write the historical out-of-order data D8, D9, and D10, as well as the out-of-order data D11 and D12, into the second database.
[0094] In an optional embodiment, when the difference between the insertion time of the first piece of data into the transmission queue and the current time reaches a first preset difference, the electronic device can directly write all the data in the transmission queue into the second database. The first preset difference can be set by the user or determined by the electronic device based on certain rules.
[0095] For example, assuming the first preset difference is 20 seconds, the electronic device places the out-of-order data D13 to be written into the sending queue at 19:20:50 on June 1, 2022. This out-of-order data D13 is the first data in the sending queue. However, between 19:20:50 and 19:21:10 on June 1, 2022, the electronic device does not receive the out-of-order data to be written. Therefore, the electronic device can write the out-of-order data D13 into the second database at 19:21:10 on June 1, 2022.
[0096] It is understandable that electronic devices can convert the data format of the out-of-order data to be written and the historical out-of-order data to be written into the data format supported by the second database, obtain the converted out-of-order data and the historical converted out-of-order data, and then write the converted out-of-order data and the historical converted out-of-order data into the second database.
[0097] In an optional embodiment, the electronic device can write received data, such as converted out-of-order data and historical converted out-of-order data, into a second database via a streaming communication method through a sending device.
[0098] Specifically, stream processing can be:
[0099] 1. The Protocol Buffer file, specifically the transfer.proto file, defines the interface and data types between the sending device and the second database. The key data streaming interface is AppendStream, and the streaming data type is message AppendStreamReq{
[0100] ReqID = 1;
[0101] string Database = 2;
[0102] Repeated SSamples Reqs = 3;
[0103] Note that the SSamples data format here refers to the Sample structure that the second database can recognize. It is defined as Repeated here, indicating that this is a batch of data.
[0104] 2. Compile the proto file, i.e., transfer.proto, using the Protocol Buffer compiler. This is the code file for the corresponding platform. Here, it is compiled into Go language code, i.e., transfer.pb.go.
[0105] After the conversion, AppendStreamReq will be converted to the following format:
[0106] type AppendStreamReq struct{
[0107] ReqID uint64
[0108] Database string
[0109] Reqs[]*SSamples
[0110] };
[0111] 3. The sending device is the client, and the second database is the server. The client binds to the server's IP and port, initiates a long-connection request via gRPC, and in a separate coroutine, polls and sends AppendStreamReq to the server's AppendStream.
[0112] 4. The server listens locally on the specified IP and port, waiting for gRPC requests to connect. It also receives the AppendStreamReq data sent by the client through the AppendStream interface.
[0113] Using the gRPC streaming processing method, data is transmitted in packets with a high compression ratio and performance. For different platforms and languages, the original transfer.proto file can be directly compiled to generate transfer.proto.xx files for the target platform and language, where xx represents the language type, such as transfer.proto.go file.
[0114] In an optional embodiment, the data processing method may further include:
[0115] (1) Receive a first data query request for the first database;
[0116] (2) Send the first data query request to the first database and receive the first query data returned by the first database in response to the first data query request;
[0117] (3) Convert the first data query request into a second data query request supported by the second database;
[0118] (4) Send the second data query request to the second database and receive the second query data returned by the second database in response to the second data query request;
[0119] (5) Based on the first query data and the second query data, obtain the query results of the first data query request.
[0120] Since most of the data collected by IoT devices is sequential data, it is typically written to a first database. Therefore, during a query, a first data query request is usually made to the first database, and the electronic device receives this first data query request. The electronic device can send this first data query request to the first database and receive the first query data returned by the first database in response. Considering that some data may also be written to the second database, to avoid missing data, the electronic device can also convert the first data query request into a second data query request supported by the second database. Subsequently, the electronic device sends the second data query request to the second database and receives the second query data returned by the second database in response. Finally, the electronic device obtains the query result of the first data query request based on the first and second query data.
[0121] For example, the first data query request (the query request supported by the first database) can be:
[0122] query=temperature{homeid=1, deviceid=C2}&starttime=2022-06-0119:56:0s&endtime=2022-06-01 19:56:30s.
[0123] Then, the electronic device can send the first data query request to the first database and receive the temperature data collected by the temperature sensor C2 from 19:56:00 to 19:56:30 on June 1, 2022.
[0124] The query format of the first data query request is different from that of the second database. Therefore, the electronic device can convert the first data query request into a second data query request according to the query format supported by the second database.
[0125] Specifically, the first data query request is parsed and assigned values according to the following structure format.
[0126] Tag structure:
[0127] type LabelMatcher struct{
[0128] Type int32
[0129] Name string
[0130] Value string
[0131] }
[0132] The query query specifies the indicator name, temperature, with Name="__name__" and Value="temperature". Type indicates the assignment type of Name and Value, with options for "equal", "unequal", and "OR". Other query tags include Name="homeid" and Value="1" and Name="deviceid" and Value="C2".
[0133] Query structure: The query metric has a start time (StartTimestamp), an end time (EndTimestamp), and a label structure.
[0134] type Query struct{
[0135] StartTimestamp int64
[0136] EndTimestamp int64
[0137] Matchers[]*LabelMatcher
[0138] }
[0139] Based on the content of this structure, the electronic device obtains the following second data query request supported by the second database:
[0140] SELECT value FROM / .+ / WHERE "__name__"='temperature'AND"homeid"='1'AND"deviceid"='2'AND time>=1654084560000ms AND time<=1654084590000ms.
[0141] Then, the electronic device can send the second data query request to the second database and receive the temperature data collected by the temperature sensor C2 from 19:56:00 on June 1, 2022 to 19:56:30 on June 1, 2022.
[0142] The first data query request can request the output of corresponding data, such as temperature data collected by temperature sensor C2 from 19:56:00 to 19:56:30 on June 1, 2022. Assuming the first query data is temperature data D21, D22, and D23, and the second query data is temperature data D24 and D25, the electronic device can convert the data format of temperature data D24 and D25 into a format supported by the first database, obtaining converted temperature data D26 and D27. The electronic device can then output these temperature data D21, D22, D23, D26, and D27, such as displaying them on the screen as the query result of the first data query request. Alternatively, the electronic device can sort the temperature data D21, D22, D23, D26, and D27 according to their collection time from earliest to latest, and then output the sorted temperature data as the query result of the first data query request.
[0143] In some embodiments, the first data query request may also request the result of a corresponding function operation, such as calculating the mean, maximum, or minimum value, etc. For example, the first data query request may be to query the average value of the temperature data collected by temperature sensor C2 from 19:56:00 on June 1, 2022 to 19:56:30 on June 1, 2022. Then, after querying the temperature data D21, D22, D23, D26, and D27, the electronic device may calculate the mean of the temperature data D21, D22, D23, D26, and D27 to obtain the query result of the first data query request, etc.
[0144] In an optional embodiment, writing the out-of-order data to be written to the second database may include:
[0145] (1) Determine the data file to be written containing the out-of-order data from the data files included in the second database;
[0146] (2) If the data file to be written is the data file to be uploaded, then create a new data file to be written;
[0147] (3) Write the out-of-order data to be written into a new data file to be written.
[0148] For example, the second database includes at least one data file. Out-of-order data acquired by the electronic device is written to the corresponding data file in the second database. The electronic device saves the data files locally to disk at an hourly / dayly granularity, with each save marked with a timestamp. Due to limited local disk space, data files saved a certain period, such as one day ago, need to be uploaded to cloud storage or other persistent storage to reduce local disk space usage.
[0149] For example, suppose the second database includes data files M1, M2, and M3. Data file M1 stores temperature data for June 1, 2022; data file M2 stores temperature data for June 2, 2022; and data file M3 stores humidity data for June 1, 2022. Assume the out-of-order data to be written is temperature data D2 collected by temperature sensor C2 at 19:56:00 on June 1, 2022. The electronic device can determine that the data file to be written is data file M1. Assume the timestamp of data file M1 is 5:00:00 on June 2, 2022, and the current time is 6:03:12 on June 3, 2022. The electronic device needs to upload a data file saved one day ago to the cloud, and the electronic device can determine that data file M1 is the data file to be uploaded. To prevent the electronic device from uploading data file M1 before the temperature data D2 is fully written to it, the device will create a new data file (e.g., data file M4) upon determining that data file M1 is the file to be uploaded, and then write the temperature data D2 to data file M4. However, assuming the timestamp of data file M1 is 5:00:00 AM on June 2, 2022, and the current time is 6:03:12 AM on June 2, 2022, the electronic device can determine that data file M1 is not the file to be uploaded, and can therefore write the temperature data D2 to data file M1.
[0150] In an optional embodiment, the data processing method may further include:
[0151] If the data file to be written is the same as the data file to be uploaded and is being written, then after the data file to be uploaded is written, the written data file will be uploaded to the cloud.
[0152] It is understandable that, considering the possibility that when a portion of the data to be written to the data file is written out of order, the electronic device may determine that the data file to be written is the data file to be uploaded. In this case, directly uploading the data file to the cloud would result in data loss. Therefore, in this embodiment, the electronic device uploads the data file to the cloud only after the data to be uploaded has been written.
[0153] For example, electronic devices can determine that the data file to be written has been written for a preset time after identifying it as the data file to be uploaded, and then upload the data file to the cloud.
[0154] For example, after an electronic device identifies a data file to be written as a data file to be uploaded, it can detect in real time whether data is being written to the data file to be uploaded; when it detects that no data is being written to the data file to be uploaded, the electronic device determines that the data writing is complete, and the electronic device can upload the data file to the cloud.
[0155] In an alternative embodiment, such as Figure 3 As shown, the second database includes a scanning module, a writing module, and a query module.
[0156] The scanning module is responsible for scanning the timestamps of data files in the second database. When a data file's timestamp exceeds the configured threshold for uploading to the cloud, it identifies the file as a data file to be uploaded and records its filename in the `history.ocs` configuration file. For example, for data older than one month that needs to be uploaded to the cloud, if the difference between the data file's timestamp and the current time exceeds one month, the filename is recorded in the `history.ocs` configuration file. The recorded data format is: "filename; scan timestamp", such as:
[0157] / home / service / var / data / database / data_ocs / v3f1836.data; 1654161017
[0158] Write module: When out-of-order data to be written is to be written to a data file, if the data file is already recorded in the history.ocs configuration file, a new data file is created and the out-of-order data to be written is written; if the data file is not in the history.ocs configuration file, the original process is followed, that is, the out-of-order data to be written is written to the data file.
[0159] The electronic device also includes an adaptation module. This module iterates through the `history.ocs` configuration file to check if there exists a data file whose filename is not in the `uploaded.ocs` configuration file, and the difference between the recorded time and the current time is greater than a configured threshold (e.g., 2 days). If such a file exists, the data file is uploaded to the cloud. After successful upload, the local file is deleted, and the filename is recorded in the `uploaded.ocs` configuration file in the format: "filename; adaptation module cluster domain name".
[0160] / home / service / var / data / database / data_ocs / v3f1836.data; https: / / ocs_adaptor:9099
[0161] The adaptation module has both read and write capabilities, and can also be horizontally scaled to adapt to the cluster size and data size of the time-series database.
[0162] Considering that when the scanning module records the filename of a data file in the history.ocs configuration file, there may still be data being written to that file (i.e., the data file is not yet fully written), if the adaptation module directly uploads the currently being written data file to the cloud without any checks, data loss will occur, and the original database write process will abnormally exit. Therefore, the adaptation module needs to delay uploading the data files in the history.ocs configuration file based on the recorded timestamp, giving the write module sufficient time to detect this. For example, if the difference between the time the data file's filename was recorded in the history.ocs configuration file and the current time is greater than two days, then the data file will be uploaded. Based on this, when the next batch of out-of-order data to be written arrives, the write module detects that the filename of the data file to be written to in this batch of out-of-order data, such as v3f1836.data, exists in the history.ocs configuration file, and then creates a new data file, such as v3f1837.data, and continues the subsequent writing. In this way, the files recorded in the history.ocs configuration file will no longer have any write operations, and the adaptation module can upload or delete the files as needed.
[0163] Query Module: When querying data, if the data file storing the data to be queried is not found in the uploaded.ocs configuration file, it means the data file is still local, and the query can be performed locally. If the data file storing the data to be queried exists in the uploaded.ocs configuration file, the module notifies the adaptation module to read the data from the cloud based on the "adapter module cluster domain name" recorded in the uploaded.ocs configuration file. The adaptation module then returns the read data to the query module.
[0164] The above four parts revolve around the `history.ocs` and `uploaded.ocs` configuration files, ensuring data file read / write separation and reducing the coupling between modules. The contents of the `history.ocs` and `uploaded.ocs` configuration files are editable, greatly aiding in intervention for abnormal events. The scanning and adaptation modules can be deployed independently and horizontally scaled, allowing for flexible deployment based on actual business needs.
[0165] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application. The data processing apparatus 200 includes: a data acquisition module 201, a data splitting module 202, a sequential data writing module 203, and an out-of-order data writing module 204.
[0166] The data acquisition module 201 is used to acquire the timing data to be written.
[0167] The data splitting module 202 is used to split the time-series data to be written into out-of-order data to be written and sequential data to be written if the time-series data to be written includes out-of-order data.
[0168] The sequential data writing module 203 is used to write sequential data to be written into the first database.
[0169] The out-of-order data writing module 204 is used to write out-of-order data to be written into the second database.
[0170] In an optional embodiment, the out-of-order data writing module 204 can be used to: determine the data format supported by the second database; convert the data format of the out-of-order data to be written into the data format supported by the second database to obtain the converted out-of-order data; and write the converted out-of-order data into the second database.
[0171] In an optional embodiment, the sequential data writing module 203 can be used to: determine the target sequential data to be written from the sequential data to be written, wherein the target sequential data to be written is data whose difference between the acquisition time and the current time is less than a preset difference; and write the target sequential data to be written into the first database.
[0172] In an optional embodiment, the data processing device 200 may further include other data writing modules, which may be used to write other data to be written in the order of data to be written, excluding the target data to be written, into a third database.
[0173] In an optional embodiment, the out-of-order data writing module 204 can be used to: write the out-of-order data to be written into the second database if the number of out-of-order data to be written and the number of historical out-of-order data to be written reach a preset number.
[0174] In an optional embodiment, the out-of-order data writing module 204 can be used to write historical out-of-order data to a second database.
[0175] In an optional embodiment, the data processing device 200 may further include a data query module, which can be used to: receive a first data query request for a first database; send the first data query request to the first database and receive first query data returned by the first database in response to the first data query request; convert the first data query request into a second data query request supported by a second database; send the second data query request to the second database and receive second query data returned by the second database in response to the second data query request; and obtain the query result of the first data query request based on the first query data and the second query data.
[0176] In an optional embodiment, the out-of-order data writing module 204 can be used to: determine a data file to be written from the data files included in the second database; if the data file to be written is a data file to be uploaded, create a new data file to be written; and write the out-of-order data to be written into the new data file to be written.
[0177] In an optional embodiment, the data processing device 200 may further include a data upload module, which may be used to: if the data file to be written is a data file to be uploaded and is being written, then after the data file to be uploaded is written, upload the written data file to the cloud.
[0178] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed on a computer, causes the computer to perform the data processing method provided in this embodiment.
[0179] This application also provides an electronic device, including a memory and a processor, wherein the processor executes the data processing method provided in this embodiment by calling a computer program stored in the memory.
[0180] For example, the aforementioned electronic device could be a mobile terminal such as a tablet or smartphone. See also... Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0181] The electronic device 300 may include components such as a processor 301 and a memory 302. Those skilled in the art will understand that... Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0182] The processor 301 is the control center of the electronic device. It connects various parts of the electronic device through various interfaces and lines. By running or executing the application program stored in the memory 302 and calling the data stored in the memory 302, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole.
[0183] Memory 302 can be used to store applications and data. The applications stored in memory 302 contain executable code. Applications can be composed of various functional modules. Processor 301 executes various functional applications and data processing by running the applications stored in memory 302.
[0184] In this embodiment, the processor 301 in the electronic device loads the executable code corresponding to the processes of one or more applications into the memory 302 according to the following instructions, and the processor 301 runs the applications stored in the memory 302, thereby realizing the process:
[0185] Obtain the timing data to be written;
[0186] If the timing data to be written includes out-of-order data, then the timing data to be written will be split into out-of-order data to be written and sequential data to be written.
[0187] Write the sequential data to be written into the first database;
[0188] Write the out-of-order data to be written into the second database.
[0189] Please see Figure 6 The electronic device 300 may include components such as a processor 301, a memory 302, an input unit 303, and an output unit 304.
[0190] The processor 301 is the control center of the electronic device. It connects various parts of the electronic device through various interfaces and lines. By running or executing the application program stored in the memory 302 and calling the data stored in the memory 302, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole.
[0191] Memory 302 can be used to store applications and data. The applications stored in memory 302 contain executable code. Applications can be composed of various functional modules. Processor 301 executes various functional applications and data processing by running the applications stored in memory 302.
[0192] The input unit 303 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.
[0193] The output unit 304 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of electronic devices. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The output unit may include a display screen, which may include a display area.
[0194] In this embodiment, the processor 301 in the electronic device loads the executable code corresponding to the processes of one or more applications into the memory 302 according to the following instructions, and the processor 301 runs the applications stored in the memory 302, thereby realizing the process:
[0195] Obtain the timing data to be written;
[0196] If the timing data to be written includes out-of-order data, then the timing data to be written will be split into out-of-order data to be written and sequential data to be written.
[0197] Write the sequential data to be written into the first database;
[0198] Write the out-of-order data to be written into the second database.
[0199] In an optional embodiment, when processor 301 executes the process of writing out-of-order data to the second database, it may perform the following steps: determining the data format supported by the second database; converting the data format of the out-of-order data to be written into the data format supported by the second database to obtain the converted out-of-order data; and writing the converted out-of-order data into the second database.
[0200] In an optional embodiment, when the processor 301 executes the writing of the sequential data to be written to the first database, it may perform the following: determine the target sequential data to be written from the sequential data to be written, wherein the target sequential data to be written is data whose difference between the acquisition time and the current time is less than a preset difference; and write the target sequential data to be written to the first database.
[0201] In an alternative embodiment, the processor 301 may also perform the following: writing other write-order data besides the target write-order data into a third database.
[0202] In an optional embodiment, the processor 301 executes writing out-of-order data to be written to the second database, and may also execute: if the number of out-of-order data to be written and the number of historical out-of-order data to be written reach a preset number, write the out-of-order data to be written to the second database.
[0203] In an alternative embodiment, the processor 301 may also perform the following: writing historical out-of-order data to a second database.
[0204] In an optional embodiment, the processor 301 may further perform the following actions: receiving a first data query request for a first database; sending the first data query request to the first database and receiving first query data returned by the first database in response to the first data query request; converting the first data query request into a second data query request supported by a second database; sending the second data query request to the second database and receiving second query data returned by the second database in response to the second data query request; and obtaining the query result of the first data query request based on the first query data and the second query data.
[0205] In an optional embodiment, when processor 301 executes the process of writing out-of-order data to the second database, it may perform the following: determine the data file to be written from the data files included in the second database; if the data file to be written is a data file to be uploaded, create a new data file to be written; and write the out-of-order data to be written into the new data file to be written.
[0206] In an optional embodiment, the processor 301 may also perform the following: if the data file to be written is a data file to be uploaded and is being written, then after the data file to be uploaded is written, the written data file to be uploaded is uploaded to the cloud.
[0207] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed description of the data processing method above, which will not be repeated here.
[0208] The data processing device provided in this application embodiment belongs to the same concept as the data processing method in the above embodiment. Any method provided in the data processing method embodiment can be run on the data processing device. For details of its implementation process, please refer to the data processing method embodiment, which will not be repeated here.
[0209] It should be noted that, for the data processing method of the embodiments of this application, those skilled in the art will understand that all or part of the process of implementing the data processing method of the embodiments of this application can be accomplished by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium, such as a memory, and executed by at least one processor. During execution, it can include the process of the embodiments of the data processing method. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), etc.
[0210] For the data processing apparatus of this application embodiment, its functional modules can be integrated into a single processing chip, or each module can exist physically separately, or two or more modules can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0211] The data processing method, apparatus, storage medium, and electronic device provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A data processing method, characterized by, include: Obtain the timing data to be written; If the timing data to be written includes out-of-order data, then the timing data to be written is split into out-of-order data to be written and sequential data to be written. Write the sequential data to be written into the first database; Write the out-of-order data to be written into the second database; Receive a first data query request for the first database; Send the first data query request to the first database and receive the first query data returned by the first database in response to the first data query request; Convert the first data query request into a second data query request supported by the second database; Send the second data query request to the second database and receive the second query data returned by the second database in response to the second data query request; Based on the first query data and the second query data, the query result of the first data query request is obtained; The step of writing the out-of-order data to the second database includes: Determine the data file to be written from the data files included in the second database; If the data file to be written is the same as the data file to be uploaded, then a new data file to be written is created. Write the out-of-order data to be written into the new data file to be written. If the data file to be written is a data file to be uploaded and is being written, then after the data file to be uploaded is written, the written data file to be uploaded will be uploaded to the cloud. The second database records the data files to be uploaded through the history .ocs configuration file and the filenames of the data files already uploaded to the cloud through the uploaded .ocs configuration file.
2. The data processing method according to claim 1, characterized in that, The step of writing the out-of-order data to the second database includes: Determine the data formats supported by the second database; The data format of the out-of-order data to be written is converted into the data format supported by the second database to obtain the converted out-of-order data. The transformed, out-of-order data is written into the second database.
3. The data processing method of claim 1, wherein, The step of writing the sequential data to be written into the first database includes: The target data to be written is determined from the data to be written in sequence, wherein the target data to be written in sequence is data whose difference between the acquisition time and the current time is less than a preset difference; The target data to be written is written into the first database.
4. The data processing method according to claim 3, characterized in that, The method further includes: Write the other data to be written in the order of data to be written, excluding the target data to be written, into the third database.
5. The data processing method of claim 1, wherein, The step of writing the out-of-order data to the second database includes: If the number of out-of-order data to be written and the number of historical out-of-order data to be written reach a preset number, the out-of-order data to be written will be written to the second database.
6. The data processing method according to claim 5, characterized in that, The method further includes: The historical data to be written out of order is written into the second database.
7. A data processing apparatus, characterized by, include: The data acquisition module is used to acquire the timing data to be written. The data splitting module is used to split the time-series data to be written into out-of-order data to be written and sequential data to be written if the time-series data to be written includes out-of-order data. A sequential data writing module is used to write the sequential data to be written into a first database; The out-of-order data writing module is used to write the out-of-order data to be written into the second database; The data query module is used to receive a first data query request for the first database; Send the first data query request to the first database and receive the first query data returned by the first database in response to the first data query request; Convert the first data query request into a second data query request supported by the second database; Send the second data query request to the second database and receive the second query data returned by the second database in response to the second data query request; Based on the first query data and the second query data, the query result of the first data query request is obtained; The step of writing the out-of-order data to the second database includes: Determine the data file to be written from the data files included in the second database; If the data file to be written is the same as the data file to be uploaded, then a new data file to be written is created. Write the out-of-order data to be written into the new data file to be written. If the data file to be written is a data file to be uploaded and is being written, then after the data file to be uploaded is written, the written data file to be uploaded will be uploaded to the cloud. The second database records the data files to be uploaded through the history .ocs configuration file and the filenames of the data files already uploaded to the cloud through the uploaded .ocs configuration file.
8. A storage medium, characterized by The storage medium stores a computer program that, when run on a computer, causes the computer to perform the data processing method according to any one of claims 1 to 6.
9. An electronic device, comprising: The electronic device includes a processor and a memory, the memory storing a computer program, and the processor executing the data processing method according to any one of claims 1 to 6 by calling the computer program stored in the memory.