Data storage method, device and system and storage medium

Abstract

The embodiment of the invention provides a data storage method, device and system and a storage medium. In the embodiment of the invention, on one hand, the latest time sequence data and the historical time sequence data of the target object are stored by adopting the isomorphic time sequence database, and only processor resources and memory resources need to be allocated to the time sequence database, so that compared with a heterogeneous storage scheme, two databases with different structures do not need to be maintained; processor resources and memory resources occupied by the database can be reduced, namely the processor resources and the memory resources are saved. And on the other hand, the latest time series data with high query frequency adopts the data table which is provided with the timestamp and is independently stored in the time series database with the permanent storage strategy, compared with a traditional heterogeneous storage mode, the relation between the data tables does not need to be maintained in a relational database, and storage resources occupied by the latest time series data can be reduced.

Description

Technical Field

[0001] This application relates to the technical field of data storage, and in particular, to a data storage method, device, system, and storage medium. Background Art

[0002] Time-series data is a series of data generated by entity objects over time, which is a series of data with time tags, also known as time series data. There is a large amount of time-series data in fields such as application performance monitoring, Internet of Things, and industrial Internet, which poses a great challenge to the storage of time-series data.

[0003] Time-series data can be divided into the latest data and historical data. In order to monitor or analyze the performance of entity objects, it is necessary to store both the latest data and historical data of entity objects. However, the existing time-series data storage methods occupy more resources. Summary of the Invention

[0004] Multiple aspects of this application provide a data storage method, device, system, and storage medium to reduce the resources occupied by time-series data storage.

[0005] An embodiment of this application provides a data storage method, including:

[0006] Obtain the latest time-series data of the target object;

[0007] Configure a set timestamp for the latest time-series data to obtain the configured latest time-series data;

[0008] Generate a first write request to write the configured latest time-series data into the first data table of the time-series database and write the latest time-series data into the second database of the time-series database as historical time-series data;

[0009] Send the first write request to the database engine of the time-series database, so that the database engine, in response to the first write request, writes the configured latest time-series data into the first data table and writes the latest time-series data into the second data table;

[0010] Wherein, the storage policy of the data in the first data table is permanent storage; the storage policy of the data in the second data table is expired deletion.

[0011] An embodiment of this application also provides a data storage system, including: a server device and the database engine of the time-series database;

[0012] The server device is used to execute the steps in the above data storage method;

[0013] The database engine is configured to write the configured latest time-series data into the first data table and write the latest time-series data into the second data table in response to a first write request sent by the server device; wherein, the storage policy for the data in the first data table is permanent storage; the storage policy for the data in the second data table is deletion upon expiration.

[0014] An embodiment of the present application further provides an electronic device, including: a memory, a processor, and a communication component; wherein, the memory is configured to store a computer program.

[0015] The processor is coupled to the memory and the communication component and is configured to execute the computer program to perform the steps in the above data storage method.

[0016] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps in the above data storage method.

[0017] In an embodiment of the present application, on the one hand, the latest time-series data and historical time-series data of the target object are stored in a homogeneous time-series database. Only processor resources and memory resources need to be allocated for the time-series database. Compared with a heterogeneous storage solution, there is no need to maintain two databases with different structures, which can reduce the processor resources and memory resources occupied by the database, that is, save processor and memory resources. On the other hand, the latest time-series data with a high query frequency uses a set timestamp and is separately stored in a data table of the time-series database with a permanent storage policy. Compared with the traditional heterogeneous storage method, there is no need to maintain the relationship between data tables in a relational database, which can reduce the storage resources occupied by the latest time-series data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0019] Figure 1 is a schematic structural diagram of a data processing system provided by an embodiment of the present application;

[0020] Figure 2 is a schematic structural diagram of a data storage system provided by an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of a data model of the latest time-series data and historical time-series data provided by an embodiment of the present application;

[0022] Figure 4Schematic diagram of the specific process of data storage for the data storage system provided by the embodiments of the present application;

[0023] Figure 5 Flow chart of the data storage method provided by the embodiments of the present application;

[0024] Figure 6 Schematic diagram of the structure of the electronic device provided by the embodiments of the present application. Detailed implementation manners

[0025] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0026] The physical model is a digital abstraction of an entity object in the natural world in the digital world, and is the digital model of the entity object. In the present application, the entity object refers to an entity objectively existing in the natural world, and can generate a series of data over time, that is, time-series data. For example, the entity object can be an Internet of Things (IoT) device, such as a sensor, an electrical appliance, or a vehicle-mounted device, etc., and the entity object can also be any industrial device or facility, such as a boiler, a computer room, or a data center, etc.

[0027] In the present application, the entity object can be described from three dimensions of attributes, services, and events, namely what it is, what it can do, and what information it can provide externally. Defining these three dimensions of the physical model completes the definition of the physical model of the entity object and obtains the physical model definition information of the entity object. Among them, the physical model definition information can include information in three dimensions of attributes, services, and events. Attributes are used to describe the specific information and status of the entity object during operation. For example, the current environmental temperature read by an environmental monitoring device, the on / off state of an electrical appliance, or the wind power level of an electric fan, etc. A service refers to an instruction or method that can be externally called by the entity object. Input and output parameters can be set in the service call. The input parameter is the parameter during the execution of the service, and the output parameter is the result after the execution of the service. An event is information actively reported when the entity object is running, and generally includes information that needs to be perceived and processed externally, alarms, and faults. Multiple output parameters can be included in the event. For example, the notification information after a certain task is completed, or the temperature and time information when the entity object fails, or the running state when the device alarms, etc.

[0028] The data generated by the entity object according to the above object model (such as temperature, humidity, or switch status, etc.) is called object model data. For example, the attributes of the entity object may include: humidity, temperature, and switch status; the services of the entity object include: voice broadcast service; the events of the entity object include: fault events, etc.; then at a certain moment, the object model data of the entity object may include: data such as the temperature, humidity, or switch status of the entity object at that moment. If a voice broadcast call is made at that moment, the object model data may also include: the call result data of the voice broadcast service corresponding to that moment. If a fault event occurs to the entity object at that moment, the object model data of the entity object may also include: the fault event description information of the entity object at that moment.

[0029] Among them, object model data is generally time-series data, that is, a series of data generated as time changes. For entity objects, time-series data is generally divided into: the latest time-series data and historical time-series data. Among them, the latest time-series data is the object model data generated by the entity object most recently; the historical time-series data is the object model data generated by the entity object in the past.

[0030] The inventors of this application have found through research that users have a higher query frequency for the latest time-series data of entity objects; while the query frequency for historical time-series data is lower. Therefore, the latest time-series data of entity objects generally needs to be permanently retained to facilitate users to view the latest running status of entity objects at any time. The historical time-series data of entity objects generally expires and is deleted to save storage space.

[0031] In the traditional solution, a heterogeneous storage method is generally used to store the latest time-series data and historical time-series data of entity objects respectively. Among them, historical time-series data is generally stored in a time-series database, and the latest time-series data is generally stored in a relational database. Among them, a relational database organizes data in the form of tables, and connections can be established between tables through relationships, and complex queries and data operations are supported. The Structured Query Language (SQL) can be used to create, modify, query, and manage data, etc. This heterogeneous storage method requires maintaining 2 databases (i.e., a time-series database and a relational database) respectively, and allocating resources such as a Central Processing Unit (CPU) and memory to each database separately, resulting in a relatively large overall resource occupation of the heterogeneous storage method. On the other hand, in addition to maintaining data tables, a relational database also needs to maintain the relationships between data tables, increasing the storage resources occupied by time-series data.

[0032] In order to reduce the resources occupied by the storage of time-series data, in some embodiments of the present application, on the one hand, the latest time-series data and historical time-series data of the target object are stored in a homogeneous time-series database. Only the processor resources and memory resources need to be allocated for the time-series database. Compared with the heterogeneous storage solution, there is no need to maintain two databases with different structures, which can reduce the processor resources and memory resources occupied by the database, that is, save the processor and memory resources. On the other hand, the latest time-series data with a high query frequency is stored separately in a data table of the time-series database with a save policy of permanent storage using a set timestamp. Compared with the traditional heterogeneous storage method, there is no need to maintain the relationship between data tables in a relational database, which can reduce the storage resources occupied by the latest time-series data.

[0033] The following will describe in detail the technical solutions provided by the embodiments of the present application with reference to the accompanying drawings.

[0034] It should be noted that the same reference numerals represent the same object in the following drawings and embodiments. Therefore, once an object is defined in one drawing or embodiment, it does not need to be further discussed in the subsequent drawings and embodiments.

[0035] Figure 1 It is a schematic diagram of the overall structure of the data processing system provided by the embodiment of the present application. As Figure 1 shown, the data processing system includes: a terminal device 10 and a server device 20. Among them, the terminal device 10 can be the terminal device of the developer of the server device 20, such as a desktop computer or a laptop computer, etc. The developer can define the physical model structure of the target object 30 according to actual needs to obtain the physical model definition information of the target object 30. For the description of the physical model definition information, reference can be made to the foregoing related content, which will not be elaborated here. Among them, the target object 30 can be any entity object in the natural world, such as an IoT device, any industrial device or facility, etc., but not limited thereto. In this embodiment, the target object 30 has the function of communicating with the server device 20.

[0036] The server device 20 can communicate with the target object 30 through the access layer. The communication protocols supported by the access layer include but are not limited to: Message Queuing Telemetry Transport (MQTT) protocol, Constrained Application Protocol (CoAP) protocol, and / or Hypertext Transfer Protocol (HTTP) protocol, etc. Correspondingly, the access layer can provide relevant plugins for these protocols, such as MQTT protocol plugin, CoAP protocol plugin, and / or HTTP protocol plugin, etc.

[0037] The server device 20 can perform data management and generally has the ability to undertake and guarantee services. The server device 20 can be a single server device, or a cloudified server array, or a virtual machine (VM) running in the cloudified server array. In addition, the server device can also refer to other computing devices with corresponding service capabilities, such as terminal devices (running service programs) like computers. In this embodiment, the server device 20 can provide an IoT platform to manage the physical model data generated by the target object 30.

[0038] Specifically, developers can define the physical model of the target object 30 through the terminal device 10 to obtain the physical model definition information of the target object 30. Further, the terminal device 10 can send the physical model definition information to the server device 20 by calling the Application Programming Interface (API) provided by the server device 20 (corresponding to Figure 1 Step 1 "Define the physical model").

[0039] The target object 30 can request the physical model definition information from the server device 20; and generate physical model data based on the physical model definition information, that is, generate time-series data (corresponding to Figure 1 Step 2). The physical model data carries time information and can also be called time-series data. In the embodiments of this application, the physical model data is referred to as time-series data. The time-series data includes information such as tags, timestamps, metrics, and metric values. Among them, tags are used to represent monitored objects. Metrics are attributes in the physical model definition information and can represent monitoring indicators; metric values represent the specific values of the monitoring indicators. The timestamp can represent the time when the target object 30 collects the metric value. The time-series data can also include events and / or service information defined by the physical model definition information. Now, with reference to the time-series data shown in Table 1, the information included in the time-series data will be described.

[0040] Table 1 Example of time-series data

[0041]

[0042] In Table 1, the label is: Device + Region; the label value is "Device No. D0001 in North China Region", that is, the monitoring object (i.e., the target object) is "Device No. D0001 in North China Region". The metrics, i.e., attributes, are: temperature and humidity. The metric values (or attribute values) are temperature of 12.1 °C and humidity of 45. The service is the voice playback service; the event is the fault event. 12.1 and 45 are the values under temperature and humidity respectively, which are the metric values. The call result of the voice playback service is a successful call; the description information of the fault event is that the software with number 500 sends an error message. Therefore, Table 1 can represent the time-series data of the target object (Device No. D0001 in North China Region). Among them, a data point (Point) in the time-series database consists of tags, timestamp, and data (Field). Data (Field) refers to other data in the time-series data except for tags and timestamps, such as the above-mentioned metrics and metric values, services and service call results, and events and event description information, etc.

[0043] Furthermore, the target object 30 can report the time-series data to the server device 20. In the embodiments of the present application, the specific data format of the time-series data reported by the target object 30 is not limited. In some embodiments, the target object 30 can adopt a binary data format to report the time-series data; or report the time-series data in the form of an SQL statement, etc. The data formats of the time-series data reported by different target objects may be the same or different.

[0044] The server device 20 can perform data structuring processing on the time-series data reported by the target object 30 through a pre-configured script, that is, convert the time-series data reported by the target object 30 into a target data structure supported by the server device 20. The server device 20 can also provide the time-series data with the target data structure to the database engine 40 of the time-series database, and the database engine 40 writes the time-series data with the target data structure into the time-series database. Among them, the database engine 40 is one of the core components in the database system, responsible for managing the storage, indexing, transactions, and concurrency of the time-series database. Of course, the server device 20 can also provide the structured time-series data to the user application or middleware through the rule engine (corresponding to Figure 1 Step 3). Among them, the user is the user who uses the services provided by the server device 20.

[0045] Of course, the server device 20 can also provide a data query service. The server device 20 can generate a query request according to the user's query requirement information; and send the query request to the database engine 40. The database engine 40 can perform a query in the time-series database according to the query request to obtain the query result.

[0046] The above Figure 1From the perspective of the architecture of the data processing system, an exemplary description is given of the time-series data, the generation and processing process of the time-series data, and the application scenarios of the data storage method provided in the embodiments of the present application. The embodiments of the present application focus on protecting and describing the storage process of the time-series data. The following combines Figure 2 The data storage system shown is used to give an exemplary description of the data storage method provided in the embodiments of the present application.

[0047] It should be noted that the data storage solution provided in the embodiments of the present application is applicable to both cold storage and hot storage. For the purpose of improving the data writing and reading speeds, it is mainly used for hot storage. Among them, hot storage and cold storage are relative. Hot storage means that frequently used data is stored in storage media with relatively fast reading and writing speeds, such as high-speed disks, solid-state drives (Solid State Disk, SSD) or random access memories (Random Access Memory, RAM), so as to enable fast access. In contrast, cold storage means that infrequently accessed data is stored in low-cost storage media with relatively slow reading and writing speeds (such as magnetic tapes), and generally does not need to be accessed frequently.

[0048] As Figure 2 shown, the data storage system may include: a server device 20 and a database engine 40 of the time-series database. The time-series data received by the server device 20 may be the time-series data generated by the target object 30 according to the foregoing physical model definition information, or may of course be other forms of time-series data generated by the target object 30, which is not limited in the present application.

[0049] In order to reduce the resources occupied by the storage of time-series data, in the embodiments of the present application, it is proposed that the latest time-series data and the historical time-series data are stored in a homogeneous time-series database, but the data models of the latest time-series data and the historical time-series data are separately modeled. Specifically, as Figure 3 shown, the latest time-series data and the historical time-series data of the target object are respectively stored in different data tables. For the convenience of description and distinction, the data table storing the latest historical time-series data is defined as the first data table; the data table storing the historical time-series data is defined as the second data table. Among them, both the first data table and the second data table can be one or more. Multiple means two or more. The number of the first data table and the second data table is determined by the capacity of the data table and the size of the specifically written data volume.

[0050] Since the first data table is used to store the latest time-series data of the target object, and the query frequency of the latest time-series data is relatively high. Users often need to understand the recent operating conditions of the target object based on the latest time-series data of the target object. Therefore, the saving policy of the first data table can be configured to be permanently saved, which can also be referred to as non-expiring. The second data table is used to store the historical time-series data of the target object. The historical time-series data has relatively low reference significance for the recent operating conditions of the target object, or even no reference significance. Therefore, the query frequency of the historical time-series data is relatively low. Even some historical time-series data will not be queried by users after a certain period of time. Therefore, in order to save storage space, the saving policy of the second data table can be configured to be deleted when it expires. The retention duration of the historical time-series data is determined by the duration between the timestamp carried by the historical time-series data and the current time. Deleting when it expires means that if the retention duration of the historical time-series data is greater than or equal to the set expiration duration, then the historical time-series data is deleted. Among them, the expiration duration can be flexibly set according to actual needs. For example, the expiration duration can be 1 month, 2 months, half a year, 1 year, 2 years, etc.

[0051] In some embodiments, the server device 20 can configure the saving policy of the data table during the data table creation process. For example, the server device 20 can send a table creation request to the database engine 40. The table creation request can include: the identifier of the data table to be created and the saving policy of the data table to be created. Correspondingly, the database engine 40 can respond to the table creation request and create the data table. The name of the data table is the identifier of the data table to be created and the saving policy is the saving policy of the data table to be created. Based on this, the server device 20 can send a table creation request to the database engine 40. The table creation request includes: the identifier of the first data table and the saving policy of the first data table is permanently saved. The database engine 40 can respond to the table creation request, create the first data table, and configure the saving policy of the first data table to be permanently saved. Similarly, the server device 20 can send another table creation request to the database engine 40. The table creation request includes: the identifier of the second data table, the saving policy of the second data table is deleted when it expires, and the set expiration duration. The database engine 40 can respond to the table creation request, create the second data table, and configure the saving policy of the first data table to be deleted when it expires, and the expiration duration is the set expiration duration.

[0052] Such as Figure 3As shown in the figure, the latest time-series data for the target object 30 is stored in the data model with a set timestamp, that is, a set timestamp is configured for the latest time-series data to obtain the configured latest time-series data. The same target object 30 uses the same set timestamp. Different target objects can use the same set timestamp or different set timestamps. In the embodiments of the present application, the format of the set timestamp is different from the timestamp carried by the time-series data. The timestamp carried by the time-series data is generally the time that actually exists in the natural world, and the timestamp has a set format, such as 2023 / 08 / 01 00:00:00 (that is, 0:00:00 on August 1, 2023); the set timestamp generally uses set numbers, such as 0, 1, 2, 3, etc., without actual time meaning, etc. Figure 3 Only the case where the set timestamp is the number 0 is illustrated in the figure, but it does not constitute a limitation.

[0053] The timestamp of the configured latest time-series data is the set timestamp; the latest time-series data reported by the target object 30 is the data (Field) corresponding to the set timestamp. For example, as Figure 3 shown, the latest time-series data reported by the target object includes: device name (i.e., the identifier of the target object), attributes and attribute values (i.e., metrics and metric values), and reporting time. Among them, the reporting time can be the time when the target object generates the latest time-series data, or the time when the target object reports the latest time-series data, etc. The storage data model of the latest time-series data can be implemented as Figure 3 shown, including: the timestamp is the set timestamp, the label is the identifier of the target object, and the data (Field) is the attribute and attribute value and reporting time, etc.

[0054] For example, as Figure 3 shown, the latest time-series data reported by the target object 2 includes: "Device name: identifier of target object 2; Switch: off; Switch - reporting time: 2023 / 08 / 01 00:00:00; Temperature: -100; Temperature - reporting time: 2023 / 08 / 01 00:00:00", then the configured latest time-series data of the target object 2 includes: "Timestamp: 0; Identifier of target object 2; Switch: off; Switch - reporting time: 2023 / 08 / 01 00:00:00; Temperature: -100; Temperature - reporting time: 2023 / 08 / 01 00:00:00".

[0055] If different target objects use different set timestamps, the server device 20 can pre-store the correspondence between the identifier of the target object and the set timestamp. In this way, when configuring the set timestamp for the latest time-series data of the target object, the set timestamp corresponding to the current target object can be obtained from the correspondence between the identifier of the target object and the set timestamp; and the set timestamp corresponding to the current target object can be configured for the latest time-series data of the current target object, etc.

[0056] Since the latest time-series data after configuration for the same target object has the same timestamp and tags (such as device name or identification of the target object, etc.), when the time-series database saves the latest time-series data after configuration of the target object in the first data table, it will overwrite the original latest time-series data after configuration of the target object, so that there is only one data point for the latest time-series data of each target object in the first data table.

[0057] For historical time-series data, a traditional time-series database data model is used for storage, that is, the time information carried by the time-series data reported by the target object 30 (i.e., the reporting time) is used as the timestamp, the device name in the time-series data reported by the target object 30 (such as the identification of the target object) is used as the tag, and the attributes and attribute values carried by the time-series data are used as data (Field) for storing historical time-series data. The historical time-series data stored in the second data table includes: the timestamp is the reporting time carried by the time-series data, the tag is the identification of the target object, and the data (Field) is the attribute and attribute value, etc.

[0058] For example, as Figure 3 shown, the latest time-series data reported by the target object 2 includes: "Device name: Identification of target object 2; Switch: Off; Switch - Reporting time: 2023 / 08 / 01 00:00:00; Temperature: -100; Temperature - Reporting time: 2023 / 08 / 01 00:00:00", then the latest time-series data of the target object 2 stored in the second data table includes: "Timestamp: 2023 / 08 / 01 00:00:00; Device name: Identification of target object 2; Switch: Off; Temperature: -100".

[0059] Since the timestamp of the historical time-series data stored in the second data table is the reporting time carried by the time-series data, for the same target object, there are multiple data points with different timestamps. Multiple means 2 or more than 2. For example, as Figure 3 shown, the historical time-series data of the target object 2, in addition to the above data points, also includes the data point "Timestamp: 2023 / 10 / 01 00:00:00; Device name: Identification of target object 2; Switch: On; Temperature: -110".

[0060] Combined with Figure 2 and Figure 3 , based on the data models of the latest time-series data and historical time-series data shown in the above embodiments, the server device 20 can obtain the latest time-series data of the target object 30 (corresponding to Figure 2 Step 1). In some embodiments, the server device 20 can receive the time-series data currently reported by the target object 30 as the latest time-series data of the target object 30.

[0061] Further, based on the data model of the latest time series data and historical time series data shown in the above embodiments, the server device 20 may configure a set timestamp for the latest time series data of the target object 30 to obtain the configured latest time series data (corresponding to Figure 2 Step 2). Further, the server device 20 may generate a write request for writing the configured latest time series data into the first data table and writing the latest time series data (the latest time series data before configuration) of the target object 30 into the second data table as historical time series data (corresponding to Figure 2 Step 3). The write request includes: the configured latest time series data and the identifier of the data table to be written corresponding to the configured latest time series data (i.e., the identifier of the first data table), and the latest time series data (i.e., the latest time series data before configuration) and the identifier of the data table to be written corresponding to the latest time series data (i.e., the identifier of the second data table). Further, the server device 20 may send the write request to the database engine 40 (corresponding to Figure 2 Step 4).

[0062] Correspondingly, the database engine 40 may receive the write request; and in response to the write request, write the configured latest time series data into the first data table; and write the latest time series data of the target object 30 into the second data table (corresponding to Figure 2 Step 5 "store data").

[0063] Specifically, the database engine 40 may write the set timestamp carried by the configured latest time series data into the timestamp column of the first data table, and write the latest time series data into the corresponding data (Field) column of the first data table, so as to write the configured latest time series data into the first data table. Of course, the database engine 40 also writes the target time information carried by the latest time series data of the target object into the timestamp column of the second data table, and writes the other data in the latest time series data except the target time information into the corresponding data (Field) column of the second data table, so as to write the latest time series data of the target object into the second data table, realizing the storage of the latest time series data as historical time series data.

[0064] In this embodiment, on the one hand, the latest time series data and historical time series data of the target object are stored in a homogeneous time series database. Only the processor resources and memory resources need to be allocated for the time series database. Compared with the heterogeneous storage scheme, there is no need to maintain two databases with different structures, which can reduce the processor resources and memory resources occupied by the database, that is, save the processor and memory resources. On the other hand, the latest time series data with a high query frequency uses a set timestamp and is separately stored in the data table of the time series database with the save policy of permanent storage. Compared with the traditional heterogeneous storage method, there is no need to maintain the relationship between data tables in a relational database, which can reduce the storage resources occupied by the latest time series data.

[0065] Moreover, in this embodiment, the latest time-series data with a high query frequency adopts a set timestamp and is separately stored in a data table of a time-series database with a permanent storage policy, so that each target object has only one data point in this data table, which can ensure that the data of the target object does not expand.

[0066] In addition, in some traditional solutions, only historical time-series data is stored in the time-series database, and the latest time-series data is not stored separately. When querying the latest time-series data, the latest time-series data is traversed and queried in the historical time-series data. In order to be able to query the latest time-series data, all historical time-series data needs to be permanently stored, that is, the historical time-series data needs to always occupy storage resources, and the storage resource consumption is relatively large. Compared with this method, in this embodiment, the historical time-series data with a low query frequency is separately stored in a data table with an expiration deletion storage policy, and the expired historical time-series data can be deleted, reducing the storage resources occupied by the historical time-series data.

[0067] Since the latest time-series data and the historical time-series data are stored in a homogeneous time-series database, therefore, in the embodiment of the present application, there is only an operation of writing to the time-series database, and there is no operation of reading and writing a relational database, which can reduce the probability of the occurrence of a write bottleneck. Since there is only an operation of writing to the time-series database in the embodiment of the present application, therefore, operations such as data aggregation and asynchronous batch processing can be adopted to achieve high-concurrency point writing. The following combines Figure 4 for specific description.

[0068] As Figure 4 shown, the server device 20 can receive the time-series data reported by the target object 30 (corresponding to Figure 4 step 1). The time-series data reported by the target object 30 is the latest time-series data at the current reporting moment. The target object 30 can be one or more, and multiple means two or more.

[0069] In order to reduce the probability of the occurrence of a write bottleneck in the time-series database, in this embodiment, the server device 20 can perform data aggregation on the time-series data reported by the target object 30. Specifically, the server device 20 can set a storage queue. This storage queue can be set in the memory of the server device 20, and can also be called a memory queue. Correspondingly, the server device 20 can store the received time-series data in the storage queue.

[0070] In some embodiments, the data structure of the time-series data reported by the target object 30 may be different from the target data structure supported by the server device 20. In order to implement the management of the time-series data by the server device 20, before storing the time-series data reported by the target object 30 received by the server device 20 in the storage queue, the server device 20 can also convert the time-series data reported by the target object 30 into the target data structure supported by the server device 20 (corresponding to Figure 4Step 2 "Data Structuring"); and store the time-series data with the target data structure into the storage queue (corresponding to Figure 4 Step 3).

[0071] It should be noted that in each embodiment of the present application, the time-series data stored by the server device 20 into the storage queue and sent to the database engine 40 can both be the time-series data with the target data structure obtained through the above data structure conversion. This will not be emphasized one by one in each embodiment.

[0072] For high-concurrency data points, that is, a large number of target objects report time-series data simultaneously at the same moment, after the server device 20 converts the time-series data reported concurrently by multiple target objects into the target data structure supported by the server device 20, it stores the time-series data reported concurrently by multiple target objects into the storage queue. Optionally, the server device 20 can adopt an asynchronous method to store the time-series data reported by multiple target objects into the storage queue (corresponding to Figure 4 Step 3 "Asynchronous Enqueue"); and after storing the time-series data reported by the target object into the storage queue, asynchronously return a storage success message to the target object (corresponding to Figure 4 Step 5). In this way, the server device 20 can return a storage success message to the target object without waiting for the time-series data to be written into the time-series database, which can reduce the storage latency perceived by the user side of the target object and make the storage process of the server device 20 imperceptible to the user.

[0073] The time-series data reported by the target object 30 is aggregated by the server device 20 in the storage queue. Further, when the time-series data of the target object 30 stored in the storage queue by the server device 20 meets the set conditions, the server device 20 can obtain the time-series data with the latest time information carried from the time-series data of the same target object 30 stored in the storage queue as the latest time-series data of the target object 30.

[0074] In the embodiments of the present application, the specific implementation manner of the set conditions is not limited. In some embodiments, the set conditions may include: the data volume of the time series data stored in the storage queue reaches a set data volume. Accordingly, the server device 20 may determine that the time series data of the target object 30 stored in the storage queue meets the set conditions when the data volume of the time series data stored in the storage queue reaches the set data volume. In other embodiments, the set conditions may include: the time interval of the time series data stored in the storage queue reaches a set time interval. Accordingly, the server device 20 may determine that the time series data of the target object 30 stored in the storage queue meets the set conditions when the time interval of the time series data stored in the storage queue reaches the set time interval. In still other embodiments, the set conditions may include: the data volume of the time series data stored in the storage queue reaches a set data volume, and the time interval of the time series data stored in the storage queue reaches a set time interval. Accordingly, the server device 20 may determine that the time series data of the target object 30 stored in the storage queue meets the set conditions when the data volume of the time series data stored in the storage queue reaches the set data volume and the time interval of the time series data stored in the storage queue reaches the set time interval.

[0075] Further, when the time series data of the target object 30 stored in the storage queue meets the set conditions, for any target object 30, the time series data with the latest carried time information may be obtained from the time series data of the target object 30 stored in the storage queue as the latest time series data of the target object 30. And, all the time series data of the target object 30 may be obtained from the storage queue as the historical time series data of the target object 30 (corresponding to Figure 4 "data dequeue" in step 4).

[0076] Further, the server device 20 may configure a set timestamp for the latest time series data of the target object to obtain the configured latest time series data of the target object. In some embodiments, there are multiple target objects. The server device 20 may configure the same set timestamp for the latest time series data of the multiple target objects 30 to obtain the configured latest time series data corresponding to each of the multiple target objects.

[0077] Alternatively, the server device 20 may configure different set timestamps for the latest time-series data of multiple target objects 30 respectively to obtain the configured latest time-series data corresponding to each target object. In this embodiment, the server device 20 may pre-store the correspondence between the identifier of the target object and the set timestamp. In this correspondence, the set timestamps corresponding to target objects with different identifiers are different. Based on the correspondence between the identifier of the target object and the set timestamp, the server device 20 may obtain the set timestamps corresponding to each of the multiple target objects 30 from this correspondence; and configure the set timestamp corresponding to its identifier in the correspondence for the latest time-series data of each target object 30 to obtain the configured latest time-series data of this target object 30.

[0078] Further, a write request is generated based on the configured latest time-series data and historical time-series data of the target object 30 (i.e., all the time-series data of the target object 30 stored in the storage queue). Specifically, the server device 20 may generate a write request to write the configured latest time-series data of the target object 30 into the first data table and write the time-series data of the target object stored in the storage queue as historical time-series data into the second data table. Among them, the time-series data of the target object stored in the storage queue includes: the latest time-series data of each target object, and the generated write request also includes a request to write the latest time-series data of the target object as historical time-series data into the second data table.

[0079] Specifically, the server device 20 may use the set timestamp corresponding to the target object 30 as the timestamp to be written, use the configured latest time-series data of the target object 30 as the data (Field) to be written corresponding to the set timestamp corresponding to this target object 30, and, use the target time information included in the time-series data of the target object 30 stored in the storage queue as the timestamp to be written, and use the other data in the time-series data of the target object 30 stored in the storage queue except the target time information as the data (Field) to be written corresponding to the target time information, to generate a write request to write the configured latest time-series data of the target object 30 into the first data table and write the time-series data of the target object stored in the storage queue as historical time-series data into the second data table.

[0080] The write request may include: the configured latest time-series data of the target object 30 and the identifier of the data table to be written corresponding to the configured latest time-series data of the target object 30 (i.e., the identifier of the first data table), and, the historical time-series data of the target object 30 (i.e., the time-series data of the target object stored in the storage queue) and the identifier of the data table to be written corresponding to the historical time-series data (the identifier of the second data table).

[0081] Further, the server device 20 may send the above write request to the database engine 40. Since the time-series data of the target object stored in the storage queue by the server device 20 is aggregated together before the time-series data meets the set conditions, and a write request, that is, an input / output (IO) request, is generated when the time-series data of the target object stored in the storage queue meets the set conditions, the aggregated time-series data in the storage queue can be stored in the time-series database together, which can reduce the frequency of IO access to the time-series database.

[0082] Optionally, the server device 20 may asynchronously send a write request to the database engine 40 (corresponding to Figure 4 step 4 "asynchronously issue a write request"); and after sending the write request to the database engine, asynchronously return a storage success message to the target object (corresponding to Figure 4 step 5). In this way, the server device 20 does not need to wait for the time-series data to be written into the time-series database to return a storage success message to the target object, which can reduce the storage delay perceived by the user side of the target object.

[0083] Correspondingly, for the database engine 40, it can receive the above write request, and in response to the write request, write the set timestamp corresponding to the target object 30 into the timestamp column of the first data table, and write the latest time-series data of the target object into the data column corresponding to the timestamp column of the first data table; and write the target time information included in the time-series data of the target object stored in the storage queue into the timestamp column of the second data table, and write the other data except the target time information in the time-series data of the target object stored in the storage queue into the data column corresponding to the timestamp column of the second data table, so as to write the configured time-series data of the target object 30 into the first data table, and write the time-series data of the target object stored in the storage queue as historical time-series data into the second data table (corresponding to Figure 4 step 5 "store data").

[0084] In the traditional heterogeneous storage method of using two heterogeneous databases, namely a time-series database and a relational database, to store historical time-series data and the latest time-series data respectively, for concurrent points, write requests need to be generated for each target object, and a large number of concurrent write requests are sent to the relational database, resulting in a write bottleneck in the relational database. In the embodiment of the present application, data aggregation and asynchronous methods are used to batch store time-series data. For the latest time-series data concurrent for multiple target objects, it can also be batch stored to achieve concurrent point writing; at the same time, the batch storage method can reduce the number of IO accesses to the time-series database, reduce the probability of the write bottleneck occurring in the time-series database, and even avoid the write bottleneck.

[0085] Under the same resource conditions, the inventors of this application tested the performance of the heterogeneous storage method provided by the traditional solution, which stores historical time-series data and the latest time-series data in two heterogeneous databases, namely a time-series database and a relational database, and the storage method provided by the embodiments of this application, which stores historical time-series data and the latest time-series data in a homogeneous time-series database. Among them, both the traditional heterogeneous storage method and the homogeneous storage method provided by the embodiments of this application used three servers with 4-core CPUs and 8G of memory for data storage testing. The test results showed that the traditional heterogeneous storage method could support concurrent writes of 30,000 data points, and the homogeneous storage method provided by this application could support concurrent writes of 40,000 data points. Therefore, the homogeneous storage method for time-series data provided by the embodiments of this application can support high-concurrency point writes and has good concurrent processing performance.

[0086] In some embodiments, the server device 20 also supports write retries. Specifically, when the database engine 40 fails to write time-series data in response to the write request sent by the server device 20, it can return a write failure message to the server device 20 (corresponding to Figure 4 step 6). The server device 20 can receive the write failure message and, in response to the write failure message, resend the aforementioned write request to the database engine (corresponding to Figure 4 "retry" in step 7). Correspondingly, the database engine 40 can, in response to the resent write request, write the latest time-series data after configuring the target object into the first data table; and write the time-series data of the target object stored in the storage queue as historical time-series data into the second data table.

[0087] Optionally, in addition to supporting write retries, the server device 20 also supports fault tolerance to improve the robustness of data storage (corresponding to Figure 4"Fault tolerance" in step 7). Specifically, when the number of write failure messages received by the server device 20 for the foregoing write request reaches a set number threshold, a write request for writing the configured latest time-series data corresponding to the target object and the time-series data of the target object stored in the storage queue into the third data table of the time-series database may be generated. For the convenience of description and distinction, the foregoing write request for requesting to write the configured latest time-series data corresponding to the target object into the first data table and the time-series data of the target object stored in the storage queue into the second data table is defined as the first write request; the write request for requesting to write the configured latest time-series data corresponding to the target object and the time-series data of the target object stored in the storage queue into the third data table of the time-series database here is defined as the second write request. Among them, the second write request may include: the configured latest time-series data corresponding to the target object, the time-series data of the target object stored in the storage queue, and the identifier of the data table to which these data are to be written (i.e., the identifier of the third data table). Among them, the third data table may be a temporary data table in the time-series database. During the process of creating the third data table, the save policy of the third data table may be set to delete upon expiration, and the expiration duration corresponding to the third data table is less than the expiration duration of the foregoing second data table. Optionally, the expiration duration corresponding to the third data table may be in the order of minutes, hours, or days. The expiration duration corresponding to the second data table may be in the order of months or years, etc.

[0088] The database engine 40 may receive the foregoing second write request and, in response to the second write request, write the configured latest time-series data corresponding to the target object and the time-series data of the target object stored in the storage queue into the third data table, realizing the fault-tolerant storage of time-series data and improving the storage robustness of the time-series database.

[0089] In the embodiment of the present application, the latest time-series data and the historical time-series data are independently stored in the time-series database using different data tables, and the latest time-series data is stored using a set timestamp, so that each target object has only one data point with the set timestamp, which can ensure that the time complexity of querying the latest time-series data is controllable and will not deteriorate as the data volume of the time-series data reported by the target object increases. Compared with the traditional solution of simply using the time-series database to store historical time-series data for querying the latest time-series data, when querying the latest time-series data in the embodiment of the present application, only querying in the first data table is required, and there is no need to traverse all the historical time-series data, which can improve the query efficiency of the latest time-series data.

[0090] In addition to the data storage system provided in the foregoing embodiments, the embodiment of the present application also provides a data storage method. The data storage method provided in the embodiment of the present application will be exemplarily described below.

[0091] Figure 5 It is a schematic flowchart of the data storage method provided in the embodiment of the present application. AsFigure 5 As shown in Figure 5 , the data storage method mainly includes:

[0092] 501. Obtain the latest time-series data of the target object.

[0093] 502. Configure a set timestamp for the latest time-series data to obtain the configured latest time-series data.

[0094] 503. Generate a first write request to write the configured latest time-series data into the first data table of the time-series database and write the latest time-series data as historical time-series data into the second database of the time-series database.

[0095] 504. Send the first write request to the database engine of the time-series database, so that the database engine, in response to the first write request, writes the configured latest time-series data into the first data table and writes the latest time-series data into the second data table; wherein, the storage policy of the data in the first data table is permanent storage; the storage policy of the data in the second data table is expired deletion.

[0096] In order to reduce the resources occupied by time-series data storage, in the embodiments of the present application, it is proposed that the latest time-series data and historical time-series data are stored in a homogeneous time-series database, but the data models of the latest time-series data and historical time-series data are separately modeled. Specifically, the latest time-series data and historical time-series data of the target object are respectively stored in different data tables. For the convenience of description and distinction, the data table storing the latest historical time-series data is defined as the first data table; the data table storing historical time-series data is defined as the second data table. Among them, both the first data table and the second data table can be one or more. Multiple means two or more.

[0097] Since the first data table is used to store the latest time-series data of the target object, the query frequency of the latest time-series data is relatively high, and users often need to understand the recent running status of the target object according to the latest time-series data of the target object. Therefore, the storage policy of the first data table can be configured as permanent storage, which can also be called non-expired. The second data table is used to store the historical time-series data of the target object. The historical time-series data has relatively low reference significance for the recent running status of the target object, or even no reference significance. Therefore, the query frequency of the historical time-series data is relatively low, and even some historical time-series data will not be queried by users after a certain period of time. Therefore, in order to save storage space, the storage policy of the second data table can be configured as expired deletion. In some embodiments, the storage policy of the data table can be configured during the data table creation process. For the implementation manner of configuring the storage policy of the data table, reference can be made to the relevant content of the foregoing system embodiments, which will not be elaborated here.

[0098] The latest time-series data for the target object is stored on the data model with a set timestamp, that is, a set timestamp is configured for the latest time-series data to obtain the configured latest time-series data. The same target object uses the same set timestamp. Different target objects can use the same set timestamp or different set timestamps. In the embodiments of the present application, the format of the set timestamp is different from the timestamp carried by the time-series data.

[0099] The timestamp of the configured latest time-series data is the set timestamp; the latest time-series data reported by the target object is the data (Field) corresponding to the set timestamp. If different target objects use different set timestamps, the corresponding relationship between the identifier of the target object and the set timestamp can be stored in advance. In this way, when configuring the set timestamp for the latest time-series data of the target object, the set timestamp corresponding to the current target object can be obtained from the corresponding relationship between the identifier of the target object and the set timestamp; and the set timestamp corresponding to the target object is configured for the latest time-series data of the current target object, etc.

[0100] Since the configured latest time-series data of the same target object has the same timestamp and tags (such as device name or identifier of the target object, etc.), when the time-series database stores the configured latest time-series data of the target object in the first data table, it will overwrite the original configured latest time-series data of the target object, so that there is only one data point for the latest time-series data of each target object in the first data table.

[0101] For historical time-series data, the data model of the traditional time-series database is used for storage, that is, the time information carried by the time-series data reported by the target object (i.e., the reporting time) is used as the timestamp, the device name in the time-series data reported by the target object (such as the identifier of the target object) is used as the tag, and the attributes and attribute values carried by the time-series data are used as the data (Field) for storing historical time-series data. The historical time-series data stored in the second data table includes: the timestamp is the reporting time carried by the time-series data, the tag is the identifier of the target object, and the data (Field) is the attribute and attribute value, etc.

[0102] Based on the data models of the latest time-series data and historical time-series data shown in the above embodiments, in step 501, the latest time-series data of the target object can be obtained. In some embodiments, the time-series data currently reported by the target object can be received as the latest time-series data of the target object.

[0103] Further, based on the data model of the latest time-series data and historical time-series data shown in the above embodiments, in step 502, a set timestamp can be configured for the latest time-series data of the target object to obtain the configured latest time-series data. Further, in step 503, a write request can be generated to write the configured latest time-series data into the first data table and write the latest time-series data of the target object (the latest time-series data before configuration) into the second data table as historical time-series data. The write request includes: the configured latest time-series data and the identifier of the data table to be written corresponding to the configured latest time-series data (i.e., the identifier of the first data table), and the latest time-series data (i.e., the latest time-series data before configuration) and the identifier of the data table to be written corresponding to the latest time-series data (i.e., the identifier of the second data table).

[0104] Further, in step 504, the write request can be sent to the database engine of the time-series database. Correspondingly, the database engine can respond to the write request, write the configured latest time-series data into the first data table, and write the latest time-series data of the target object into the second data table. For the specific implementation of the database engine writing the configured latest time-series data into the first data table and writing the latest time-series data of the target object into the second data table, reference can be made to the relevant content of the foregoing system embodiments, which will not be elaborated here.

[0105] In this embodiment, on the one hand, the latest time-series data and historical time-series data of the target object are stored using a homogeneous time-series database. Only processor resources and memory resources need to be allocated for the time-series database. Compared with the heterogeneous storage solution, there is no need to maintain two databases with different structures, which can reduce the processor resources and memory resources occupied by the database, that is, save processor and memory resources. On the other hand, the latest time-series data with a high query frequency uses a set timestamp and is separately stored in a data table of the time-series database with a save policy of permanent storage. Compared with the traditional heterogeneous storage method, there is no need to maintain the relationship between data tables in a relational database, which can reduce the storage resources occupied by the latest time-series data.

[0106] Moreover, in this embodiment, the latest time-series data with a high query frequency uses a set timestamp and is separately stored in a data table of the time-series database with a save policy of permanent storage, so that each target object has only one data point in this data table, which can ensure that the data of the target object does not expand.

[0107] In addition, in some traditional solutions, only historical time-series data is stored in a time-series database, and the latest time-series data is not stored separately. When querying the latest time-series data, it is traversed and queried in the historical time-series data. In order to be able to query the latest time-series data, all historical time-series data needs to be permanently saved, that is, the historical time-series data needs to always occupy storage resources, resulting in a relatively large consumption of storage resources. Compared with this method, in this embodiment, the historical time-series data with a lower query frequency is separately stored in a data table with a save policy of expiration deletion, and the expired historical time-series data can be deleted, reducing the storage resources occupied by the historical time-series data.

[0108] Since the latest time-series data and historical time-series data are stored in a homogeneous time-series database, therefore, in the embodiments of the present application, there is only an operation of writing to the time-series database, and there is no operation of reading and writing a relational database, which can reduce the probability of the occurrence of a write bottleneck. Since there is only an operation of writing to the time-series database in the embodiments of the present application, therefore, operations such as data aggregation and asynchronous batch processing can be adopted to achieve high-concurrency point writing. Specifically, the time-series data reported by the target object can be received. The time-series data reported by the target object is the latest time-series data at the current reporting moment. The target object can be one or more, and multiple means two or more.

[0109] In order to reduce the probability of the occurrence of a write bottleneck in the time-series database, in this embodiment, the time-series data reported by the target object can be aggregated. Specifically, a storage queue can be set. This storage queue can be set in the memory of the device that executes the data storage method, and can also be called a memory queue. Correspondingly, the received time-series data can be stored in the storage queue.

[0110] In some embodiments, the data structure of the time-series data reported by the target object may be different from the target data structure supported by the device that executes the data storage method. In order to implement the management of the time-series data by the device that executes the data storage method, before storing the received time-series data reported by the target object in the storage queue, the time-series data reported by the target object can also be converted into the target data structure supported by the server device; and the time-series data with the target data structure is stored in the storage queue.

[0111] It should be noted that in the embodiments of the present application, the time-series data stored in the storage queue and sent to the database engine can both be the time-series data with the target data structure obtained through the above data structure conversion. This will not be emphasized one by one in each embodiment.

[0112] For high-concurrency data points, that is, a large number of target objects reporting time-series data simultaneously at the same moment, after converting the time-series data reported concurrently by multiple target objects into a target data structure supported by the server device, the time-series data reported concurrently by multiple target objects can be stored in a storage queue. Optionally, an asynchronous method can be adopted to store the time-series data reported by multiple target objects in the storage queue; and after storing the time-series data reported by the target object in the storage queue, an asynchronous storage success message can be returned to the target object. In this way, the storage success message can be returned to the target object without waiting for the time-series data to be written into the time-series database, which can reduce the storage delay perceived by the user side of the target object and make the storage process imperceptible to the user.

[0113] The time-series data reported by the target object is aggregated by the server device in the storage queue. Further, when the time-series data of the target object stored in the storage queue meets the set conditions, the time-series data with the latest time information carried can be obtained from the time-series data of the same target object stored in the storage queue as the latest time-series data of the target object.

[0114] In the embodiments of the present application, the specific implementation manner of the set conditions is not limited. In some embodiments, the set conditions may include: the data volume of the time-series data stored in the storage queue reaches the set data volume. Accordingly, when the data volume of the time-series data stored in the storage queue reaches the set data volume, it can be determined that the time-series data of the target object stored in the storage queue meets the set conditions. In other embodiments, the set conditions may include: the time interval of the time-series data stored in the storage queue reaches the set time interval. Accordingly, when the time interval of the time-series data stored in the storage queue reaches the set time interval, it can be determined that the time-series data of the target object 30 stored in the storage queue meets the set conditions. In still other embodiments, the set conditions may include: the data volume of the time-series data stored in the storage queue reaches the set data volume, and the time interval of the time-series data stored in the storage queue reaches the set time interval. Accordingly, when the data volume of the time-series data stored in the storage queue reaches the set data volume and the time interval of the time-series data stored in the storage queue reaches the set time interval, it can be determined that the time-series data of the target object stored in the storage queue meets the set conditions.

[0115] Further, the above step 501 can be implemented as: when the time-series data of the target object stored in the storage queue meets the set conditions, for any target object, the time-series data with the latest time information carried can be obtained from the time-series data of the target object stored in the storage queue as the latest time-series data of the target object. And all the time-series data of the target object can be obtained from the storage queue as the historical time-series data of the target object.

[0116] Further, a set timestamp may be configured for the latest time-series data of the target object to obtain the configured latest time-series data of the target object. In some embodiments, there are multiple target objects, and the same set timestamp may be configured for the latest time-series data of the multiple target objects to obtain the configured latest time-series data corresponding to each of the multiple target objects.

[0117] Alternatively, different set timestamps may be configured for the latest time-series data of the multiple target objects respectively to obtain the configured latest time-series data corresponding to each of the multiple target objects. In this embodiment, the corresponding relationship between the identifier of the target object and the set timestamp may be pre-stored. In this corresponding relationship, the set timestamps corresponding to the target objects with different identifiers are different. Based on the corresponding relationship between the identifier of the target object and the set timestamp, the set timestamps corresponding to each of the multiple target objects may be obtained from this corresponding relationship; and for the latest time-series data of each target object, the set timestamp corresponding to its identifier in the corresponding relationship is configured to obtain the configured latest time-series data of this target object.

[0118] Further, based on the configured latest time-series data of the target object and the historical time-series data (i.e., all the time-series data of the target object stored in the storage queue), a write request is generated. Specifically, a write request may be generated to write the configured latest time-series data of the target object into the first data table and write the time-series data of the target object stored in the storage queue as historical time-series data into the second data table. The time-series data of the target object stored in the storage queue includes: the latest time-series data of each target object, and the generated write request also includes a request to write the latest time-series data of the target object as historical time-series data into the second data table.

[0119] Specifically, the set timestamp corresponding to the target object may be used as the timestamp to be written, the configured latest time-series data of the target object may be used as the data (Field) corresponding to the set timestamp of this target object, and further, the target time information included in the time-series data of the target object stored in the storage queue may be used as the timestamp to be written, and the other data in the time-series data of the target object stored in the storage queue except the target time information may be used as the data (Field) to be written corresponding to the target time information, to generate a write request to write the configured latest time-series data of the target object into the first data table and write the time-series data of the target object stored in the storage queue as historical time-series data into the second data table.

[0120] The write request may include: the configured latest time-series data of the target object and the identifier of the data table to be written corresponding to the configured latest time-series data of the target object (i.e., the identifier of the first data table), and the historical time-series data of the target object (i.e., the time-series data of the target object stored in the storage queue) and the identifier of the data table to be written corresponding to the historical time-series data (the identifier of the second data table).

[0121] Further, the above write request can be sent to the database engine. Since the time-series data of the target object stored in the storage queue by the server device aggregates the time-series data before the time-series data of the target object stored in the storage queue meets the set condition, and when the time-series data of the target object stored in the storage queue meets the set condition, a write request is generated once, that is, an I / O request, so that the aggregated time-series data in the storage queue can be stored in the time-series database together, which can reduce the I / O access frequency to the time-series database.

[0122] Optionally, the write request can be sent to the database engine asynchronously; and after the write request is sent to the database engine, a storage success message is returned to the target object asynchronously. In this way, the storage success message can be returned to the target object without waiting for the time-series data to be written into the time-series database, which can reduce the storage delay perceived by the user side of the target object.

[0123] Correspondingly, for the database engine, the above write request can be received, and in response to the write request, the set timestamp corresponding to the target object is written into the timestamp column of the first data table, and the latest time-series data of the target object is written into the data column corresponding to the timestamp column of the first data table; and, the target time information included in the time-series data of the target object stored in the storage queue is written into the timestamp column of the second data table, and the other data except the target time information in the time-series data of the target object stored in the storage queue is written into the data column corresponding to the timestamp column of the second data table, so as to write the configured time-series data of the target object 30 into the first data table, and write the time-series data of the target object stored in the storage queue as historical time-series data into the second data table.

[0124] In the traditional heterogeneous storage method of storing historical time-series data and the latest time-series data separately using two heterogeneous databases, namely a time-series database and a relational database, for concurrent points, write requests need to be generated for each target object, and a large number of concurrent write requests are sent to the relational database, resulting in a write bottleneck in the relational database. In the embodiment of the present application, data aggregation and asynchronous methods are used to batch store time-series data. For the latest time-series data of multiple concurrent target objects, it can also be batch stored to achieve concurrent point writing; at the same time, the batch storage method can reduce the number of I / O accesses to the time-series database, reduce the probability of the write bottleneck occurring in the time-series database, and even avoid the write bottleneck.

[0125] In some embodiments, write retry is also supported. Specifically, when the database engine fails to write timing data for the write request sent to the aforementioned server device, it can return a write failure message to the device executing the data storage method. The device executing the data storage method can receive the write failure message and, in response to the write failure message, resend the aforementioned write request to the database engine. Accordingly, the database engine can, in response to the resent write request, write the latest timing data after configuring the target object into the first data table; and write the timing data of the target object stored in the storage queue as historical timing data into the second data table.

[0126] Optionally, in addition to supporting write retry, the embodiments of the present application also support fault tolerance to improve the robustness of data storage. Specifically, when the number of received write failure messages for the aforementioned write request reaches a set number threshold, a write request for writing the latest configured timing data corresponding to the target object and the timing data of the target object stored in the storage queue into the third data table of the timing database can be generated. For the sake of easy description and distinction, the aforementioned write request for requesting to write the latest configured timing data corresponding to the target object into the first data table and the timing data of the target object stored in the storage queue into the second data table is defined as the first write request; the write request here for requesting to write the latest configured timing data corresponding to the target object and the timing data of the target object stored in the storage queue into the third data table of the timing database is defined as the second write request. Among them, the second write request may include: the latest configured timing data corresponding to the target object, the timing data of the target object stored in the storage queue, and the identifier of the data table to which these data are to be written (i.e., the identifier of the third data table). Among them, the third data table can be a temporary data table in the timing database. During the process of creating the third data table, the save policy of the third data table can be set to expire and be deleted, and the expiration duration corresponding to the third data table is less than the expiration duration of the aforementioned second data table.

[0127] The database engine can receive the above second write request and, in response to the second write request, write the latest configured timing data corresponding to the target object and the timing data of the target object stored in the storage queue into the third data table, realizing fault-tolerant storage of timing data and improving the storage robustness of the timing database.

[0128] In the embodiments of the present application, the latest time-series data and historical time-series data are independently stored in a time-series database using different data tables, and the latest time-series data is stored using a set timestamp, so that each target object has only one data point with the set timestamp, which can ensure that the time complexity of querying the latest time-series data is controllable and will not deteriorate as the amount of time-series data reported by the target object increases. Compared with the conventional solution of querying the latest time-series data by simply storing historical time-series data in a time-series database, when querying the latest time-series data in the embodiments of the present application, it is only necessary to query in the first data table, and there is no need to traverse all historical time-series data, which can improve the query efficiency of the latest time-series data.

[0129] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subjects of steps 501 and 502 can be device A; for another example, the execution subject of step 501 can be device A, and the execution subject of step 502 can be device B; and so on.

[0130] In addition, in some of the processes described in the above embodiments and the accompanying drawings, a plurality of operations appear in a specific order, but it should be clearly understood that these operations can be executed not in the order in which they appear in this document or in parallel. The operation numbers such as 501 and 502 are only used to distinguish different operations, and the numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and these operations can be executed in sequence or in parallel.

[0131] Correspondingly, the embodiments of the present application further provide a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, cause the one or more processors to execute the steps in the above data storage method.

[0132] Figure 6 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 6 shown, the electronic device includes: a memory 60a, a processor 60b, and a communication component 60c. Among them, the memory 60a is used to store computer programs.

[0133] The processor 60b is coupled to the memory 15a and the communication component 60c, and is used to execute a computer program to execute the steps in the data storage methods provided in the foregoing embodiments. For the specific implementation manners of the respective steps, reference may be made to the relevant descriptions in the foregoing embodiments, and details are not described herein again.

[0134] In some alternative embodiments, as Figure 6As shown, the electronic device may further include optional components such as a power supply component 60d, a display component 60e, and an audio component 60f. Figure 6 Only some components are schematically shown, which does not mean that the electronic device must include Figure 6 all the components shown, nor does it mean that the electronic device can only include Figure 6 the components shown.

[0135] In addition, Figure 6 the components within the dashed box are optional components, not mandatory components, and can be determined according to the product form of the electronic device. The computing device in this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a mobile phone, or an Internet of Things device; it can also be various server devices such as a traditional server, a cloud server, or a server cluster.

[0136] In the embodiment of the present application, the memory is used to store computer programs and can be configured to store various other data to support operations on the device where it is located. Among them, the processor can execute the computer programs stored in the memory to implement corresponding control logic. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), electrically programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0137] In the embodiments of the present application, the processor may be any hardware processing device capable of executing the above method logic. Optionally, the processor may be a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), or a Microcontroller Unit (MCU); it may also be a programmable device such as a Field-Programmable Gate Array (FPGA), a Programmable Array Logic (PAL), a General Array Logic (GAL), or a Complex Programmable Logic Device (CPLD); or an Advanced RISC Machines (ARM) processor or a System on Chip (SoC), etc., but not limited thereto.

[0138] In the embodiments of the present application, the communication component is configured to facilitate communication between the device where it is located and other devices in a wired or wireless manner. The device where the communication component is located may access a wireless network based on a communication standard, such as Wireless Fidelity (WiFi), 2G or 3G, 4G, 5G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component may also be implemented based on Near Field Communication (NFC) technology, Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology, or other technologies.

[0139] In an embodiment of the present application, the display component may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the display component includes a touch panel, the display component can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operations.

[0140] In an embodiment of the present application, the power supply component is configured to provide power to various components of the device where it is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device where the power supply component is located.

[0141] In an embodiment of the present application, the audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC). When the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive external audio signals. The received audio signals can be further stored in the memory or sent via the communication component. In some embodiments, the audio component further includes a speaker for outputting audio signals. For example, for a device with a language interaction function, voice interaction with the user can be implemented through the audio component.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data that have been authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0143] It should also be noted that the descriptions such as "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, nor do they limit that "first" and "second" are of different types.

[0144] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, compact disc read-only memory (CD-ROM), optical memory, etc.) that contain computer-usable program code.

[0145] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices produce means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0146] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0148] In a typical configuration, a computing device includes one or more processors (such as CPUs), an input / output interface, a network interface, and a memory.

[0149] Memory may include non-permanent memory in the form of computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0150] The storage medium of a computer is a readable storage medium, also known as a readable medium. Readable storage media include permanent and non-permanent, removable and non-removable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of the storage medium of a computer include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassette tapes, disk storage, or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0151] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the above elements.

[0152] The above content is only an embodiment of the present application and is not used to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A data storage method, It is characterized in that include: Get the latest time series data of the target object; Setting a timestamp for the latest time series data configuration to obtain the latest time series data after configuration; Generate a first write request to write the configured latest time series data into a first data table of the time series database and to write the latest time series data as historical time series data into a second database of the time series database; Sending the first write request to a database engine of the time series database, so that the database engine writes the configured latest time series data into the first data table and writes the latest time series data into the second data table in response to the first write request; The data storage policy of the first data table is permanent storage; the data storage policy of the second data table is expiration deletion.

2. The method according to claim 1, It is characterized in that The target object is at least one; the method further comprises: Before acquiring the latest time series data of the target object, receiving time series data reported by at least one target object, and storing the time series data reported by the at least one target object in a storage queue; The obtaining of the latest time series data of the target object includes: When the time series data of at least one target object stored in the storage queue meets a set condition, the time series data with the latest time information is obtained from the time series data of the same target object stored in the storage queue as the latest time series data of the same target object; The method further includes: acquiring the time series data of the same target object from the storage queue as the historical time series data of the same data object.

3. The method according to claim 2, It is characterized in that Also includes: If the data volume of the time series data of the at least one target object stored in the storage queue reaches the set data volume, and / or the time interval of the time series data of the at least one target object stored in the storage queue reaches the set time interval, it is determined that the time series data of the at least one target object stored in the storage queue meets the set conditions.

4. The method according to claim 2, It is characterized in that The target objects are multiple; the timestamp is configured for the latest time series data to obtain the latest data after configuration, including: Configuring the same set timestamp for the latest time series data of multiple target objects to obtain the configured latest time series data corresponding to each of the multiple target objects; or, Different set time stamps are respectively configured for the latest time series data of the multiple target objects to obtain the latest time series data corresponding to each of the multiple target objects.

5. The method according to claim 2, It is characterized in that The generating of a first write request to write the configured latest time series data into a first data table of the time series database and to write the latest time series data as historical time series data into a second database of the time series database includes: Generate a first write request to write the latest time series data of the at least one target object after configuration into the first data table, and write the time series data of the at least one target object stored in the storage queue as historical time series data into the second data table; The time series data of the at least one target object stored in the storage queue includes the latest time series data of the at least one target object.

6. The method according to claim 5, It is characterized in that The generating of a first write request to write the latest time series data of the at least one target object after configuration into the first data table, and to write the time series data of the at least one target object stored in the storage queue into the second data table as historical time series data, comprises: A first write request is generated to write the latest configured time series data corresponding to the at least one target object into the first data table, and to write the time series data of the at least one target object stored in the storage queue as the historical time series data into the second data table, by using the set timestamp corresponding to the at least one target object and the latest time series data after configuration corresponding to the at least one target object as the data to be written corresponding to the set timestamp corresponding to the at least one target object, and by using the target time information contained in the time series data of the at least one target object stored in the storage queue as the timestamp to be written and other data except the target time information in the time series data of the at least one target object stored in the storage queue as the data to be written corresponding to the target time information.

7. The method according to any one of claims 2 to 6, It is characterized in that The storing the time series data reported by the at least one target object into a storage queue includes: Converting the time series data reported by the at least one target object into a target data structure supported by the server device; The time series data having the target data structure is stored in the storage queue.

8. The method according to claim 5 or 6, It is characterized in that The storing the time series data reported by the at least one target object into a storage queue includes: For any target object among the at least one target object, storing the received time series data reported by the any target object in a storage queue in an asynchronous manner; The method further comprises: after storing the time series data reported by any target object in a storage queue, asynchronously returning a storage success message to any target object; or, The sending the first write request to the database engine comprises: asynchronously sending the first write request to the database engine; The method further includes: after sending the first write request to the database engine, asynchronously returning a storage success message to any target object.

9. The method according to claim 8, It is characterized in that Also includes: receiving a write failure message returned by the database engine in response to the first write request; In response to the write failure message, the first write request is resent to the database engine.

10. The method according to claim 9, It is characterized in that Also includes: When the number of times the write failure message is received reaches a set number threshold, a second write request is generated to write the latest configured time series data corresponding to the at least one target object and the time series data of the at least one target object stored in the storage queue into the third data table of the time series database; The second write request is sent to the database engine, so that the database engine responds to the second write request and writes the latest configured time series data corresponding to the at least one target object and the time series data of the at least one target object stored in the storage queue into the third data table.

11. The method according to claim 1, It is characterized in that Also includes: Obtaining object model definition information defined for the target object; The object model definition information is provided to the target object so that the target object generates the time series data of the target object according to the object model definition information.

12. A data storage system, It is characterized in that include: Database engine for server-side devices and time series databases; The server device is used to execute the steps in the method according to any one of claims 1 to 11; The database engine is used to respond to the first write request sent by the server device, write the configured latest time series data into the first data table, and write the latest time series data into the second data table; wherein the storage policy of the data in the first data table is permanent storage; and the storage policy of the data in the second data table is expiration deletion.

13. An electronic device, It is characterized in that include: Memory, processor and communication components; wherein the memory is used to store computer programs; The processor is coupled to the memory and the communication component, and is configured to execute the computer program to perform the steps in the method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing computer instructions, It is characterized in that When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the method according to any one of claims 1 to 11.

Images