Time series data storage method and system, electronic equipment and storage medium

By acquiring metadata information of time-series data, determining storage strategies and parallel storage instances, the performance bottleneck of writing massive amounts of time-series data with high concurrency is solved, achieving efficient, real-time, and reliable time-series data storage, and adapting to storage needs of different sources and types.

CN121233644APending Publication Date: 2025-12-30CISDI INFORMATION TECH CO LTD

Patent Information

Application Number
CN202511610359.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-11-05
Publication Date
2025-12-30

AI Technical Summary

Technical Problem

Existing real-time data storage solutions, when processing massive amounts of time-series data, lead to frequent thread contention and context switching, resulting in high CPU utilization and memory overhead, making it difficult to meet the real-time requirements of millisecond-level response in industrial scenarios.

Method used

By acquiring metadata information of time-series data, including location identifiers, data sources, and types, a storage strategy is determined. A dynamic storage strategy is adopted, and time-series data is stored in the time-series database through target storage instances. By using a metadata-driven dynamic storage strategy and parallel storage instances, thread contention and lock contention are avoided, achieving high throughput, strong consistency, low latency, and traceability storage.

Benefits of technology

It effectively reduces system resource consumption, improves storage efficiency, ensures the real-time performance and reliability of time-series data storage, adapts to storage needs of different sources and types, and completes high-throughput, strong consistency, low latency, and traceable time-series data storage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121233644A_ABST
    Figure CN121233644A_ABST
Patent Text Reader

Abstract

The invention provides a time sequence data storage method and system, electronic equipment and a storage medium, and the method comprises the steps: obtaining time sequence data and metadata information of the time sequence data, the metadata information comprising a first point location identifier, a data source and a data type corresponding to the time sequence data; according to the data source and the data type, determining a storage strategy of the time series data, including a target storage period and a target storage position; according to the storage strategy and the first point position identification, a target storage instance corresponding to the time sequence data is matched from a storage instance set, all storage instances in the storage instance set are combined and constructed according to different storage strategies and the point position identification, and all the storage instances are parallel storage instances; and storing the time sequence data to a time sequence database through the target storage instance. By determining a dynamic storage strategy driven by metadata and a parallel storage instance, the problem of performance bottleneck during high-concurrency writing of massive time series data is solved, and the real-time performance and reliability of time series data storage are ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a time-series data storage method, system, electronic device, and storage medium. Background Technology

[0002] With the profound transformation of modern industry towards intelligence and digitalization, data acquisition and processing capabilities in intelligent manufacturing environments have become core elements for improving production efficiency, ensuring product quality, and achieving predictive maintenance. In industrial settings, tens of thousands of sensors and intelligent terminal devices are widely deployed to monitor equipment operating status, environmental parameters, and production processes in real time. These devices continuously collect data at high frequency, resulting in the system processing tens of millions or even hundreds of millions of data points per second, forming massive time-series data streams. This data not only presents real-time requirements such as "high-concurrency writes, millisecond-level queries, and second-level analysis," but also carries industrial-grade reliability requirements of "strong consistency, zero data loss, and traceability."

[0003] In related technologies, common real-time data storage solutions often adopt a multi-threaded concurrent processing architecture, which manages a large number of concurrent requests through thread pools. To ensure data consistency and thread safety, synchronization mechanisms (such as mutexes, read-write locks, atomic operations, etc.) are introduced. However, when dealing with massive amounts of time-series data, this approach leads to frequent thread contention and context switching, which not only causes extremely high CPU (Central Processing Unit) utilization and memory overhead, severely consuming system resources, but also results in data write latency, making it difficult to meet the millisecond-level response real-time requirements in industrial scenarios. Summary of the Invention

[0004] This application discloses a time-series data storage method, system, electronic device, and storage medium to solve the technical problems of low efficiency and performance in time-series data storage.

[0005] This application provides a time-series data storage method, the method comprising: acquiring time-series data and metadata information of the time-series data, wherein the metadata information includes a first location identifier corresponding to the time-series data, a data source, and a data type; determining a storage strategy for the time-series data based on the data source and the data type, wherein the storage strategy includes a target storage period and a target storage location of the time-series data in a time-series database; matching a target storage instance corresponding to the time-series data from a pre-constructed storage instance set based on the storage strategy and the first location identifier, wherein each storage instance in the storage instance set is constructed according to different combinations of storage strategies and location identifiers, and each storage instance is a parallel storage instance; and storing the time-series data into the time-series database through the target storage instance.

[0006] In one embodiment of this application, determining the storage strategy for the time-series data based on the data source and the data type includes: determining the target storage period based on the data source, wherein the data source represents the entity that generates the time-series data, and the entity that generates the data includes a data acquisition device, a data processing device, and a data application device; and determining the target storage location based on a mapping relationship between the data source, the data type, the storage period, and a preset storage location, wherein the mapping relationship between the storage location and different combinations of data sources, data types, and storage periods.

[0007] In one embodiment of this application, determining the target storage period based on the data source includes: detecting whether a user-set storage period exists in the metadata information; if it exists, determining the user-set storage period as the target storage period; if it does not exist, determining the generating entity of the time-series data based on the data source; if the generating entity is the data acquisition device, determining the target storage period based on the real-time acquisition frequency of the time-series data; if the generating entity is the data processing device, determining the target storage period based on the processing window length of the time-series data and a preset storage period mapping relationship, wherein the storage period mapping relationship characterizes the correspondence between different processing window lengths and storage periods; if the generating entity is the data application device, determining a preset first storage period as the target storage period.

[0008] In one embodiment of this application, before determining the storage strategy of the time-series data based on the data source and the data type, the method further includes: obtaining audit information of the time-series data, wherein the audit information includes the data generation time and data correction information corresponding to the time-series data, and the data correction information includes the original data value, the data corrector, the data correction time, and the data correction method; converting the time-series data into a preset data format, and binding the audit information to the format-converted time-series data.

[0009] In one embodiment of this application, storing the time-series data into a time-series database through the target storage instance includes: detecting whether a first write task is currently in a write state; if not, creating a second write task through the target storage instance to write the time-series data into the time-series database; if so, writing the time-series data into a cache through the target storage instance, and when the cache time length reaches a preset time length and / or the cache data length reaches a preset data length, obtaining the cached data in the cache, blocking the first write task, and creating a third write task to write the cached data into the time-series database.

[0010] In one embodiment of this application, determining the target storage period based on the real-time acquisition frequency of the time-series data includes: comparing the real-time acquisition frequency with a preset frequency threshold; if the real-time acquisition frequency is less than the preset frequency threshold, then determining a preset second storage period as the target storage period, wherein the second storage period is less than the first storage period; if the real-time acquisition frequency is greater than or equal to the preset frequency threshold, then determining the first storage period as the target storage period.

[0011] In one embodiment of this application, the data composition of the time series data includes a timestamp, an indicator value, and a tag, wherein the tag includes a second location identifier corresponding to the time series data; the method of obtaining the metadata information includes: extracting the second location identifier from the tag of the time series data; matching the second location identifier with the first location identifier in different metadata information to obtain the metadata information corresponding to the time series data.

[0012] This application also provides a time-series data storage system, the system comprising: a data acquisition module, configured to acquire time-series data and metadata information of the time-series data, wherein the metadata information includes a first location identifier corresponding to the time-series data, a data source, and a data type; a strategy determination module, configured to determine a storage strategy for the time-series data based on the data source and the data type, wherein the storage strategy includes a target storage period and a target storage location for the time-series data in a time-series database; an instance matching module, configured to match a target storage instance corresponding to the time-series data from a pre-constructed storage instance set based on the storage strategy and the first location identifier, wherein each storage instance in the storage instance set is constructed according to different combinations of storage strategies and location identifiers, and each storage instance is a parallel storage instance; and a data storage module, configured to store the time-series data into the time-series database through the target storage instance.

[0013] This application also provides an electronic device, including: a processor; and a storage device for storing a program, which, when executed by the processor, causes the electronic device to implement the timing data storage method described above.

[0014] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer's processor, causes the computer to perform the timing data storage method as described above.

[0015] The beneficial effects of this application are as follows: This application provides a time-series data storage method, system, electronic device, and storage medium. First, it acquires the time-series data and its metadata information, including the first point identifier, data source, and data type. Then, based on the data source and data type, it determines the storage strategy for the time-series data, including the target storage period and target storage location in the time-series database. Next, based on the storage strategy and the first point identifier, it matches the corresponding target storage instance from a pre-constructed storage instance set. Each storage instance in this set is constructed according to different storage strategies and point identifier combinations, and each storage instance is a parallel storage instance. Finally, the time-series data is stored in the time-series database through the target storage instance. By using a metadata-driven dynamic storage strategy and determining parallel storage instances, it solves the performance bottleneck problem of high-concurrency writing of massive amounts of time-series data, avoids thread contention and lock contention, effectively reduces system resource consumption, and allows for flexible storage strategy configuration to adapt to the storage needs of time-series data from different sources and of different types. This achieves high-throughput, strong consistency, low latency, and traceable time-series data storage, improving storage efficiency and ensuring the real-time performance and reliability of time-series data storage. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0017] In the attached diagram: Figure 1 This is a schematic diagram illustrating an implementation environment of a time-series data storage system, as shown in an exemplary embodiment of this application. Figure 2 This is a flowchart illustrating a time-series data storage method in an exemplary embodiment of this application; Figure 3 This is a flowchart illustrating a method for determining a storage strategy, as shown in an exemplary embodiment of this application; Figure 4 This is a block diagram illustrating a time-series data storage system as shown in an exemplary embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0018] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0019] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the shape, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0020] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0021] With the profound transformation of modern industry towards intelligence and digitalization, data acquisition and processing capabilities in intelligent manufacturing environments have become core elements for improving production efficiency, ensuring product quality, and achieving predictive maintenance. In industrial settings, tens of thousands of sensors and intelligent terminal devices are widely deployed to monitor equipment operating status, environmental parameters, and production processes in real time. These devices continuously collect data at high frequency, resulting in the system processing tens of millions or even hundreds of millions of data points per second, forming massive time-series data streams. This data not only presents real-time requirements such as "high-concurrency writes, millisecond-level queries, and second-level analysis," but also carries industrial-grade reliability requirements of "strong consistency, zero data loss, and traceability."

[0022] Current real-time data storage solutions mostly adopt a multi-threaded concurrent processing architecture, which manages a large number of concurrent requests through thread pools. To ensure data consistency and thread safety, synchronization mechanisms (such as mutexes, read-write locks, atomic operations, etc.) are introduced. However, the inventors of this application have found that this approach leads to frequent thread contention and context switching when dealing with massive amounts of time-series data. This not only causes extremely high CPU utilization and memory overhead, severely consuming system resources, but also results in data write latency, making it difficult to meet the real-time requirements of millisecond-level response in industrial scenarios.

[0023] Therefore, please see Figure 1 , Figure 1This is a schematic diagram illustrating an implementation environment of a time-series data storage system, as shown in an exemplary embodiment of this application. Figure 1 As shown, the implementation environment may include a time-series data storage system 110 and a computer device 120. The time-series data storage system 110 can be located within the computer device 120 for implementing time-series data storage. The computer device 120 can be at least one of a desktop graphics processing unit (GPU) computer, a GPU computing cluster, or a neural network computer. This time-series data storage system 110, through a metadata-driven dynamic storage strategy and the determination of parallel storage instances, solves the performance bottleneck problem of high-concurrency writing of massive amounts of time-series data, avoids thread contention and lock contention, effectively reduces system resource consumption, and its flexible storage strategy configuration can adapt to the storage needs of time-series data from different sources and of different types, achieving high-throughput, strong consistency, low latency, and traceable time-series data storage, improving storage efficiency, and ensuring the real-time performance and reliability of time-series data storage.

[0024] Please see Figure 2 , Figure 2 This is a flowchart illustrating a time-series data storage method in an exemplary embodiment of this application. This method can be applied to... Figure 1 The implementation environment shown is specifically executed by the time-series data storage system 110 within that implementation environment. It should be understood that this method can also be applied to other exemplary implementation environments and executed by devices in other implementation environments; this embodiment does not limit the implementation environment to which the method is applicable.

[0025] like Figure 2 As shown, in an exemplary embodiment, the time-series data storage method includes at least steps S210 to S240, which are described in detail below: Step S210: Obtain time series data and metadata information of time series data, wherein the metadata information includes the first point identifier corresponding to the time series data, the data source and the data type.

[0026] The first point identifier is used to uniquely identify the data, including but not limited to equipment number, sensor number or other unique identifier; the data source includes but is not limited to data acquisition equipment, data processing equipment or data application equipment, and the data type can be physical quantities such as temperature, pressure, vibration or intermediate data that has been processed, used to describe the data value type of the index data in the time series data.

[0027] In one embodiment, the data components of the time series data include timestamps, indicator values, and tags, and the tags include a second location identifier corresponding to the time series data; the method of obtaining metadata information includes: extracting the second location identifier from the tags of the time series data; matching the second location identifier with a first location identifier in different metadata information to obtain the metadata information corresponding to the time series data.

[0028] The time series data consists of, but is not limited to, three parts: timestamp data, indicator data, and tag data. Timestamp data is a set of ordered time-series data. Indicator data refers to the specific values ​​or states measured or recorded at a specific point in time. Tag data contains the second point identifier corresponding to the time series data, which is used to uniquely identify the data, including but not limited to device number, sensor number, or other unique identifiers.

[0029] For example, the second point identifier is matched with the first point identifier in different metadata information to obtain the metadata information corresponding to the time series data. That is, each first point identifier is traversed, and the metadata information corresponding to the first point identifier that is the same as the second point identifier is determined as the metadata information of the time series data.

[0030] In this embodiment, by embedding the location identifier into the tags of the time-series data, the acquisition of metadata information can be achieved by directly extracting key identifiers from the data ontology and completing the matching. By pre-embedding the second location identifier in the tag, the metadata acquisition process is simplified to two steps: first, parsing the tag structure to extract the identifier, and then performing a matching query in the local metadata cache. This reduces the network overhead caused by obtaining metadata information by accessing external metadata services. Especially in high-concurrency write scenarios, it effectively reduces system resource pressure and response latency, and achieves efficient acquisition of metadata.

[0031] Step S220: Determine the storage strategy for time-series data based on the data source and data type. The storage strategy includes the target storage period and target storage location of the time-series data in the time-series database.

[0032] The target storage period refers to the longest time that time-series data is retained and available for querying at the target storage location in the time-series database; the target storage location refers to the specific physical or logical destination where time-series data is written and stored in the time-series database, which can be different partitions or tablespaces in the time-series database.

[0033] In one embodiment, determining a storage strategy for time-series data based on the data source and data type includes: determining a target storage period based on the data source, wherein the data source represents the entity that generates the time-series data, and the generating entity includes data acquisition equipment, data processing equipment, and data application equipment; and determining a target storage location based on a mapping relationship between the data source, data type, storage period, and a preset storage location, wherein the storage location mapping relationship represents the correspondence between different combinations of data sources, data types, and storage periods and storage locations.

[0034] The storage location mapping can be implemented based on a hash algorithm, using the data source, data type, and storage period as input keys to output the corresponding physical storage path. Furthermore, the storage location mapping can be designed in layers: the first layer divides storage areas according to data source, the second layer creates subdirectories according to data type, and the third layer sets partitioning strategies based on storage period.

[0035] In this embodiment, the technical solution addresses the problem of unreasonable storage resource allocation caused by the complex sources of massive time-series data by establishing a correlation mechanism between data sources and storage strategies. The working principle is as follows: first, the basic storage cycle is automatically matched based on the device type that generated the entity; then, the physical storage location is determined by combining multi-dimensional attribute mapping, thereby achieving fine-grained scheduling of storage resources. Compared with existing technologies, this avoids resource waste caused by a globally unified storage strategy, reduces the complexity of storage location calculation through a hierarchical mapping mechanism, and simultaneously ensures the access efficiency of different types of data.

[0036] Further, determining the target storage period based on the data source includes: detecting whether a user-defined storage period exists in the metadata information; if it exists, the user-defined storage period is determined as the target storage period; if it does not exist, the entity that generates the time-series data is determined based on the data source; if the generating entity is a data acquisition device, the target storage period is determined based on the real-time acquisition frequency of the time-series data; if the generating entity is a data processing device, the target storage period is determined based on the mapping relationship between the processing window length of the time-series data and the preset storage period, wherein the storage period mapping relationship represents the correspondence between different processing window lengths and storage periods; if the generating entity is a data application device, the preset first storage period is determined as the target storage period.

[0037] The processing window length includes, but is not limited to, the time length and the data volume length, such as how many seconds or how many bytes.

[0038] It should be noted that the metadata information also includes a storage period, which includes a preset first storage period and a preset second storage period. The first storage period is a long storage period, and the second storage period is a short storage period. The specific duration of the first and second storage periods can be set according to specific needs or circumstances. In addition, the storage period can also include a user-defined storage period. Of course, this will only exist if the user has set a storage period; otherwise, it will not exist.

[0039] For example, the correlation between real-time acquisition frequency and storage period can be controlled hierarchically through frequency thresholds.

[0040] In this embodiment, a three-level decision-making mechanism is used to achieve precise matching of storage cycles. First, user-defined configurations are used first. Second, calculation logic is selected based on data type. Finally, dynamic decision-making is achieved through quantitative indicators. This mechanism can dynamically balance data real-time performance and storage costs to establish the optimal storage configuration, taking into account the characteristics of different data sources in industrial scenarios.

[0041] In one possible embodiment, the metadata information also includes a fixed acquisition frequency, used to characterize the acquisition time interval of the data processing device.

[0042] Specifically, determining the target storage period based on the real-time acquisition frequency of time-series data includes: comparing the real-time acquisition frequency with a preset frequency threshold; if the real-time acquisition frequency is less than the preset frequency threshold, then determining the preset second storage period as the target storage period, wherein the second storage period is less than the first storage period; if the real-time acquisition frequency is greater than or equal to the preset frequency threshold, then determining the first storage period as the target storage period.

[0043] The preset frequency threshold can be configured according to the actual application scenario, and the calculation of the real-time acquisition frequency can be achieved by statistically analyzing the amount of data received per unit time.

[0044] In this embodiment, by dynamically matching the acquisition frequency and storage cycle, the problem of excessive consumption of storage resources in high-frequency acquisition scenarios is solved. While ensuring data timeliness, the utilization rate of storage resources is optimized, and more refined storage management is achieved according to different acquisition frequencies.

[0045] Please see Figure 3 , Figure 3 This is a flowchart illustrating a method for determining a storage strategy, as shown in an exemplary embodiment of this application. Figure 3As shown, the steps for determining the storage strategy are as follows: First, if the generating entity is a data acquisition device, the target storage period is determined based on the acquisition frequency of the time-series data. That is, if the acquisition frequency is a continuous high-frequency acquisition state, it is set to long-period storage; otherwise, it is set to short-period storage. If the generating entity is a data processing device, the target storage period is determined based on the window length of the processed data. If the generating entity is a data application device, it is set to long-period storage. Then, if it is detected that the user has set a storage period, the user setting is used. Finally, the target storage location is determined from the storage location mapping relationship based on the data source, data type, and storage period, thereby completing the determination of the storage strategy.

[0046] In one embodiment, before determining the storage strategy for time-series data based on the data source and data type, the method further includes: obtaining audit information for the time-series data, wherein the audit information includes the data generation time and data correction information corresponding to the time-series data, and the data correction information includes the original data value, the person who corrected the data, the data correction time, and the data correction method; converting the time-series data into a preset data format, and binding the audit information to the converted time-series data.

[0047] Among them, the data generation time represents the time when the time-series data was generated; the data correction information is used to record the data modification traces; the original data value is the value before the data was modified; the data corrector represents the user or system identifier that performed the modification operation; the data correction time represents the time point when the modification occurred; the data correction method represents the type of modification operation, such as manual correction, automatic calibration, or system completion; the preset data format can be a structured data format such as JSON (a lightweight data exchange format), Protocol Buffers (a binary serialization protocol), or Avro (an open source data serialization system).

[0048] For example, when binding audit information, an embedded storage method can be used to attach the audit information as a metadata field to the time series data, or an external storage method can be used to establish the association between the time series data and the audit information through a unique identifier.

[0049] In this embodiment, by introducing an audit information acquisition and binding mechanism, the generation, flow and change history of data can be fully preserved during subsequent storage, thus solving the problem of insufficient traceability of time-series data in industrial scenarios.

[0050] Step S230: Based on the storage strategy and the first location identifier, match the target storage instance corresponding to the time series data from the pre-built storage instance set. Each storage instance in the storage instance set is constructed according to different combinations of storage strategies and location identifiers, and each storage instance is a parallel storage instance.

[0051] In this embodiment, a parallel processing model is pre-adopted to create at least one storage instance under different storage strategies and location identifier combinations to obtain a storage instance set. The parallel processing model includes, but is not limited to, the Actor model (a parallel computing model), and the storage instances include, but are not limited to, Actor storage instances. Each storage instance corresponds to a specific storage configuration and write channel, so that the Actor storage instance corresponding to the time series data can be uniquely determined according to the corresponding storage strategy and location identifier to ensure the smooth writing of data.

[0052] Step S240: Store the time series data into the time series database through the target storage instance.

[0053] The target storage instance performs micro-batch processing on the time-series data and stores it in the time-series database.

[0054] Specifically, storing time-series data into a time-series database through a target storage instance includes: detecting whether a first write task is currently in a write state; if not, creating a second write task through the target storage instance to write the time-series data into the time-series database; if so, writing the time-series data into a cache through the target storage instance, and when the cache time length reaches a preset time length and / or the cache data length reaches a preset data length, retrieving the cached data in the cache, blocking the first write task, and creating a third write task to write the cached data into the time-series database.

[0055] Specifically, detecting whether there is a first write task in the write state can be achieved by querying the task queue or the write status identifier; caching can be done using a memory buffer or a distributed message queue; the preset time length and preset data length can be set according to specific needs or circumstances.

[0056] In this embodiment, by dynamically detecting the write status and introducing a caching mechanism, the resource contention problem in high-concurrency write scenarios is effectively solved. When there are active write tasks in the system, new data is temporarily stored in the cache to avoid direct competition for database resources. When the cache reaches the trigger condition, batch writing is achieved by blocking existing tasks and creating new tasks, which significantly reduces the consumption of CPU and memory resources. At the same time, the data throughput is improved by batch processing through caching, and the real-time requirements are met in industrial-grade massive time-series data scenarios.

[0057] The aforementioned time-series data storage method first acquires the time-series data and its metadata, including the first point identifier, data source, and data type. Next, based on the data source and data type, it determines the storage strategy for the time-series data, including the target storage period and target storage location in the time-series database. Then, based on the storage strategy and the first point identifier, it matches the corresponding target storage instance from a pre-built storage instance set. Each storage instance in this set is constructed according to different storage strategies and point identifier combinations, and each storage instance is a parallel storage instance. Finally, the time-series data is stored in the time-series database through the target storage instance. By using a metadata-driven dynamic storage strategy and determining parallel storage instances, it solves the performance bottleneck problem of high-concurrency writing of massive amounts of time-series data, avoids thread contention and lock contention, effectively reduces system resource consumption, and allows flexible storage strategy configuration to adapt to the storage needs of time-series data from different sources and of different types. This achieves high-throughput, strong consistency, low latency, and traceable time-series data storage, improving storage efficiency and ensuring the real-time performance and reliability of time-series data storage.

[0058] Please see Figure 4 , Figure 4 This is a block diagram illustrating a time-series data storage system as an exemplary embodiment of this application. The system can be applied to... Figure 1 The implementation environment shown is intended to illustrate the system, but it should be understood that the system can also be applied to other exemplary implementation environments. This embodiment does not limit the implementation environment to which the system is applicable.

[0059] like Figure 4 As shown, in an exemplary embodiment, the time-series data storage system 400 includes at least a data acquisition module 410, a strategy determination module 420, an instance matching module 430, and a data storage module 440, which are described in detail below: The data acquisition module 410 is used to acquire time series data and metadata information of the time series data. The metadata information includes the first point identifier corresponding to the time series data, the data source and the data type. The strategy determination module 420 is used to determine the storage strategy of time series data based on the data source and data type. The storage strategy includes the target storage period and target storage location of the time series data in the time series database. The instance matching module 430 is used to match the target storage instance corresponding to the time series data from a pre-built storage instance set according to the storage strategy and the first point identifier. The storage instances in the storage instance set are constructed according to different combinations of storage strategies and point identifiers, and each storage instance is a parallel storage instance. Data storage module 440 is used to store time-series data to a time-series database through a target storage instance.

[0060] It should be noted that the time-series data storage system provided in the above embodiments and the time-series data storage method provided in the above embodiments belong to the same concept. The content of the operation performed by each module has been described in detail in the method embodiments, and will not be repeated here.

[0061] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Figure 5 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 5 The computer system 500 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0062] like Figure 5 As shown, the computer system 500 includes a Central Processing Unit (CPU) 501, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 502 or programs loaded from storage portion 508 into Random Access Memory (RAM) 503. The RAM 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An Input / Output (I / O) interface 505 is also connected to the bus 504.

[0063] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 510 as needed so that computer programs read from them can be installed into storage section 508 as needed.

[0064] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs various functions defined in the system of this application.

[0065] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the timing data storage method described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not deployed within that electronic device.

[0066] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0067] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A time-series data storage method, characterized in that, The method includes: Obtain time-series data and metadata information of the time-series data, wherein the metadata information includes the first point identifier, data source, and data type corresponding to the time-series data; Based on the data source and the data type, a storage strategy for the time-series data is determined, wherein the storage strategy includes the target storage period and target storage location of the time-series data in the time-series database; According to the storage strategy and the first location identifier, the target storage instance corresponding to the time series data is matched from the pre-constructed storage instance set, wherein each storage instance in the storage instance set is constructed according to different combinations of storage strategies and location identifiers, and each storage instance is a parallel storage instance. The time-series data is stored in the time-series database through the target storage instance.

2. The time-series data storage method according to claim 1, characterized in that, The step of determining the storage strategy for the time-series data based on the data source and the data type includes: The target storage period is determined based on the data source, wherein the data source represents the entity that generates the time-series data, and the entity that generates the data includes data acquisition equipment, data processing equipment, and data application equipment; The target storage location is determined based on the data source, the data type, the storage period, and the preset storage location mapping relationship, wherein the storage location mapping relationship represents the correspondence between different combinations of data sources, data types, and storage periods and the storage location.

3. The time-series data storage method according to claim 2, characterized in that, Determining the target storage period based on the data source includes: Detect whether the metadata information contains a user-defined storage period; If it exists, the storage period set by the user is determined as the target storage period; if it does not exist, the generating entity of the time-series data is determined according to the data source. If the generating entity is the data acquisition device, then the target storage period is determined based on the real-time acquisition frequency of the time-series data; If the generating entity is the data processing device, the target storage period is determined according to the processing window length of the time-series data and the preset storage period mapping relationship, wherein the storage period mapping relationship represents the correspondence between different processing window lengths and storage periods; If the generating entity is the data application device, then the preset first storage period is determined as the target storage period.

4. The time-series data storage method according to claim 1, characterized in that, Before determining the storage strategy for the time-series data based on the data source and the data type, the method further includes: Obtain audit information of the time series data, wherein the audit information includes the data generation time and data correction information corresponding to the time series data, and the data correction information includes the original data value, the person who corrected the data, the data correction time, and the data correction method; The time series data is converted into a preset data format, and the audit information is bound to the converted time series data.

5. The time-series data storage method according to claim 1, characterized in that, The step of storing the time-series data into the time-series database through the target storage instance includes: Check if there is currently a first write task in a write state; If it does not exist, a second write task is created through the target storage instance to write the time series data into the time series database; If it exists, the time-series data is written to the cache through the target storage instance, and when the cache time length reaches the preset time length and / or the cache data length reaches the preset data length, the cached data in the cache is obtained, the first write task is blocked, and a third write task is created to write the cached data to the time-series database.

6. The time-series data storage method according to claim 3, characterized in that, Determining the target storage period based on the real-time acquisition frequency of the time-series data includes: The real-time acquisition frequency is compared with a preset frequency threshold. If the real-time acquisition frequency is less than the preset frequency threshold, then the preset second storage period is determined as the target storage period, wherein the second storage period is less than the first storage period; If the real-time acquisition frequency is greater than or equal to the preset frequency threshold, then the first storage period is determined as the target storage period.

7. The time-series data storage method according to any one of claims 1 to 6, characterized in that, The data components of the time series data include timestamps, indicator values, and tags, and the tags include the second point identifier corresponding to the time series data. The methods for obtaining the metadata information include: Extract the second point identifier from the labels of the time series data; The second location identifier is matched with the first location identifier in different metadata information to obtain the metadata information corresponding to the time series data.

8. A time-series data storage system, characterized in that, The system includes: The data acquisition module is used to acquire time-series data and metadata information of the time-series data, wherein the metadata information includes the first point identifier corresponding to the time-series data, the data source, and the data type; The strategy determination module is used to determine the storage strategy of the time series data based on the data source and the data type, wherein the storage strategy includes the target storage period and target storage location of the time series data in the time series database; The instance matching module is used to match the target storage instance corresponding to the time series data from a pre-constructed storage instance set according to the storage strategy and the first location identifier, wherein each storage instance in the storage instance set is constructed according to different combinations of storage strategies and location identifiers, and each storage instance is a parallel storage instance. The data storage module is used to store the time-series data into the time-series database through the target storage instance.

9. An electronic device, characterized in that: include: processor; A storage device for storing a program, which, when executed by the processor, causes the electronic device to implement the timing data storage method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the time-series data storage method as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Storage method of massive time series data

    CN103605805A

  • Cryptographic data integrity protection

    CN113127893A

  • Distributed time sequence database system, electronic equipment and storage medium

    CN113641763A

  • Related content storage and retrieval systems and methods

    US20090248731A1

Cited By

  • Data storage and acquisition method and system based on point location identification

    CN121560893A

  • A data storage and acquisition method and system based on point location identification

    CN121560893B

  • Time series data writing method, system and equipment and medium

    CN121561543A