An efficient time series database data collection method and system
Through dynamic adjustment of conditional thresholds and multi-level data compression technology, the network bandwidth burden and blocking problems caused by synchronous data acquisition and transmission of timing databases in high concurrency environments are solved, efficient and reliable data transmission is achieved, and the concurrent processing of the system is improved Capacity and resource utilization efficiency.
Patent Information
- Application Number
- CN202411211992.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-30
AI Technical Summary
The existing timing databases are prone to loss and low efficiency in high concurrency environments due to synchronous data acquisition and transmission, and are prone to loss and low efficiency during data transmission.
The asynchronous data acquisition method is selected using dynamically adjusted conditional thresholds, batch storage and batch transmission of sensor data, and during the transmission process, multi-level data compression and a multiplexed transmission protocol are used to monitor and retry the transmission of fault data in real time.
It reduces network transmission frequency, avoids blocking problems caused by synchronous acquisition and transmission, improves the system's concurrent processing capabilities, improves data transmission efficiency and reliability, and reduces bandwidth usage and CPU resource consumption.
Smart Images

Figure CN118733679B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of communication technology, and in particular, to an efficient time series database data acquisition method and system. Background Art
[0002] Some existing time series databases (such as InfluxDB) use synchronous data collection and transmission to collect and transmit data, so that each time the data is collected, it will be transmitted separately, resulting in frequent network requests, increasing the burden on network bandwidth and transmission delays, and blocking threads during I / O operations. Especially in a high-concurrency environment, it is easy to cause the overall system performance to decline and affect the processing operations of other concurrent data. In addition, the transmission process generally follows the HTTP / 1.1 protocol and does not compress the data. If a failure occurs during the data transmission process and the data still cannot be successfully transmitted after multiple retries, the data will be discarded, resulting in data loss and low transmission efficiency. Summary of the invention
[0003] The purpose of this application is to provide an efficient time series database data acquisition method and system to solve at least one of the above technical problems.
[0004] In a first aspect, the present application provides an efficient time series database data collection method, which is applied to edge computing nodes and includes the following steps:
[0005] A1. Real-time acquisition of system status information for data collection mode selection; the data collection mode includes synchronous data collection mode and asynchronous data collection mode;
[0006] A2. If the synchronous data collection method is selected, the sensor data collected by the sensor in real time is transmitted to the time series database server in real time;
[0007] A3. If the asynchronous data collection method is selected, the sensor data collected by the sensor in real time is temporarily stored, and when it is determined based on the dynamically adjusted condition threshold that the batch transmission condition is currently met, the temporarily stored sensor data is batch-transmitted to the time series database server.
[0008] When performing synchronous data collection, this method transmits the real-time collected sensor data to the time series database server in real time to ensure the consistency and real-time nature of the data; when performing asynchronous data collection, batch storage and batch transmission of sensor data are performed based on dynamically adjusted conditional thresholds, which can reduce the frequency of network transmission, avoid blocking problems caused by synchronous collection and transmission, and improve the system's concurrent processing capabilities.
[0009] Preferably, the system status information includes data concurrency, network stability and data consistency requirements;
[0010] Step A1 includes:
[0011] The data collection method is selected based on the data concurrency, the network stability and the data consistency requirements.
[0012] Preferably, step A3 comprises:
[0013] A301. Temporarily store the sensor data collected by the sensor in real time in a local data buffer;
[0014] A302 dynamically adjusts the condition threshold according to the system status information; the condition threshold includes a data volume threshold and a time interval threshold;
[0015] A303. According to the data volume threshold, the time interval threshold, the temporarily stored data volume and the transmission time interval, determine whether the batch transmission condition is currently met; the temporarily stored data volume is the data volume of the sensor data temporarily stored in the data buffer; the transmission time interval is the time interval between the current time and the last time data was transmitted to the time series database server;
[0016] A304. When the batch transmission conditions are currently met, the sensor data temporarily stored in the data buffer is batch transmitted to the time series database server.
[0017] Compared with adopting a fixed condition threshold, dynamically adjusting the condition threshold is beneficial to optimizing transmission efficiency.
[0018] Preferably, the system status information includes network transmission efficiency and system load;
[0019] In step A302, the data volume threshold is adjusted according to the following formula:
[0020] ;
[0021] in, is the adjusted data volume threshold, is the data volume threshold before adjustment, is the network transmission efficiency, is the target network transmission efficiency, is the system load, The target system load.
[0022] Preferably, the system status information includes system response time;
[0023] In step A302, the time interval threshold is adjusted according to the following formula:
[0024] ;
[0025] in, is the adjusted time interval threshold, is the time interval threshold before adjustment, is the system response time, is the target system response time.
[0026] Preferably, step A303 includes:
[0027] If the temporarily stored data volume reaches the data volume threshold, or the transmission time interval reaches the time interval threshold, it is determined that the batch transmission condition is currently met; otherwise, it is determined that the batch transmission condition is currently not met.
[0028] Preferably, in step A304, when the sensor data temporarily stored in the data buffer is transmitted in batches to the time series database server, a multi-level data compression strategy is adopted to compress the data to be transmitted, and then the data is transmitted to the time series database server using a transmission protocol that supports multiplexing.
[0029] Using a multi-level data compression strategy to compress the data to be transmitted can effectively reduce the amount of transmitted data, increase the transmission speed, and reduce bandwidth occupancy; using a transmission protocol that supports multiplexing for data transmission can fully utilize network bandwidth and reduce transmission delays.
[0030] Preferably, after compressing the data to be transmitted using a multi-level data compression strategy, the data is transmitted to the time series database server using a transmission protocol supporting multiplexing, specifically including:
[0031] Using a first preset compression algorithm to preliminarily compress the data to be transmitted to obtain first compressed data;
[0032] Acquire current network status reference information, and determine whether the first compressed data is adapted to the current network status according to the network status reference information and the size of the first compressed data;
[0033] If the first compressed data is adapted to the current network condition, transmitting the first compressed data to the time series database server using a transmission protocol supporting multiplexing;
[0034] If the first compressed data is not suitable for the current network condition, a second preset compression algorithm is used to deeply compress the first compressed data to obtain second compressed data, and the second compressed data is transmitted to the time series database server using a transmission protocol that supports multiplexing.
[0035] Preferably, in step A304, when the sensor data temporarily stored in the data buffer is transferred in batches to the time series database server, the data transmission status is monitored in real time. If a transmission failure is detected, the data that has not been successfully transmitted is recorded in the local persistent cache, and the data that has not been successfully transmitted is periodically retried to be transmitted to the time series database server. If the retried transmission fails, after the transmission failure is recovered, the data that has not been successfully transmitted is extracted from the local persistent cache and transmitted to the time series database server.
[0036] In a second aspect, the present application provides an efficient time series database data acquisition system, comprising a plurality of sensors, an edge computing node and a time series database server; the sensors and the time series database server are both communicatively connected to the edge computing node;
[0037] The sensor is used to collect sensor data;
[0038] The edge computing node is used to obtain system status information in real time, so as to select a data collection method, wherein the data collection method includes a synchronous data collection method and an asynchronous data collection method. If the synchronous data collection method is selected, the sensor data collected by the sensor in real time is transmitted to the time series database server in real time. If the asynchronous data collection method is selected, the sensor data collected by the sensor in real time is temporarily stored, and when it is determined that the batch transmission condition is currently met based on the dynamically adjusted condition threshold, the temporarily stored sensor data is batch-transmitted to the time series database server;
[0039] The time series database server is used to store the received sensor data.
[0040] Beneficial effects: The efficient time series database data acquisition method and system provided by the present application selects the data acquisition method according to the real-time system status information; when performing synchronous data acquisition, the real-time collected sensor data is transmitted to the time series database server in real time to ensure the consistency and real-time nature of the data; when performing asynchronous data acquisition, batch temporary storage and batch transmission of sensor data are performed based on dynamically adjusted conditional thresholds, which can reduce the frequency of network transmission, avoid blocking problems caused by synchronous acquisition and transmission, and improve the concurrent processing capability of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 A flowchart of an efficient time series database data acquisition method provided in an embodiment of the present application.
[0042] Figure 2 A schematic diagram of the structure of an efficient time series database data acquisition system provided in an embodiment of the present application.
[0043] Figure 3 A schematic diagram of the structure of an edge computing node.
[0044] Explanation of reference numerals: 1. sensor; 2. edge computing node; 3. time series database server; 301. processor module; 302. memory module; 303. storage module; 304. network module; 305. I / O interface module; 306. power management module. DETAILED DESCRIPTION
[0045] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0046] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0047] Please refer to Figure 1 , Figure 1 It is an efficient time series database data collection method in some embodiments of the present application, applied to edge computing nodes, including the steps of:
[0048] A1. Obtain system status information in real time to select data collection methods; data collection methods include synchronous data collection methods and asynchronous data collection methods;
[0049] A2. If the synchronous data collection method is selected, the sensor data collected by the sensor in real time is transmitted to the time series database server in real time;
[0050] A3. If the asynchronous data collection method is selected, the sensor data collected by the sensor in real time is temporarily stored, and when it is determined based on the dynamically adjusted condition threshold that the batch transmission conditions are currently met, the temporarily stored sensor data is batch-transmitted to the time series database server.
[0051] When performing synchronous data collection, this method transmits the real-time collected sensor data to the time series database server in real time to ensure the consistency and real-time nature of the data; when performing asynchronous data collection, batch storage and batch transmission of sensor data are performed based on dynamically adjusted conditional thresholds, which can reduce the frequency of network transmission, avoid blocking problems caused by synchronous collection and transmission, and improve the system's concurrent processing capabilities.
[0052] Among them, the edge computing node can be Figure 2 The edge computing node 2 in the efficient time series database data acquisition system is shown.
[0053] Among them, system status information includes data concurrency, network stability and data consistency requirements.
[0054] Data concurrency refers to the number of tasks that an edge computing node needs to process per unit time. The current data concurrency can be obtained by dividing the total number of tasks processed in the first preset time period (set according to actual needs) before the current moment by the length of the first preset time period.
[0055] Network stability can be evaluated through multiple network performance indicators (for example, a weighted sum of multiple network performance indicators can be calculated as a numerical value of network stability), such as bandwidth, delay, packet loss rate, network throughput, and network reliability.
[0056] Data consistency requirements refer to the requirements for consistency between the sensor data obtained from the time series database and the sensor data collected by the sensor. They are pre-set by the user based on actual needs. Data consistency requirements can be expressed by data consistency levels (the higher the data consistency level, the higher the data consistency requirements).
[0057] Since the system status information is obtained in real time in step A1, the obtained system status information is all current system status information, such as the current data concurrency, the current network stability, the current data consistency requirements, and the like.
[0058] Thus, step A1 includes:
[0059] The data collection method is selected based on the data concurrency, network stability and data consistency requirements (referring to the current data concurrency, current network stability and current data consistency requirements, etc.).
[0060] For example, in some implementations, the data collection method is selected based on the data concurrency, network stability, and data consistency requirements, specifically including:
[0061] The network resource adequacy is calculated based on the data concurrency and network stability. For example, the network resource adequacy is calculated using the following formula: , Y is the network resource abundance, B is the data concurrency, w is the network stability, a and b are weight factors; but the calculation method is not limited to this;
[0062] If the data consistency requirement is higher than the preset data consistency level threshold, then when the network resource abundance is higher than the first network resource abundance threshold (set according to actual needs), the synchronous data collection mode is selected; when the network resource abundance is not higher than the first network resource abundance threshold (set according to actual needs), the asynchronous data collection mode is selected;
[0063] If the data consistency requirement is not higher than the preset data consistency level threshold, then when the network resource abundance is higher than the second network resource abundance threshold (set according to actual needs), the synchronous data collection method is selected; when the network resource abundance is not higher than the second network resource abundance threshold (set according to actual needs), the asynchronous data collection method is selected; the first network resource abundance threshold is less than the second network resource abundance threshold.
[0064] Therefore, when the data consistency requirement is high, the network resource abundance only needs to meet relatively low requirements (higher than the first network resource abundance threshold), and the synchronous data collection method is used to transmit data to the time series database server to give priority to meeting the higher data consistency requirements; when the data consistency requirement is low, the network resource abundance needs to meet relatively high requirements (higher than the second network resource abundance threshold), and the synchronous data collection method is used to transmit data to the time series database server to ensure that there are sufficient network resources to support the synchronous data collection method, avoid blocking threads and affecting other data processing operations.
[0065] Among them, in step A2, when transmitting sensor data to the time series database server, the HTTP / 1.1 protocol is used for transmission and the sensor data is not compressed to ensure the originality of the data, so that the sensor data obtained by the time series database and the sensor data collected by the sensor are highly consistent.
[0066] Further, step A3 includes:
[0067] A301. Temporarily store the sensor data collected by the sensor in real time in a local data buffer;
[0068] A302. Dynamically adjust the condition threshold according to the system status information; the condition threshold includes the data volume threshold and the time interval threshold;
[0069] A303. According to the data volume threshold, time interval threshold, temporary storage data volume and transmission time interval, determine whether the batch transmission conditions are currently met; the temporary storage data volume is the amount of sensor data temporarily stored in the data buffer; the transmission time interval is the time interval between the current time and the last time data was transmitted to the time series database server;
[0070] A304. When the batch transmission conditions are currently met, the sensor data temporarily stored in the data buffer is batch transmitted to the time series database server.
[0071] Compared with adopting a fixed condition threshold, dynamically adjusting the condition threshold is more conducive to optimizing transmission efficiency.
[0072] The data buffer is an area set in the local memory of the edge computing node for temporarily storing sensor data.
[0073] The system status information also includes network transmission efficiency and system load (current network transmission efficiency and current system load). Network transmission efficiency refers to the amount of data successfully transmitted per unit time. System load refers to the system resource usage per unit time.
[0074] In step A302, the data volume threshold is adjusted according to the following formula:
[0075] ;
[0076] in, is the adjusted data volume threshold, is the data volume threshold before adjustment, For network transmission efficiency, is the target network transmission efficiency (it is actually the ideal transmission efficiency and can be set according to actual needs), is the system load, is the target system load (it is actually the ideal system load and can be set according to actual needs). The initial value is a preset value and can be set according to actual needs. and The ratio reflects the gap between the current network transmission efficiency and the ideal state. If the current network transmission efficiency is higher than the target network transmission efficiency, the data volume threshold should be reduced, otherwise, the data volume threshold should be increased. and The ratio reflects the gap between the current system load and the ideal state. If the current system load is higher than the target system load, the data volume threshold should be increased, otherwise, the data volume threshold should be reduced.
[0077] Furthermore, the system status information also includes the system response time (the current system response time). The system response time is the time required for the edge computing node to process a batch processing task; wherein, the processing task performed in the process of temporarily storing the sensor data in the data buffer and transferring the temporarily stored sensor data in batches to the time series database server is a batch processing task; the sum of the time taken for each batch processing task completed within the second preset time period (which can be set according to actual needs) before the current moment divided by the total number of batch processing tasks completed within the second preset time period can be used as the current system response time.
[0078] Therefore, in step A302, the time interval threshold is adjusted according to the following formula:
[0079] ;
[0080] in, is the adjusted time interval threshold, is the time interval threshold before adjustment, is the system response time, is the target system response time (it is actually the ideal system response time and can be set according to actual needs). The initial value is a preset value and can be set according to actual needs. and The ratio reflects the gap between the current system response time and the ideal state. If the current system response time is longer than the target system response time, the time interval threshold should be increased, otherwise, the time interval threshold should be decreased.
[0081] Specifically, step A303 includes:
[0082] If the temporarily stored data volume reaches the data volume threshold, or the transmission time interval reaches the time interval threshold, it is determined that the batch transmission condition is currently met; otherwise, it is determined that the batch transmission condition is currently not met.
[0083] Among them, in order to improve the transmission efficiency, in step A304, when the sensor data temporarily stored in the data buffer is transferred in batches to the time series database server, the data to be transferred can be compressed before being sent to the time series database server. In order to further improve the transmission efficiency and data transmission security, in step A304, when the sensor data temporarily stored in the data buffer is transferred in batches to the time series database server, a transmission protocol that supports multiplexing (such as HTTP / 2 protocol, gRPC protocol, etc., but not limited to this) can also be used for data transmission.
[0084] For example, in a preferred embodiment, in step A304, when the sensor data temporarily stored in the data buffer is transmitted in batches to the time series database server, a multi-level data compression strategy is used to compress the data to be transmitted, and then the data is transmitted to the time series database server using a transmission protocol that supports multiplexing.
[0085] Using a multi-level data compression strategy to compress the data to be transmitted can effectively reduce the amount of transmitted data, increase the transmission speed, and reduce bandwidth occupancy; using a transmission protocol that supports multiplexing for data transmission can fully utilize network bandwidth and reduce transmission delays.
[0086] Specifically, after the data to be transmitted is compressed by adopting a multi-level data compression strategy, it is transmitted to the time series database server using a transmission protocol supporting multiplexing, including:
[0087] Using a first preset compression algorithm to preliminarily compress the data to be transmitted to obtain first compressed data;
[0088] Acquire current network status reference information, and determine whether the first compressed data is suitable for the current network status according to the network status reference information and the size of the first compressed data;
[0089] If the first compressed data is adapted to the current network condition, transmitting the first compressed data to the time series database server using a transmission protocol supporting multiplexing;
[0090] If the first compressed data is not suitable for the current network condition, a second preset compression algorithm is used to deeply compress the first compressed data to obtain second compressed data, and the second compressed data is transmitted to the time series database server using a transmission protocol supporting multiplexing.
[0091] Through the above method, after completing the preliminary compression, if the data volume of the first compressed data is within an acceptable range, the first compressed data is directly transmitted to the time series database server, thereby saving the compression processing time of the edge computing node and the decompression processing time of the time series database server; if the data volume of the first compressed data is not within an acceptable range, it is compressed again and then transmitted to ensure the transmission speed.
[0092] Among them, an existing compression algorithm can be selected as the first preset compression algorithm according to actual needs.
[0093] The network status reference information may include bandwidth, delay, and packet loss rate; judging whether the first compressed data is adapted to the current network status according to the network status reference information and the size of the first compressed data specifically includes:
[0094] Calculate the network status quality according to the network status reference information; for example, calculate the weighted sum of various network status reference information as the network status quality, but not limited to this;
[0095] Obtaining the adaptation data volume threshold value corresponding to the network condition quality; for example, different network condition quality and corresponding adaptation data volume threshold values may be determined in advance through experiments, forming an adaptation data volume threshold query table or fitting to generate an adaptation data volume threshold calculation formula. In practical applications, the adaptation data volume threshold query table is queried according to the actual network condition quality to obtain the corresponding adaptation data volume threshold value, or the actual network condition quality is substituted into the adaptation data volume threshold calculation formula to calculate the corresponding adaptation data volume threshold value;
[0096] If the size of the first compressed data does not exceed the corresponding adaptation data amount threshold, it is determined that the first compressed data is adapted to the current network condition; otherwise, it is determined that the first compressed data is not adapted to the current network condition.
[0097] Among them, an existing compression algorithm can be selected as the second preset compression algorithm according to actual needs, such as a gzip compression algorithm, an Lz4 compression algorithm, etc., but is not limited thereto.
[0098] Furthermore, in step A304, when the sensor data temporarily stored in the data buffer is transferred in batches to the time series database server, the data transmission status is monitored in real time. If a transmission failure is detected, the data that has not been successfully transmitted is recorded in the local persistent cache, and the data that has not been successfully transmitted is periodically retried to be transmitted to the time series database server. If the retried transmission fails, after the transmission failure is recovered, the data that has not been successfully transmitted is extracted from the local persistent cache and transmitted to the time series database server.
[0099] Among them, when the data that has not been successfully transmitted is retried to the time series database server at a scheduled time, the data that has not been successfully transmitted is periodically retried according to a preset time interval (set according to actual needs) until the transmission is successful or the number of retries reaches the upper limit. If the number of retries reaches the upper limit and the transmission is still not successful, the retry transmission is determined to have failed. Among them, the upper limit can be preset according to actual needs, or calculated based on the quality of the network and the importance of the data that has not been successfully transmitted. The importance of the data that has not been successfully transmitted can be represented by an importance coefficient, and the range of the importance coefficient is 0-1. The importance coefficient can be determined based on some or all of the factors such as the type of sensor that collects the corresponding data, the detection object and detection location, and the data collection time (the specific determination rules can be set according to actual needs and are not limited here); for example, the upper limit can be calculated according to the following formula:
[0100] ;
[0101] in, is the upper limit of the number of times, y is the quality of the network condition, z is the importance coefficient of the data that was not successfully transmitted, , is the weight coefficient (set according to actual needs), is the rounding function, The maximum number of retries (can be set according to actual needs). The better the network condition is and the more important the data that has not been successfully transmitted is, the greater the number of retries allowed will be, so as to minimize data loss (especially loss of important data) without affecting other data processing.
[0102] After the retry transmission fails, the transmission of the corresponding unsuccessfully transmitted data is suspended until the transmission failure is recovered, and then the unsuccessfully transmitted data is extracted from the local persistent cache again and transmitted to the time series database server.
[0103] The transmission failure may be caused by poor network conditions or device failure. When retrying the transmission, execute:
[0104] Compare the importance coefficient of the data that was not successfully transmitted with a preset importance coefficient threshold (which can be set according to actual needs) to determine whether the data that was not successfully transmitted is important data; for example, if the importance coefficient exceeds the importance coefficient threshold, the data that was not successfully transmitted is determined to be important data, otherwise, the data that was not successfully transmitted is determined to be not important data;
[0105] If the data that was not successfully transmitted is important data, the whole packet is retransmitted to retry the transmission of the data that was not successfully transmitted;
[0106] If the data that was not successfully transmitted is not important data, determine whether the current network status is poor. If so, use incremental transmission to retry transmission of the data that was not successfully transmitted. If not, when the data that was not successfully transmitted is too large, use incremental transmission to retry transmission of the data that was not successfully transmitted. When the data that was not successfully transmitted is not too large, use whole packet retransmission to retry transmission of the data that was not successfully transmitted. In the incremental transmission mode, only the untransmitted part of the unsuccessful transmission is retried (thereby improving transmission efficiency and reducing the amount of data that is repeatedly transmitted). In the whole packet retransmission mode, the entire unsuccessful transmission of the data is retried (thereby ensuring that data is not lost).
[0107] Therefore, priority is given to ensuring that important data is not lost. For non-important data, the retry transmission method is selected by comprehensively considering the real-time network status and the amount of data that has not been successfully transmitted, so as to balance the transmission efficiency, data retransmission rate and data loss rate.
[0108] Among them, it can be used to judge whether the current network state is poor according to the network state quality. For example, if the network state quality exceeds the quality threshold, the current network state is determined to be good, otherwise, the current network state is determined to be poor; the quality threshold can be set according to actual needs.
[0109] If the amount of data that is not successfully transmitted exceeds a preset data amount threshold, it is considered that the data that is not successfully transmitted is too large, and the data amount threshold can be set according to actual needs.
[0110] In some preferred implementations, in step A3, based on the asynchronous I / O operation, the processes of temporarily storing sensor data and transmitting sensor data in batches are executed in parallel, thereby effectively avoiding blocking the main thread.
[0111] refer to Figure 2 , the present application provides an efficient time series database data acquisition system, including a plurality of sensors 1, an edge computing node 2 and a time series database server 3; the sensors 1 and the time series database server 3 are both communicatively connected with the edge computing node 2;
[0112] Sensor 1 is used to collect sensor data;
[0113] The edge computing node 2 is used to obtain system status information in real time, and is used to select a data collection method. The data collection methods include synchronous data collection methods and asynchronous data collection methods. If the synchronous data collection method is selected, the sensor data collected by the sensor 1 in real time is transmitted to the time series database server 3 in real time. If the asynchronous data collection method is selected, the sensor data collected by the sensor 1 in real time is temporarily stored, and when it is determined that the batch transmission conditions are currently met based on the dynamically adjusted condition threshold, the temporarily stored sensor data is batch-transmitted to the time series database server 3 (for the specific process, refer to the steps of the efficient time series database data collection method in the previous text);
[0114] The time series database server 3 is used to store the received sensor data.
[0115] The number and type of sensors 1 can be set according to actual needs.
[0116] The time series database server 3 is, for example, a server of the InfluxDB database, but is not limited thereto.
[0117] Among them, reference Figure 3, the edge computing node 2 includes a processor module 301, a memory module 302, a storage module 303, a network module 304, an I / O interface module 305 and a power management module 306, and the memory module 302, the storage module 303, the network module 304, the I / O interface module 305 and the power management module 306 are all connected to the processor module 301. Among them, the data buffer is set in the memory module 302, and the local persistent cache is set in the storage module 303. Among them, the processor module 301 is used to execute the efficient time series database data acquisition method mentioned above.
[0118] In summary, the efficient time series database data collection method and system provided by the present application have at least the following advantages:
[0119] 1. Improve data transmission efficiency: Dynamic batch processing strategy reduces frequent network transmission, data compression and multiplexing technology improves transmission efficiency and reduces network bandwidth occupancy;
[0120] 2. Reduce system blocking: Asynchronous data collection and transmission avoids the blocking problem in synchronous methods and improves the concurrent processing capability of the system;
[0121] 3. Enhanced data reliability: The perfect fault recovery mechanism ensures that data is not lost during transmission, improving the reliability and stability of the system;
[0122] 4. Optimize resource usage: Through batch processing and data compression technology, the consumption of CPU (ie, processor module 301) and memory resources is reduced, and the resource utilization efficiency of the system is improved.
[0123] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0124] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0125] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0126] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.
[0127] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. An efficient time series database data collection method, applied to edge computing nodes, characterized in that: Includes steps: A1. Real-time acquisition of system status information for data collection mode selection; the data collection mode includes synchronous data collection mode and asynchronous data collection mode; A2. If the synchronous data collection method is selected, the sensor data collected by the sensor in real time is transmitted to the time series database server in real time; A3. If the asynchronous data collection method is selected, the sensor data collected by the sensor in real time is temporarily stored, and when it is determined that the batch transmission conditions are currently met based on the dynamically adjusted condition threshold, the temporarily stored sensor data is batch-transmitted to the time series database server; Step A3 includes: A301. Temporarily store the sensor data collected by the sensor in real time in a local data buffer; A302 dynamically adjusts the condition threshold according to the system status information; the condition threshold includes a data volume threshold and a time interval threshold; A303. According to the data volume threshold, the time interval threshold, the temporarily stored data volume and the transmission time interval, determine whether the batch transmission condition is currently met; the temporarily stored data volume is the data volume of the sensor data temporarily stored in the data buffer; the transmission time interval is the time interval between the current time and the last time data was transmitted to the time series database server; A304. When the batch transmission conditions are currently met, the sensor data temporarily stored in the data buffer is batch transmitted to the time series database server; The system status information includes network transmission efficiency and system load; In step A302, the data volume threshold is adjusted according to the following formula: ; in, is the adjusted data volume threshold, is the data volume threshold before adjustment, is the network transmission efficiency, is the target network transmission efficiency, is the system load, is the target system load; Step A303 includes: If the temporarily stored data volume reaches the data volume threshold, or the transmission time interval reaches the time interval threshold, it is determined that the batch transmission condition is currently met; otherwise, it is determined that the batch transmission condition is currently not met.
2. The efficient time series database data collection method according to claim 1 is characterized in that: The system status information includes data concurrency, network stability and data consistency requirements; Step A1 includes: The data collection method is selected based on the data concurrency, the network stability and the data consistency requirements.
3. The efficient time series database data collection method according to claim 1 is characterized in that: The system status information includes system response time; In step A302, the time interval threshold is adjusted according to the following formula: ; in, is the adjusted time interval threshold, is the time interval threshold before adjustment, is the system response time, is the target system response time.
4. The efficient time series database data collection method according to claim 1 is characterized in that: In step A304, when the sensor data temporarily stored in the data buffer is transmitted in batches to the time series database server, a multi-level data compression strategy is used to compress the data to be transmitted, and then the data is transmitted to the time series database server using a transmission protocol that supports multiplexing.
5. The efficient time series database data collection method according to claim 4 is characterized in that: After compressing the data to be transmitted using a multi-level data compression strategy, the data is transmitted to the time series database server using a transmission protocol supporting multiplexing, specifically including: Using a first preset compression algorithm to preliminarily compress the data to be transmitted to obtain first compressed data; Acquire current network status reference information, and determine whether the first compressed data is adapted to the current network status according to the network status reference information and the size of the first compressed data; If the first compressed data is adapted to the current network condition, transmitting the first compressed data to the time series database server using a transmission protocol supporting multiplexing; If the first compressed data is not suitable for the current network condition, a second preset compression algorithm is used to deeply compress the first compressed data to obtain second compressed data, and the second compressed data is transmitted to the time series database server using a transmission protocol that supports multiplexing.
6. The efficient time series database data collection method according to claim 1 is characterized in that: In step A304, when the sensor data temporarily stored in the data buffer is transferred in batches to the time series database server, the data transmission status is monitored in real time. If a transmission failure is detected, the data that has not been successfully transmitted is recorded in the local persistent cache, and the data that has not been successfully transmitted is retried to be transmitted to the time series database server at a fixed time. If the retried transmission fails, after the transmission failure is recovered, the data that has not been successfully transmitted is extracted from the local persistent cache and transmitted to the time series database server.
7. An efficient time series database data acquisition system, characterized in that: It includes several sensors, edge computing nodes and a time series database server; the sensors and the time series database server are both communicatively connected with the edge computing nodes; The sensor is used to collect sensor data; The edge computing node is used to obtain system status information in real time, so as to select a data collection method, wherein the data collection method includes a synchronous data collection method and an asynchronous data collection method. If the synchronous data collection method is selected, the sensor data collected by the sensor in real time is transmitted to the time series database server in real time. If the asynchronous data collection method is selected, the sensor data collected by the sensor in real time is temporarily stored, and when it is determined that the batch transmission condition is currently met based on the dynamically adjusted condition threshold, the temporarily stored sensor data is batch-transmitted to the time series database server; The time series database server is used to store the received sensor data; When the asynchronous data collection method is selected, the edge computing node temporarily stores the sensor data collected by the sensor in real time, and when it is determined based on the dynamically adjusted condition threshold that the batch transmission condition is currently met, the temporarily stored sensor data is batch-transmitted to the time series database server, specifically performing: A301. Temporarily store the sensor data collected by the sensor in real time in a local data buffer; A302 dynamically adjusts the condition threshold according to the system status information; the condition threshold includes a data volume threshold and a time interval threshold; A303. According to the data volume threshold, the time interval threshold, the temporarily stored data volume and the transmission time interval, determine whether the batch transmission condition is currently met; the temporarily stored data volume is the data volume of the sensor data temporarily stored in the data buffer; the transmission time interval is the time interval between the current time and the last time data was transmitted to the time series database server; A304. When the batch transmission conditions are currently met, the sensor data temporarily stored in the data buffer is batch transmitted to the time series database server; The system status information includes network transmission efficiency and system load; In step A302, the data volume threshold is adjusted according to the following formula: ; in, is the adjusted data volume threshold, is the data volume threshold before adjustment, is the network transmission efficiency, is the target network transmission efficiency, is the system load, is the target system load; Step A303 includes: If the temporarily stored data volume reaches the data volume threshold, or the transmission time interval reaches the time interval threshold, it is determined that the batch transmission condition is currently met; otherwise, it is determined that the batch transmission condition is currently not met.
