A time series data storage method and device, computer equipment and storage medium

Abstract

The application relates to the technical field of data storage, and discloses a time series data storage method and device, computer equipment and a storage medium. The application stores time series data to be stored into a to-be-imported queue according to a preset length, and stores the time series data into a time series database through the to-be-imported queue, so that the time series data is not loaded into the memory all at once, and at most the data according to the preset length exists in the memory at the same time, therefore, the memory cost is small and does not increase with the data volume. Meanwhile, the time series data to be stored is checked before storage, so that the accuracy of data storage is improved.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and specifically to a time-series data storage method, apparatus, computer equipment, and storage medium. Background Technology

[0002] In practical applications, there is a common need for batch data import. When importing small amounts of data, the general approach is to first read the data into memory, parse it into a list of objects, process each data item individually, and finally return the processing result of each item to the caller. This solution is simple and convenient, and in most scenarios, it can meet business needs.

[0003] Time series data is characterized by large data volume and few data fields. Using the above methods to process time series data import will result in excessive memory usage. Summary of the Invention

[0004] In view of this, the present invention provides a time-series data storage method, apparatus, computer device, and storage medium to solve the problem of excessive memory usage when using existing solutions to process time-series data import.

[0005] In a first aspect, the present invention provides a time-series data storage method for a time-series database; the method includes:

[0006] Obtain the time series dataset to be stored; based on the time series data to be stored, and through a preset processing and verification method, obtain the import queue and the first error queue; when the queue length of the import queue is equal to the preset length, based on the first error queue, store the time series data stored in the import queue into the time series database; clear the import queue and the first error queue; repeat the steps from obtaining the import queue and the first error queue based on the time series data to be stored, through the preset processing and verification method, to clearing the import queue and the first error queue, until the time series dataset to be stored is stored in the time series database.

[0007] The time-series data storage method provided by this invention stores the time-series data to be stored in an import queue according to a preset length, and then stores it in a time-series database through the import queue. This prevents the time-series data from being loaded into memory all at once, ensuring that memory holds a maximum of the preset length of data at any given time. Therefore, memory overhead is low and does not increase with the amount of data. Furthermore, verifying the time-series data before storage improves the accuracy of data storage.

[0008] In one optional implementation, based on the time-series data to be stored, a queue to be imported and a first error queue are obtained through a preset processing and verification method, including:

[0009] Read the time series dataset to be stored line by line in sequence, and verify each line of data read; store the time series data that pass the verification in sequence to form an import queue; store the line number and error type of the time series data that fail the verification in sequence to form the first error queue.

[0010] This invention improves the accuracy of data storage by sequentially verifying each row of data in the time-series dataset to be stored and storing it in the corresponding error queue.

[0011] In an optional implementation, when the queue length of the queue to be imported is equal to a preset length, after storing the time-series data stored in the queue to be imported into a time-series database, the method further includes:

[0012] When the time series database returns error information, obtain the error information set; based on the error information set and the first error queue, obtain the second error queue through a classification and merging method; clear the import queue and the second error queue; repeat the steps from obtaining the import queue and the first error queue based on the time series data to be stored through a preset processing and verification method to clear the import queue and the second error queue, until the time series dataset to be stored is stored in the time series database.

[0013] This invention categorizes and merges verification errors and storage errors in the time-series data to be stored, resulting in smaller and clearer storage response results. This solves the problem of existing solutions where returning import results one by one leads to an extremely large result set. Furthermore, by combining the storage of the time-series data with an import queue, the time-series data is not loaded into memory all at once. Only a maximum of a preset length of data is stored in memory at a time, thus minimizing memory overhead and preventing it from increasing with the amount of data.

[0014] In one optional implementation, based on the error information set and the first error queue, a second error queue is obtained through a classification and merging method, including:

[0015] Obtain the initial result stack; store the error information set into the first error queue and obtain the third error queue; sort the elements in the third error queue according to the row number to obtain the fourth error queue; compare the error code of each element in the fourth error queue with the error code of the top element in the initial result stack, and determine the second error queue based on the comparison result.

[0016] This invention classifies and merges the verification errors and storage errors of the time-series data to be stored, making the storage response results of the time-series data to be stored smaller and clearer, thus solving the problem that the result set becomes very large when the import results are returned one by one in the existing solution.

[0017] In one optional implementation, the error code of each element in the fourth error queue is compared sequentially with the error code of the top element in the initial result stack, and the second error queue is determined based on the comparison results, including:

[0018] When the error code of the first element in the fourth error queue is not equal to the error code of the top element in the initial result stack, add the error code of the first element to the initial result stack, and assign the line number of the first element to the start line parameter and end line parameter of the top element of the stack. Iterate through each element in the fourth error queue in turn, and repeat the steps of adding the error code of the first element to the initial result stack and assigning the line number of the first element to the start line parameter and end line parameter of the top element of the stack when the error code of the first element in the fourth error queue is not equal to the error code of the top element in the initial result stack, until the second error queue is obtained.

[0019] In one optional implementation, the error code of each element in the fourth error queue is compared sequentially with the error code of the top element in the initial result stack, and the second error queue is determined based on the comparison results. The method further includes:

[0020] When the error code of the first element in the fourth error queue is equal to the error code of the top element in the initial result stack, the line number of the first element is assigned to the termination line parameter of the top element. This process is repeated for each element in the fourth error queue, and the step of assigning the line number of the first element to the termination line parameter of the top element is repeated until the second error queue is obtained.

[0021] In a second aspect, the present invention provides a time-series data storage device for a time-series database; the device includes:

[0022] The first acquisition module is used to acquire the time series dataset to be stored; the processing module is used to obtain the import queue and the first error queue based on the time series data to be stored through a preset processing and verification method; the first storage module is used to store the time series data stored in the import queue into the time series database when the queue length of the import queue is equal to the preset length; the first clearing processing module is used to clear the import queue and the first error queue; the first repeating and storage module is used to repeat the steps from obtaining the import queue and the first error queue based on the time series data to be stored through the preset processing and verification method to clearing the import queue and the first error queue, until the time series dataset to be stored is stored in the time series database.

[0023] In one alternative implementation, the processing module includes:

[0024] The read and verification submodule is used to read the time series dataset to be stored line by line in sequence and verify each line of data read; the first storage submodule is used to store the time series data that has passed verification in sequence and form an import queue; the second storage submodule is used to store the line number and error type of the time series data that has failed verification in sequence and form a first error queue.

[0025] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the timing data storage method of the first aspect or any corresponding embodiment described above.

[0026] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions, which are used to cause a computer to execute the timing data storage method of the first aspect or any corresponding embodiment described above. Attached Figure Description

[0027] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating a time-series data storage method according to an embodiment of the present invention;

[0029] Figure 2 This is a flowchart illustrating another time-series data storage method according to an embodiment of the present invention;

[0030] Figure 3 This is a flowchart illustrating another time-series data storage method according to an embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram of the import process corresponding to the time-series data import method according to an embodiment of the present invention;

[0032] Figure 5 This is a schematic diagram of the merging logic corresponding to the time-series data import method according to an embodiment of the present invention;

[0033] Figure 6 This is a structural block diagram of a time-series data storage device according to an embodiment of the present invention;

[0034] Figure 7 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] In the field of industrial big data storage, in addition to traditional relational databases, there is also a type of time-series database used to store large amounts of measurement data. A time-series database is not just a database, but a system that includes: various industrial interface acquisition protocols, massive amounts of measurement data storage, and external interfaces.

[0037] In practical applications, there is a common need for batch data import. When importing small amounts of data, the general approach is to first read the data into memory, parse it into a list of objects, process each data item individually, and finally return the processing result of each item to the caller. This solution is simple and convenient, and in most scenarios, it can meet business needs.

[0038] Time series data is characterized by large data volume and few data fields. Using the above methods to process time series data import will result in excessive memory usage. In addition, returning the import results one by one will result in a very large result set.

[0039] According to an embodiment of the present invention, a time-series data storage method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0040] This embodiment provides a time-series data storage method, which can be used in the aforementioned time-series database. Figure 1 This is a flowchart of a time-series data storage method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0041] Step S101: Obtain the time series dataset to be stored.

[0042] Among them, time series data refers to data in time sequence, which is a data column recorded in chronological order according to the same unified indicator.

[0043] Step S102: Based on the time-series data to be stored, and after passing through a preset processing and verification method, the import queue and the first error queue are obtained.

[0044] Specifically, by using preset processing and verification methods, the processed and verified time-series data to be stored can be stored in the corresponding import queue and the first error queue, respectively.

[0045] Step S103: When the queue length of the queue to be imported is equal to the preset length, the time series data stored in the queue to be imported is stored in the time series database based on the first error queue.

[0046] The preset length can be set according to actual needs. For example, when there are 1 million rows of time series data in the time series dataset to be stored, the preset length can be set to 1000.

[0047] Specifically, when the length of the queue to be imported is equal to the preset length, the correct time series data stored in the queue to be imported is imported into the corresponding time series database according to the first error queue.

[0048] Step S104: Clear the import queue and the first error queue.

[0049] Specifically, after importing the time series data stored in the import queue into the corresponding time series database, the import queue and the first error queue are cleared to support the subsequent import of time series data.

[0050] Step S105: Repeat the steps of obtaining the import queue and the first error queue based on the time series data to be stored through the preset processing and verification method, until the import queue and the first error queue are cleared, until the time series dataset to be stored is stored in the time series database.

[0051] Specifically, continue repeating steps S101 to S104 until the time series dataset to be stored is empty and the import queue is also empty, indicating that all the obtained time series datasets to be stored have been stored in the corresponding time series database.

[0052] The time-series data storage method provided in this embodiment stores the time-series data to be stored in an import queue according to a preset length, and then stores it in the time-series database through the import queue. This prevents the time-series data from being loaded into memory all at once, ensuring that memory holds a maximum of the preset length of data at any given time. Therefore, memory overhead is low and does not increase with the amount of data. Furthermore, verifying the time-series data before storage improves the accuracy of data storage.

[0053] This embodiment provides a time-series data storage method, which can be used in the aforementioned time-series database. Figure 2 This is a flowchart of a time-series data storage method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0054] Step S201: Obtain the time-series dataset to be stored. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0055] Step S202: Based on the time-series data to be stored, and after passing through a preset processing and verification method, the import queue and the first error queue are obtained.

[0056] Specifically, step S202 includes:

[0057] Step S2021: Read the time series dataset to be stored line by line in sequence, and verify each line of data read.

[0058] Specifically, each row of data in the time series dataset to be stored is read sequentially according to the row number, and each time series data read is checked, such as for illegal characters and formatting.

[0059] Step S2022: The verified time-series data are stored sequentially and formed into a queue to be imported.

[0060] Specifically, if the read time series data passes the verification, all the corresponding time series data that passed the verification are written into the corresponding import queue.

[0061] Step S2023: Store the row number and error type of the time sequence data that failed the verification in sequence and form the first error queue.

[0062] Specifically, if the read time series data fails the test, the row number and error type of all the time series data that failed the test are stored in sequence to form the first error queue.

[0063] Step S203: When the queue length of the queue to be imported is equal to the preset length, the time series data stored in the queue to be imported is stored in the time series database based on the first error queue. For details, please refer to [link to relevant documentation]. Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0064] Step S204: Clear the import queue and the first error queue. For details, please refer to [link to relevant documentation]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0065] Step S205 repeats the process from obtaining the import queue and the first error queue based on the time-series data to be stored using a preset processing and verification method, to clearing the import queue and the first error queue, until the time-series dataset to be stored is stored in the time-series database. For details, please refer to [link to relevant documentation]. Figure 1 Step S105 of the illustrated embodiment will not be described again here.

[0066] The time-series data storage method provided in this embodiment stores the time-series data to be stored in an import queue according to a preset length, and then stores it in the time-series database through the import queue. This prevents the time-series data from being loaded into memory all at once, ensuring that memory holds a maximum of the preset length of data at any given time. Therefore, memory overhead is low and does not increase with the amount of data. Furthermore, by sequentially validating each row of data in the time-series dataset and storing the results in the corresponding error queue, the accuracy of data storage can be improved.

[0067] This embodiment provides a time-series data storage method, which can be used in the aforementioned time-series database. Figure 3 This is a flowchart of a time-series data storage method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:

[0068] Step S301: Obtain the time-series dataset to be stored. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0069] Step S302: Based on the time-series data to be stored, and after processing and verification using a preset method, the import queue and the first error queue are obtained. For details, please refer to [link to relevant documentation]. Figure 2 Step S202 of the illustrated embodiment will not be described again here.

[0070] Step S303: When the queue length of the queue to be imported is equal to the preset length, the time series data stored in the queue to be imported is stored in the time series database based on the first error queue. For details, please refer to [link to relevant documentation]. Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0071] Step S304: When the time-series database returns error information, obtain the error information set.

[0072] Specifically, when storing time-series data to be imported into the queue into the time-series database, the time-series database may return corresponding error messages, such as the error message that the data already exists.

[0073] Therefore, when the time-series database returns error information, the corresponding error information set can be obtained.

[0074] Step S305: Based on the error information set and the first error queue, a second error queue is obtained through a classification and merging method.

[0075] Specifically, the error messages obtained from the two tests are merged to form a corresponding second error queue.

[0076] Specifically, step S305 includes:

[0077] Step S3051: Obtain the initial result stack.

[0078] The initial result stack is empty.

[0079] Step S3052: Store the error information set in the first error queue and obtain the third error queue.

[0080] Specifically, the set of error messages returned when the time-series data to be imported into the time-series database is also stored in the first error queue, forming a corresponding third error queue.

[0081] Step S3053: Sort the elements in the third error queue according to the row number to obtain the fourth error queue.

[0082] Specifically, the elements in the third error queue are sorted according to their row numbers to obtain the sorted fourth error queue.

[0083] Step S3054: Compare the error code of each element in the fourth error queue with the error code of the top element in the initial result stack, and determine the second error queue based on the comparison results.

[0084] Specifically, the initial result stack stores triples, with the structure [error code, start line, end line]. During initialization, the stack is empty. The first line of the data file to be imported is the header, and a data entry [ok, 1, 1] is pushed onto the stack. After each batch of data is imported, the following logic is executed:

[0085] Stack top [ok, 7, 10]

[0086] [Measurement point does not exist, 4, 6]

[0087] The bottom of the stack is [ok, 1, 3].

[0088] Here, the starting line refers to the line number where an error begins to occur, and the ending line refers to the last line where this error occurs consecutively. For example, suppose there are 10 lines of data in total, lines 1 to 3 are normal data, lines 4 to 6 are errors where the measurement point does not exist, and lines 7 to 10 are normal data. Then the starting line for the error "the measurement point does not exist" is line 4, and the ending line is line 6.

[0089] Furthermore, iterate through each element in the fourth error queue and compare the error code of each element with the error code of the top element in the initial result stack.

[0090] Furthermore, the merged second error queue can be determined based on the comparison results.

[0091] In some optional implementations, step S3054 above includes:

[0092] Step a1: When the error code of the first element in the fourth error queue is not equal to the error code of the top element in the initial result stack, add the error code of the first element to the initial result stack, and assign the line number of the first element to the start line parameter and end line parameter of the top element.

[0093] Step a2: Iterate through each element in the fourth error queue in turn, and repeat the steps of adding the error code of the first element to the initial result stack and assigning the row number of the first element to the start row parameter and end row parameter of the top element of the stack when the error code of the first element in the fourth error queue is not equal to the error code of the top element of the initial result stack, until the second error queue is obtained.

[0094] Specifically, the first element in the fourth error queue is read, and if the error code of the first element is not equal to the error code of the top element in the initial result stack, the error code of the first element is added to the first error queue, and the row number of the first element is set as the start row parameter and end row parameter of the top element.

[0095] Furthermore, repeat step a1. After traversing all elements in the fourth error queue, the corresponding second error queue can be obtained.

[0096] In some optional implementations, step S3054 above further includes:

[0097] Step a3: When the error code of the first element in the fourth error queue is equal to the error code of the top element in the initial result stack, assign the line number of the first element to the termination line parameter of the top element.

[0098] Step a4: Iterate through each element in the fourth error queue in turn, and repeat the step of assigning the row number of the first element to the termination row parameter of the top element of the stack when the error code of the first element in the fourth error queue is equal to the error code of the top element of the initial result stack, until the second error queue is obtained.

[0099] Specifically, when the error code of the first element in the fourth error queue is read and determined to be equal to the error code of the top element in the first error queue, the line number of the first element is set as the termination line parameter of the top element.

[0100] Furthermore, by iterating through each element in the fourth error queue and repeating step a1 until each element in the fourth error queue has been traversed, a corresponding second error queue can be formed.

[0101] Step S306: Clear the import queue and the second error queue.

[0102] Specifically, after importing the time series data stored in the import queue into the corresponding time series database, the import queue and the second error queue are cleared to support the subsequent import of time series data.

[0103] Step S307: Repeat the steps of obtaining the import queue and the first error queue based on the time series data to be stored through the preset processing and verification method, until the import queue and the second error queue are cleared, until the time series dataset to be stored is stored in the time series database.

[0104] Specifically, continue repeating steps S301 to S307 until the time series dataset to be stored is empty and the import queue is also empty, indicating that all the obtained time series datasets to be stored have been stored in the corresponding time series database.

[0105] The time-series data storage method provided in this embodiment stores the time-series data to be stored in an import queue according to a preset length, and then stores it in the time-series database through the import queue. This prevents the time-series data from being loaded into memory all at once, ensuring that memory holds a maximum of the preset length of data at any given time. Therefore, memory overhead is low and does not increase with the amount of data. Furthermore, by classifying and merging the verification errors and storage errors of the time-series data to be stored, the storage response results are small and clear, solving the problem that returning the import results one by one in existing solutions leads to a very large result set.

[0106] In one example, a method for importing time-series data with low memory overhead and small response time is provided.

[0107] Specifically, such as Figure 4 As shown, to import 1 million rows of time-series data, follow these steps:

[0108] 1. Read a line of data and validate it, such as validating illegal characters and formatting. If the validation fails, add the line number and error type to the error queue; otherwise, add it to the import queue.

[0109] 2. If the length of the queue to be imported is 1000, then perform the data entry operation; otherwise, repeat step 1.

[0110] 3. Store the data to be imported into the time series database, and put the error information returned by the time series database, such as the row number and error type, into the error queue;

[0111] 4. Merge the line numbers and errors from steps 1 and 3, and clear the import queue and error queue;

[0112] 5. Repeat step 1;

[0113] 6. After reading to the end of the file, check the queue to be inserted. If it is not empty, execute steps 3 and 4.

[0114] Among them, such as Figure 5 As shown, the merging logic in step 4 above is as follows:

[0115] Iterate through the error queue, performing the following operations for each element i:

[0116] 1. Compare with the error code of the top element of the result stack;

[0117] 2. If the error codes are the same, the line number of the element is set to the termination line parameter of the top element of the result stack; otherwise, the error code is pushed onto the result stack, and the line number of the element is set to the start and termination line parameters of the top element of the result stack.

[0118] The result stack stores triples, with the structure [error code, start line, end line]. Initially, the stack is empty. The first line of the data file to be imported is the header, and a [ok, 1, 1] entry is pushed onto the stack. After each batch of data is imported, the following logic is executed:

[0119] Stack top [ok, 7, 10]

[0120] [Measurement point does not exist, 4, 6]

[0121] The bottom of the stack is [ok, 1, 3].

[0122] Here, the starting line refers to the line number where an error begins to occur, and the ending line refers to the last line where this error occurs consecutively. For example, suppose there are 10 lines of data in total, lines 1 to 3 are normal data, lines 4 to 6 are errors where the measurement point does not exist, and lines 7 to 10 are normal data. Then the starting line for the error "the measurement point does not exist" is line 4, and the ending line is line 6.

[0123] The time-series data import method provided in this example does not load all the imported data into memory at once; a maximum of one thousand rows of data can exist in memory at a time, thus minimizing memory overhead and not increasing with the amount of data. Furthermore, this solution categorizes and merges the response results based on error codes, resulting in concise and clear response output.

[0124] This embodiment also provides a time-series data storage device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0125] This embodiment provides a time-series data storage device, which can be used in the aforementioned time-series database, such as... Figure 6 As shown, the device includes:

[0126] The first acquisition module 601 is used to acquire the time series dataset to be stored.

[0127] The processing module 602 is used to obtain the import queue and the first error queue based on the time-series data to be stored through a preset processing and verification method.

[0128] The first storage module 603 is used to store the time-series data stored in the queue to be imported into the time-series database based on the first error queue when the queue length of the queue to be imported is equal to the preset length.

[0129] The first clearing processing module 604 is used to clear the import queue and the first error queue.

[0130] The first repeating and storage module 605 is used to repeat the steps from obtaining the import queue and the first error queue based on the time series data to be stored through a preset processing and verification method, to clearing the import queue and the first error queue, until the time series dataset to be stored is stored in the time series database.

[0131] In some alternative implementations, the processing module 602 includes:

[0132] The read and verify submodule is used to read the time series dataset to be stored line by line in sequence and to verify each line of data read.

[0133] The first storage submodule is used to store the verified time-series data sequentially and form a queue to be imported.

[0134] The second storage submodule is used to store the row numbers and error types of the time-series data that failed the verification in sequence and form the first error queue.

[0135] In some alternative embodiments, the device further includes:

[0136] The second acquisition module is used to acquire the error information set when the time-series database returns error information.

[0137] The merging module is used to obtain the second error queue based on the error information set and the first error queue through a classification and merging method.

[0138] The second storage module is used to store the queue to be imported into the time series database based on the second error queue.

[0139] The second clearing module is used to clear the import queue and the second error queue.

[0140] The second repeat and storage module is used to repeat the steps from obtaining the import queue and the first error queue based on the time series data to be stored through a preset processing and verification method, to clearing the import queue and the second error queue, until the time series dataset to be stored is stored in the time series database.

[0141] In some alternative implementations, the merging module includes:

[0142] The sorting submodule is used to sort the error message set according to the line number to obtain the third error queue.

[0143] The comparison and determination submodule is used to compare the error code of each element in the third error queue with the error code of the top element in the first error queue in turn, and determine the second error queue based on the comparison results.

[0144] In some optional implementations, the comparison and determination submodule includes:

[0145] The assignment unit is used to assign the row number of the first element to the termination row parameter of the top element of the stack when the error code of the first element in the third error queue is equal to the error code of the top element of the stack in the first error queue.

[0146] The first traversal and repetition unit is used to traverse each element in the third error queue in turn, and repeat the step of assigning the row number of the first element to the termination row parameter of the stack top element when the error code of the first element in the third error queue is equal to the error code of the stack top element in the first error queue, until the second error queue is obtained.

[0147] In some optional implementations, the comparison and determination submodule further includes:

[0148] The addition and assignment unit is used to add the error code of the first element to the first error queue when the error code of the first element in the third error queue is not equal to the error code of the top element of the stack in the first error queue, and to assign the line number of the first element to the start line parameter and end line parameter of the top element of the stack.

[0149] The second traversal and repetition unit is used to traverse each element in the third error queue in turn, and repeat the steps of adding the error code of the first element to the first error queue and assigning the row number of the first element to the start row parameter and end row parameter of the stack top element when the error code of the first element in the third error queue is not equal to the error code of the top element in the first error queue, until the second error queue is obtained.

[0150] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0151] In this embodiment, the timing data storage device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0152] This invention also provides a computer device having the above-described features. Figure 6 The time-series data storage device shown.

[0153] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 7 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 7 Take a processor 10 as an example.

[0154] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0155] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0156] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0157] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0158] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0159] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0160] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A time-series data storage method, characterized in that, For use in time-series databases; the method includes: Obtain the time series dataset to be stored; Based on the time-series data to be stored, after a preset processing and verification method, an import queue and a first error queue are obtained. The first error queue stores the row number and error type of the time-series data that failed verification. When the queue length of the queue to be imported is equal to the preset length, the time series data stored in the queue to be imported is stored in the time series database based on the first error queue. The queue to be imported and the first error queue are cleared. The process repeats the steps of obtaining the import queue and the first error queue based on the time series data to be stored through a preset processing and verification method, until the import queue and the first error queue are cleared, until the time series dataset to be stored is stored in the time series database. When the time series dataset to be stored is empty and the import queue is also empty, it means that all the obtained time series datasets to be stored have been stored in the corresponding time series database. Wherein, when the queue length of the queue to be imported is equal to a preset length, after storing the time-series data stored in the queue to be imported into the time-series database, the method further includes: When the time-series database returns error information, the error information set is obtained; Based on the error information set and the first error queue, a second error queue is obtained through a classification and merging method; The queue to be imported and the second error queue are cleared. The process of repeatedly processing and verifying the time series data to be stored using a preset method to obtain the import queue and the first error queue, and then clearing the import queue and the second error queue, continues until the time series dataset to be stored is stored in the time series database. Based on the error information set and the first error queue, a second error queue is obtained through a classification and merging method, including: Get the initial result stack; The error information set is stored in the first error queue, and a third error queue is obtained; The elements in the third error queue are sorted according to their row numbers to obtain the fourth error queue; The error code of each element in the fourth error queue is compared with the error code of the top element in the initial result stack, and the second error queue is determined based on the comparison result.

2. The method according to claim 1, characterized in that, Based on the time-series data to be stored, after processing and verification using a preset method, an import queue and a first error queue are obtained, including: The time-series dataset to be stored is read sequentially, line by line, and each line of data read is verified. The verified time-series data are stored sequentially and formed into the import queue; The row numbers and error types of the time-series data that failed the verification are stored sequentially to form the first error queue.

3. The method according to claim 1, characterized in that, The error code of each element in the fourth error queue is compared sequentially with the error code of the top element in the initial result stack, and the second error queue is determined based on the comparison results, including: When the error code of the first element in the fourth error queue is not equal to the error code of the top element in the initial result stack, the error code of the first element is added to the initial result stack, and the row number of the first element is assigned to the start row parameter and end row parameter of the top element. The process involves iterating through each element in the fourth error queue and repeating the steps of adding the error code of the first element to the initial result stack and assigning the row number of the first element to the start row parameter and end row parameter of the top element of the stack when the error code of the first element in the fourth error queue is not equal to the error code of the top element of the initial result stack, until the second error queue is obtained.

4. The method according to claim 1, characterized in that, The process further includes sequentially comparing the error code of each element in the fourth error queue with the error code of the top element in the initial result stack, and determining the second error queue based on the comparison results. When the error code of the first element in the fourth error queue is equal to the error code of the top element in the initial result stack, the row number of the first element is assigned to the termination row parameter of the top element. The process involves iterating through each element in the fourth error queue and repeating the step of assigning the row number of the first element to the termination row parameter of the top element of the stack when the error code of the first element in the fourth error queue is equal to the error code of the top element of the initial result stack, until the second error queue is obtained.

5. A time-series data storage device, characterized in that, Used to perform the time-series data storage method according to any one of claims 1-4; The device is used for a time-series database; the device includes: The first acquisition module is used to acquire the time series dataset to be stored. The processing module is used to obtain an import queue and a first error queue based on the time-series data to be stored through a preset processing and verification method. The first error queue stores the row number and error type of the time-series data that failed verification. The first storage module is used to store the time-series data stored in the queue to be imported into the time-series database based on the first error queue when the queue length of the queue to be imported is equal to a preset length. The first clearing processing module is used to clear the queue to be imported and the first error queue; The first repeating and storage module is used to repeat the steps of obtaining an import queue and a first error queue based on the time series data to be stored through a preset processing and verification method, up to the step of clearing the import queue and the first error queue, until the time series dataset to be stored is stored in the time series database. When the time series dataset to be stored is empty and the import queue is also empty, it means that all the obtained time series datasets to be stored have been stored in the corresponding time series database.

6. The apparatus according to claim 5, characterized in that, The processing module includes: The read and verify submodule is used to read the time series dataset to be stored line by line in sequence, and to verify each line of data read. The first storage submodule is used to store the verified time-series data sequentially and form the queue to be imported. The second storage submodule is used to store the row numbers and error types of the time-series data that failed the verification in sequence and form the first error queue.

7. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the timing data storage method according to any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the timing data storage method according to any one of claims 1 to 4.

Images