Neural network-based time sequence library multi-thread rapid import and export method

CN118964467AActive Publication Date: 2024-11-15GUODIAN NANJING AUTOMATION

Patent Information

Application Number
CN202411184492.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-15
Estimated Expiration
2044-08-27

AI Technical Summary

Technical Problem

The existing InfluxDB timing database has problems such as excessive resource consumption, high complexity, reduced performance and insufficient scalability in data backup, import and export, and it is difficult to meet the needs of large-scale data processing and complex conditional query.

Method used

Using a multi-threaded rapid import and export method based on neural networks, processing time sequence data through sharding, using a nonlinear fitting model to predict the time efficiency of import and export and resource usage status, optimize the import and export parameter configuration, and combine encryption technology and hash algorithm to ensure data Integrity and resource allocation.

Benefits of technology

It improves the efficiency of importing and exporting timing libraries, adapts to different operating environments and task requirements, reduces system resource consumption and complexity, and improves the flexibility and scalability of data processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964467A_ABST
    Figure CN118964467A_ABST
Patent Text Reader

Abstract

The invention discloses a time sequence library multi-thread rapid import and export method based on a neural network, and relates to the field of data import and export, and the method comprises the following steps: carrying out fragmentation processing on time sequence data based on a preset definition rule, and determining a thread processing rule according to a fragmentation processing result to execute time sequence library import and export operation; obtaining business data corresponding to the import and export operation, extracting statistical characteristics of the business data, and combining the statistical characteristics into a characteristic data set to reflect the performance and load condition of the time sequence data manager; and constructing a nonlinear fitting model by using a neural network on the basis of the feature data set, and predicting import and export time efficiency and resource occupation conditions by using the nonlinear fitting model. According to the method, a multi-thread self-defined fragmentation mode is adopted, time sequence data is divided into a plurality of fragments according to self-defined rules such as a time range, the multiple fragments are processed at the same time through multiple threads, and import and export operation is conducted on a time sequence library.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data import and export, and in particular to a multi-threaded fast import and export method of a timing library based on a neural network. Background Art

[0002] With the rapid expansion of Internet services, the deployment of large-scale storage systems and computing systems faces increasingly severe challenges. Effective monitoring and analysis of such a large scale of machines, systems, and services has become a very challenging task. In order to meet this challenge, the industry has continuously proposed various solutions, including the time series data management system (TSMS) with time series database as the core. These systems are designed to optimize the storage and analysis of time series data and make up for the shortcomings of traditional database management systems in this area. Although relational database management systems have performed well in past applications, they are not suitable for processing the massive time series data generated by various monitoring systems.

[0003] InfluxDB is an open source time series database written in GO language. It is particularly suitable for processing and analyzing resource monitoring data. It provides many special functions, such as standard deviation calculation, random data sampling, and data change ratio statistics, making it very convenient to perform statistics and real-time analysis on time series data.

[0004] However, because InfluxDB consumes a lot of memory and CPU resources, and the minimum unit of its backup operation is a single database, the granularity of the backup operation is too large to meet the requirements of filtering backup of specific resource tables. At the same time, the resources required for read and write operations or their combination during the backup process are not reasonably restricted, which may lead to resource contention and affect the normal operation of InfluxDB and other components.

[0005] For example, Chinese patent CN111611104B discloses an InfluxDB data backup method, apparatus and terminal device, including: receiving external instructions and loading initialization configuration information; using the initialization configuration information to pull resources from the resource library, judging the occupancy rate of the resources in the resource library and issuing execution instructions to the read operation module and the write operation module in combination with the initialization configuration information; the read operation module initiates a data backup request to InfluxDB, and pulls backup data corresponding to the backup request from InfluxDB; the write operation module writes the execution information formed by the backup data into the database in the form of a log file.

[0006] The above data backup method performs read and write operations on the InfluxDB time series library by receiving external instructions and loading initialization configuration information to issue execution instructions to the read operation module and the write operation module. However, since it involves interactions and operations in multiple aspects such as external instructions, initialization configuration information, resource library, and backup information table, the complexity of the system is increased, and the operation and maintenance are difficult. In addition, a large amount of system resources will be consumed when judging resource occupancy and performing data backup operations, which may cause system performance to deteriorate, especially when backing up large-scale data.

[0007] For example, Chinese patent CN115048390B discloses a data storage method and device based on InfluxDB, which includes: determining target processing data based on the read configuration information combined with the device configuration table, and constructing key data based on the target processing data; determining the scheduled key information in the key data based on the method parameters of the scheduled task, and determining the business data based on the scheduled key information; encapsulating the business data into preset objects, and converting all preset objects within a preset time period into the InfluxDB data format and storing them in the InfluxDB database.

[0008] The above data storage method constructs business data by combining configuration information with the device configuration table, and utilizes the high throughput and high compression rate of the influxdb database to improve the query efficiency during aggregate query operations. However, due to its reliance on the characteristics of the Influxdb database, this method lacks universality, and in the process of determining the target processing data and constructing the key data, a large amount of data processing and processing is required, which makes it not flexible enough for queries that require complex conditions or across multiple data sources.

[0009] For example, Chinese patent CN108897796B discloses a method, storage medium and server for a business system to call an influxdb database. The method constructs a processing logic on the server side where the influxdb database is located. When a data call request from the business system is obtained, the processing logic is used to call the original data table of the influxdb database, and the original data table is aggregated to obtain the target data table required by the business system. The business system can then directly call the data in the target data table.

[0010] The above data call method builds processing logic, receives data call requests from the business system, and calls the original data table of the InfluxDB database according to the request, and directly aggregates the called original data table in the processing logic. However, this will cause the real-time performance of the business system in obtaining data to decrease. In the case of high concurrency or large data volume, the processing logic may become the performance bottleneck of the system, and the processing logic may need to be constantly modified and adjusted due to changes in business needs, and lacks scalability and flexibility.

[0011] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0012] In order to solve the above problems, the present invention proposes a multi-threaded fast import and export method of a timing library based on a neural network, so as to achieve the purpose of adapting to different operating environments and task requirements.

[0013] In order to achieve the above object, the present invention adopts the following technical solution:

[0014] The present invention provides a multi-threaded fast import and export method for a timing library based on a neural network, and the multi-threaded fast import and export method for a timing library comprises the following steps:

[0015] Slice the time series data based on the preset definition rules, and determine the thread processing rules according to the slicing processing results to execute the time series library import and export operations;

[0016] Obtain the business data corresponding to the import and export operations, extract the statistical features of the business data, and merge the statistical features into a feature data set to reflect the performance and load of the time series data manager;

[0017] Based on the feature data set, a nonlinear fitting model is constructed using a neural network, and the nonlinear fitting model is used to predict the import and export time efficiency and resource usage;

[0018] Optimize the parameter configuration when importing and exporting the timing library according to the prediction results, and adjust the thread processing rules based on the optimization results.

[0019] Preferably, performing sharding processing on the time series data based on a preset definition rule, and determining a thread processing rule to perform the import and export operation of the time series library according to the sharding processing result includes the following steps:

[0020] Divide the time series data into slices according to the timestamp, use encryption technology to establish the initial checksum for each group of slices, and implement resource allocation according to the thread processing rules to determine the export process;

[0021] After establishing a monitoring thread to track the thread status based on the thread processing rules, the slice is regarded as a separate task and the thread is started to process the slice to execute the export task;

[0022] Store the exported time series data in the memory buffer, use data import technology and thread processing rules to import the exported data into the database, and use the monitoring thread to track the thread status during the import process;

[0023] After the import work is completed, encryption technology is used to calculate the checksum of the imported data, and the initial checksum is compared with the checksum of the imported data, and the data integrity is verified based on the comparison result.

[0024] Preferably, the time series data is divided into slices according to the timestamp, an initial checksum is established for each group of slices using encryption technology, and resource allocation is implemented according to the thread processing rule to determine the export process, including the following steps:

[0025] Divide the time series data into slices based on the date timestamp and device ID;

[0026] Design a data storage virtual space using hash algorithm and associative mapping, improve the data scrambling rules in the virtual space, and build a time series data verification mechanism based on the scrambling rules;

[0027] Use the time series data encryption mechanism to calculate the checksum for the slices, and store the checksum of each group of slices in the checksum file;

[0028] Limit the amount of data in a single query through paging query, and set the corresponding export buffer, thread pool and asynchronous queue according to the thread processing rules.

[0029] Preferably, a data storage virtual space is designed by using a hash algorithm and an associative mapping, and a scrambling rule of data in the virtual space is improved. A time series data verification mechanism is constructed based on the scrambling rule, including the following steps:

[0030] Use the hash algorithm to access the sharded time series data to calculate the data hash value, and select the data storage server in the data virtual space in order of the data hash value;

[0031] In the data virtual space, a data storage server is defined as a hash volume, and the sharded time series data is stored according to the location of the data storage server;

[0032] Design data virtual storage space according to the storage capacity of the data storage server, and use the principle of spatial association mapping to divert the sharded time series data to generate data virtual storage space;

[0033] The sharded time series data in the data virtual storage space is analyzed, and based on the analysis results, the data virtual storage space is split to build a time series data verification mechanism.

[0034] Preferably, analyzing the fragmented time series data in the data virtual storage space, and splitting the data virtual storage space based on the analysis result to construct a time series data verification mechanism includes the following steps:

[0035] Conduct confidentiality analysis on the time series data after sharding in the data virtual storage space, and split the data virtual storage space into planes according to the analysis results and splitting principles;

[0036] According to the splitting results and the confidentiality of the time series data, the scrambling rules for data distribution in the data virtual storage space are set, and the scrambling rules are reset and adjusted to obtain new scrambling rules;

[0037] A prime number generator is used to generate strong prime numbers to determine the modulus of the public key and the private key in the data verification mechanism, and then the Euler function is calculated, and the encryption mechanism of the time series data is defined according to the calculation results.

[0038] Preferably, the expression of the new scrambling rule is:

[0039]

[0040] Where W D (X) represents the scrambling degree of the total amount of time series data X after sharding, D represents the data virtual storage space, T represents the original scrambling rule of the data virtual storage space, M represents the confidentiality degree of the time series data after sharding, x and y represent the horizontal and vertical coordinates of the storage nodes of the time series data in the data virtual storage space after sharding, a and b represent the size of the plane split, r m Indicates the mapping control factor of the time series data after the mth shard in the data virtual storage space, where m represents the number of slices.

[0041] Preferably, after establishing a monitoring thread to track the thread state based on the thread processing rule, treating the slice as a separate task and starting the thread processing slice to execute the export task includes the following steps:

[0042] Based on the thread processing rules, a monitoring thread is established to track the thread status in real time, and a failover function is defined to check the thread status results;

[0043] After treating the slice as a separate task, start the thread method to process the slice data to perform the export task, and regularly update the thread status during the export process;

[0044] Analyze whether there are abnormal conditions in the export process according to the thread status check result of the failover function, and use the failover function to find and identify the corresponding failed thread when an abnormal condition occurs;

[0045] Restart the failed thread to execute the export task and check the thread status again. If there is no abnormality in the thread status, continue to run.

[0046] If an abnormal situation occurs in the thread state, the export task of the failed thread will be transferred to the idle thread to ensure the continued execution of the export task.

[0047] Preferably, establishing a monitoring thread to track the status of a thread in real time based on a thread processing rule and defining a failover function to check the thread status result includes the following steps:

[0048] Determine the monitoring indicators of thread status based on thread processing rules, and use the polling mechanism to set the monitoring mechanism of monitoring threads, and use the monitoring software development platform and thread processing rules to establish thread running status animation simulation;

[0049] The thread status and monitoring indicators are used as monitoring input data, the relationship between the monitoring input data and the state animation simulation is established, and the corresponding monitoring mechanism when the relationship result is optimal is selected as the final monitoring thread. The thread execution function is defined to regularly update the thread status when the thread executes the task;

[0050] After defining the thread execution function, define the failover function and set the failover function to check the thread status regularly.

[0051] Preferably, obtaining business data corresponding to the import and export operations, extracting statistical features of the business data, and merging the statistical features into a feature data set to reflect the performance and load of the time series data manager includes the following steps:

[0052] Use the command line tool corresponding to the time series data manager to obtain the corresponding processor usage and memory usage during import and export operations, and use the performance monitoring tool to obtain disk read and write speed data;

[0053] Extract network transmission speed based on network traffic monitoring tools, and obtain import and export parameters from the time series data manager input;

[0054] Processor usage, memory occupancy, disk read and write speed data, network transmission speed and import and export parameters are used as business data corresponding to the target node;

[0055] Calculate the maximum, minimum and average values ​​based on business data, and use the maximum, minimum and average values ​​as statistical features;

[0056] Combine statistical features into feature data sets, and use feature data sets to reflect the performance and load of the time series data manager.

[0057] Preferably, building a nonlinear fitting model using a neural network based on a feature data set, and using the nonlinear fitting model to predict the import and export time efficiency and resource occupancy status includes the following steps:

[0058] The work efficiency and resource usage load in the import and export process are set as influencing parameters, and the influencing parameters are combined with the feature data set to obtain the training set as the input of the neural network, and the import and export time efficiency and resource usage status are defined as the output;

[0059] After determining the number of nodes in the input layer, hidden layer, and output layer, initialize the weights and biases of the neural network, pass the training set into the neural network through the input layer, and perform linear transformation using the weights and biases through the hidden layer;

[0060] Based on the linear change results, the activation function is used to make nonlinear changes. After the training set reaches the output layer, the predicted output of the neural network is obtained to obtain the import and export time efficiency and resource occupancy status;

[0061] The loss function is used to calculate the difference between the predicted output and the actual data to obtain the loss value. The error gradient of the neuron is calculated based on the loss value to determine the weight and bias update value.

[0062] The gradient descent algorithm is used to correct the weights and biases of the neural network, and the neural network structure corresponding to the optimal weights and biases is selected as the final nonlinear fitting model. The nonlinear fitting model is used to predict the import and export time efficiency and resource occupancy.

[0063] The beneficial effects of the present invention are:

[0064] The present invention adopts a multi-threaded custom sharding method, divides the time series data into multiple shards according to custom rules such as time range, uses multi-threading to process multiple shards simultaneously, imports and exports the time series library, and generates a training set with the import and export parameters and import and export results. The neural network model is used to optimize the import and export parameters of the time series library, improve the import and export efficiency, and adapt to different operating environments and task requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0066] Figure 1 This is a flow chart of a multi-threaded fast import and export method of a timing library based on a neural network according to an embodiment of the present invention;

[0067] Figure 2 This is a schematic diagram of the import and export process in a multi-threaded fast import and export method for a timing library based on a neural network according to an embodiment of the present invention;

[0068] Figure 3 It is a schematic diagram of BP neural network training in a multi-threaded fast import and export method of a timing library based on a neural network according to an embodiment of the present invention;

[0069] Figure 4 The invention discloses a neural network prediction model data flow diagram in a multi-threaded fast import and export method of a timing library based on a neural network according to an embodiment of the invention. DETAILED DESCRIPTION

[0070] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0071] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0072] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0073] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0074] See also Figure 1 The present invention provides a multi-threaded fast import and export method for a timing library based on a neural network, and the multi-threaded fast import and export method for a timing library includes the following steps:

[0075] S1. Slice the time series data based on the preset definition rules, and determine the thread processing rules according to the slicing processing results to execute the time series library import and export operations.

[0076] In this embodiment, when the time series data is sliced ​​based on the preset definition rules and the thread processing rules are determined according to the slice processing results to execute the time series library import and export operations, the time series data can be divided into slices according to the timestamp, and the initial checksum is established for each group of slices using encryption technology, and the export process is determined according to the thread processing rules. After the monitoring thread is established based on the thread processing rules to track the thread status, the slice is regarded as a separate task to start the thread processing slice to execute the export task; the exported time series data is stored in the memory buffer, and the exported data is imported into the database using data import technology and thread processing rules, and the monitoring thread is used to track the thread status during the import process; after the import work is completed, the encryption technology is used to calculate the checksum of the imported data, and the initial checksum is compared with the checksum of the imported data, and the data integrity is verified according to the comparison result.

[0077] See also Figure 2 It should be explained that when performing the export work, the time series data is first divided into multiple slices according to the date timestamp and device ID, and the amount of data for a single query is limited through paging query to facilitate parallel processing. Then, each slice is treated as a task through multi-threading, and multiple threads are started to process multiple slice export tasks at the same time. The exported data is first stored in the memory buffer and written to the file after reaching the set data capacity.

[0078] When executing the import process, define data files, perform partition optimization, classify data files according to date and time stamps, set the maximum data capacity of a single file to 20,000, and allocate sufficient resources for write operations, including buffers, thread pools, asynchronous queues, etc. The buffer is a memory area that temporarily stores data to balance the speed difference between data producers and consumers. A buffer is established in the timing library from reading data from files to writing data to avoid frequent resource application and release during the writing process and reduce resource competition during the writing process.

[0079] Both thread pools and asynchronous queues are used to efficiently utilize multiple threads. Thread pools manage multiple threads to avoid frequent creation and destruction of threads, thereby improving the system's response speed and resource utilization. Asynchronous queues provide a series of non-blocking operation interfaces to improve the system's concurrency and corresponding performance. The buffer size calculation formula is as follows:

[0080] buffersize=datatransrate×transtime;

[0081] In the formula, buffersize represents the buffer size, datatransrate represents the data transmission rate, and transition represents the transmission time.

[0082] In order to reduce the frequency of write operations, a batch write method is adopted. First, the data in a single file is uniformly read into the memory, and classified and packaged according to the date and time stamp and data device ID. Then the packaged data is written into the buffer. After all the data is successfully written into the buffer, it is uniformly submitted and written into the time series library.

[0083] When performing write operations, a multi-threaded approach is used. First, the compressed file is decompressed, and the data in the file is parsed and then segmented. The granularity is the date and timestamp. Each segment is read and imported by a thread. The number of database connections is pre-set, and the child threads are managed through the thread pool, which can significantly improve the concurrency and efficiency of writing.

[0084] Specifically, the time series data is divided into slices according to the timestamp, the encryption technology is used to establish an initial checksum for each group of slices, and the resource allocation is implemented according to the thread processing rules to determine the export process. The time series data can be divided into slices according to the date and time stamp and the device identification; the data storage virtual space is designed by using the hash algorithm and the association mapping, and the scrambling rules of the data in the virtual space are improved, and the time series data verification mechanism is constructed based on the scrambling rules; the time series data encryption mechanism is used to calculate the checksum for the slices, and the checksum of each group of slices is stored in the verification file; the amount of data for a single query is limited by paging query, and the corresponding export buffer, thread pool and asynchronous queue are set according to the thread processing rules.

[0085] Among them, the data storage virtual space is designed by using hash algorithm and associative mapping, and the scrambling rules of data in the virtual space are improved. When the time series data verification mechanism is constructed based on the scrambling rules, the hash algorithm can be used to access the sharded time series data to calculate the data hash value, and the data storage server is selected in sequence in the data virtual space according to the data hash value; the data storage server is defined as a hash volume in the data virtual space, and the sharded time series data is stored according to the location of the data storage server; the data virtual storage space is designed according to the storage capacity of the data storage server, and the space associative mapping principle is used to divert the sharded time series data to generate a data virtual storage space; the sharded time series data in the data virtual storage space is analyzed, and the data virtual storage space is split based on the analysis results to construct a time series data verification mechanism.

[0086] Among them, when analyzing the sharded time series data in the data virtual storage space and splitting the data virtual storage space based on the analysis results to construct a time series data verification mechanism, the confidentiality level of the sharded time series data in the data virtual storage space can be analyzed, and the data virtual storage space can be plane-split according to the analysis results and the splitting principles; the scrambling rules for data distribution in the data virtual storage space are set according to the splitting results and the confidentiality level of the time series data, and the scrambling rules are reset and adjusted to obtain new scrambling rules; a prime number generator is used to generate a strong prime number to determine the modulus of the public key and the private key in the data verification mechanism, and the encryption mechanism of the time series data is defined according to the calculation results.

[0087] Specifically, the expression of the new scrambling rule is:

[0088]

[0089] Where W D(X) represents the scrambling degree of the total amount of time series data X after sharding, D represents the data virtual storage space, T represents the original scrambling rule of the data virtual storage space, M represents the confidentiality degree of the time series data after sharding, x and y represent the horizontal and vertical coordinates of the storage nodes of the time series data in the data virtual storage space after sharding, a and b represent the size of the plane split, r m Indicates the mapping control factor of the time series data after the mth shard in the data virtual storage space, where m represents the number of slices.

[0090] It should be explained that in order to ensure the accuracy and integrity of the data import and export process, a data integrity verification step is designed. The goal of data integrity verification is to ensure that the data remains unchanged during the transmission and storage process from the source to the target file, and thus the data needs to be verified before exporting. Specifically, the time series data encryption mechanism can be combined with MD5 or SHA-256. The verification and calculation method is explained using the MD5 code as an example. The specific steps are as follows:

[0091] Step 1. Pad the data slice: To ensure that the data length (in bits) modulo 512 is 448, that is, the data length after padding is 64 bits (8 bytes) less than 512, the MD5 algorithm adds padding bits at the end of the data, that is, first adds a "1" bit, and then adds several "0" bits until the data length modulo 512 is 448, and then adds length bits, that is, a 64-bit integer, which represents the data length before padding.

[0092] Step 2: Initialize the MD buffer: The MD5 algorithm uses four 32-bit registers (A, B, C, D) to store intermediate and final results. Then, initialize the registers to the following constants:

[0093] A=0x67452301, B=0xEFCDAB89, C=0x98BADCFE, D=0x10325476;

[0094] Step 3, process the data block: divide the padded data into small blocks of 512 bits (64 bytes), process each data block in turn, and each data block undergoes the following operations: copy the values ​​of the four registers (A, B, C, D) to another variable; perform four rounds of operations on each data block, each round using a different nonlinear function (F, G, H, I) and constant table (T) to mix the 16 32-bit sub-blocks in the data block with the values ​​in the four registers; add the operation results back to the initial registers (A, B, C, D).

[0095] Step 4. Output result: After processing all data blocks, the values ​​in the four registers (A, B, C, D) are concatenated to form the final 128-bit hash value.

[0096] Then, after the data is imported, the data integrity is checked to ensure that the imported data is consistent with the exported data. The specific steps are as follows:

[0097] Step 1: Recalculate the checksum for the imported data slice.

[0098] Step 2: Compare the recalculated checksum with the checksum stored before exporting.

[0099] Step 3: If the checksums are consistent, the data has not been tampered with and the import is successful. If the checksums are inconsistent, the data may have been tampered with during transmission or storage and needs to be re-imported or troubleshooted.

[0100] Among them, after the monitoring thread is established based on the thread processing rules to track the thread status, the slice is regarded as a separate task to start the thread processing slice to execute the export task. A monitoring thread can be established based on the thread processing rules to track the thread status in real time, and a fault transfer function can be defined to check the thread status result; after treating the slice as a separate task, the slice data is started in a thread manner to execute the export task, and the thread status is regularly updated during the export process; according to the thread status check result of the fault transfer function, whether there is an abnormal condition in the export process is analyzed, and when an abnormal condition occurs, the fault transfer function is used to find and identify the corresponding failed thread; restart the failed thread to execute the export task, and judge the thread status again. If there is no abnormal condition in the thread status, continue to run; if there is an abnormal condition in the thread status, transfer the export task of the failed thread to the idle thread to ensure the continued execution of the export task.

[0101] Specifically, when the monitoring thread is established based on the thread processing rules to track the status of the thread in real time and a fault transfer function is defined to check the thread status result, the monitoring index of the thread status can be determined based on the thread processing rules, and the monitoring mechanism of the monitoring thread can be set using the polling mechanism, and the thread running status animation simulation can be established using the monitoring software development platform and the thread processing rules; the thread status and the monitoring index are used as monitoring input data, and the relationship between the monitoring input data and the status animation simulation is established, and the corresponding monitoring mechanism when the relationship result is optimal is selected as the final monitoring thread, and the thread execution function is defined to periodically update the thread status when the thread executes a task; after the thread execution function is defined, a fault transfer function is defined, and the fault transfer function is set to periodically check the thread status.

[0102] It should be explained that in order to ensure the accuracy and integrity of the data import and export process, a failover step needs to be designed. A monitoring thread needs to be established in advance to track the status of each thread in real time. This monitoring thread uses a shared data structure to store the status and task information of each thread. At this time, it is necessary to ensure that this shared data structure is thread-safe to ensure that data competition does not occur when multiple threads access it at the same time. A thread execution function needs to be defined to regularly update the status of the thread when it executes a task. This function should contain the following logic:

[0103] (1) Set the thread state to "running" during initialization;

[0104] (2) Execute data import or export operations;

[0105] (3) Periodically update the thread status to “running” during task execution;

[0106] (4) If the task is successfully completed, update the status to "Completed";

[0107] (5) If an exception occurs, capture the exception and update the thread status to "failed", and record the error information.

[0108] After defining the thread execution function, you also need to define a failover function to periodically check the status of all threads. The failover function contains the following logic:

[0109] (1) Regularly check the status of all threads;

[0110] (2) If a thread status is found to be "failed", a failover operation is performed:

[0111] Find and identify the task slice processed by the failed thread;

[0112] Try to restart the thread and re-execute its task slice;

[0113] If the restart fails, the task of the failed thread is transferred to an idle thread to ensure that the task continues to be executed.

[0114] (3) Update the thread status in the monitoring system to ensure that the status of the restarted thread or the newly assigned thread correctly reflects the current progress.

[0115] S2. Obtain the business data corresponding to the import and export operations, extract the statistical features of the business data, and merge the statistical features into a feature data set to reflect the performance and load of the time series data manager.

[0116] In this embodiment, when obtaining the business data corresponding to the import and export operations, extracting the statistical features of the business data, and merging the statistical features into a feature data set to reflect the performance and load of the time series data manager, the command line tool corresponding to the time series data manager can be used to obtain the processor usage and memory occupancy corresponding to the import and export operations, and the performance monitoring tool can be used to obtain the disk read and write speed data; the network transmission speed is extracted based on the network traffic monitoring tool, and the import and export parameters are obtained from the time series data manager input; the processor usage, memory occupancy, disk read and write speed data, network transmission speed and import and export parameters are used as the business data corresponding to the target node; the maximum, minimum and average values ​​are calculated based on the business data, and the maximum, minimum and average values ​​are used as statistical features; the statistical features are merged into a feature data set, and the feature data set is used to reflect the performance and load of the time series data manager.

[0117] It needs to be explained that in the process of establishing the CPU processing and I / O operation model, data collection is carried out. The data collection phase is designed to obtain data on the target node related to CPU usage, memory usage, disk read and write speeds, network transmission speeds, and export data granularity, import data granularity, file data capacity, number of database connections, thread pool parameters, etc.

[0118] The parameters for designing target node performance are obtained by using system performance monitoring tools or directly accessing the interfaces provided by the operating system. For example, tools such as top and free are used to obtain data on CPU usage and memory occupancy. iostat and iotop are used to obtain disk read and write speed data. nload and iftop are used to obtain network transmission speed data. For import and export related parameters, they are directly obtained from software input.

[0119] Specifically, in computer system management, top is a commonly used command line tool for displaying a dynamic real-time view of running processes, and the free command is used to display the total amount of free and used physical memory and swap space in the system, as well as kernel buffer and cache memory.

[0120] iostat is a powerful tool for monitoring system disk I / O statistics. It can display disk read / write operations, amount of data transferred, average transfer rate, and waiting time. iotop is a tool similar to top that focuses on displaying real-time I / O usage, sorting by I / O usage of processes or threads, and displaying the I / O operations they are performing and related disk activities.

[0121] nload is a network traffic monitoring tool that can display the inbound and outbound traffic of a network interface in real time. iftop is a command line tool for monitoring network interface traffic in real time.

[0122] The second is the feature extraction stage, which aims to extract useful features from the processed data. For indicators such as CPU usage, memory usage, disk read and write speed, and network transmission speed, various statistical features are extracted, including average, maximum, minimum, etc. These features can reflect the performance and load of the system and provide an important information basis for building the model. The formulas for average, maximum, and minimum are as follows:

[0123]

[0124] Maximum=max(x1,x2,Λ,x n );

[0125] Minimum=min(x1,x2,Λ,x n );

[0126] In the formula, x i represents the i-th sample in the data set, and n represents the number of samples.

[0127] The feature data set obtained through data collection and feature extraction is used to establish a neural network model for CPU processing and I / O operations, helping the system to perform intelligent resource allocation and task scheduling according to real-time environment and task requirements to optimize system performance and resource utilization.

[0128] S3. Use neural networks to build a nonlinear fitting model based on feature data sets, and use the nonlinear fitting model to predict import and export time efficiency and resource usage.

[0129] In this embodiment, when a nonlinear fitting model is constructed using a neural network based on a feature data set and the nonlinear fitting model is used to predict the import and export time efficiency and resource occupancy status, the work efficiency and resource occupancy load during the import and export execution process can be set as influencing parameters, and the influencing parameters are combined with the feature data set to obtain a training set as the input of the neural network, and the import and export time efficiency and resource occupancy status are defined as the output; after determining the number of nodes in the input layer, the hidden layer and the output layer, the weights and biases of the neural network are initialized, and the training set enters the neural network through the input layer and passes through the hidden layer to perform a linear transformation using the weights and biases; based on the linear change result, a nonlinear change is performed using an activation function, and after the training set reaches the output layer, the predicted output of the neural network is obtained to obtain the import and export time efficiency and resource occupancy status; the loss function is used to calculate the difference between the predicted output and the actual data to obtain the loss value, and the error gradient of the neuron is calculated based on the loss value to determine the weight and bias update value; the gradient descent algorithm is used to correct the weight and bias of the neural network, and the neural network structure corresponding to the optimal weight and bias is selected as the final nonlinear fitting model, and the nonlinear fitting model is used to predict the import and export time efficiency and resource occupancy status.

[0130] It should be explained that the parameters that need to be set during the import and export process include export data granularity, import data granularity, file data capacity, number of database connections, thread pool parameters, etc. These parameters will directly affect the efficiency of import and export and the load on system resources. However, in actual situations, the impact of these parameters on efficiency and the system presents a nonlinear trend. Therefore, the BP neural network model can be used to tune the import and export performance, and the parameters that meet actual needs can be selected through the nonlinear fitting ability of the neural network.

[0131] See also Figure 3 , BP neural network is a feedforward network trained based on the error back propagation algorithm, which includes input layer, output layer and hidden layer. In each layer, there are many neurons that process the signal transmitted from the previous layer through the activation function. After the deviation is calculated by the loss function, the gradient search technology is used for back propagation, and finally the error between the output value and the target value is minimized. The BP neural network model includes three basic elements: weight W, bias b and activation function f. If the input vector is X, the hidden layer vector is: Z = f1 (W1X + b1), and the output layer vector is Y = f2 (W2Z + b2).

[0132] See also Figure 4The five groups of parameters, namely, export data granularity, import data granularity, file data capacity, number of database connections, and thread pool parameters, are taken as input, and the import and export time efficiency and system resource occupancy are output. By adjusting different parameter configurations, the corresponding import and export time efficiency and system resource occupancy are obtained after import and export. The data is preprocessed, including statistical feature processing, to obtain the training set of the neural network model.

[0133] Construct a neural network model, predetermine the number of nodes in the input layer, hidden layer, and output layer. The number of input layers is temporarily set to the number of parameters mentioned above, that is, 5 layers. The number of nodes in the hidden layer and output layer is adjusted according to the actual situation, and the activation function and loss function are determined. The hidden layer selects the ReLU activation function and the mean square error loss function. The specific formula is expressed as follows:

[0134] f(x)=max(0,x);

[0135]

[0136] In the formula, x represents the input data of the input layer, represents the predicted output of the model, Y i represents the actual output, m represents the number of training samples, J represents the mean square error loss value, and f represents the activation function value.

[0137] The neural network model consists of three basic elements: weight W, bias b and activation function f. The specific design of the neural network training model with multiple hidden layers has the following calculation process:

[0138] Step 1: Initialize the neural network parameters: the input layer has 5 neurons and the output layer has 4 neurons;

[0139] Step 2: Calculate the output data of the hidden layer: Combine the input variable x, the connection parameters of the hidden layer, and the activation function f to obtain the hidden layer data Z, as follows:

[0140]

[0141] Where d represents the index of the last input point involved in the summation process. represents the weight from the i-th input node to the j-th hidden layer node, x i represents the input data of the input layer, b j represents the bias of the jth hidden layer node, Z j Represents the weighted sum from the input layer to the hidden layer after being processed by the activation function.

[0142] Step 3: Output layer data calculation: Based on the connection parameters between the hidden layer and the output layer and the calculated Z value, the model output system efficiency and resource usage can be calculated as follows:

[0143]

[0144] In the formula, y k represents the weighted sum from the hidden layer to the output layer after being processed by the activation function, b k represents the bias of the kth output layer node, and n represents the number of nodes in the hidden layer;

[0145] Represents the jth hidden layer node to the kth output layer node

[0146] Step 4: Estimate the loss function using the mean square error loss function.

[0147] Step 5: Weight update: When updating weights, the weight parameters are modified using the gradient descent method. In each iteration of the gradient descent algorithm, the parameter W of the lth layer is (l) The update method of and b(l) is:

[0148]

[0149]

[0150] In the formula, η represents the learning rate, R(W, b) represents the regularization term of weight and bias, represents the error derivative, N represents the number of samples in the training data set, λ represents the regularization coefficient, It represents the error between the predicted value and the actual value of the nth sample.

[0151] After the neural network training is completed, a set of optimal weights and biases are finally obtained, which is the trained neural network model. Based on the artificial neural network method, which has powerful nonlinear fitting functions in modeling, a multi-hidden layer error back propagation neural network structure is used to construct a prediction model for the import and export parameters, efficiency and resource occupancy of the time series library. The trained model can dynamically adjust the efficiency and resource occupancy of import and export according to the user's actual operating environment, flexibly respond to various complex environments, and meet the different import and export task requirements of customers.

[0152] S4. Optimize the parameter configuration when importing and exporting the timing library according to the prediction results, and adjust the thread processing rules based on the optimization results.

[0153] In summary, with the help of the above technical solution of the present invention, the present invention adopts a multi-threaded custom slicing method, by dividing the time series data into multiple slices according to custom rules such as time range, using multiple threads to process multiple slices at the same time, importing and exporting the time series library, and generating a training set with the import and export parameters and the import and export results, optimizing the import and export parameters of the time series library through a neural network model, and finding the optimal parameter configuration for the current operating environment, so that the import and export process is adapted to the current environment, saving time and system resources. The present invention adopts a multi-threaded custom slicing method, custom slices the data according to the timestamp range, sets the export multi-thread granularity to the time range slice, and sets the import multi-thread granularity to the file, and the data capacity of a single file can be set, and the read and write efficiency can be improved through parallel means, and the data device identifier can be defined as an index condition at the same time, and the query granularity can be further reduced in a single thread, and the neural network model is used to predict the import and export results of the user's operating environment, find the relatively optimal import and export parameters, and improve the import and export efficiency.

[0154] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in conjunction with this embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0155] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A multi-threaded fast import and export method for timing library based on neural network, characterized in that: The timing library multi-threaded fast import and export method includes the following steps: Slice the time series data based on the preset definition rules, and determine the thread processing rules according to the slicing processing results to execute the time series library import and export operations; Obtain the business data corresponding to the import and export operations, extract the statistical features of the business data, and merge the statistical features into a feature data set to reflect the performance and load of the time series data manager; Based on the feature data set, a nonlinear fitting model is constructed using a neural network, and the nonlinear fitting model is used to predict the import and export time efficiency and resource usage; Optimize the parameter configuration when importing and exporting the timing library according to the prediction results, and adjust the thread processing rules based on the optimization results.

2. According to the neural network-based timing library multi-threaded fast import and export method of claim 1, it is characterized in that: The method of performing sharding processing on the time series data based on the preset definition rules and determining the thread processing rules to perform the import and export operations of the time series library according to the sharding processing results includes the following steps: Divide the time series data into slices according to the timestamp, use encryption technology to establish the initial checksum for each group of slices, and implement resource allocation according to the thread processing rules to determine the export process; After establishing a monitoring thread to track the thread status based on the thread processing rules, the slice is regarded as a separate task and the thread is started to process the slice to execute the export task; Store the exported time series data in the memory buffer, use data import technology and thread processing rules to import the exported data into the database, and use the monitoring thread to track the thread status during the import process; After the import work is completed, encryption technology is used to calculate the checksum of the imported data, and the initial checksum is compared with the checksum of the imported data, and the data integrity is verified based on the comparison result.

3. According to the neural network-based timing library multi-threaded fast import and export method of claim 2, it is characterized in that: The process of dividing the time series data into slices according to the timestamp, establishing an initial checksum for each group of slices by using encryption technology, and implementing resource allocation according to the thread processing rules to determine the export process includes the following steps: Divide the time series data into slices based on the date timestamp and device ID; Design a data storage virtual space using hash algorithm and associative mapping, improve the data scrambling rules in the virtual space, and build a time series data verification mechanism based on the scrambling rules; Use the time series data encryption mechanism to calculate the checksum for the slices, and store the checksum of each group of slices in the checksum file; Limit the amount of data in a single query through paging query, and set the corresponding export buffer, thread pool and asynchronous queue according to the thread processing rules.

4. According to the neural network-based timing library multi-threaded fast import and export method of claim 3, it is characterized in that: The method of designing a data storage virtual space by using a hash algorithm and an associative mapping, improving the scrambling rules of data in the virtual space, and constructing a time series data verification mechanism based on the scrambling rules includes the following steps: Use the hash algorithm to access the sharded time series data to calculate the data hash value, and select the data storage server in the data virtual space in order of the data hash value; In the data virtual space, a data storage server is defined as a hash volume, and the sharded time series data is stored according to the location of the data storage server; Design data virtual storage space according to the storage capacity of the data storage server, and use the principle of spatial association mapping to divert the sharded time series data to generate data virtual storage space; The sharded time series data in the data virtual storage space is analyzed, and based on the analysis results, the data virtual storage space is split to build a time series data verification mechanism.

5. According to the neural network-based timing library multi-threaded fast import and export method of claim 4, it is characterized in that: The analyzing of the fragmented time series data in the data virtual storage space and the splitting of the data virtual storage space based on the analysis result to construct a time series data verification mechanism comprises the following steps: Conduct confidentiality analysis on the time series data after sharding in the data virtual storage space, and split the data virtual storage space into planes according to the analysis results and splitting principles; According to the splitting results and the confidentiality of the time series data, the scrambling rules for data distribution in the data virtual storage space are set, and the scrambling rules are reset and adjusted to obtain new scrambling rules; A prime number generator is used to generate strong prime numbers to determine the modulus of the public key and the private key in the data verification mechanism, and then the Euler function is calculated, and the encryption mechanism of the time series data is defined according to the calculation results.

6. The method for fast multi-threaded import and export of timing libraries based on neural networks according to claim 5 is characterized in that: The expression of the new scrambling rule is: Where W D (X) represents the scrambling degree of the total amount of time series data X after sharding, D represents the data virtual storage space, T represents the original scrambling rule of the data virtual storage space, M represents the confidentiality degree of the time series data after sharding, x and y represent the horizontal and vertical coordinates of the storage nodes of the time series data in the data virtual storage space after sharding, a and b represent the size of the plane split, r m Indicates the mapping control factor of the time series data after the mth shard in the data virtual storage space, where m represents the number of slices.

7. The neural network-based time series library multi-threaded fast import and export method according to claim 6, characterized in that: After the monitoring thread is established based on the thread processing rule to track the thread status, the slice is regarded as a separate task and the thread is started to process the slice to execute the export task, which includes the following steps: Based on the thread processing rules, a monitoring thread is established to track the thread status in real time, and a failover function is defined to check the thread status results; After treating the slice as a separate task, start the thread method to process the slice data to perform the export task, and regularly update the thread status during the export process; Analyze whether there are abnormal conditions in the export process according to the thread status check result of the failover function, and use the failover function to find and identify the corresponding failed thread when an abnormal condition occurs; Restart the failed thread to execute the export task and check the thread status again. If there is no abnormality in the thread status, continue to run. If an abnormal situation occurs in the thread state, the export task of the failed thread will be transferred to the idle thread to ensure the continued execution of the export task.

8. The method for fast multi-threaded import and export of timing libraries based on neural networks according to claim 7 is characterized in that: The method of establishing a monitoring thread based on thread processing rules to track the status of a thread in real time and defining a failover function to check the status of a thread includes the following steps: Determine the monitoring indicators of thread status based on thread processing rules, and use the polling mechanism to set the monitoring mechanism of monitoring threads, and use the monitoring software development platform and thread processing rules to establish thread running status animation simulation; The thread status and monitoring indicators are used as monitoring input data, the relationship between the monitoring input data and the state animation simulation is established, and the corresponding monitoring mechanism when the relationship result is optimal is selected as the final monitoring thread. The thread execution function is defined to regularly update the thread status when the thread executes the task; After defining the thread execution function, define the failover function and set the failover function to check the thread status regularly.

9. The method for fast multi-threaded import and export of timing libraries based on neural networks according to claim 1, characterized in that: The process of obtaining the business data corresponding to the import and export operations, extracting the statistical features of the business data, and merging the statistical features into a feature data set to reflect the performance and load of the time series data manager includes the following steps: Use the command line tool corresponding to the time series data manager to obtain the corresponding processor usage and memory usage during import and export operations, and use the performance monitoring tool to obtain disk read and write speed data; Extract network transmission speed based on network traffic monitoring tools, and obtain import and export parameters from the time series data manager input; Processor usage, memory occupancy, disk read and write speed data, network transmission speed and import and export parameters are used as business data corresponding to the target node; Calculate the maximum, minimum and average values ​​based on business data, and use the maximum, minimum and average values ​​as statistical features; Combine statistical features into feature data sets, and use feature data sets to reflect the performance and load of the time series data manager.

10. The method for fast multi-threaded import and export of timing libraries based on neural networks according to claim 1, characterized in that: The method of constructing a nonlinear fitting model using a neural network based on a feature data set and predicting the import and export time efficiency and resource occupancy using the nonlinear fitting model includes the following steps: The work efficiency and resource usage load in the import and export process are set as influencing parameters, and the influencing parameters are combined with the feature data set to obtain the training set as the input of the neural network, and the import and export time efficiency and resource usage status are defined as the output; After determining the number of nodes in the input layer, hidden layer, and output layer, initialize the weights and biases of the neural network, pass the training set into the neural network through the input layer, and perform linear transformation using the weights and biases through the hidden layer; Based on the linear change results, the activation function is used to perform nonlinear changes. After the training set reaches the output layer, the predicted output of the neural network is obtained to obtain the import and export time efficiency and resource occupancy status; The loss function is used to calculate the difference between the predicted output and the actual data to obtain the loss value. The error gradient of the neuron is calculated based on the loss value to determine the weight and bias update value. The gradient descent algorithm is used to correct the weights and biases of the neural network, and the neural network structure corresponding to the optimal weights and biases is selected as the final nonlinear fitting model. The nonlinear fitting model is used to predict the import and export time efficiency and resource occupancy.

Citation Information

Patent Citations

  • A method, storage medium, and server for a business system to access an InfluxDB database.

    CN108897796B

  • An InfluxDB data backup method, system, and terminal device

    CN111611104B

  • Data storage method and device based on InfluxDB

    CN115048390B

  • A data multi-thread export method and system

    CN106407231A

  • Tool wear state detection method and apparatus based on convolutional neural network and device

    CN109048492A

Cited By

  • Method and system for processing multi-thread data in sequence in frequency regulation control

    CN119025284A

  • Query method and device for multi-source time series data

    CN120872963A

  • Database data importing method and device, electronic equipment and storage medium

    CN120950487A

  • Database data import method and device, electronic equipment and storage medium

    CN120950487B

  • Relational database query method, system and equipment supporting fault tolerance and storage medium

    CN121277977A