Real-time storage management method for mass dynamic data

Abstract

The invention discloses a real-time storage management method for a large amount of dynamic data, and relates to the technical field of data management, and the method comprises the steps: carrying out the real-time analysis of a dynamic data flow through a storage configuration matrix by using a pulse neural network, generating a data feature vector, and dividing the cold and hot categories of the data through an intelligent classifier, meanwhile, optimization is carried out according to a compression strategy in the storage configuration matrix, a preprocessed data packet is output, intelligent routing distribution is carried out according to a coding label of the preprocessed data packet, hot data is routed to a high-speed storage layer, and cold data is routed to a capacity storage layer; executing data storage and index construction in each storage layer through a plastic queue management mechanism, and generating a storage position mapping table; according to the method, the pulse distribution rate is calculated through the pulse neural network to generate the spatio-temporal characteristics, and the spatio-temporal characteristics are input into the gradient boosting tree model to execute weighted voting to divide cold and hot categories, so that deep spatio-temporal analysis and intelligent classification of dynamic data streams are realized.

Description

Technical Field

[0001] This invention relates to the field of data management technology, and in particular to a method for real-time storage management of large amounts of dynamic data. Background Technology

[0002] Due to the rapid development of big data and IoT technologies, the demand for dynamic data generation and processing has exploded. Dynamic data, characterized by high frequency, high concurrency, and diversity, is widely used in important fields such as real-time log analysis, IoT sensor data processing, financial transaction systems, and intelligent traffic monitoring. Existing methods include distributed databases and time-series databases, which achieve large-scale data storage and management through sharding and distributed architectures. Meanwhile, stream processing frameworks provide efficient pipelines for the real-time ingestion and processing of dynamic data. Furthermore, memory-based caching technologies and hybrid indexing mechanisms have significantly improved data access speed.

[0003] However, there are still some areas for improvement in existing storage management methods. First, when dealing with dynamic data, there is a lack of in-depth analysis of the spatiotemporal characteristics of data streams, resulting in low accuracy in classifying hot and cold data. Moreover, existing compression strategies are basically fixed algorithms or single thresholds, making it difficult to dynamically adjust them according to data characteristics. Second, data routing mechanisms rely heavily on static rules and lack adaptability, failing to effectively cope with changes in data access patterns, further leading to latency bottlenecks in high-concurrency scenarios. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a real-time storage management method for large amounts of dynamic data to solve the problem of low accuracy in classifying hot and cold data due to a lack of in-depth analysis of the spatiotemporal characteristics of data streams.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a method for real-time storage management of large amounts of dynamic data, comprising: Construct a tiered storage architecture and divide storage resources into a high-speed storage layer and a capacity storage layer, generating a storage configuration matrix; The dynamic data stream is analyzed in real time using a spiking neural network through a storage configuration matrix to generate data feature vectors. The data is then divided into hot and cold categories by an intelligent classifier. At the same time, the compression strategy in the storage configuration matrix is ​​used for optimization, and preprocessed data packets are output. Intelligent routing is performed based on the encoded tags of the preprocessed data packets, routing hot data to the high-speed storage layer and cold data to the capacity storage layer. Within each storage layer, data storage and index construction are performed through a flexible queue management mechanism to generate a storage location mapping table. A pulse-triggered query interface is established based on the storage location mapping table. Query requests are parsed and routed for optimization. Performance indicators are monitored in real time and storage configuration matrix parameters are dynamically adjusted to complete real-time storage management.

[0007] As a preferred embodiment of the real-time storage management method for large amounts of dynamic data described in this invention, the step of constructing a hierarchical storage architecture and dividing storage resources into a high-speed storage layer and a capacity storage layer, and generating a storage configuration matrix, specifically involves: The performance of the storage media is probed, and read / write latency, throughput and persistence parameters are collected to obtain a set of performance indicators. The performance score of each storage medium is calculated, and the performance thresholds of the high-speed storage layer and the capacity storage layer are set. Based on performance scores, high-speed storage layer performance thresholds, and capacity storage layer performance thresholds, high-speed storage layer and capacity storage layer are divided, and the available capacity ratio of each storage layer is calculated and storage resource quotas are allocated. A storage configuration matrix is ​​established by dividing and allocating storage resource quotas at the storage layer.

[0008] As a preferred embodiment of the real-time storage management method for large amounts of dynamic data described in this invention, the step of using a spiking neural network to perform real-time analysis of the dynamic data stream and generate data feature vectors specifically involves... Receive dynamic data streams, preprocess them into standardized data packets, and extract timestamps, data size, and access frequency to generate initial feature vectors; The data routing rules are retrieved from the storage configuration matrix, and the real-time requirement weights of the initial feature vector are determined and then input into the spiking neural network. The spiking neural network is used to calculate the firing rate of each neuron and generate temporal dynamic features and spatial distribution features. The temporal dynamic features and spatial distribution features are weighted and fused to generate a spatiotemporal feature vector, which is then normalized to obtain a standardized data feature vector.

[0009] As a preferred embodiment of the real-time storage management method for large amounts of dynamic data described in this invention, the method involves: classifying data into hot and cold categories using an intelligent classifier, specifically... The system receives standardized data feature vectors, extracts time dynamic features and access frequency features as classification criteria, and generates hot and cold data classification thresholds based on the data routing rules. A rule set is constructed using the hot and cold data classification threshold. The rule set is then input into a pre-trained gradient boosting tree model to generate standardized hot and cold classification probabilities for data feature vectors. Weighted voting is then performed to determine the final hot and cold categories. Assign coded labels, including hot data identifiers and cold data identifiers, to the final hot / cold categories.

[0010] As a preferred embodiment of the real-time storage management method for large amounts of dynamic data described in this invention, the method involves: optimizing according to the compression strategy in the storage configuration matrix and outputting preprocessed data packets, specifically... Compression strategy parameters are extracted from the storage configuration matrix. The compression strategy parameters include the compression algorithm type and the compression ratio threshold. Based on the compression strategy parameters and the encoding label, the compression algorithm type is selected and compression is performed to generate compressed data blocks. Calculate the compression ratio and entropy value of the compressed data block, verify whether the compression ratio is less than the compression ratio threshold, and generate compressed metadata if the threshold is met. Compressed metadata is appended to the compressed data block, and a preprocessed data packet is generated after verifying its integrity.

[0011] As a preferred embodiment of the real-time storage management method for large amounts of dynamic data described in this invention, the step of intelligent routing allocation based on the encoded tags of preprocessed data packets specifically involves: Parse the preprocessed data packets and determine the storage layer priority based on the routing rules in the storage configuration matrix; Based on routing rules and encoding labels, priority scores are calculated to determine the target storage layer. According to the target storage layer, data packets are preprocessed into fragments to generate data fragments. The data fragments are then sent to the target storage layer through a distributed router to complete the routing allocation.

[0012] As a preferred embodiment of the real-time storage management method for large amounts of dynamic data described in this invention, the step of performing data storage and index construction through a plastic queue management mechanism to generate a storage location mapping table specifically involves: Extract the storage layer identifier and IOPS metric from the storage configuration matrix, and initialize the plastic queue; Data sent by the distributed router is fragmented, received, and stored in a plastic queue. The plastic queue buffer and priority are adjusted according to the IOPS metric. Using the adjusted queue, perform batch writes, generate storage data blocks, and construct an index matrix; Based on the index matrix, a storage location mapping table is generated by recording physical addresses and timestamps.

[0013] As a preferred embodiment of the real-time storage management method for large amounts of dynamic data described in this invention, the step of establishing a pulse-triggered query interface based on a storage location mapping table specifically involves: The physical address and index information of the storage location mapping table are parsed using the storage layer identifier; Using the physical address and index information of the storage location mapping table, initialize the query thread pool and configure a pulse triggering mechanism to listen for query requests, thus completing the construction of the pulse-triggered query interface.

[0014] As a preferred embodiment of the real-time storage management method for large amounts of dynamic data described in this invention, the real-time monitoring of performance indicators and dynamic adjustment of storage configuration matrix parameters specifically includes: Based on performance metrics, calculate storage layer resource utilization and bottleneck coefficient to determine the storage layer that needs adjustment. Update the weights and IOPS metrics of the storage configuration matrix by adjusting the storage layer that needs adjustment.

[0015] As a preferred embodiment of the real-time storage management method for large amounts of dynamic data described in this invention, the construction of the index matrix refers to extracting the logical identifier, storage layer identifier, and storage offset of each stored data block, constructing the row records of the index matrix, grouping the row records according to the storage layer identifier, and generating the index matrix.

[0016] The beneficial effects of this invention are as follows: By calculating the pulse firing rate using a spiking neural network to generate spatiotemporal features and inputting them into a gradient boosting tree model to perform weighted voting to classify hot and cold data into categories, deep spatiotemporal analysis and intelligent classification of dynamic data streams are achieved. This captures temporal dependence and spatial patterns, avoids misjudgments caused by a single threshold, and ultimately improves classification accuracy and reduces the latency of cold storage of hot data. In addition, by calling the compression strategy parameter selection algorithm based on the encoded label and verifying the compression ratio and entropy value to generate preprocessed data packets, dynamic adjustment of compression features is achieved, and hot and cold data are processed differently. This approach is superior to a fixed algorithm, achieving the goal of saving space and reducing transmission consumption. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a method for real-time storage management of large amounts of dynamic data.

[0019] Figure 2 A flowchart for spatiotemporal feature analysis and classification.

[0020] Figure 3 A flowchart for intelligent routing and storage management.

[0021] Figure 4 A flowchart for adjustments to the query interface and monitoring. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 This is one embodiment of the present invention, which provides a real-time storage management method for large amounts of dynamic data, including the following steps: S1: Construct a tiered storage architecture and divide storage resources into a high-speed storage layer and a capacity storage layer, generating a storage configuration matrix.

[0026] The performance of storage media is probed by collecting read / write latency, throughput, and persistence parameters of non-volatile memory, high-speed solid-state drives, and large-capacity hard disk drives using standard benchmark tools, resulting in a performance metric set that includes read / write latency, throughput, and persistence parameters.

[0027] A weighted scoring method is used to calculate the performance score of each storage medium in the performance index set. The weights are determined based on the performance requirements of the storage medium in real-time storage management. For example, the comprehensive score is calculated with read / write latency accounting for 40%, throughput accounting for 40%, and persistence metrics accounting for 20% (the value range is 0-100 points), reflecting the priority of real-time performance and throughput. A performance threshold of 80 points is set for the high-speed storage layer and 50 points for the capacity storage layer. The high-speed storage layer performance threshold is a comprehensive score greater than 80 points. The scores of 80 and 50 are set based on the statistics of the performance distribution of storage media. As a high-speed storage layer threshold, 80 points can ensure the selection of high-performance media with a response time of less than 10ms, while a score below 80 points may lead to a latency increase of more than 20%, affecting real-time queries. Selecting 90 points would be too strict, resulting in insufficient available high-speed storage layer resources, causing resource waste and poor scalability. As a capacity storage layer threshold, 50 points can accommodate media with strong persistence, balancing cost and capacity.

[0028] Based on the comprehensive score of each storage medium, the performance threshold of the high-speed storage layer, and the performance threshold of the capacity storage layer, those with a comprehensive score greater than 80 are classified as high-speed storage layers, and those with a comprehensive score between 50 and 80 are classified as capacity storage layers.

[0029] Based on the division results of high-speed storage layer and capacity storage layer, the total available capacity of each storage layer is calculated, and storage resource quotas are allocated according to the total storage capacity. Based on the division results of high-speed storage layer and capacity storage layer and storage resource quotas, a storage configuration matrix is ​​constructed. The storage configuration matrix records the storage layer identifier, performance score, available capacity ratio, storage resource quota and IOPS index in row and column form, thus completing the establishment of the storage configuration matrix.

[0030] S2: Real-time analysis of dynamic data streams is performed using a spiking neural network through a storage configuration matrix, generating data feature vectors. Data is then categorized into hot and cold categories using an intelligent classifier. Simultaneously, optimization is performed based on the compression strategy in the storage configuration matrix, and preprocessed data packets are output.

[0031] S2.1: A spiking neural network is used to perform real-time analysis of the dynamic data stream and generate data feature vectors, specifically, Receive dynamic data streams, such as IoT sensor data streams, and remove invalid timestamps or missing values ​​from the dynamic data streams through data cleaning; divide the dynamic data streams into fixed time windows, such as generating a data packet every 1 second; extract the timestamp, data size, and access frequency of each data packet to generate an initial feature vector; perform Z-score normalization on the initial feature vectors to eliminate dimensional differences and generate standardized data packets.

[0032] Data routing rules are extracted from the storage configuration matrix, including storage layer identifiers, IOPS metrics, and priority weights. Real-time requirement parameters of the data routing rules are parsed, and real-time requirement weights are calculated using a weighted average method based on the IOPS metric and the access frequency of the initial feature vector. For example, a weighting method is used, with IOPS accounting for 60% of the weight and access frequency accounting for 40%, to generate real-time requirement weight values. The 60% and 40% weights are determined based on the priority of storage performance and data popularity in real-time storage management. The real-time requirement weight values ​​are then appended to standardized data packets to form a weighted feature vector.

[0033] Historical dynamic data streams, such as one month's worth of IoT sensor data, are collected to form a sample dataset containing timestamps, data size, and access frequency. A portion of this dataset is used as a test dataset. A three-layer spiking neural network is constructed using the Leaky-Integrate-and-Fire neuron model, consisting of an input layer (128 neurons), a hidden layer (256 neurons), and an output layer (64 neurons). Fully connected layers ensure complete information transmission. The sample dataset is divided into training data packets by time windows (1 second). Initial feature vectors are extracted and normalized for each training data packet, serving as training inputs. Based on the training inputs, supervised learning is used to adjust the neuron connection weights of the three-layer spiking neural network. Gradient descent (with a learning rate of 0.001, set to balance convergence speed and stability, avoiding excessive oscillations or excessively small learning times) is employed to minimize feature extraction errors based on the temporal and spatial distribution features of the historical dynamic data stream. The test dataset is then input into the three-layer spiking neural network. Training is considered complete when the feature extraction accuracy reaches 95% or higher, resulting in a pre-trained three-layer spiking neural network. The requirement for feature extraction accuracy to reach over 95% is based on the precision requirements for spatiotemporal feature extraction in real-time IoT storage scenarios. If it is below 95%, it may lead to amplified feature deviations, affecting the judgment of hot and cold features. While an accuracy above 98% may provide a slight improvement, it will increase the consumption of training resources.

[0034] Weighted feature vectors are input into a pre-trained three-layer spiking neural network. The input layer distributes the weighted feature vectors to hidden layer neurons. Each neuron updates its membrane potential based on the weighted feature vector. When the membrane potential exceeds a pulse threshold, a pulse is fired. The pulse threshold is set to 0.8. This 0.8 is based on the sensitivity requirements of the Leaky-Integrate-and-Fire neuron model. Setting it to 0.8 balances the real-time performance and stability of feature extraction, ensuring that the pulse firing rate reflects more than 80% of significant feature changes while filtering out noise. Setting it to 0.9 would be too strict, leading to a sparse pulse sequence and increased time loss. Dependency details affect the accuracy of spatiotemporal features, while values ​​below 0.7 indicate oversensitivity, generating noise pulses and increasing computational overhead, thus generating pulse sequences. The number of pulses per second for each hidden layer neuron in the pulse sequence is counted to obtain the pulse firing rate. Temporal dynamic features are extracted from the pulse firing rate. By calculating the difference in pulse firing rates within adjacent time windows, the pulse frequency change rate is generated to reflect the temporal dependence of the dynamic data stream, i.e., the temporal dynamic features. Spatial distribution features are extracted from the pulse firing rate to generate a spatial pattern vector containing the pulse firing rate of each hidden layer neuron. These vectors are combined to form a feature matrix containing both temporal dynamic features and spatial distribution features.

[0035] Priority weights are extracted from the storage configuration matrix, and temporal dynamic features and spatial distribution features are linearly combined to generate a 128-dimensional spatiotemporal feature vector, prioritizing responses to the time-varying needs of IoT scenarios. The spatiotemporal feature vector is then L2 normalized to ensure its magnitude is 1, eliminating dimensional differences and forming a standardized data feature vector. The stability of the standardized data feature vector is verified, for example, by checking if the magnitude is 1. Once verified, the standardized data feature vector is output to the intelligent classifier.

[0036] S2.2: Data is categorized into hot and cold categories using an intelligent classifier. Specifically, The system receives standardized data feature vectors and extracts time-dynamic features and requests per second (RPD) as the basis for hot and cold data classification. It then retrieves data routing rules from the storage configuration matrix and generates hot and cold data classification thresholds based on the real-time demand weights in these rules. These thresholds are also known as access frequency thresholds and time-dynamic feature thresholds. The access frequency threshold is set to 10 times / second, and the time-dynamic feature threshold is set to 0.5. The 10 times / second threshold is based on the statistical distribution of high-frequency access data in IoT scenarios, capturing 80% of high-frequency data to ensure rapid response. Setting it to 20 times / second would be too strict, leading to misclassification of hot data as cold and increasing response latency. A threshold below 5 times / second would excessively tolerate noisy accesses, wasting high-speed storage resources. The 0.5 threshold is set based on the real-time requirements of rapidly changing data, balancing sensitivity to change and stability to ensure the capture of significant dynamic patterns. A threshold of 0.7 would ignore the impact of moderate changes on classification accuracy, while a threshold below 0.3 would increase misclassification due to sensitive noise.

[0037] Based on the hot and cold data classification thresholds, a rule set is constructed. The rules can include "hot data" where "access frequency greater than the access frequency threshold and time dynamic feature greater than the time dynamic feature threshold" and "cold data" where "access frequency less than or equal to the access frequency threshold or time dynamic feature less than or equal to the time dynamic feature threshold". The rule set is then organized into a decision tree format, including condition nodes and classification results, to ensure that the rule set covers all input cases.

[0038] Historical IoT sensor data is collected to form a sample dataset containing timestamps, data size, and access frequency, which is then divided into training and test datasets. The training dataset is further divided into training data packets using a 1-second time window. Standardized feature vectors are extracted from each training data packet and used as input to a pre-trained gradient boosting tree model. Based on the real hot / cold category labels of the training dataset, a pre-trained gradient boosting tree model is constructed, for example, containing 500 decision trees. The weights of each decision tree are iteratively optimized using gradient descent. After minimizing the classification error using the standardized feature vectors from the training data packets, the test dataset is input into the pre-trained gradient boosting tree model to calculate the hot / cold classification probability. The calculation method involves each decision tree independently predicting the hot / cold category probability of the test sample, and then taking a weighted average of the predicted probabilities from all decision trees to obtain the hot / cold classification probability. When the classification accuracy reaches 95% or higher, the pre-trained gradient boosting tree model is output. The 95% accuracy is set based on empirical standards for gradient boosting trees in IoT hot / cold classification tasks. Accuracy below 95% may amplify probability bias and affect routing accuracy, while accuracy above 98% offers only a slight improvement but requires additional trees, increasing the complexity of the gradient boosting tree model.

[0039] The rule set and standardized data feature vectors are input into a pre-trained gradient boosting tree model. The pre-trained gradient boosting tree model evaluates the matching degree between the time dynamic features and access frequency features and the rule set through decision trees. Each decision tree outputs an initial hot / cold classification probability. The initial hot / cold classification probabilities of all decision trees are integrated, and voting is performed based on the weight of each tree to generate the final hot / cold classification probability. The weight of each tree is calculated based on the reciprocal of the classification error of the decision tree during training. The smaller the error, the higher the weight. An encoding label is assigned to each standardized data feature vector. If the final hot / cold category is hot data, a hot data label is assigned; if it is cold data, a cold data label is assigned. The consistency between the label and the classification probability is checked to ensure that the hot data label corresponds to high-probability hot data and the cold data label corresponds to high-probability cold data, resulting in an encoding label containing both hot and cold data labels.

[0040] S2.3: Optimize according to the compression strategy in the storage configuration matrix, and output preprocessed data packets, specifically, Extract compression strategy parameters from the storage configuration matrix, including compression algorithm type and compression ratio threshold; parse the encoded tags to determine the hot and cold categories of data packets; select a compression algorithm based on the compression strategy parameters and encoded tags, for example, select LZ4 for hot data identifiers to prioritize compression speed, and select Zstd for cold data identifiers to prioritize compression ratio; execute the selected compression algorithm on the input data packets to generate compressed data blocks.

[0041] Based on the compressed data blocks and the original data packets, the compression ratio is calculated by dividing the compressed data size by the original data size. For example, if the compressed size is 100KB and the original size is 150KB, the compression ratio is 0.667. The probability of occurrence of each byte value in the compressed data block is statistically analyzed, and the entropy value is calculated based on the distribution of the probability of occurrence. The effectiveness of the information density after data compression is evaluated by comparing the ratio of the entropy value after compression to the entropy value of the original data. If the entropy value after compression is less than 80% of the original entropy value, the compression is considered effective. This is because choosing 80% ensures high information density and sufficient redundancy removal. Values ​​other than 80% will result in over-compression or under-compression.

[0042] Verify whether the compression ratio meets the preset compression ratio threshold. If it does, generate compression metadata, which includes the compression algorithm identifier, compression time (in milliseconds), and original data size. To further explain, the compression ratio threshold is 0.7, which is set based on the compression strategy parameters in the storage configuration matrix and the need for a balance between storage efficiency and decompression speed in IoT scenarios. Statistical calculations show that 0.7 can ensure that compressed data blocks save storage space while maintaining fast decompression. Setting it to 0.8 would be too lenient, resulting in insufficient space saving and affecting efficiency, while a value below 0.6 would be too strict and may increase decompression time, affecting real-time performance. Meeting the compression ratio threshold means that the compression ratio is less than the compression ratio threshold.

[0043] Compressed metadata is appended to compressed data blocks to generate a combined structure containing metadata and compressed data, such as storing it in a format with metadata appended to the header; the integrity of compressed data blocks and compressed metadata is checked by a checksum algorithm, and if the check passes, a preprocessed data packet containing compressed data blocks and compressed metadata is output.

[0044] Preferably, traditional methods rely on static feature extraction or single access frequency analysis, which makes it difficult to capture spatiotemporal characteristics, resulting in low accuracy in hot and cold data classification and wasted storage resources. However, this invention utilizes a three-layer spiking neural network combined with the real-time requirement weights of the storage configuration matrix to generate high-precision standardized data feature vectors, improving the accuracy of hot and cold data classification. This is particularly suitable for the time-sensitive requirements of IoT scenarios. Moreover, compared to fixed compression strategies, this invention dynamically selects compression algorithms based on encoded tags and optimizes storage efficiency through compression ratio thresholds and information entropy verification, balancing space saving and decompression speed. In summary, this invention achieves higher real-time performance, classification accuracy, and resource utilization through spatiotemporal feature analysis, dynamic classification, and adaptive compression, providing an efficient solution for large-scale dynamic data management.

[0045] S3: Based on the encoded tags of the preprocessed data packets, intelligent routing is performed, routing hot data to the high-speed storage layer and cold data to the capacity storage layer. Within each storage layer, data storage and index construction are performed through a flexible queue management mechanism to generate a storage location mapping table.

[0046] S3.1: Intelligent route allocation is performed based on the encoded labels of the preprocessed data packets, specifically as follows: Parse the preprocessed data packets, extract the encoded tags, and determine the storage layer priority based on the priority weights in the routing rules of the storage configuration matrix. For example, the high-speed storage layer has a higher priority than the capacity storage layer. Priority is allocated based on IOPS.

[0047] Based on the storage layer identifier and encoding label in the routing rules, the priority score of the preprocessed data packet is calculated. Specifically, a priority score table containing the mapping between storage layer identifiers and encoding labels is pre-constructed using a lookup table method. For example, the hot data identifier corresponds to the high-speed storage layer and is mapped to a priority score of 1, while the cold data identifier corresponds to the capacity storage layer and is mapped to a priority score of 0. Based on the encoding label of the preprocessed data packet and the storage layer identifier in the routing rules, the priority score table is queried to directly obtain the corresponding priority score, and the priority score of the preprocessed data packet is generated. The priority score of the preprocessed data packet is compared with the storage layer priority to determine the target storage layer. The hot data identifier corresponds to the high-speed storage layer, and the cold data identifier 0 corresponds to the capacity storage layer.

[0048] Based on the storage capacity and fragmentation strategy of the target storage layer, the preprocessed data packets are divided into data fragments of a fixed size (1MB, based on the multiple expansion setting of the network MTU in the distributed storage tool) to optimize transmission efficiency; a target storage layer identifier is attached to each data fragment; the data fragments are sent to the corresponding high-speed storage layer or capacity storage layer through the distributed router according to the target storage layer identifier; and the received data fragments are checked using CRC32 to ensure that there is no loss or corruption, thus completing the routing allocation.

[0049] S3.2: Data storage and index construction are performed through a plastic queue management mechanism, generating a storage location mapping table, specifically, Extract storage layer identifiers and IOPS metrics from the storage configuration matrix; initialize a plastic queue based on the target storage layer and set the buffer size; receive data fragments sent by the distributed router and store them in the corresponding plastic queues; dynamically adjust the plastic queue buffer size and write priority based on the IOPS metrics, for example, increasing the buffer to 2GB when the IOPS metrics are higher than 50K; use the adjusted plastic queues to write data fragments in batches according to priority, generating storage data blocks; construct an index matrix based on the storage data blocks, recording the storage layer identifier, storage data block size, and physical storage address; extract the physical storage address and write timestamp from the index matrix to obtain a storage location mapping table, completing storage allocation.

[0050] Preferably, this invention achieves intelligent routing allocation by preprocessing the encoded tags of data packets, prioritizing hot data to high-speed storage layers to ensure fast access, and routing cold data to capacity storage layers to optimize space utilization. Within the storage layers, a flexible queue management mechanism is introduced to dynamically adjust buffers and write priorities, improving data storage throughput and resource adaptability. However, traditional routing methods often ignore the difference between hot and cold data, leading to uneven load and I / O bottlenecks. This invention, by combining routing rules of the storage configuration matrix and priority scores calculated using a lookup table, accurately matches the target storage layer, avoiding latency caused by blind allocation. Simultaneously, the flexible queue mechanism supports batch writing and index matrix construction, simplifying physical address tracking. The generated storage location mapping table further improves data retrieval efficiency and overall stability. In summary, this invention achieves greater storage flexibility and performance balance through the collaboration of routing and queues, providing a more reliable solution for dynamic big data management.

[0051] S4: Based on the storage location mapping table, a pulse-triggered query interface is established to parse and optimize the routing of query requests, monitor performance indicators in real time, and dynamically adjust the storage configuration matrix parameters to complete real-time storage management.

[0052] S4.1: Establish a pulse-triggered query interface based on the storage location mapping table, specifically as follows: The system extracts storage layer identifiers from the storage configuration matrix, parses the physical address and index information from the storage location mapping table, identifies the physical address as the specific location of the stored data block, and identifies the index information as the unique identifier of the stored data block. The parsed physical address and index information are used to preload the query path. A query thread pool is initialized using the extracted storage layer identifiers and the parsed physical address and index information, allocating thread resources to handle concurrent queries. A pulse triggering mechanism is configured using the initialized query thread pool to listen for query requests via an event queue, triggering thread wake-up when a query request arrives. Thread wake-up is correlated with physical address preloading to speed up response time. Pulse triggering is integrated. The mechanism and query thread pool utilize an event-driven framework to bind query request entry points, ensuring that storage layer identifiers directly guide thread scheduling and avoid resource waste. A pulse-triggered query interface is constructed by directly connecting to the index information of the storage location mapping table to optimize query routing paths, thus providing a real-time path foundation for subsequent scheduling. Based on the constructed pulse-triggered query interface, a scheduling strategy supporting real-time parsing is generated. Query requests are managed through a priority queue, and physical addresses are preloaded based on the index information of the storage location mapping table. Simultaneously, thread resources are dynamically allocated to implement parsing responses, completing the construction of the pulse-triggered query interface.

[0053] S4.2: Real-time monitoring of performance metrics and dynamic adjustment of storage configuration matrix parameters, specifically, Real-time acquisition of storage layer performance metrics, including throughput, query latency, and load metrics, is achieved through sensor interfaces from the high-speed storage layer and the capacity storage layer.

[0054] Based on the collected performance metrics, the storage layer resource utilization rate and bottleneck coefficient are calculated. The ratio is obtained by dividing the current throughput and IOPS metrics, and the ratio is also the resource utilization rate. The bottleneck coefficient is obtained by multiplying the query latency and the load metric.

[0055] By combining storage layer resource utilization and bottleneck coefficient, the storage layers that need adjustment are determined. For example, if the high-speed storage layer ratio is higher than 80% and the bottleneck coefficient is greater than 1, it is marked as an object that needs adjustment so as to prioritize the optimization of performance bottlenecks. The 80% setting is based on storage load balancing standards to avoid excessive resource consumption leading to increased latency. The setting of 1 is to quantify the boundary between normal and abnormal states. The combination of 80% and 1 can ensure timely intervention when the utilization is too high.

[0056] For storage layers that need adjustment, update the capacity allocation weights of the storage configuration matrix, such as increasing the weight of high-speed storage layers to guide more resource allocation, and adjusting IOPS metrics, such as to higher values ​​to improve processing capabilities, thereby completing dynamic adjustment and real-time storage management.

[0057] This embodiment also provides a computer device applicable to the real-time storage management method for large amounts of dynamic data, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the real-time storage management method for large amounts of dynamic data as proposed in the above embodiment.

[0058] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0059] This embodiment also provides a storage medium on which a computer program is stored. When executed by a processor, the program implements the real-time storage management method for large amounts of dynamic data as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0060] In summary, this invention achieves deep spatiotemporal analysis and intelligent classification of dynamic data streams by calculating the pulse firing rate using a spiking neural network to generate spatiotemporal features and inputting them into a gradient boosting tree model to perform weighted voting for hot and cold data classification. This captures temporal dependencies and spatial patterns, avoiding misjudgments caused by a single threshold, ultimately improving classification accuracy and reducing the latency of cold storage for hot data. Furthermore, by using an algorithm to select compression strategy parameters based on encoded tags and calling the storage configuration matrix, and verifying the compression ratio and entropy value to generate preprocessed data packets, this invention achieves dynamic adjustment of compression based on characteristics, providing differentiated processing for hot and cold data. This approach is superior to fixed algorithms, achieving the goals of saving space and reducing transmission consumption.

[0061] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for managing real-time storage of a large amount of dynamic data, characterized by: comprising, a hierarchical storage architecture is constructed, and storage resources are divided into a high-speed storage layer and a capacity storage layer to generate a storage configuration matrix; a dynamic data stream is analyzed in real time using a spiking neural network based on the storage configuration matrix to generate a data feature vector, and an intelligent classifier is used to divide data into hot and cold categories, and a compression strategy in the storage configuration matrix is used for optimization to output a preprocessed data packet; intelligent routing is performed according to an encoding label of the preprocessed data packet, hot data is routed to the high-speed storage layer, and cold data is routed to the capacity storage layer, and a plasticity queue management mechanism is used to perform data storage and index construction in each storage layer to generate a storage location mapping table; a spiking trigger type query interface is established based on the storage location mapping table, a query request is analyzed and routed, performance indicators are monitored in real time, and storage configuration matrix parameters are dynamically adjusted to complete real-time storage management.

2. The method for managing real-time storage of a large amount of dynamic data according to claim 1, wherein: The hierarchical storage architecture is constructed, and storage resources are divided into a high-speed storage layer and a capacity storage layer to generate a storage configuration matrix, specifically, performance detection is performed on a storage medium to collect read-write delay, throughput, and persistence index parameters to obtain a performance index set, and a performance score of each storage medium is calculated, and a high-speed storage layer performance threshold and a capacity storage layer performance threshold are set; based on the performance score, the high-speed storage layer performance threshold, and the capacity storage layer performance threshold, a high-speed storage layer and a capacity storage layer are divided, and the available capacity ratio of each storage layer is calculated to allocate a storage resource quota; a storage configuration matrix is established based on the storage resource quota allocated by the storage layer division and allocation.

3. The method for managing real-time storage of a large amount of dynamic data according to claim 2, wherein: The dynamic data stream is analyzed in real time using a spiking neural network to generate a data feature vector, specifically, a dynamic data stream is received, preprocessed into a standardized data packet, and a timestamp, data size, and access frequency are extracted to generate an initial feature vector; a data routing rule is called from the storage configuration matrix, the real-time requirement weight of the initial feature vector is determined, and then input into a spiking neural network, the spiking neural network is used to calculate the firing rate of each neuron to generate time dynamic features and spatial distribution features; the time dynamic features and the spatial distribution features are weighted and fused to generate a spatio-temporal feature vector, and normalized to obtain a standardized data feature vector.

4. The method for managing real-time storage of a large amount of dynamic data according to claim 3, wherein: The data is divided into hot and cold categories by an intelligent classifier, specifically, the standardized data feature vector is received, time dynamic features and access frequency features are extracted as classification criteria, and a hot and cold data classification threshold is generated based on the data routing rule; a rule set is constructed based on the hot and cold data classification threshold, the rule set is input into a pre-trained gradient boosting tree model to generate a hot and cold classification probability of the standardized data feature vector, and a weighted vote is performed to determine the final hot and cold category; an encoding label including a hot data identifier and a cold data identifier is assigned to the final hot and cold category.

5. The method for managing real-time storage of a large amount of dynamic data according to claim 4, wherein: The compression strategy in the storage configuration matrix is optimized to output a preprocessed data packet, specifically, compression strategy parameters including a compression algorithm type and a compression ratio threshold are extracted from the storage configuration matrix, a compression algorithm type is selected based on the compression strategy parameters and the encoding label, and compression is performed to generate a compressed data block; The compression ratio and entropy value of the compressed data block are calculated, and it is verified whether the compression ratio is less than the compression ratio threshold value, and if it is satisfied, the compressed metadata is generated; The compressed metadata is attached to the compressed data block, and the preprocessed data packet is generated after verifying the integrity.

6. The method for managing real-time storage of a large amount of dynamic data according to claim 5, wherein: The intelligent routing allocation is performed according to the encoding label of the preprocessed data packet, specifically, The preprocessed data packet is parsed, and the storage layer priority is determined through the routing rule in the storage configuration matrix; According to the routing rule and the encoding label, the priority score is calculated, the target storage layer is determined, and the preprocessed data packet is fragmented to generate data fragments, which are sent to the target storage layer through the distributed router to complete the routing allocation.

7. The method for managing real-time storage of a large amount of dynamic data according to claim 6, wherein: The data storage and index construction are performed through the plasticity queue management mechanism to generate a storage location mapping table, specifically, The storage layer identifier and IOPS indicator are extracted from the storage configuration matrix to initialize the plasticity queue; The data fragments sent by the distributed router are received and stored in the plasticity queue, and the plasticity queue buffer and priority are adjusted according to the IOPS indicator; Batch writing is performed using the adjusted queue to generate storage data blocks and construct an index matrix; According to the index matrix, the physical address and timestamp are recorded to generate a storage location mapping table.

8. The method for managing real-time storage of a large amount of dynamic data according to claim 7, wherein: The pulse trigger type query interface is established based on the storage location mapping table, specifically, The physical address and index information of the storage location mapping table are parsed through the storage layer identifier; The physical address and index information of the storage location mapping table are used to initialize the query thread pool and configure the pulse trigger mechanism to listen to the query request and complete the construction of the pulse trigger type query interface.

9. The method for managing real-time storage of a large amount of dynamic data according to claim 8, wherein: The performance indicators are monitored in real time and the storage configuration matrix parameters are dynamically adjusted, specifically, Based on the performance indicators, the storage layer resource utilization rate and bottleneck coefficient are calculated to determine the storage layer to be adjusted; The weight and IOPS indicator of the storage configuration matrix are updated through the storage layer to be adjusted.

10. The method for managing real-time storage of a large amount of dynamic data according to claim 9, wherein: The index matrix is constructed by extracting the logical identifier, storage layer identifier and storage offset of each storage data block, constructing the row record of the index matrix, grouping the row records according to the storage layer identifier, and generating the index matrix.

Images