Optical fiber data storage management system and method based on big data

Abstract

The invention discloses an optical fiber data storage management system and method based on big data, and relates to the technical field of storage management. According to the optical fiber data storage management system and method based on big data, the spatial-temporal characteristics of the historical access records are extracted through the LSTM network, the data heat value is calculated in combination with the time decay function, and the thermal distribution diagram of the data blocks is generated. And based on the thermal distribution map, constructing a storage resource allocation model by using a deep reinforcement learning algorithm, and dynamically adjusting the optical path priority and copy distribution. A data prefetching strategy is optimized through the spatio-temporal joint index and the vector similarity index, and a high-frequency data cache index distribution diagram is generated. And comprehensively considering the data access delay, the index hit rate and the copy migration frequency, obtaining an optical fiber storage efficiency index, and adjusting a storage strategy according to the index. According to the method, the dynamic scheduling and utilization efficiency of data storage resources is effectively improved, the data access delay is reduced, the copy management is optimized, and the overall performance of a storage system is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of storage management, and particularly to a fiber optic data storage management system and method based on big data. Background Art

[0002] With the advent of the big data era, the demand for data storage has shown an explosive growth. The traditional storage architecture is difficult to meet the growing high-throughput and low-latency access requirements. Fiber optic storage technology has been widely used in fields such as data centers, cloud computing, and edge computing due to its high bandwidth, low latency, and high reliability. However, in the face of complex data access patterns and dynamic load changes, there are still many challenges in resource management, data scheduling, and storage optimization in existing fiber optic storage systems. There is an urgent need for a more intelligent and dynamic management mechanism to improve storage efficiency and optimize data access performance.

[0003] Currently, fiber optic storage management mainly relies on traditional rule-based scheduling strategies, such as static replica allocation, fixed bandwidth allocation, and linear index management. These methods perform well when the data access pattern changes little, but in the face of large-scale concurrent access, uneven data heat and cold, sudden access hotspots, etc., they are prone to cause a decline in resource utilization and low storage efficiency. In addition, the existing index mechanism is usually based on static data structures such as B+ trees and hash indexes, which are difficult to adapt to complex query requirements, resulting in a decline in query efficiency. At the same time, the lack of an intelligent storage efficiency evaluation mechanism makes storage optimization rely on manual adjustment, with a slow response speed and unable to meet the requirements of efficient management. Therefore, there is an urgent need for a fiber optic storage management system based on big data analysis and intelligent optimization to improve data scheduling capabilities, optimize the index structure, and dynamically evaluate storage efficiency to ensure the high efficiency and stability of the system. Summary of the Invention

[0004] Aiming at the deficiencies of the prior art, the present invention provides a fiber optic data storage management system and method based on big data, which solves the problems in the above background art.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: A fiber optic data storage management system based on big data, comprising the following modules: a data access analysis module, a data scheduling module, an index optimization module, and a storage efficiency evaluation module; the data access analysis module is used to extract the spatio-temporal features of the historical access records of the fiber optic storage system through an LSTM network, calculate the data heat value in combination with a time decay function, and generate a heat distribution map identifying high-frequency and low-frequency data blocks; the data scheduling module is used to construct a storage resource allocation model through a deep reinforcement learning algorithm according to the heat distribution map, define the bandwidth utilization rate, node load rate, and predicted access volume as state space parameters, output an optical path priority matrix and a replica distribution topology instruction, and monitor the data flow fluctuation through a sliding window mechanism to trigger the migration of high-frequency data to low-load nodes; the index optimization module is used to construct a spatio-temporal joint index for structured data and a vector similarity index for unstructured data based on the heat distribution map and the replica distribution topology instruction, and optimize the prefetch strategy according to the data heat value to generate a high-frequency data cache index distribution map; the storage efficiency evaluation module is used to comprehensively consider the data access latency, index hit rate, and replica migration frequency to obtain a fiber optic storage efficiency index, and when the fiber optic storage efficiency index is lower than a preset threshold, generate a structure optimization instruction set and feedback it to the data scheduling module and the index optimization module.

[0006] Further, the specific process of extracting the spatio-temporal features of the historical access records of the fiber optic storage system through an LSTM network and calculating the data heat value in combination with a time decay function is as follows: Perform spatio-temporal encoding on the historical access records, extract the timestamp, geographical coordinates, and concurrent request volume, and construct a spatio-temporal feature vector; Model the time dimension features through a bidirectional LSTM network to capture the periodic pattern of data access, and at the same time extract the spatial dimension features through a convolutional network; Input the spatio-temporal features into a spatio-temporal attention mechanism to dynamically allocate time decay weights and generate a spatio-temporal correlation weight matrix; Input the spatio-temporal correlation weight matrix into the time decay function to dynamically decay the influence of historical data according to the time distance, and perform normalization processing on the decayed data to generate a globally comparable data heat value.

[0007] Further, the specific process of generating a heat distribution map identifying high-frequency and low-frequency data blocks is as follows: Perform geographical grid division on the storage space based on the data heat value, and map the access location to a hierarchical spatial grid through geographical hash coding; Identify high-frequency access grid clusters through a density clustering algorithm, and merge adjacent high-density grids to form hot blocks; Separate low-frequency cold data regions through a spatio-temporal joint clustering algorithm, and dynamically adjust the block boundary in combination with the stability of the access interval; Mark the high-frequency blocks as red hot spots and the low-frequency blocks as blue cold regions to generate a visual heat distribution map.

[0008] Furthermore, according to the heat distribution map, the specific process of constructing a storage resource allocation model through a deep reinforcement learning algorithm is as follows: Construct a hierarchical state space, and encode the block heat values, fiber optic network topology, and load data in the heat distribution map into a multi-dimensional state vector; Design a multi-objective reward function, with minimizing transmission delay, balancing node load, and maximizing throughput as the joint optimization objectives, and increasing the delay penalty weight for high-frequency blocks; Train the model through the deep deterministic policy gradient algorithm, and the action decision-making network generates a resource allocation strategy to construct a storage resource allocation model.

[0009] Furthermore, define the bandwidth utilization rate, node load rate, and predicted access volume as state space parameters, and the specific process of outputting the optical path priority matrix and replica distribution topology instructions is as follows: Jointly encode the bandwidth utilization rate, node load rate, and predicted access volume to construct a dynamically perceivable state space matrix; Calculate the fiber optic path priority through the optical path game model, dynamically adjust the path weight in combination with the real-time load, and generate an optical path priority matrix; Design a replica distribution algorithm with spatio-temporal constraints, generate replica topology instructions in combination with the heat value distribution and node geographical distance, and dynamically correct the instruction parameters through a sliding window verification mechanism.

[0010] Furthermore, based on the heat distribution map and replica distribution topology instructions, the specific process of constructing a spatio-temporal joint index for structured data and a vector similarity index for unstructured data is as follows: Analyze the high-frequency block coordinates and heat values in the heat map, and generate spatio-temporal key values in combination with the replica topology instructions; Based on the spatio-temporal key values, construct a multi-level B+ tree index, extract the semantic features of unstructured data, generate a weighted vector space in combination with the heat values, construct a vector index through the distributed HNSW algorithm, and preferentially store high-heat data to low-latency nodes; Dynamically associate the spatio-temporal joint index and the vector similarity index to construct a cross-modal retrieval path.

[0011] Furthermore, optimize the prefetch policy according to the data heat value, and the specific process of generating a high-frequency data cache index distribution map is as follows: Divide the cache levels according to the data heat value, and define the rules for preloading high-frequency data into memory; Design a dynamic prefetch window to predict future hot data and generate a prefetch task queue; Dynamically adjust the cache weight through a spatio-temporal attention mechanism, and map the cache location and validity period into a visual distribution map.

[0012] Furthermore, comprehensively consider the data access delay, index hit rate, and replica migration frequency to obtain the fiber optic storage efficiency index, and the specific process is as follows: Comprehensively adjust the weight ratios of the access delay, index hit rate, and replica migration frequency through the analytic hierarchy process; Introduce a time decay factor to perform exponential smoothing on historical performance data, and generate a fiber optic storage efficiency index through linear weighted fusion.

[0013] Further, when the fiber optic storage efficiency index is lower than the preset threshold, the specific process of generating a structure optimization instruction set and feeding it back to the data scheduling module and the index optimization module is as follows: Establish a multi-objective decision tree, and select optimization strategies according to the branches of the fiber optic storage efficiency index, including: Excessive latency: Trigger an optical path priority reset instruction and an edge cache expansion instruction; Low hit rate: Generate an index level migration instruction and a prefetch policy update instruction; Excessive migration frequency: Activate a replica distribution topology reconstruction instruction and a load balancing policy.

[0014] A fiber optic data storage management method based on big data includes the following steps: S1. Extract the spatio-temporal features of the historical access records of the fiber optic storage system through an LSTM network, calculate the data heat value in combination with a time decay function, and generate a heat distribution map identifying high-frequency and low-frequency data blocks; S2. According to the heat distribution map, construct a storage resource allocation model through a deep reinforcement learning algorithm, define the bandwidth utilization rate, node load rate, and predicted access volume as state space parameters, output an optical path priority matrix and a replica distribution topology instruction, and monitor the data flow fluctuation through a sliding window mechanism to trigger the migration of high-frequency data to low-load nodes; S3. Based on the heat distribution map and the replica distribution topology instruction, construct a spatio-temporal joint index for structured data and a vector similarity index for unstructured data, and optimize the prefetch policy according to the data heat value to generate a high-frequency data cache index distribution map; S4. Synthesize the data access latency, index hit rate, and replica migration frequency to obtain the fiber optic storage efficiency index. When the fiber optic storage efficiency index is lower than the preset threshold, generate a structure optimization instruction set and feed it back to the data scheduling module and the index optimization module.

[0015] The present invention has the following beneficial effects: (1) A fiber optic data storage management system based on big data uses the LSTM network of the data access analysis module to model historical access records, calculates the data heat value in combination with a time decay function, and generates a heat distribution map, thereby accurately identifying high-frequency access data blocks and low-frequency cold data regions, providing an optimization basis for data scheduling. The data scheduling module uses a deep reinforcement learning algorithm to construct a storage resource allocation model, combines state space parameters such as bandwidth utilization rate, node load rate, and predicted access volume, dynamically adjusts the optical path priority and replica allocation strategy, realizes the adaptive optimization of the storage path, and improves the system throughput capacity.

[0016] (2)A fiber optic data storage management method based on big data constructs a spatio-temporal joint index for structured data and a vector similarity index for unstructured data, combines data popularity to optimize the prefetching strategy, realizes fast indexing and cache optimization of high-frequency data, and improves retrieval efficiency. By comprehensively analyzing data access latency, index hit rate, and replica migration frequency, calculates the fiber optic storage efficiency index, and when the index is lower than a preset threshold, automatically generates a structure optimization instruction set to dynamically adjust data scheduling and indexing strategies to ensure the long-term efficient and stable operation of the storage system.

[0017] Of course, it is not necessary for any product implementing the present invention to simultaneously achieve all the above-mentioned advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flowchart of a fiber optic data storage management system based on big data according to the present invention.

[0019] Figure 2 It is a flowchart of a fiber optic data storage management method based on big data according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] The embodiments of the present application solve the problems of low data access efficiency, uneven storage resource allocation, insufficient index retrieval performance, and poor system stability in the existing fiber optic storage system through a fiber optic data storage management system and method based on big data.

[0021] The general idea of the solution in the embodiments of the present application is as follows: Extract the spatio-temporal features of the historical access records of the fiber optic storage system through the LSTM network, calculate the data popularity value in combination with the time decay function, and generate a heat distribution map indicating high-frequency and low-frequency data blocks.

[0022] According to the heat distribution map, construct a storage resource allocation model through the deep reinforcement learning algorithm, define the bandwidth utilization rate, node load rate, and predicted access volume as state space parameters, output the optical path priority matrix and replica distribution topology instructions, and monitor the data flow fluctuation through the sliding window mechanism to trigger the migration of high-frequency data to low-load nodes.

[0023] Based on the heat distribution map and replica distribution topology instructions, construct a spatio-temporal joint index for structured data and a vector similarity index for unstructured data, and optimize the prefetching strategy according to the data popularity value to generate a high-frequency data cache index distribution map.

[0024] Comprehensively consider data access latency, index hit rate, and replica migration frequency to obtain the fiber optic storage efficiency index. When the fiber optic storage efficiency index is lower than a preset threshold, generate a structure optimization instruction set and feedback it to the data scheduling module and index optimization module.

[0025] Please refer toFigure 1 , an embodiment of the present invention provides a technical solution: a fiber optic data storage management system based on big data, including the following modules: a data access analysis module, a data scheduling module, an index optimization module, and a storage efficiency evaluation module; the data access analysis module is used to extract the spatio-temporal features of the historical access records of the fiber optic storage system through an LSTM network, calculate the data heat value in combination with a time decay function, and generate a heat distribution map identifying high-frequency and low-frequency data blocks; the data scheduling module is used to construct a storage resource allocation model through a deep reinforcement learning algorithm according to the heat distribution map, define the bandwidth utilization rate, node load rate, and predicted access volume as state space parameters, output an optical path priority matrix and a replica distribution topology instruction, and monitor the data flow fluctuation through a sliding window mechanism to trigger the migration of high-frequency data to low-load nodes; the index optimization module is used to construct a spatio-temporal joint index for structured data and a vector similarity index for unstructured data based on the heat distribution map and the replica distribution topology instruction, and optimize the prefetch policy according to the data heat value to generate a high-frequency data cache index distribution map; the storage efficiency evaluation module is used to comprehensively consider the data access latency, index hit rate, and replica migration frequency to obtain the fiber optic storage efficiency index. When the fiber optic storage efficiency index is lower than a preset threshold, a structure optimization instruction set is generated and fed back to the data scheduling module and the index optimization module.

[0026] In this implementation plan, the data access analysis module: By analyzing historical access data, it identifies the hot regions with high-frequency access and the cold data regions with low-frequency access in the system, providing data support for resource scheduling and storage optimization. LSTM network: A type of recurrent neural network that is good at capturing the long-term dependencies of time series data and is used to extract the time periodicity features of data access (such as daily traffic peaks). Spatiotemporal features: Comprehensive features that combine the time dimension (such as access timestamps) and the space dimension (such as access geographical coordinates). Time decay function: A mathematical function that dynamically adjusts the weights of historical data, with high weights for recent data and low weights for distant data, reflecting the timeliness of data heat. Heat distribution map: A visual map of data heat, which uses color gradients (such as red for high frequency and blue for low frequency) to identify the access heat of different blocks. Spatiotemporal feature fusion modeling: For the first time, it combines the time decay function with the LSTM network to dynamically fuse spatiotemporal features and solve the problem of the separation of time and space analysis in traditional methods. Nonlinear decay mechanism: Design a function with an adaptive decay rate to flexibly handle sudden traffic scenarios (such as a sudden increase in data heat caused by emergencies). Data scheduling module: Dynamically allocate fiber bandwidth and storage node resources to ensure that high-heat data is given priority in transmission and disaster-tolerant storage. Deep reinforcement learning algorithm: A machine learning method that automatically optimizes decision-making strategies through simulating environment interactions and is used to build a resource allocation model. State space parameters: Input variables on which the model's decisions depend, including bandwidth utilization rate (the proportion of fiber bandwidth used), node load rate (the occupancy rate of storage node resources), and predicted access volume (prediction of future data request volume). Optical path priority matrix: Define the transmission priority rules for fiber paths (such as paths with low load have high weights) and dynamically optimize the data transmission path. Replica distribution topology instruction: Specify the rules for the storage location distribution of data replicas (such as storing across geographical regions) to ensure data disaster tolerance and access efficiency. Sliding window mechanism: A real-time monitoring technology that triggers resource adjustment strategies through the analysis of data flow fluctuations within a fixed time window (such as 5 minutes). Multi-objective joint optimization: Simultaneously optimize latency, throughput, and load balancing through the reinforcement learning model, breaking through the limitations of traditional single-objective scheduling strategies. Dynamic disaster tolerance rule: Combine geographical distance and network topology to generate replica distribution instructions to improve the data survival rate under regional disasters (such as earthquakes and power outages). Index optimization module: Improve data retrieval efficiency and support efficient queries for structured data (such as logs) and unstructured data (such as images and gene sequences). Structured data: Data with a fixed format (such as database tables and time series logs). Unstructured data: Data without a fixed format (such as text, images, and videos). Spatiotemporal joint index: An index structure that combines the time and space dimensions and supports joint queries of "time range + geographical region". Vector similarity index: Generate vectors by extracting data features and build an index to support similarity retrieval.Prefetch strategy: Predict future access demands based on data popularity, and load high-frequency data into the cache (such as memory, SSD) in advance. Cross-modal index fusion: First unify the retrieval frameworks for structured and unstructured data, and support "time-space-semantics" compound queries. Heat-driven dynamic prefetch: Dynamically adjust prefetch rules based on heat values to reduce redundant cache occupancy and improve cache hit rate. Storage efficiency evaluation module: Monitor system performance in real time, trigger adaptive optimization strategies, and ensure the long-term efficient operation of the system. Fiber storage efficiency index (FSEI): A quantitative performance metric generated by comprehensively considering access latency (data retrieval time), index hit rate (cache hit ratio), and replica migration frequency (data replica scheduling times). Structure optimization instruction set: A set of executable instructions including index migration, replica expansion, path reset, etc., used to dynamically optimize system configurations. Multi-objective decision tree: A logical model that matches optimization strategies based on the reasons for the decline of performance metrics (such as excessive latency, low hit rate). Closed-loop feedback mechanism: Feed back the execution effects of optimization instructions to the evaluation module in real time to form a closed loop of self-iterative optimization. Dynamic weighted evaluation model: Dynamically adjust the metric weights according to the real-time load (such as focusing on latency optimization under high load), breaking through the rigidity of the fixed-weight model.

[0027] Specifically, the process of calculating the data heat value by extracting the spatio-temporal features of the historical access records of the fiber storage system through the LSTM network and combining with the time decay function is as follows: Perform spatio-temporal encoding on the historical access records, extract timestamps, geographical coordinates, and the number of concurrent requests, and construct a spatio-temporal feature vector; Model the time dimension features through a bidirectional LSTM network to capture the periodic patterns of data access, and at the same time extract the spatial dimension features through a convolutional network; Input the spatio-temporal features into the spatio-temporal attention mechanism to dynamically allocate time decay weights and generate a spatio-temporal correlation weight matrix; Input the spatio-temporal correlation weight matrix into the time decay function to dynamically decay the influence of historical data according to time distance, and perform normalization on the decayed data to generate a globally comparable data heat value.

[0028] In this implementation, the historical access records of the optical fiber storage system are extracted through the LSTM network to obtain the spatio-temporal characteristics of the data, and the time decay function is combined to calculate the popularity value of the data, thereby generating the heat distribution of data access. The specific process is as follows. Spatio-temporal feature encoding: The historical access records contain information about time and space. The main encoding contents include: Time information: The timestamp of data access. Space information: The storage location of the accessed data (such as the geographical coordinates of the storage node or the location of the data center). Access features: The concurrent request volume and the access frequency. These information are combined to form a spatio-temporal feature vector. Time dimension feature extraction (bidirectional LSTM), Data access usually has periodic patterns (such as more frequent data access during peak hours). Therefore, the bidirectional LSTM is used to model the time dimension features to capture long-term dependencies and access patterns. Through the memory gating mechanism, LSTM can effectively extract the evolution trend of time series data and learn the influence of different time points on data access behavior. Space dimension feature extraction, Since data access may show an aggregation effect in space (such as some storage nodes being accessed more frequently than others), a convolutional network (CNN) is used to extract spatial features to identify access hotspots. Through convolutional operations, CNN can extract the access patterns of different data blocks in the storage system, identify local hotspots, and enhance the expressive ability of spatio-temporal feature learning. Spatio-temporal attention mechanism to calculate the spatio-temporal correlation weight matrix, In order to dynamically measure the influence of time and space factors on data access patterns, this method uses the spatio-temporal attention mechanism to calculate the weights of data access. This mechanism can capture the influence degree of data access at different time points and different spatial regions and assign corresponding weights. The spatio-temporal attention mechanism forms a spatio-temporal correlation weight matrix by calculating the correlation between data points: ; where: represents the historical data point the influence degree of the access behavior on the data point . This weight matrix is used to measure the spatio-temporal correlation between data at different time points and storage locations. In order to ensure that the data access records in the relatively recent time have a greater impact on the popularity value, the time decay function is used to weight and decay the historical data. The time decay weight is calculated as follows: ; where: represents the data point time decay weight. is the decay coefficient, which determines the influence degree of the data access history. is the current time, is the timestamp of data point a. When is closer to the current time , is larger; conversely, the farther the time, the more obvious the decay. The data popularity value is calculated as follows: ; where: represents the location The data heat value at Calculated by the spatio-temporal attention mechanism. Calculated by the time decay function. Is the access request volume. The calculated heat value is further normalized for global comparison and visualization analysis. Based on the calculated data heat value, a heat distribution map of data access is constructed to visually display the hot data area of the storage system and provide a basis for subsequent data scheduling and index optimization.

[0029] Specifically, the specific process of generating the heat distribution map identifying high-frequency and low-frequency data blocks is as follows: Geographically grid the storage space based on the data heat value, and map the access location to a hierarchical spatial grid through Geohash encoding; Identify high-frequency access grid clusters through the density clustering algorithm, and merge adjacent high-density grids to form hot blocks; Separate low-frequency cold data areas through the spatio-temporal joint clustering algorithm, and dynamically adjust the block boundaries in combination with the stability of the access interval; Mark the high-frequency blocks as red hot spots and the low-frequency blocks as blue cold areas to generate a visual heat distribution map.

[0030] In this implementation plan, analyze the access pattern of the storage system through the data heat value, and use technologies such as spatial gridization, density clustering, high- and low-frequency block identification, and spatio-temporal joint clustering to generate a heat distribution map to realize the visualization of data access hot spots. The specific steps are as follows: Geographical grid division of the storage space. In order to accurately identify the spatial distribution of data access, use Geohash encoding to perform hierarchical grid division of the storage space. Geohash encoding (Geohash encoding): Divide the storage space into L-level grids, and the size of each level of grid is determined by the length of the Geohash code. Higher-level Geohash codes represent larger areas, and lower-level Geohash codes represent finer areas. Through the Geohash code, map each access location (x, y) to a grid cell for subsequent clustering analysis. Identify high-frequency access blocks (density clustering). On the basis of grid division, use the density clustering algorithm (DBSCAN) to identify high-frequency access blocks. Calculate the access density of each grid: Where: Represents the network Access density of. Is the grid Inside the The heat value of the data point. Is the number of data points in the grid. Set the density threshold , When the network density Greater than When this occurs, the network is marked as a high-frequency grid. The DBSCAN algorithm is used to aggregate adjacent high-frequency grids to form a hotspot area. Identify low-frequency cold data blocks (spatiotemporal joint clustering). To distinguish low-frequency data blocks, a spatiotemporal joint clustering algorithm is used to identify cold data areas: Calculate the access interval stability of the grid: ; where: represents the access interval stability of the grid . is the number of data accesses within this grid. represents the th access timestamp. is the average access time of this grid. Set the stability threshold . When , it indicates that the access frequency of this grid is low and the time distribution is unstable, and it is determined as a low-frequency cold data block. Combining with the density clustering algorithm, low-frequency cold data grids are aggregated to form a cold data area.

[0031] Specifically, according to the heat distribution map, the specific process of constructing a storage resource allocation model through the deep reinforcement learning algorithm is as follows: Construct a hierarchical state space, and encode the block heat value, fiber optic network topology structure, and load data in the heat distribution map into a multi-dimensional state vector; Design a multi-objective reward function, with minimizing transmission delay, balancing node load, and maximizing throughput as the joint optimization objectives, and increasing the delay penalty weight of high-frequency blocks; Train the model through the deep deterministic policy gradient algorithm, and the action decision-making network generates a resource allocation strategy to construct a storage resource allocation model.

[0032] In this implementation plan, a hierarchical state space is constructed. The state space of the storage resource allocation model consists of multiple factors, including data heat value, fiber optic network topology structure, node load data, etc. This information needs to be encoded as a multi-dimensional vector to represent the state of the current system. The specific steps are as follows: Data heat value: Obtain the heat value of each block from the heat distribution map to measure the high or low data access frequency. A high heat value corresponds to a high-frequency data block, and a low heat value corresponds to a low-frequency data block. Fiber optic network topology structure: Encode the connection situation between each node and optical path in the network to reflect the structure of the fiber optic network. The network topology structure helps the model understand the communication delay and bandwidth resources between nodes. Node load data: The load situation of each node (such as storage load, processing capacity, etc.) affects the priority of nodes when allocating data. Integrate these factors to form a multi-dimensional state vector , where: represents the heat value distribution vector at the current time point t. represents the fiber optic network topology structure. Represents the load data of each current node. A multi-objective reward function is designed. To balance the optimization performance of storage resource allocation among different objectives, a multi-objective reward function is designed. Its goal is to jointly minimize the transmission delay, balance the node load, and maximize the throughput. Specifically as follows: Minimize the transmission delay: Minimize the delay during data transmission as much as possible. The goal is to allocate the frequently accessed data to the nodes with minimized delay to improve the system's response speed. Balance the node load: Ensure the load balance of each node and avoid overloading some nodes while other nodes are idle. Load balance can improve the overall processing capacity of the system. Maximize the throughput: Improve the overall throughput of the network and increase the amount of data that can be processed. During this process, a higher penalty weight is given to the delay of high-frequency blocks, that is, the optimization intensity of high-frequency data is increased to ensure the minimum transmission delay. Multi-objective reward function Is expressed as: Where: Is a measure of the transmission delay in the current state. Is a measure of the node load balance. Is the current network throughput. Is the weight coefficient, used to control the relative importance of different objectives, and . Through the Deep Deterministic Policy Gradient algorithm (DDPG), a deep reinforcement chemical model is trained to optimize the storage resource allocation. This algorithm is used for optimization problems in continuous action spaces and is suitable for tasks such as storage resource allocation that require selecting specific resource allocation strategies. Actor network: Generates the storage resource allocation strategy. By continuously updating the strategy to maximize the long-term reward, the optimal resource allocation scheme is selected. The output of the Actor network is an allocation strategy π(S) in a continuous action space, that is, how much resources should be allocated to each node. Critic network: Evaluates the value function of the current state-action pair( , ). Through this evaluation, the Critic network can give the predicted value of the long-term reward of the current action and assist the Actor network in optimization. In each training cycle, the Actor network generates an action according to the current state =π( ), and then the Critic network calculates the Q value of this action, representing the expectation of the long-term reward. Update formula: Update of the Actor network: By the gradient ascent method, maximize the Q value given by the Critic network. ; Update of the Critic network: Optimize the Q value by minimizing the Bellman error. Where: , The learning rates for the Actor network and the Critic network. is the reward value at the current time step. γ is the discount factor used to calculate the long-term return. After training through deep reinforcement learning, the storage resource allocation model can dynamically adjust the resource allocation strategy according to the real-time state. Through continuous exploration and optimization, the model can find the optimal solution in multi-objective optimization, and intelligently allocate storage and bandwidth resources, thereby improving the performance and efficiency of the entire optical fiber storage system.

[0033] Specifically, defining the bandwidth utilization rate, node load rate, and predicted access volume as state space parameters, the specific process of outputting the optical path priority matrix and the replica distribution topology instruction is as follows: jointly encode the bandwidth utilization rate, node load rate, and predicted access volume to construct a dynamically perceiving state space matrix; calculate the optical fiber path priority through the optical path game model, dynamically adjust the path weights in combination with the real-time load, and generate the optical path priority matrix; design a replica distribution algorithm with spatio-temporal constraints, generate the replica topology instruction in combination with the heat value distribution and the geographical distance between nodes, and dynamically correct the instruction parameters through the sliding window verification mechanism.

[0034] In this implementation plan, first, the state space matrix is constructed by jointly encoding the bandwidth utilization rate, node load rate, and predicted access volume. These state parameters reflect the current resource usage of the storage system and can dynamically perceive and adapt to changes in the system state. Bandwidth utilization rate: It reflects the degree of bandwidth usage on the optical fiber path. The bandwidth usage directly affects the priority of optical path selection. The higher the bandwidth utilization rate, the more likely it is that the path is in a high-load state and its priority may need to be reduced. Node load rate: It indicates the current processing load of each storage node. Nodes with a high load rate may need to transfer a part of the traffic to other nodes to prevent overload. Predicted access volume: It is the future access demand predicted based on historical data or models. The predicted access volume can help the system schedule and allocate resources in advance to cope with future access peaks. Generation of the optical path priority matrix. In this model, the design of the optical path priority depends on the optical path game model. This model dynamically adjusts the real-time data of the bandwidth utilization rate and the node load in the middle to generate the optical path priority matrix, determines the weight of each optical fiber path, and ensures efficient and balanced path selection. The key point of the optical path game model is to analyze the competition relationship between different paths through game theory and calculate the priority of each optical path. This priority is not only based on the bandwidth usage but also needs to be dynamically adjusted according to the load situation and transmission delay on each optical path. The calculation formula for the optical path priority matrix P is: ; where: is a function of the bandwidth utilization rate, describing the impact of the bandwidth utilization rate on the priority. is a function of the node load rate, describing the impact of the load on path selection. It is a function for predicting the access volume and describes the impact of the access volume on the path. is the weight coefficient, which is used to adjust the influence degree of bandwidth, load, and access volume on the priority. This formula dynamically adjusts the path weight by comprehensively considering bandwidth, load, and access requirements, thereby optimizing the priority of the optical path. The goal of replica distribution is to reasonably select the replica storage location according to the data popularity value and the geographical distance between nodes, so as to improve the storage efficiency of the system and reduce the access latency. The spatio-temporal constrained replica distribution algorithm combines the popularity value distribution with the geographical distance information of nodes and generates replica distribution topology instructions through the following steps: Popularity value distribution: High-popularity data should be as close as possible to the nodes with frequent access to reduce access latency. Node geographical distance: According to the geographical distribution of nodes, replicas are preferentially stored in nodes closer to users or high-access-frequency regions, thereby improving the efficiency of data access. To ensure the effectiveness and adaptability of the replica distribution instructions, a sliding window verification mechanism is adopted. This mechanism can dynamically adjust the replica storage scheme according to the actual access pattern, popularity change, and network topology. The core of sliding window verification is to verify the replica distribution instructions in batches, and the replica strategy within each window will be dynamically corrected according to the actual data access situation. By regularly adjusting the replica storage scheme, it is ensured that data can be efficiently stored and quickly respond to access requests.

[0035] Specifically, based on the heat distribution map and the replica distribution topology instructions, the specific process of constructing the spatio-temporal joint index for structured data and the vector similarity index for unstructured data is as follows: Analyze the high-frequency block coordinates and popularity values in the heat map, and generate spatio-temporal key values in combination with the replica topology instructions; Based on the spatio-temporal key values, construct a multi-level B+ tree index, extract the semantic features of unstructured data, generate a weighted vector space in combination with the popularity value, construct a vector index through the distributed HNSW algorithm, and preferentially store high-popularity data to low-latency nodes; Dynamically associate the spatio-temporal joint index with the vector similarity index to construct a cross-modal retrieval path.

[0036] In this implementation, the thermal map and replica topology instructions are parsed to generate spatio-temporal key values. The system identifies the blocks with higher access frequencies by parsing the high-frequency block coordinates and heat values in the thermal map. These high-frequency blocks represent the most active areas in the system with high data access demands. By combining the replica distribution topology instructions, the storage locations and access paths of the data can be determined. This information will be used to generate spatio-temporal key values, which serve as the basis for subsequent index construction. The spatio-temporal key value is a combined identifier consisting of the geographical coordinates of the block (representing the storage location) and the heat value (representing the access frequency of the data), used to distinguish and index different data blocks. In this process, the spatio-temporal key value not only contains the geographical location of the data but also reflects the heat of the data, thus improving the data retrieval efficiency. Build a multi-level B+ tree index. After generating the spatio-temporal key values, use these spatio-temporal key values to build a multi-level B+ tree index. The B+ tree is an index structure widely used in databases and file systems, which has high query efficiency and good range query characteristics. By constructing a B+ tree index, specific spatio-temporal blocks can be quickly located, improving the access efficiency of structured data. Each node of the B+ tree contains spatio-temporal key value information, and the child nodes of each node point to the data blocks with similar spatio-temporal key values, thus supporting efficient query operations, especially when data needs to be retrieved in spatio-temporal order. Extract the semantic features of unstructured data and generate a weighted vector space. For unstructured data such as text, pictures or videos, semantic feature extraction is first required. This process extracts the semantic hierarchical features of the data through natural language processing (NLP) or computer vision (CV) techniques to form a vector representation that can express the data content. After obtaining the semantic feature vectors, the system combines the heat values and assigns weights to each vector. The heat value, as an important weight parameter, reflects the access frequency of the data. Combining the heat value with the semantic feature vectors forms a weighted vector space, thus enhancing the weight of high-heat data in the index. This means that data with high-frequency access will be given priority and stored in nodes with higher performance and lower latency to improve the overall performance of the system. Build a vector similarity index. Using the weighted vector space, the system constructs a vector similarity index through the distributed HNSW algorithm (Hierarchical Navigable Small World). HNSW is a graph-based approximate nearest neighbor search algorithm that can effectively process high-dimensional vectors in large-scale datasets. Through the HNSW algorithm, fast similarity search for unstructured data can be achieved to find high-heat data similar to the query vector. The HNSW algorithm constructs a hierarchical graph structure, enabling the query to find data blocks with higher similarity in a shorter time. This method is efficient and scalable, especially suitable for systems that need to process massive data. Dynamically associate the spatio-temporal joint index with the vector similarity index. After constructing the spatio-temporal joint index and the vector similarity index, the system will perform dynamic association.The spatio-temporal joint index mainly processes structured data, indexing and retrieving data through spatio-temporal key-value pairs. The vector similarity index, on the other hand, processes unstructured data and provides a retrieval path based on semantic similarity. By dynamically associating these two indexes, the system can provide users with a cross-modal retrieval path, that is, it supports retrieval requests based on both spatio-temporal features and semantic similarity. Users can, through a single query request, locate the storage location of the data in the spatio-temporal joint index and then find the most relevant unstructured data through the vector similarity index, thus achieving efficient cross-modal data retrieval.

[0037] Specifically, the specific process of optimizing the prefetch strategy according to the data heat value and generating the cache index distribution map of high-frequency data is as follows: divide the cache levels according to the data heat value and define the rules for preloading high-frequency data into memory; design a dynamic prefetch window to predict future hot data and generate a prefetch task queue; dynamically adjust the cache weights through a spatio-temporal attention mechanism and map the cache location and validity period into a visual distribution map.

[0038] In this implementation, the cache levels are divided according to the data heat value, and the system classifies and hierarchically divides the data through the data heat value. The data heat value is calculated based on the data access frequency or access pattern, reflecting the activity degree of the data. The part with a higher data heat value represents the hot data with frequent access, while the part with a lower data heat value is the cold data or the data with less access. According to the high or low heat value, the system divides the data into different cache levels. For example, the following levels can be set: High-frequency data: These data are frequently accessed and should be pre-loaded into the memory cache to ensure fast access. Medium-frequency data: These data are not frequently accessed but have a certain access frequency and can be stored in the local disk cache. Low-frequency data: Data with a lower access frequency is stored in a remote storage or archival system. This division is achieved by setting heat value thresholds. The thresholds can be dynamically adjusted according to the actual application and requirements to ensure that the data can be reasonably stored according to the access frequency. Define the rules for pre-loading high-frequency data into memory. After the cache level division, the system needs to define the rules for pre-loading high-frequency data into memory. For high-frequency data, the system must ensure that it is in the memory for fast access. Therefore, the rules are defined as follows: Rule 1: Loading based on the heat threshold: If the heat value of a certain data exceeds a certain threshold, it will be loaded into the memory cache. Rule 2: Prediction based on access frequency: According to the trend of historical access records, the system can predict the data that will become hot in the future and pre-load these data into the memory cache in advance. The goal of these rules is to ensure that high-frequency data can quickly respond to access requests and improve the overall performance of the system. Design a dynamic prefetch window to predict future hot data. To cope with future hot data, the system designs a dynamic prefetch window. The dynamic prefetch window is based on the current data access history and predicts the data that may become hot in a future period. The specific process is as follows: Data access pattern analysis: By analyzing the access patterns of historical data, identify which data have a high access trend. Time series prediction: Use time series analysis methods (such as LSTM, ARIMA, etc.) to predict which data will have an increasing access volume in a future period, thus forming future hot data. Task queue generation: According to the prediction results, the system will generate a prefetch task queue, and these tasks will pre-load the predicted hot data into the cache in advance for subsequent access. Through the dynamic prefetch window, the delay caused by cache misses can be effectively reduced, and the data processing efficiency can be improved. Dynamically adjust the cache weights through the spatio-temporal attention mechanism. To further optimize the cache efficiency, the system introduces the spatio-temporal attention mechanism to dynamically adjust the cache weights. The spatio-temporal attention mechanism combines the time and space dimensions to dynamically adjust the data distribution in the cache. The specific process is as follows: Time dimension: As time goes by, the access frequency of some data may change. For example, some data may be frequently accessed during a specific period and less accessed during other periods.The spatio-temporal attention mechanism can dynamically adjust the caching strategy according to the changes in time, increase the weight of hot data, and reduce the weight of low-frequency data. Spatial dimension: The access of data is usually related to its storage location. Spatially, some data may need to be stored and accessed at specific geographical locations (e.g., locations close to users). The spatio-temporal attention mechanism can dynamically adjust the caching location according to the spatial distance and the load conditions of storage nodes. By combining time and space information, the spatio-temporal attention mechanism can automatically adjust the weight of each data block in the cache to achieve an optimal storage strategy. Mapping the caching location and the expiration period into a visualization distribution map, the system displays the distribution map of the caching location and the expiration period through visualization means. This map intuitively shows the storage location and expiration period of each data block in the cache, enabling the system administrator to quickly understand the cache distribution and adjust the strategy in a timely manner. Caching location mapping: Visualize the location of data blocks according to the location where the data is stored (such as memory, disk, remote storage, etc.), showing which data is stored on efficient storage media and which is stored on slower media. Expiration period mapping: Show the expiration period of the data in the cache, indicating which data needs to be updated and which data can be cleared, thereby improving the utilization rate of the cache space. This visualization distribution map provides an intuitive basis for data management, facilitating the administrator to make optimization adjustments.

[0039] Specifically, the specific process of obtaining the fiber optic storage efficiency index by comprehensively considering data access latency, index hit rate, and replica migration frequency is as follows: Adjust the weight ratio of access latency, index hit rate, and replica migration frequency through the Analytic Hierarchy Process (AHP); introduce a time decay factor to perform exponential smoothing on historical performance data, and generate the fiber optic storage efficiency index through linear weighted fusion.

[0040] In this implementation plan, the formula for the fiber optic storage efficiency index is: ; Explanation of the formula: : The weight coefficients obtained through the Analytic Hierarchy Process (AHP), corresponding to data access latency, index hit rate, and replica migration rate respectively. These weights reflect the relative importance of different indicators in storage efficiency. : The data access latency at the current moment. : The data access latency at the previous moment. : The index hit rate at the current moment. : The replica migration rate at the current moment. : The decay factors, representing the contribution ratios of the data access latency at the current moment and the previous moment to the efficiency respectively. Usually, will be relatively large, indicating that the latency at the current moment has a greater impact on the efficiency index. : The introduced time decay factor, where is the decay rate, is the time step. This factor decays historical data to ensure that the storage efficiency index mainly focuses on current performance data and reduces the impact of historical data.

[0041] Specifically, when the fiber optic storage efficiency index is lower than a preset threshold, the specific process of generating a structure optimization instruction set and feeding it back to the data scheduling module and the index optimization module is as follows: Establish a multi-objective decision tree, and select optimization strategies according to the branches of the fiber optic storage efficiency index, including: Excessive latency: Trigger optical path priority reset instructions and edge cache expansion instructions; Low hit rate: Generate index level migration instructions and prefetch policy update instructions; Excessive migration: Start replica distribution topology reconstruction instructions and load balancing strategies.

[0042] In this implementation, a multi-objective decision tree is established: The multi-objective decision tree is a decision-making model constructed through various storage performance objectives (such as latency, hit rate, migration frequency, etc.). This tree makes branch selections based on the change of the fiber optic storage efficiency index according to a predetermined threshold. When the efficiency index is lower than the threshold, each branch of the decision tree corresponds to different optimization strategies. Decision tree branches and strategy selection: Excessive latency: If the access latency of the fiber optic storage system exceeds the preset threshold, the system will trigger two main instructions: Optical path priority reset instruction: By readjusting the priority of the fiber optic path, optimize the access path of the fiber optic storage data and reduce the transmission latency. Edge cache expansion instruction: Increase the cache capacity of the edge nodes, reduce the frequency of data transmission to the storage center, and thus reduce the latency. Low hit rate: If the index hit rate is low, it means that the cache or storage hierarchy is often not hit during data access. At this time, the system will take the following optimization measures: Index level migration instruction: Adjust the data level structure of the index, migrate the index level of hot data upward, and improve the access hit rate. Prefetch strategy update instruction: Update the prefetch strategy, predict the data that may be needed in the future based on historical access data, and preload it into the cache in advance, thereby improving the data hit rate. Excessive migration: If the replica migration frequency is too high, it means that there are frequent replica relocations in the system, which may lead to waste of network bandwidth or uneven load. In response to this, the system will trigger the following optimization instructions: Replica distribution topology reconstruction instruction: Reconstruct the replica distribution, reasonably distribute the data replicas according to characteristics such as geographical location and access frequency, and reduce unnecessary data migrations. Load balancing strategy: Reallocate resources according to the load situation, avoid overloading of some nodes, ensure system load balancing, and reduce performance degradation caused by uneven resources. Feedback to the data scheduling module and the index optimization module: The above optimization instructions are input into the data scheduling module and the index optimization module through a feedback mechanism. These modules will make adjustments according to the optimization instructions and perform operations such as resource reallocation, path optimization, and cache policy update. Data scheduling module: Dynamically optimize data scheduling according to the optical path priority and cache expansion instructions to ensure the minimum system response latency. Index optimization module: Optimize the data index strategy according to the index level migration and prefetch strategy update instructions, improve the cache hit rate, and enhance the overall performance of the system.

[0043] Please refer to Figure 2, A fiber optic data storage management method based on big data, comprising the following steps: S1. Extract the spatio-temporal features of the historical access records of the fiber optic storage system through an LSTM network, calculate the data heat value in combination with a time decay function, and generate a heat distribution map identifying high-frequency and low-frequency data blocks; S2. According to the heat distribution map, construct a storage resource allocation model through a deep reinforcement learning algorithm, define the bandwidth utilization rate, node load rate, and predicted access volume as state space parameters, output an optical path priority matrix and a replica distribution topology instruction, and monitor the data flow fluctuation through a sliding window mechanism to trigger the migration of high-frequency data to low-load nodes; S3. Based on the heat distribution map and the replica distribution topology instruction, construct a spatio-temporal joint index for structured data and a vector similarity index for unstructured data, and optimize the prefetch strategy according to the data heat value to generate a high-frequency data cache index distribution map; S4. Comprehensively consider the data access latency, index hit rate, and replica migration frequency, obtain the fiber optic storage efficiency index, and when the fiber optic storage efficiency index is lower than a preset threshold, generate a structure optimization instruction set and feedback it to the data scheduling module and the index optimization module.

[0044] In this implementation, a fiber optic data storage management method based on big data aims to optimize the performance of the fiber optic storage system by intelligently managing and dynamically adjusting aspects such as historical data access records, heat distribution, storage resource allocation, and index optimization. The following is a specific explanation of the functions of each step: S1: The goal of this step is to perform spatio-temporal analysis on the historical access records of the fiber optic storage system to identify high-frequency and low-frequency data blocks in the system. LSTM (Long Short-Term Memory Network) is used to extract the temporal features in the access records to more accurately predict the heat changes of the data. Combining with the time decay function, the timeliness of data access can be better considered, causing the heat values of historical access data to decay, thereby generating a heat distribution map reflecting the data access frequency. These maps can help identify which data are high-frequency access data (hot data) and which are low-frequency access data (cold data), providing a basis for subsequent storage resource optimization. S2: In this step, combined with the heat distribution map, the deep reinforcement learning (DRL) algorithm is used to optimize the storage resource allocation. The bandwidth utilization rate, node load rate, and predicted access volume are used as state space parameters and input into the model. The model will generate resource allocation strategies based on these input parameters, including the optical path priority matrix and the replica distribution topology instruction. The optical path priority matrix determines the transmission priority of data streams between different paths, and the replica distribution topology instruction optimizes the storage location and migration strategy of data replicas. The sliding window mechanism can monitor the changes in data streams, timely identify high-frequency data and migrate it to low-load nodes, thereby improving the overall load balance and storage efficiency of the system. S3: This step focuses on data storage and index optimization. By according to the heat distribution map and the replica distribution topology instruction, a spatio-temporal joint index and a vector similarity index are constructed, which are used for the management of structured data and unstructured data respectively. The spatio-temporal joint index can effectively associate the time and space characteristics of data, optimizing data retrieval and storage efficiency; the vector similarity index is optimized by calculating the similarity between data, especially suitable for the storage and query of unstructured data (such as images, texts, etc.). In addition, combined with the data heat value, the prefetch strategy can be optimized to ensure that high-frequency data maintains priority in the system, thereby generating a high-frequency data index distribution map in the cache, further improving the response speed and storage efficiency of data access. S4: The function of this step is to calculate a fiber optic storage efficiency index reflecting the efficiency of the fiber optic storage system by comprehensively considering data access latency, index hit rate, and replica migration frequency. When this index is lower than the set threshold, it indicates that the current system has low storage efficiency and may have performance bottlenecks. In this case, the system will automatically generate a set of structure optimization instructions and feedback them to the data scheduling module and the index optimization module to guide subsequent resource adjustment and optimization strategies. These instruction sets can involve operations such as fiber optic storage path adjustment, cache optimization, and index layer update, thereby improving the performance of the overall storage system.

[0045] In summary, the present application has at least the following effects: A fiber optic data storage management system and method based on big data realizes precise data location and optimized storage by combining spatio-temporal feature analysis, heat distribution maps, and replica distribution topology instructions, reducing unnecessary data migration and redundant storage, thereby improving the overall storage efficiency of the fiber optic storage system. Through an optimized storage resource allocation strategy, dynamic load balancing, and data migration based on deep reinforcement learning, it ensures that high-frequency data is preferentially stored on low-load nodes, reducing data access latency and improving the system response speed. By constructing spatio-temporal joint indexes and vector similarity indexes, the storage and query processes are optimized for structured and unstructured data respectively, significantly improving the data retrieval efficiency, especially the access speed of unstructured data. Through the application of a sliding window mechanism, historical performance data analysis, and a time decay function, the cache weight, data prefetching strategy, and fiber optic storage path optimization can be dynamically adjusted to ensure that the system automatically self-optimizes according to real-time data flow fluctuations. When the fiber optic storage efficiency index is lower than a preset threshold, the system can generate a structure optimization instruction set and feedback it to the data scheduling module and index optimization module to achieve targeted optimization and ensure the stability and efficient operation of the system. By combining a multi-objective decision tree and a deep reinforcement learning algorithm, the system can adjust the storage resource allocation strategy and data migration plan according to real-time situations, making it highly flexible and scalable to adapt to changing data flows and access requirements. Through the dynamic adjustment of replica distribution topology instructions, the storage location and migration strategy of replicas are optimized to avoid over-frequent replica migration and unnecessary data redundancy, thereby improving the resource utilization rate and reliability of the fiber optic storage system.

[0046] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0047] The present invention is described with reference to the flowcharts and / or block diagrams of systems, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, and the combination of processes and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate for realizing in the process Figure 1One or more processes and / or blocks Figure 1 Apparatus for the functions specified in one or more blocks

[0048] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction apparatus that implements the functions in the process Figure 1 One or more processes and / or blocks Figure 1 The functions specified in one or more blocks

[0049] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus, such that a series of operational steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions in the process Figure 1 One or more processes and / or blocks Figure 1 The steps of the functions specified in one or more blocks

[0050] Although the preferred embodiments of the present invention have been described, additional changes and modifications can be made by those skilled in the art once they learn of the basic inventive concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present invention

[0051] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the equivalent technology of the claims of the present invention, the present invention is also intended to include these modifications and variations

Claims

1. A fiber optic data storage and management system based on big data, characterized in that: It includes the following modules: data access analysis module, data scheduling module, index optimization module, and storage efficiency evaluation module; The data access analysis module is used to extract the spatiotemporal characteristics of the historical access records of the optical fiber storage system through the LSTM network, calculate the data heat value in combination with the time decay function, and generate a thermal distribution map identifying high-frequency and low-frequency data blocks; The data scheduling module is used to build a storage resource allocation model based on the thermal distribution map through a deep reinforcement learning algorithm, define bandwidth utilization, node load rate and predicted access volume as state space parameters, output the optical path priority matrix and replica distribution topology instructions, and monitor data flow fluctuations through a sliding window mechanism to trigger high-frequency data migration to low-load nodes; The index optimization module is used to construct a spatiotemporal joint index of structured data and a vector similarity index of unstructured data based on the heat distribution map and the replica distribution topology instructions, and optimize the pre-fetch strategy according to the data heat value to generate a high-frequency data cache index distribution map; The storage efficiency evaluation module is used to comprehensively consider data access delay, index hit rate and replica migration frequency to obtain the fiber storage efficiency index. When the fiber storage efficiency index is lower than a preset threshold, a structure optimization instruction set is generated and fed back to the data scheduling module and the index optimization module.

2. The optical fiber data storage and management system based on big data according to claim 1, characterized in that: The specific process of extracting the spatiotemporal characteristics of the historical access records of the optical storage system through the LSTM network and calculating the data heat value in combination with the time decay function is as follows: Perform spatiotemporal encoding on historical access records, extract timestamps, geographic coordinates, and concurrent requests, and construct spatiotemporal feature vectors; The bidirectional LSTM network is used to model the time dimension features to capture the periodicity of data access, while the convolutional network is used to extract the spatial dimension features. Input spatiotemporal features into the spatiotemporal attention mechanism, dynamically assign time decay weights, and generate a spatiotemporal correlation weight matrix; The spatiotemporal correlation weight matrix is ​​input into the time decay function, the influence of historical data is dynamically decayed according to the time distance, the decayed data is normalized, and a globally comparable data heat value is generated.

3. The optical fiber data storage and management system based on big data according to claim 2, characterized in that: The specific process of generating a thermal distribution map that identifies high-frequency and low-frequency data blocks is as follows: The storage space is divided into geographic grids based on the data heat value, and the access location is mapped into a hierarchical spatial grid through geographic hash coding; The high-frequency access grid clusters are identified through density clustering algorithm, and adjacent high-density grids are merged to form hotspot blocks; Separate low-frequency cold data areas through spatiotemporal joint clustering algorithm, and dynamically adjust block boundaries based on access interval stability; High-frequency blocks are marked as red hot spots and low-frequency blocks are marked as blue cold areas to generate a visual thermal distribution map.

4. The optical fiber data storage and management system based on big data according to claim 1, characterized in that: According to the thermal distribution map, the specific process of building a storage resource allocation model through a deep reinforcement learning algorithm is as follows: Construct a hierarchical state space to encode the block heat value, fiber network topology and load data in the thermal distribution map into a multi-dimensional state vector; Design a multi-objective reward function to jointly optimize the goals of minimizing transmission delay, balancing node load, and maximizing throughput, and increase the delay penalty weight of high-frequency blocks; The model is trained using a deep deterministic policy gradient algorithm, and the action decider network generates a resource allocation strategy to build a storage resource allocation model.

5. The optical fiber data storage and management system based on big data according to claim 4, characterized in that: The specific process of defining bandwidth utilization, node load rate and predicted access volume as state space parameters and outputting the optical path priority matrix and replica distribution topology instructions is as follows: The bandwidth utilization, node load rate and predicted access volume are jointly encoded to construct a dynamic perception state space matrix; The optical path priority is calculated through the optical path game model, and the path weight is dynamically adjusted based on the real-time load to generate the optical path priority matrix. A replica distribution algorithm with time and space constraints is designed, which combines the heat value distribution and the geographical distance of the nodes to generate replica topology instructions, and dynamically corrects the instruction parameters through the sliding window verification mechanism.

6. The optical fiber data storage and management system based on big data according to claim 1, characterized in that: Based on the heat distribution map and replica distribution topology instructions, the specific process of building the spatiotemporal joint index of structured data and the vector similarity index of unstructured data is as follows: Analyze the high-frequency block coordinates and heat values ​​in the heat map, and generate spatiotemporal key values ​​in combination with the replica topology instructions; Build a multi-level B+ tree index based on spatiotemporal key values, extract the semantic features of unstructured data, generate a weighted vector space based on the heat value, build a vector index through the distributed HNSW algorithm, and prioritize the storage of high-heat data to low-latency nodes; Dynamically associate spatiotemporal joint index with vector similarity index to build cross-modal retrieval path.

7. The optical fiber data storage and management system based on big data according to claim 6, characterized in that: The specific process of optimizing the pre-fetch strategy based on the data heat value and generating the high-frequency data cache index distribution map is as follows: Divide cache levels according to data heat values ​​and define rules for preloading high-frequency data into memory; Design a dynamic prefetch window to predict future hot data and generate a prefetch task queue; The cache weight is dynamically adjusted through the spatiotemporal attention mechanism, and the cache location and validity period are mapped into a visual distribution map.

8. The optical fiber data storage and management system based on big data according to claim 1, characterized in that: The specific process of obtaining the fiber storage efficiency index by combining data access latency, index hit rate, and replica migration frequency is as follows: The weight ratio of access delay, index hit rate and replica migration frequency is comprehensively adjusted through hierarchical analysis method; The time decay factor is introduced to perform exponential smoothing on the historical performance data, and the optical fiber storage efficiency index is generated through linear weighted fusion.

9. The optical fiber data storage and management system based on big data according to claim 8, characterized in that: When the fiber storage efficiency index is lower than the preset threshold, the specific process of generating a structure optimization instruction set and feeding it back to the data scheduling module and the index optimization module is as follows: A multi-objective decision tree is established to select the optimization strategy based on the fiber storage efficiency index branch, including: Too high latency: triggers the light path priority reset command and edge cache expansion command; Hit rate is too low: Generate index level migration instructions and prefetch strategy update instructions; Migration is too frequent: Initiate the replica distribution topology reconstruction instruction and load balancing strategy.

10. A method for managing optical fiber data storage based on big data, applied to an optical fiber data storage management system based on big data as claimed in any one of claims 1 to 9, characterized in that: The following steps are involved: S1. Extract the spatiotemporal characteristics of the historical access records of the optical storage system through the LSTM network, calculate the data heat value in combination with the time decay function, and generate a thermal distribution map that identifies high-frequency and low-frequency data blocks; S2. Based on the thermal distribution map, a storage resource allocation model is constructed through a deep reinforcement learning algorithm. The bandwidth utilization, node load rate and predicted access volume are defined as state space parameters. The optical path priority matrix and replica distribution topology instructions are output. The data flow fluctuation is monitored through a sliding window mechanism to trigger the migration of high-frequency data to low-load nodes. S3. Based on the heat distribution map and replica distribution topology instructions, build a spatiotemporal joint index of structured data and a vector similarity index of unstructured data, optimize the pre-fetch strategy according to the data heat value, and generate a high-frequency data cache index distribution map; S4. Comprehensively consider the data access delay, index hit rate and replica migration frequency to obtain the fiber storage efficiency index. When the fiber storage efficiency index is lower than the preset threshold, generate a structure optimization instruction set to feed back to the data scheduling module and the index optimization module.

Images