Distributed storage and indexing method

Abstract

The invention relates to the technical field of information retrieval, and discloses a distributed storage and indexing method, which comprises the following steps of: dynamically fragmenting an original file through an improved consistent Hash algorithm to generate a plurality of data blocks with timestamps; constructing a three-dimensional Bloom filter index matrix containing timestamps, data types and content features for the data blocks with the timestamps, associating a B + tree local index with an inverted global index through a hierarchical index structure, and establishing an index update priority queue by adopting a two-channel synchronization mechanism, performing real-time increment synchronization on the hotspot index through a heartbeat mechanism, performing batch synchronization on the cold data layer index according to a cold data synchronization period, predicting a data distribution probability through a distributed query statistics probability table during query, and initiating multi-path query in parallel based on probability weight, and the query path is dynamically optimized according to the node group storage medium type and the inter-node network transmission delay, so that rapid distribution adjustment and access of distributed storage are realized.

Description

Technical Field

[0001] The invention relates to the technical field of information retrieval and discloses a distributed storage and indexing method. Background Art

[0002] The demand for data storage and management in various industries has increased dramatically. When faced with massive amounts of data, traditional centralized storage systems usually rely on a single storage device or a few large storage nodes. This architecture has obvious deficiencies in data processing capabilities, scalability, and fault tolerance. As the amount of data continues to increase, the performance of centralized storage systems has many deficiencies, data reading and writing speeds have slowed down, and response times have increased, seriously affecting the normal operation of the business.

[0003] In terms of data sharding, there is a lack of dynamic adjustment capabilities for shard boundaries, and timely optimization is impossible. The load on some nodes is too high or too low, resulting in uneven utilization of system resources. In the process of data block allocation, the type and performance differences of the node group storage media are not fully considered, and intelligent allocation cannot be performed according to the characteristics of the data, thus affecting data access efficiency and storage costs. Existing indexing methods cannot meet the needs of complex queries. Traditional index structures have low query efficiency when processing large-scale data, and cannot guarantee the real-time and consistency of the index, resulting in inaccurate query results. Summary of the Invention

[0004] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of this application to avoid obscuring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions should not be used to limit the scope of the present invention.

[0005] In order to solve the above technical problems, the main purpose of the present invention is to provide a distributed storage and indexing method.

[0006] A distributed storage and indexing method, comprising:

[0007] The original file is dynamically sharded using an improved consistent hashing algorithm to generate multiple data blocks with timestamps. The improved consistent hashing algorithm dynamically adjusts the sharding boundaries through a two-layer mapping of virtual nodes and physical node groups, and intelligently allocates data blocks based on the storage medium type of the node group.

[0008] Constructing a three-dimensional Bloom filter index matrix containing timestamps, data types, and content features for the timestamp-carrying data blocks, and associating a B+ tree local index with an inverted global index through a hierarchical index structure;

[0009] An index update priority queue is established by using a double-channel synchronization mechanism, hot index is incrementally synchronized in real time by using a heartbeat mechanism, and cold data layer index is batch-synchronized according to a cold data synchronization period;

[0010] When querying, the data distribution probability is predicted by using a distributed query statistical probability table, a multi-path query is initiated in parallel based on a probability weight, and the query path is dynamically optimized according to the node group storage medium type and the network transmission delay between nodes.

[0011] As a preferred scheme of the distributed storage and index method, wherein:

[0012] The implementation of the dynamic adjustment of the shard boundary comprises:

[0013] The real-time storage utilization rate data and historical load fluctuation characteristics of the physical node group are taken as input data, a dynamic monitoring window is established at the virtual node layer, when it is detected that the continuous timestamp storage utilization rate of the target physical node group exceeds the upper limit of the storage utilization rate threshold, a dynamic shard instruction is triggered;

[0014] Based on the historical load fluctuation characteristics, the over-standard data block is split into a plurality of sub-data blocks along the time axis, and the original timestamp identifier is inherited;

[0015] A virtual node pointer mapped to a low-load physical node group is created for the newly generated sub-data block;

[0016] A dynamic shard topology graph containing the mapping relationship between the sub-data block and the new virtual node is formed.

[0017] As a preferred scheme of the distributed storage and index method, wherein:

[0018] The intelligent allocation of the data block comprises:

[0019] The timestamp identifier and the associated access log of the data block are taken as input data, a data timeliness evaluation model is constructed, and the data block is marked as a hot data layer, a warm data layer and a cold data layer;

[0020] By marking the data block as a hot data layer, a warm data layer and a cold data layer, a dynamically matched storage medium strategy is generated;

[0021] The dynamically matched storage medium strategy comprises that the hot data layer is forced to be written into a cache acceleration area of an SSD storage node group, the warm data layer is dynamically migrated to an SSD or an HDD storage node group according to an access frequency, and the cold data layer is stored into an archive storage area of an HDD storage node group after compression;

[0022] A storage medium configuration identifier is generated for each data block and written into a metadata descriptor, and finally a data block copy set with the storage medium configuration identifier is output;

[0023] The intelligent allocation data block pre-creates SSD and HDD dual medium copies at data block writing time, and dynamically activates the optimal copy according to the access mode.

[0024] As a preferred scheme of the distributed storage and indexing method of the application, wherein:

[0025] The method for constructing the data timeliness evaluation model comprises:

[0026] The input end receives timestamp metadata and associated access logs of the data block, medium performance benchmark parameters of the storage node group, and a real-time network topology map;

[0027] Based on the difference between the data block generation time and the current time, an exponential decay function is constructed, and a dynamic decay coefficient is set to adjust the timeliness weight of the data block;

[0028] The access time distribution characteristics of the same type of data block in the last 7 days are extracted, the periodic access law is identified, and a time sensitivity prediction curve is generated;

[0029] The time sensitivity is divided into three levels, including hot data layer, warm data layer and cold data layer, and a label containing timeliness level and medium matching suggestion is generated for each data block;

[0030] The access mode is continuously updated through a sliding time window, and the model is recalibrated when an abnormal decrease in storage medium performance parameters is detected.

[0031] As a preferred scheme of the distributed storage and indexing method of the application, wherein:

[0032] The method for pre-creating SSD and HDD dual medium copies comprises:

[0033] In the associated access log writing stage of the timestamp identifier carried by the data block, the cache copy of the SSD storage node group and the persistent copy of the HDD storage node group are generated synchronously;

[0034] The associated access log comprises a predicted access frequency and a real-time state table of the storage node group;

[0035] The cache copy and the persistent copy are created with a copy metadata descriptor containing storage medium performance parameters, node group topology location and creation time;

[0036] A dual copy mapping relationship table is established in the metadata descriptor, and the initial copy activation state is set to sleep mode;

[0037] The dynamic activation of the optimal copy comprises:

[0038] An access pattern analyzer is arranged to count the request response time and access times of the data block replica set within 1 hour in real time;

[0039] A replica performance index is calculated based on the storage medium performance benchmark parameter and the real-time inter-node network transmission delay;

[0040] An activation decision maker is arranged to select the replica with the highest replica performance index as the optimal replica, and if the performance difference between the SSD replica and the HDD replica is less than the performance difference tolerance threshold, the SSD replica is activated preferentially, and the data block replica set with the activation state marker is output.

[0041] As a preferred scheme of the distributed storage and indexing method of the application, wherein:

[0042] The construction method of the three-dimensional Bloom filter index matrix comprises:

[0043] The timestamp metadata, file type identifier and content feature vector of the timestamped data block are inputted;

[0044] The timestamp is divided into time period codes according to a preset time window, the file header information is parsed to generate classification codes containing file format and structure features, the content fingerprint hash value is extracted from the first 1024 bytes of the data block, and the time period codes and the coding values of the file format and structure features are inputted into parallel hash channels to generate a set of non-overlapping bitmap coordinates;

[0045] A three-dimensional Bloom filter index matrix formed by superimposing three independent bitmap spaces is outputted;

[0046] Each dimension bitmap of the three-dimensional Bloom filter index matrix is cross-mapped with the metadata descriptor.

[0047] As a preferred scheme of the distributed storage and indexing method of the application, wherein:

[0048] The association mechanism of the hierarchical index structure comprises:

[0049] The B+ tree local index entries and the inverted global index query request are inputted;

[0050] The index converter converts the B+ tree leaf node entries into key-value pairs of the inverted global index, establishes a reverse mapping table to record the entry correspondence relationship between the B+ tree local index and the inverted global index, and the dynamic weight distributor adjusts the index access weight according to the query request type:

[0051] The B+ tree local index is preferentially accessed for the exact match query;

[0052] The inverted global index is preferentially accessed for the fuzzy match query;

[0053] Output a joint index access instruction set with weight identification;

[0054] The instruction set is fed back to the index update priority queue in real time.

[0055] As a preferred solution of the distributed storage and indexing method of the present invention, wherein:

[0056] The implementation of real-time incremental synchronization of the hotspot index through the heartbeat mechanism includes:

[0057] The data is based on the index access frequency carried by the heartbeat packets between storage node groups and the network transmission delay data between nodes;

[0058] The dynamic priority calculator calculates the index priority coefficient based on access frequency, data freshness, and node load, establishes a multi-level cache queue, and stores index entries with index priority coefficients higher than the threshold in the high-speed synchronization channel. When the heartbeat trigger module detects a difference in index versions between adjacent nodes, it extracts the difference entries and compares them with the index priority coefficient.

[0059] The output generates a set of incremental synchronization instructions with priority tags;

[0060] The incremental synchronization instruction set includes a target node group address, an index entry list, and a transmission order strategy, and mandates that the SSD storage node group give priority to responding to high-speed channel requests.

[0061] As a preferred solution of the distributed storage and indexing method of the present invention, wherein:

[0062] The cold data layer indexing process includes:

[0063] Input the remaining capacity warning signal of the storage node group and the cold data layer access timestamp record;

[0064] The cold data layer identifier marks indexes with no access records for several months and whose storage media is HDD as data to be migrated. The batch packager aggregates the index entries of the data to be migrated according to the rack topology, generates compressed data packets, and attaches topology routing information. Based on historical network traffic patterns, batch transfer is initiated during low-load periods.

[0065] Output the migration task queue containing the target rack coordinates and storage media type requirements;

[0066] When executing the migration task in the migration task queue, the source node index shadow copy is retained until the target node group confirms that the writing is completed.

[0067] As a preferred solution of the distributed storage and indexing method of the present invention, wherein:

[0068] The training process of the distributed query statistical probability table includes:

[0069] Input historical query log, node group storage medium type and network topology state record;

[0070] Log feature extraction extracts query pattern features from historical query logs, including time correlation, data type preference and path hop count, environment simulator builds a virtual training scene, injects storage medium type read-write delay parameters and network jitter variables, model updater dynamically adjusts storage medium type weight coefficient and path selection strategy according to feedback reward value of query success rate and response time;

[0071] Output configuration file containing storage medium sensitivity parameters and network topology adaptation parameters distributed query statistical probability table;

[0072] The path selection weight of the SSD storage node group in the configuration file is set to be greater than that of the HDD storage node group, and the real-time response type query is forced to bind the SSD storage node group path.

[0073] As a preferred scheme of the distributed storage and indexing method of the application, wherein:

[0074] The execution of the dynamic optimization query path includes:

[0075] Based on the probability distribution data output by the distributed query statistical probability table and the real-time node group state message;

[0076] The path generator generates three candidate query paths according to the probability weight, and each candidate query path contains at least two storage medium type node groups, and the path evaluator calculates the composite evaluation value of each candidate query path;

[0077] Output query routing instruction package containing optimal path identification and backup path cache;

[0078] The query routing instruction package carries a storage medium type verification mark to ensure that the returned data matches the medium performance requirements of the query request.

[0079] The application has the following beneficial effects:

[0080] The application sets an improved consistent hash algorithm, which realizes dynamic adjustment of the shard boundary by virtue of the double-layer mapping of virtual nodes and physical node groups, and automatically splits and reallocates data blocks according to the real-time storage utilization rate and historical load fluctuation of the physical node group, avoids the problem of uneven node load, improves the overall utilization rate of system resources, ensures stable operation of the system, pre-creates dual-medium copies and dynamically activates the optimal copy according to the access mode, and further improves the data access efficiency.

[0081] This application creates an index structure to construct a three-dimensional Bloom filter index matrix for data blocks with timestamps, integrates timestamps, data types and content features, and generates non-overlapping bitmap coordinate sets through parallel hash channels, which reduces the storage space of the index and greatly improves the query efficiency. The bitmaps of each dimension of the index matrix establish a cross-mapping relationship with the metadata descriptor to facilitate the rapid positioning and retrieval of data, and dynamically adjust the index access weight according to the query type. Exact match queries give priority to using the B+ tree local index, while fuzzy match queries give priority to accessing the inverted global index, which improves the accuracy and efficiency of the query. At the same time, the establishment of an index update priority queue ensures the real-time and consistency of the index. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort. Among them:

[0083] Figure 1 This is an overall flow chart of a distributed storage and indexing method of the present invention;

[0084] Figure 2 A diagram of a distributed storage and indexing method of the present invention;

[0085] Figure 3 This is a flow chart of a dynamic sharding method in a distributed storage and indexing method of the present invention. DETAILED DESCRIPTION

[0086] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0087] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0088] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0089] like Figure 1 As shown, a distributed storage and indexing method includes:

[0090] As Figure 2 shown, the original file is dynamically fragmented by an improved consistent hashing algorithm to generate multiple timestamped data blocks, the improved consistent hashing algorithm realizes dynamic adjustment of fragmentation boundary through double-layer mapping of virtual nodes and physical node groups, and intelligently allocates data blocks according to the node group storage medium type;

[0091] The implementation of the dynamic adjustment of the fragmentation boundary includes:

[0092] The real-time storage utilization rate data and historical load fluctuation characteristics of the physical node group are used as input data;

[0093] A dynamic monitoring window is established at the virtual node layer, and when it is detected that the continuous timestamp storage utilization rate of the target physical node group exceeds the upper limit of the storage utilization rate threshold, a dynamic fragmentation instruction is triggered;

[0094] Each physical node group periodically reports the current storage utilization rate to form a continuous time series data stream, and the system analyzes these data in real time through stream processing.

[0095] The system maintains a fixed-length sliding time window, and if the utilization rates of a plurality of consecutive sampling points in the window exceed the threshold value and the timestamp range of the data block completely covers the current window, a dynamic fragmentation instruction is triggered. This mechanism avoids false judgments due to transient peaks and ensures the stability of the splitting decision.

[0096] Based on the historical load fluctuation characteristics, the over-standard data block is split into a plurality of sub-data blocks along the time axis, and the original timestamp identifier is inherited;

[0097] A virtual node pointer mapped to a low-load physical node group is created for the newly generated sub-data block;

[0098] A low-load physical node refers to a physical device in a distributed system or an Internet of Things environment whose current resource utilization rate is far below its processing capability upper limit. The low-load physical node usually has redundant capacity to receive additional tasks and is the main target object of task allocation in the load balancing strategy.

[0099] A dynamic fragmentation topology containing the mapping relationship between the sub-data block and the new virtual node is formed;

[0100] The splitting instruction is set to retain the hash value prefix of the original data block to ensure the index compatibility before and after splitting.

[0101] In the present application, a preferred implementation method for dynamically adjusting the fragmentation boundary includes:

[0102] The dynamic monitoring window configuration system determines the length of the monitoring window by analyzing the cyclical characteristics of historical load. For example, if historical data shows that a node group's storage load peaks typically last about two hours, the system will set the monitoring window slightly longer than the peak duration to include a buffer to avoid false splits caused by brief fluctuations. A safety factor is added to the window length based on actual scenarios to ensure that load fluctuations are covered.

[0103] Storage utilization threshold settings include static thresholds and dynamic thresholds. The dynamic threshold combines the sliding average and fluctuation range of historical load to dynamically adjust the upper threshold limit.

[0104] Static thresholds set different thresholds based on the performance differences of storage media.

[0105] Furthermore, data block splitting includes:

[0106] Based on historical load fluctuation patterns, the system divides the overload data block into multiple sub-data blocks along the time axis. For example, an original data block covering a 1-hour timestamp may be split into four 15-minute sub-data blocks.

[0107] Each sub-data block inherits the timestamp identifier of the original data block and appends the time interval suffix of the sub-data block to ensure data traceability and query compatibility.

[0108] The system prioritizes migrations for low-load physical node groups with the same storage media. For example, if the original data blocks are stored on an SSD node group, other low-load SSD node groups are prioritized to avoid performance bottlenecks caused by cross-media migration. Node group load assessment comprehensively considers real-time utilization, historical load trends, and hardware performance weights, selecting the optimal target through weighted calculation.

[0109] The system uses a graph structure to record the mapping relationship between the split sub-data blocks and virtual nodes and physical node groups. For example, the original data block points to multiple sub-data blocks, each of which is bound to a new virtual node and ultimately mapped to the target physical node group.

[0110] The metadata of the split operation is fully recorded, supporting rapid backtracking during subsequent capacity expansion or fault recovery.

[0111] The hash value of the sub-data block is generated by appending the time interval suffix to the original hash value, ensuring that the index layer can quickly locate the original data block through prefix matching during query, and then filter out the specific sub-data block based on the timestamp.

[0112] Wherein, the intelligent allocation data block includes:

[0113] Using the timestamp identifier of the data block and the associated access log as input data, the data block is marked as a hot data layer, a warm data layer, and a cold data layer by building a data timeliness evaluation model;

[0114] Furthermore, the data timeliness evaluation model construction method includes:

[0115] The input end receives the timestamp metadata of the data block and the associated access log, the media performance benchmark parameters of the storage node group and the real-time network topology map;

[0116] Based on the difference between the data block generation time and the current time, an exponential decay function is constructed, and a dynamic decay coefficient is set to adjust the timeliness weight of the data block;

[0117] Furthermore, the input of the exponential decay function includes timestamps, access logs, storage medium performance, transmission delay between nodes, and bandwidth status;

[0118] An exponential decay function is constructed based on the time difference, reducing the timeliness weight of the data over time. The decay speed is dynamically adjusted based on the storage medium performance and real-time network status. By analyzing the recent access logs, the time distribution characteristics are extracted, and a prediction curve is generated to monitor abnormal high-frequency access events. The timeliness weight of related data is temporarily increased to avoid misjudgments due to sudden demand.

[0119] Extract the access time distribution characteristics of data blocks of the same type, identify periodic access patterns, and generate time sensitivity prediction curves;

[0120] The sensitivity prediction curve divides time sensitivity into three levels, including hot data layer, warm data layer, and cold data layer. It also generates a label for each data block, including the timeliness level and medium matching suggestion.

[0121] The hot data layer has a validity period of (0, n) hours, and the response delay of the SSD storage node group is required to be no more than A1ms;

[0122] The warm data layer has a validity period of [n, 24] hours, allows for mixed deployment of SSDs and HDDs, and has a latency of no more than A2ms.

[0123] The cold data layer has a validity period of >24 hours and only requires HDD storage node groups. The latency tolerance is no less than A3ms. Among them, A1 is less than A2 and less than A3.

[0124] Access patterns are continuously updated through a sliding time window, and model recalibration is triggered when storage medium performance degradation of more than 15% is detected.

[0125] In this application, a preferred specific implementation method for constructing a data timeliness evaluation model includes:

[0126] First, the input data is preprocessed and feature extracted. Further, the data preprocessing includes integrating timestamp metadata, associated access logs and medium performance benchmark parameters of the storage node group, specifically including:

[0127] The timestamp metadata records the generation time and the last access time of the data block; the associated access log counts the access frequency and time period distribution of each data block in the time dimension; the medium performance benchmark parameters of the storage node group define the benchmark performance of SSD and HDD.

[0128] The network topology map obtains the network delay, bandwidth and connection state of the node group in real time.

[0129] Further, the feature preprocessing includes converting the time difference between the data block generation time and the current time into a time decay factor, and generating a time series feature by counting the access frequency density according to the time window of the access log; a basic decay formula of an exponential decay function is established, and a dynamic decay coefficient is set.

[0130] Further, the basic decay formula is that the timeliness weight of the data block decreases exponentially with the increase of the "data age", and the initial weight is determined by the access heat at the time of generation.

[0131] For example: the access volume of a certain type of data increases sharply from 14:00 to 16:00 every day, which is marked as "periodic peak period", and the time sensitivity prediction curve is generated by combining historical rules and real-time access trend to generate a time sensitivity prediction curve for the next 24 hours, and the periodic peak period is given a higher sensitivity weight.

[0132] The intelligent label generates an output label for each data block, including timeliness level, recommended storage medium and expected response delay; finally, the dynamic update includes a sliding time window update mechanism and model calibration.

[0133] Specifically, a sliding time window update mechanism is set, and the trigger condition is that if the actual access frequency of the data block deviates abnormally from the prediction curve, the statistical probability table is retrained; if the read-write performance of the storage node group decreases, the global label recalculation and data migration are triggered.

[0134] Further, the performance monitoring in the model calibration process is used to collect the read-write rate, delay and other indicators of the storage medium in real time.

[0135] When the performance of the SSD node group decreases, the allocation weight is reduced, and hot data is preferentially migrated to the SSD node with performance up to standard.

[0136] The time sensitivity accurately predicts the history rule and real-time data, avoiding misjudgment caused by static rules; the medium performance self-adaptation dynamically adjusts the data distribution according to the hardware state, maximizing the resource utilization; the business lossless migration ensures the smoothness of the data classification and migration process through the sliding window and gradual calibration.

[0137] The hot data layer is the data block generated in the shortest time, the warm data layer is the data block generated in the shortest time to 24 hours, and the cold data layer is the data block generated more than 24 hours;

[0138] By marking the data block as a hot data layer, a warm data layer and a cold data layer, a dynamically matched storage medium strategy is generated;

[0139] The dynamically matched storage medium strategy includes a hot data layer forced to write into a cache acceleration area of an SSD storage node group, a warm data layer dynamically migrated to an SSD or an HDD storage node group according to access frequency, and a cold data layer stored into an archival storage area of an HDD storage node group after compression;

[0140] A storage medium configuration identifier is generated for each data block and written into a metadata descriptor, and finally a data block copy set with the storage medium configuration identifier is output;

[0141] The intelligent allocation data block pre-creates an SSD and an HDD dual-medium copy when the data block is written, and dynamically activates the optimal copy according to the access mode.

[0142] Further, the method of pre-creating an SSD and an HDD dual-medium copy includes:

[0143] In the associated access log writing stage of the timestamp identification carried by the data block writing, a cache copy of an SSD storage node group and a persistent copy of an HDD storage node group are synchronously generated;

[0144] The associated access log includes a predicted access frequency and a storage node group real-time state table;

[0145] A copy metadata descriptor containing storage medium performance parameters, node group topology location and creation time is created for the cache copy and the persistent copy;

[0146] A dual-copy mapping relationship table is established in the metadata descriptor, and the initial copy activation state is set to a hibernation mode.

[0147] The dynamically activated optimal copy includes:

[0148] An access mode analyzer is set, and the request response time and the access times of the data block copy set are statistically calculated in real time;

[0149] The replica efficiency index is calculated based on the storage medium performance benchmark parameter and the real-time inter-node network transmission delay, an activation decision maker is set, the replica with the highest replica efficiency index is selected as the optimal replica, if the performance difference between the SSD replica and the HDD replica is less than the performance difference tolerance threshold, the SSD replica is activated preferentially, and the data block replica set with the activation state marker is output.

[0150] The performance difference tolerance threshold is analyzed through historical data, if the performance difference between the two replicas is less than the minimum threshold, the performance gap is not significant, the SSD is preferentially selected to utilize its low delay characteristics, when the overall system load is high, the threshold can be temporarily relaxed to avoid increasing the cost caused by frequent replica switching, the minimum threshold is set by the storage medium performance benchmark difference, and the temporarily relaxed threshold is temporarily increased when the overall system load is high, so that the HDD is preferentially selected when the performance difference between the SSD and the HDD replica is larger.

[0151] A pre-creation dual-medium replica implementation method includes:

[0152] The dual-replica pre-creation and metadata management include dual-replica synchronization generation and replica metadata descriptor design.

[0153] The dual-replica synchronization generation includes write stage triggering and associated data range.

[0154] Further, when the data block carries a timestamp to write into the system, two replicas are synchronously created according to the predicted access frequency from the historical model association and the real-time state table of the storage node group.

[0155] The SSD cache replica is used to store a high-performance SSD node group, and is used to cope with short-term high-frequency access.

[0156] The HDD persistent replica is used to store a low-cost HDD node group, and is used for long-term backup and low-frequency access scenarios.

[0157] The replica metadata descriptor includes:

[0158] The storage medium performance parameter sets the SSD read speed and the HDD seek time.

[0159] The node group topology location includes the location of the physical node group in the machine room and the network routing path.

[0160] The replica creation time is accurate to the timestamp of milliseconds, which is used for subsequent timeliness evaluation.

[0161] In the initial state of the sleep mode, both the dual replicas are marked as sleep mode, that is, the replicas are not activated as accessible state, only the metadata relationship is reserved, the replicas in the sleep mode do not participate in load balancing until they are awakened by the dynamic activation strategy.

[0162] Furthermore, the dynamic replica activation decision mechanism includes real-time analysis of access patterns, calculation of replica performance index, and setting of performance difference tolerance threshold;

[0163] Among them, real-time analysis of access patterns includes statistical indicators and data correlation:

[0164] The request response time metric is calculated by recording the access latency of two replicas of the same data block within a one-hour period. The access count metric is calculated by counting the frequency of replica calls, distinguishing between active accesses and background synchronization operations. Data relevance only analyzes the performance of the SSD and HDD replicas of the same data block to avoid cross-data interference.

[0165] The replica performance index calculation includes baseline performance weighting and network latency correction;

[0166] Among the benchmark performance weights, the SSD benchmark performance value = SSD actual read speed / maximum read speed × 100%; the HDD benchmark performance value = HDD actual read speed / minimum read speed × 100%.

[0167] In this application, network delay correction is calculated based on the real-time topology to calculate the network transmission delay from the client to the node group where the replica is located. The replica performance index = baseline performance weight × T1 + (1 / network delay) × T2, where T1 is the baseline performance weight coefficient and T2 is the network delay coefficient.

[0168] A preferred example: the SSD copy has a read speed of 600MB / s and a delay of 20ms, so the copy performance index = 120% × 0.7 + (1 / 0.02) × 0.3 = 0.84 + 15 = 15.84; the HDD copy has a read speed of 120MB / s (80% benchmark) and a delay of 50ms, and the performance index = 80% × 0.7 + (1 / 0.05) × 0.3 = 0.56 + 6 = 6.56.

[0169] Replica activation and state switching include activation decision logic and exception handling and rollback.

[0170] Among them, the activation decision maker logic priority rule is set: if the SSD copy performance index is greater than the sum of the HDD copy performance index and the threshold, the SSD copy is activated; if the performance difference is not greater than the threshold, the SSD copy is activated by default; if the load of the node group where the SSD copy is located is abnormal, it is forced to switch to the HDD copy.

[0171] The activation decision maker logic marks the activated replica as active and accepts external read and write requests; the inactivated replica remains dormant and is only used for failure recovery or batch reading.

[0172] If the response duration of the active replica exceeds the set threshold, re-evaluation is triggered immediately and the replica is switched.

[0173] Active state copy modification, asynchronous synchronization to the dormant state copy, ensure that the data is eventually consistent.

[0174] An intelligent allocation data block embodiment method comprises:

[0175] A three-dimensional Bloom filter index matrix containing timestamps, data types and content characteristics is constructed for the timestamped data block, and a B+ tree local index is associated with an inverted global index through a hierarchical index structure;

[0176] The construction method of the three-dimensional Bloom filter index matrix comprises:

[0177] The timestamp metadata, file type identifier and content feature vector of the timestamped data block are inputted;

[0178] The timestamp accurate to the millisecond level is divided into time period codes according to a preset time window, the file header information is parsed, the classification code containing the file format and structural features is generated, the content fingerprint hash value is extracted from the first 1024 bytes of the data block, and the time period code and the coding value of the file format and structural features are inputted into a parallel hash channel to generate a set of non-overlapping bitmap coordinates;

[0179] The three-dimensional Bloom filter index matrix formed by the superposition of three independent bitmap spaces is finally outputted;

[0180] Each dimension bitmap of the three-dimensional Bloom filter index matrix is cross-mapped with the metadata descriptor.

[0181] An embodiment method for constructing a three-dimensional Bloom filter index matrix comprises input data preprocessing, three-dimensional Bloom filter construction and hierarchical index association.

[0182] The input data preprocessing embodiment method comprises:

[0183] The timestamp metadata segmentation code divides the timestamp accurate to the millisecond level into time period codes according to a preset time window.

[0184] In this application, a preferred example: the timestamp range 14:00:00-15:00:00 is uniformly coded as T14, and the coding value is a hash integer, such as CRC32 generating 0x3A7F. For high-frequency writing scenarios, the window is reduced to 5 minutes, and for low-frequency scenarios, the window is expanded to 6 hours.

[0185] The file header parsing reads the fixed length of the data block header, extracts the file magic number identifier format, for example, 0x89504E47 represents PNG.

[0186] The structural feature coding generates a classification code according to the file type definition structure label.

[0187] Extract the first 1024 bytes of the data block, generate a content fingerprint value through a hash function, slice the fingerprint value, and generate a multi-segment feature hash. The specific implementation method of constructing a three-dimensional Bloom filter includes:

[0188] Parallel hash channels include time dimension hashing, type dimension hashing, and content dimension hashing. The time dimension hashing uses the time series coding value as input and maps it to the bitmap space coordinates through multiple hash functions. For example, the time period code T14 generates the coordinates (x1, y1, z1) after hashing. The type dimension hashing inputs the file type and structure coding into an independent hash channel to generate the bitmap coordinates (x2, y2, z2). The content dimension hashing is based on content fingerprint sharding and generates the coordinates (x3, y3, z3) of similar content clusters through local sensitive hashing.

[0189] The hash results of the three dimensions are filled into the time bitmap, type bitmap and content bitmap respectively.

[0190] Specifically, the time bitmap records the time window distribution of data blocks; the type bitmap records the file format and structural characteristics; and the content bitmap records the content similarity clustering.

[0191] Finally, the three bitmaps are superimposed according to coordinates to form a three-dimensional Bloom filter index matrix, and each coordinate point indicates whether there is a data block with matching features.

[0192] The metadata descriptor cross-maps the descriptive information generated when the data block is written, including storage location, access policy, dual copy status, etc.

[0193] The coordinates of each dimension bitmap are associated with the metadata descriptor through the reverse index.

[0194] In this application, a preferred example is: the time bitmap coordinates (x1, y1, z1) are mapped to the storage node group address and access frequency statistics in the metadata descriptor; the metadata is quickly located using time code + type code + content hash fragment. The specific implementation method of hierarchical index association includes:

[0195] Organize by time range and build a B+ tree with timestamp as key value. Leaf nodes store the time dimension coordinates of data blocks in a three-dimensional matrix.

[0196] Fast range query supports querying all data blocks within a set range and locating the time bitmap area through the B+ tree.

[0197] The inverted list of inverted global index content features uses the content hash fragment as the key and records the three-dimensional coordinate set of all data blocks containing the feature.

[0198] The type classification index uses the file type code as the key and associates the data block coordinates that match the type.

[0199] An implementation method of an index linkage mechanism comprises:

[0200] Locate the time window corresponding three-dimensional matrix area through B+ tree.

[0201] Filter the data block of type "picture" and resolution label "1080P" through the inverted global index.

[0202] Take the intersection of time, type, and content three-dimensional coordinates, and return the final result.

[0203] Secondarily verify the "possible existence" result returned by the Bloom filter through the metadata descriptor.

[0204] Specifically, the storage system needs to quickly retrieve the time period when a specific picture appears; the timestamp addresses the time bitmap coordinate (x, y, z).

[0205] Map the file type video to the type bitmap coordinate.

[0206] Map the content hash segment matching the picture feature to the content bitmap coordinate.

[0207] The query condition determines the query time range, the query type as video, and the query content containing the picture feature.

[0208] The B+ tree locates the time range corresponding to the time bitmap area, the inverted index filters the video type, and the content index matches the feature hash.

[0209] Take the intersection of the three-dimensional matrix, return multiple candidate data blocks, and confirm the accurate result after metadata verification.

[0210] Through the setting of the three-dimensional Bloom filter, the compound condition preliminary screening of time, type, and content can be completed in constant time, the bitmap compression technology can reduce the matrix space occupation to 1 / 10 of the traditional index, and supports mixed queries from any dimension of time range, file type, and content feature.

[0211] Further, the association mechanism of the hierarchical index structure comprises:

[0212] Input the B+ tree local index entry and the inverted global index query request;

[0213] The index converter converts the B+ tree leaf node entry into the <keyword, node position> key-value pair of the inverted global index, establishes a mapping table to record the corresponding relationship between the B+ local index and the inverted global index entry, the dynamic weight distributor adjusts the index access weight according to the query request type, the precise match query preferentially accesses the B+ tree local index; the fuzzy match query preferentially accesses the inverted global index; and outputs the joint index access instruction set with weight identification;

[0214] The instruction set is fed back to the index update priority queue in real time.

[0215] A hierarchical index structure association mechanism implementation method comprises the following steps:

[0216] The B+ tree local index entry is composed of the timestamp range and storage location of the data block, and is used to support accurate range query.

[0217] The inverted global index query request comes from the fuzzy matching demand of the upper application on the content feature and file type.

[0218] The reverse mapping table construction method comprises the following steps:

[0219] S11: The time range entry in the B+ tree leaf node is parsed into the keyword of the inverted index, and is bound to the node position as a key-value pair.

[0220] S12: When a new data block is written, the key-value pair generation of the inverted index is triggered automatically by the B+ tree new entry, so that the two are synchronized in real time.

[0221] In the dynamic weight allocation strategy, the query type identification comprises: the accurate matching query condition contains an explicit time range or storage location; the fuzzy matching condition contains a keyword, content feature or file type.

[0222] In the weight allocation rule, the accurate query path comprises: the weight is biased to the B+ tree local index, the physical node group is directly located through the time range, and the global index scanning overhead is reduced;

[0223] The fuzzy query path weight is biased to the inverted global index, the candidate set is quickly narrowed by using the keyword, and the B+ tree is combined to filter the time range; for example, when a video containing a face feature is queried, the data block coordinates of all matching contents are obtained through the inverted index, and the entries inconsistent in time are excluded through the B+ tree.

[0224] In the mixed query processing, if the query contains accurate and fuzzy conditions, the access instruction is split according to the weight proportion, the intersection of the two is taken, and the verification is performed through the metadata descriptor.

[0225] The joint instruction set generation and feedback comprises an instruction set structure and real-time feedback to a priority queue:

[0226] In the instruction set structure design, each instruction comprises: index type identification, access path weight, and target data range.

[0227] Real-time feedback to the priority queue:

[0228] The statistical feedback data records the actual execution time, result hit rate and resource consumption of each instruction.

[0229] If the response time of the inverted index exceeds the threshold, its weight is reduced, and index shard optimization is triggered.

[0230] The queue update strategy prioritizes high-frequency query instructions and queues low-weight instructions. When the storage node group is expanded or contracted, all associated instructions in the queue are forcibly refreshed.

[0231] A dual-channel synchronization mechanism is used to establish an index update priority queue. Hotspot indexes are incrementally synchronized in real time through a heartbeat mechanism, and cold data layer indexes are batch-synchronized according to a cold data synchronization period.

[0232] The implementation of the hotspot index incrementally synchronized in real time through the heartbeat mechanism includes:

[0233] The index access frequency and inter-node network transmission delay data carried by the heartbeat packet between the storage node groups are used as the data basis.

[0234] The index priority coefficient is calculated by a dynamic priority calculator based on the access frequency, data freshness, and node load. A multi-level cache queue is established, and index entries with a priority coefficient higher than a threshold are stored in a high-speed synchronization channel. When the heartbeat trigger module detects a difference in index versions between adjacent nodes, it extracts the difference entries and matches them with the index priority coefficient.

[0235] The high-speed synchronization channel is a data transmission channel based on network protocol or physical link optimization, and is used to handle real-time and high-priority synchronization tasks. In the index synchronization scenario described by the user, the channel selects entries with a priority coefficient higher than a threshold, such as high-frequency access and high-freshness data, to ensure their priority transmission.

[0236] The output generates an incremental synchronization instruction set with a priority label.

[0237] The incremental synchronization instruction set contains the target node group address, index entry list, and transmission sequence strategy, and the SSD storage node group is forced to respond to high-speed channel requests.

[0238] A specific implementation method for real-time incremental synchronization of hotspot indexes includes:

[0239] The specific implementation of real-time incremental synchronization of hotspot indexes includes storage node group definition and data basis, and a dynamic priority calculator.

[0240] Storage node group definition and data basis:

[0241] Storage node group range:

[0242] The same cluster node group includes SSD and HDD node groups interconnected within the same data center through a high-speed local area network.

[0243] The cross-cluster node group contains node groups across regions or across availability zones.

[0244] The heartbeat packet data content is used to record the number of recent access times of each index entry, the difference between the last update time of the index entry and the current time, and the node load indicators including CPU utilization and the bidirectional network delay between node groups.

[0245] Specifically, the dynamic priority calculator sets index priority coefficient calculation rules:

[0246] The calculation rules include that the access frequency weight is higher for higher access frequency; the data freshness weight is higher for newer data; and the node load weight is higher for lower load.

[0247] A preferred example: a node load of 20% corresponds to a score of (1-0.2)×0.3=0.24, and the total index priority coefficient is 0.24+0.225+0.24=0.705.

[0248] The heartbeat triggers and difference entry matching node groups exchange index version numbers through heartbeat packets. A preferred example: the version of node group A is V20231001_1420, and the version of node group B is V20231001_1415, triggering difference analysis. Further, the difference entry matching logic specifically includes the following steps:

[0249] S21: Node group A sends the difference entry list to B.

[0250] S22: For each difference entry, both sides calculate the local index priority coefficient, and a preferred example is that the index priority coefficient of entry X in A is 0.8, and the index priority coefficient of entry X in B is 0.5.

[0251] S23: If the index priority coefficients of both sides are different, mark it as a forced synchronization entry.

[0252] The target node address needs to synchronize the node group identifier.

[0253] The transmission order strategy is to arrange in descending order of index priority coefficient, and transmit in ascending order of network delay for the same priority.

[0254] SSD node group forced response rules: the SSD node group reserves 20% bandwidth for processing high-speed channel requests; if the high-speed channel queue backlog, pause the ordinary channel task, and preferentially process high-priority entries; if the SSD node does not respond within 50ms, the instruction is automatically forwarded to the HDD node group.

[0255] The high-speed channel request is a special instruction initiated for a high-speed synchronization channel.

[0256] Further, the cold data layer indexing process comprises:

[0257] Input the remaining capacity warning signal of the storage node group and the cold data layer access timestamp record;

[0258] The cold data layer identifier marks the index of the data to be migrated as the data to be migrated, which has no access record for a plurality of consecutive months and the storage medium is HDD. The batch packager aggregates the index items of the data to be migrated according to the rack topology relationship, generates a compressed data packet and attaches topology routing information, and initiates batch transmission according to the historical network traffic law.

[0259] The low-load period is a resource idle period dynamically identified by the system according to historical network traffic, storage access frequency, power consumption and other data. In this period, the network congestion probability is low, the storage device response delay is small, and the server computing resource redundancy is high.

[0260] Output the migration task queue containing the target rack coordinates and the storage medium type requirement;

[0261] The migration task in the migration task queue retains a shadow copy of the source node index until the target node group confirms the write completion.

[0262] A specific implementation method of a cold data layer index comprises:

[0263] The cold data layer refers to a data layer with low storage access frequency and low real-time requirement, which is usually deployed on large-capacity and low-cost storage media and managed in groups according to the rack topology.

[0264] Index data of the data to be migrated that meets the following conditions.

[0265] The temporary index copy retained by the source node during the migration process is used to ensure the availability of query requests during the migration period, and is deleted after the target node group confirms the data write completion.

[0266] The node group association information library is constructed based on the physical layout of the data center, and contains information such as rack number, node group physical location, link bandwidth and load balancing strategy.

[0267] When querying, the data distribution probability is predicted by a distributed query statistical probability table, multiple path queries are initiated in parallel based on the probability weight, and the query path is dynamically optimized according to the node group storage medium type and the network transmission delay between nodes.

[0268] The training process of the distributed query statistical probability table comprises:

[0269] Input the historical query log, node group storage medium type and network topology state record;

[0270] The log feature extraction extracts query pattern features from historical query logs, including time correlation, data type preference, and path hop count. The environment simulator constructs a virtual training scenario, injects storage medium type read-write delay parameters and network jitter variables, and the model updater dynamically adjusts the storage medium type weight coefficient and the path selection strategy according to the feedback reward value of query success rate and response time.

[0271] The output includes a configuration file containing storage medium sensitivity parameters and network topology adaptation parameters. A distributed query statistical probability table implementation method:

[0272] The historical query log records the timestamp, data type, access node group path, response time, and query result information of historical query operations in the distributed system.

[0273] The query pattern feature is a core attribute extracted from the historical query log, including:

[0274] Time correlation: periodicity and burstiness of query operations in the time dimension;

[0275] Data type preference: differences in the performance requirements of different types of data on storage media;

[0276] Path hop count: the number of node groups involved in the query path, reflecting the complexity of network transmission.

[0277] The environment simulator constructs a virtual training scenario containing storage medium read-write delay parameters and network jitter variables, simulating the dynamic changes of the real distributed environment.

[0278] The training process relevance constructs a data input layer based on historical query logs, node group storage medium types, and network topology state records as input, forming the original data set for model training. The feature extraction layer analyzes the historical query log through log feature extraction, separates the core features of time correlation, data type preference, and path hop count, and establishes the initial mapping relationship between query behavior and storage media and network path. The scene simulation layer environment simulator injects read-write delay parameters of different storage media and network jitter variables based on the feature extraction results, constructs a multi-dimensional virtual training scenario, simulates the query execution conditions in the real environment, and the strategy optimization layer constructs the model updater to combine the feature extraction results with the virtual scenario output data, dynamically adjust the storage medium type weight coefficient, which can include SSD path preference weight, HDD capacity weight, and path selection strategy, including preferentially selecting low-hop paths or high-medium performance paths, forming a closed-loop optimization mechanism.

[0279] The output layer generates a configuration file containing storage medium sensitivity parameters and network topology adaptation parameters through iterative training, achieving standardized definition of query path prediction strategy.

[0280] Specific implementation methods include:

[0281] S31: Collect historical query logs, including full query records, with each record marked with query time, data type, access node group sequence, response time and success status; synchronously obtain node group configuration information, including storage media type, physical location, network connection bandwidth and historical load fluctuation data; clean the original data, eliminate abnormal records, and form a standardized training data set.

[0282] S32: Use a sliding window algorithm to identify query peak periods, construct a time-query volume association matrix, and count the access ratios of different data types to SSD / HDD node groups based on data access frequency and data operation type, forming a data type-media preference mapping table. Path hop statistics calculate the number of node group jumps in historical query paths, distinguishing between single-node group direct connections with 0 hops, double-hop paths across media types with 1 hop, and complex paths with multiple node groups with 2 or more hops, and establishing a path complexity distribution model.

[0283] S33: Storage medium parameter modeling sets read and write delay baseline values, read delay baseline, write delay baseline, and fluctuation range for the SSD node group, and sets read delay baseline values, write delay baseline values, and fluctuation range for the HDD node group; simulates transmission delay in the communication link between node groups, with the baseline value set based on physical distance, such as 5ms between nodes in the same city, 50ms between nodes in different locations, and packet loss rate, to build a virtual scenario library containing more than 10 typical network environments; cross-combines data type preferences with storage medium parameters, path hop count, and network jitter variables to generate multiple sets of differentiated training scenarios.

[0284] S34: Preset the path preference weight of the SSD storage node group, initially bind the SSD path for real-time response queries, and allow mixed media paths for analysis queries.

[0285] S35: The storage medium sensitivity parameter specifies the mandatory binding strategy and weight coefficient range for different data types for SSD node groups; the network topology adaptation parameter sets the transmission delay tolerance threshold and path hop limit between node groups, and automatically switches to redundant links; the output file format uses a standardized configuration language, including version number, training timestamp, parameter validity range and policy verification rules.

[0286] Multi-dimensional feature coupling establishes a ternary mapping relationship between query behavior, storage media, and network topology through cross-analysis of time correlation, data type preference, and path hop count, improving the model's adaptability to complex scenarios.

[0287] Policy constraint enhancement uses configuration files to specify mandatory binding rules for SSD node groups for real-time response queries and the quantized range of SSD path optimization weights, ensuring performance while avoiding over-generalization of policies.

[0288] Furthermore, the execution of the dynamic optimization query path includes:

[0289] Probability distribution data and real-time node group status messages output based on distributed query statistical probability tables;

[0290] The path generator generates three candidate query paths based on probability weights. Each candidate query path contains at least two node groups of storage media types. The path evaluator calculates a composite evaluation value for each candidate query path, where the SSD storage node group contributes a performance weight and the inter-node network transmission delay contributes a weight. The execution scheduler uses the first-response-first-response principle and triggers a fast confirmation mechanism for the first path that returns valid data.

[0291] Output a query routing instruction packet containing the optimal path identifier and alternative path cache;

[0292] The query routing instruction packet carries a storage medium type verification mark to ensure that the returned data matches the medium performance requirements of the query request.

[0293] A specific implementation method for dynamically optimizing a query path includes:

[0294] The distributed query statistical probability table is used to predict the query-related probability distribution in a distributed system. The distributed query statistical probability table is trained based on various factors such as historical data and system status. The output probability distribution data can reflect information such as the probability of different queries on each node group.

[0295] The real-time node group status message contains the current real-time status information of each node group, such as the node group's storage utilization, load, network connection status, etc. This information is very important for generating reasonable query paths and evaluating the quality of paths.

[0296] The path generator generates candidate query paths based on the input probability distribution data and real-time node group status messages, combined with probability weights. Each path is required to contain node groups of at least two storage media types. This allows for the comprehensive utilization of the characteristics of different storage media to improve query performance.

[0297] The path evaluator is used to calculate the composite evaluation value of each candidate query path. SSD has high read and write speeds, and the performance of the SSD storage node group accounts for a higher proportion than the node network transmission.

[0298] The execution scheduler adopts the principle of first response priority, that is, which candidate query path returns valid data first, triggers the fast confirmation mechanism for the path to ensure that the system can obtain the required data as soon as possible and improve the query efficiency.

[0299] When a certain path returns valid data first, the execution scheduler quickly confirms the path to avoid continuing to wait for the results of other paths and reduce unnecessary waiting time.

[0300] The query routing instruction package contains the optimal path identifier and the alternative path cache. The optimal path identifier is used to indicate which path the system uses for query, and the alternative path cache is used to quickly switch to other paths when the optimal path fails, ensuring the reliability of the query.

[0301] In the query routing instruction package, the storage medium type verification marker is used to ensure that the returned data matches the medium performance requirements required by the query request. For example, if the query request requires the use of high-performance storage media, the verification marker can be used to check whether the returned data is indeed from the storage medium that meets the requirements.

[0302] Specific implementation method:

[0303] S41: Obtain probability distribution data from the distributed query statistical probability table, which reflects the likelihood of different queries on each node group, and collect real-time state messages of each node group, including storage utilization, load condition, network connection state and other information.

[0304] S42: The path generator generates three candidate query paths according to the input probability distribution data and real-time node group state messages, and combines the probability weight. When generating the path, it is ensured that each path contains at least two types of storage medium node groups, such as SSD storage node groups and HDD storage node groups.

[0305] S43: The path evaluator calculates the composite evaluation value of each candidate query path. For each path, the performance of the SSD storage node group and the network transmission delay between nodes are weighted and calculated. For example, if the performance score of the SSD storage node group in a certain path is S and the network transmission delay between nodes is N, then the composite evaluation value of the path is V = 0.7S + 0.3N.

[0306] S44: The execution scheduler sends query requests to the three candidate query paths simultaneously, adopts the principle of first response priority, and when a certain path returns valid data first, the execution scheduler triggers the fast confirmation mechanism for the path immediately to confirm that the path is the current query path.

[0307] S45: After determining the optimal path, generate a query routing instruction package containing the optimal path identifier and the alternative path cache.

[0308] S46: Execute the query and verify the data. Based on the optimal path identifier in the query routing instruction packet, the query operation is executed. After obtaining the returned data, the storage medium type verification flag is used to verify that the data meets the media performance requirements of the query request. If not, another path is selected from the alternative path cache and the query is executed again.

[0309] It is important to note that the construction and arrangement of the present application shown in a plurality of different exemplary embodiments are merely illustrative. Although only a plurality of embodiments are described in detail in this disclosure, it should be readily understood by those who refer to this disclosure that, without departing substantially from the novel teachings and advantages of the subject matter described in this application, many modifications are possible, for example, the size, scale, structure, shape and proportion of various elements, as well as parameter values ​​(e.g., temperature, pressure, etc.), mounting arrangements, use of materials, color, directional changes, etc. For example, an element shown as integrally formed can be composed of multiple parts or elements, the position of the element can be inverted or otherwise changed, and the nature or number or position of the discrete elements can be altered or changed. Therefore, all such modifications are intended to be included within the scope of the present invention. The order or sequence of any process or method steps can be changed or reordered according to alternative embodiments. Any "device plus function" clause is intended to cover the structure of the execution function described herein, and is not only structurally equivalent but also equivalent structures. Without departing from the scope of the present invention, other replacements, modifications, changes and omissions can be made in the design, operating conditions and arrangement of the exemplary embodiments. Therefore, the invention is not limited to the specific embodiments, but extends to various modifications that still fall within the scope of the appended claims.

[0310] Additionally, in order to provide a concise description of exemplary embodiments, all features of an actual embodiment (i.e., those features that are not relevant to the best mode presently contemplated for carrying out the invention or those that are not relevant to implementing the invention) may not be described.

[0311] It should be understood that in the development of any actual embodiment, as in any engineering or design project, numerous implementation-specific decisions may be made. Such a development effort may be complex and time-consuming, but for those of ordinary skill having the benefit of this disclosure, the development effort will be a routine task of design, fabrication, and production without undue experimentation.

[0312] It should be noted that the above examples are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced, without departing from the spirit and scope of the technical solutions of the present application, which should be covered in the scope of the claims of the present application.

Claims

1. A distributed storage and indexing method, characterized in that: include: The original file is dynamically sharded using an improved consistent hashing algorithm to generate multiple data blocks with timestamps. The improved consistent hashing algorithm dynamically adjusts the sharding boundaries through a two-layer mapping of virtual nodes and physical node groups, and intelligently allocates data blocks based on the storage medium type of the node group. Constructing a three-dimensional Bloom filter index matrix containing timestamps, data types, and content features for the timestamp-carrying data blocks, and associating a B+ tree local index with an inverted global index through a hierarchical index structure; A dual-channel synchronization mechanism is used to establish an index update priority queue. Hot indexes are synchronized incrementally in real time through a heartbeat mechanism, while cold data layer indexes are synchronized in batches according to the cold data synchronization cycle. During query, the data distribution probability is predicted through the distributed query statistical probability table, multi-path queries are initiated in parallel based on the probability weights, and the query path is dynamically optimized according to the node group storage medium type and the network transmission delay between nodes.

2. A distributed storage and indexing method according to claim 1, characterized in that: The implementation of dynamic adjustment of shard boundaries includes: Using the real-time storage utilization data and historical load fluctuation characteristics of the physical node group as input data, a dynamic monitoring window is established at the virtual node layer. When it is detected that the continuous timestamp storage utilization of the target physical node group exceeds the storage utilization threshold upper limit, the dynamic sharding instruction is triggered; Based on the historical load fluctuation characteristics, the excessive data block is split into multiple sub-data blocks along the time axis, and the sub-data blocks inherit the original timestamp identifier; Creating a virtual node pointer mapped to a low-load physical node group for the newly generated sub-data block; A dynamic sharding topology diagram is formed, which includes the mapping relationship between sub-data blocks and new virtual nodes.

3. A distributed storage and indexing method according to claim 2, characterized in that: The intelligent allocation data block includes: Using the timestamp identifier of the data block and the associated access log as input data, the data block is marked as a hot data layer, a warm data layer, and a cold data layer by building a data timeliness evaluation model; Generating a dynamic matching storage medium strategy by marking the data blocks as hot data tiers, warm data tiers, and cold data tiers; The dynamic matching storage medium strategy includes forcibly writing the hot data layer to the cache acceleration area of ​​the SSD storage node group, dynamically migrating the warm data layer to the SSD or HDD storage node group based on the access frequency, and compressing the cold data layer and storing it in the archive storage area of ​​the HDD storage node group; Generate a storage medium configuration identifier for each data block and write it into the metadata descriptor, and finally output a data block replica set with the storage medium configuration identifier; The intelligent allocation of data blocks pre-creates SSD and HDD dual-media copies when data blocks are written, and dynamically activates the optimal copy according to the access pattern.

4. A distributed storage and indexing method according to claim 3, characterized in that: The data timeliness evaluation model construction method includes: The input end receives the timestamp metadata of the data block and the associated access log, the media performance benchmark parameters of the storage node group and the real-time network topology map; Based on the difference between the data block generation time and the current time, an exponential decay function is constructed, and a dynamic decay coefficient is set to adjust the timeliness weight of the data block; Extract the access time distribution characteristics of data blocks of the same type, identify periodic access patterns, and generate time sensitivity prediction curves; Time sensitivity is divided into three levels: hot data layer, warm data layer, and cold data layer. At the same time, a label containing the timeliness level and medium matching suggestion is generated for each data block; Access patterns are continuously updated through a sliding time window, and model recalibration is triggered when abnormal storage medium performance degradation is detected.

5. A distributed storage and indexing method according to claim 3, characterized in that: The method for pre-creating dual media copies of SSD and HDD includes: During the access log writing phase associated with the timestamp carried by the data block write, cache copies of the SSD storage node group and persistent copies of the HDD storage node group are generated synchronously. The associated access log includes a predicted access frequency and a real-time status table of a storage node group; Create replica metadata descriptors for cached and persistent replicas, including storage media performance parameters, node group topology location, and creation time; Create a dual-copy mapping relationship table in the metadata descriptor and set the initial copy activation state to sleep mode; The dynamic activation of the optimal copy includes: Setting an access pattern analyzer to count the request response time and access count of the data block replica set in real time; Calculate the replica effectiveness index based on storage media performance benchmark parameters and real-time inter-node network transmission delay; An activation decider is set to select the replica with the highest replica performance index as the optimal replica. If the performance difference between the SSD replica and the HDD replica is less than the performance difference tolerance threshold, the SSD replica is activated first and a data block replica set with an activation status mark is output.

6. A distributed storage and indexing method according to claim 1, characterized in that: The method for constructing the three-dimensional Bloom filter index matrix includes: Input timestamp metadata, file type identifier and content feature vector of the timestamp data block; Divide the timestamp into time period codes according to the preset time window, parse the file header information, generate a classification code containing the file format and structural characteristics, extract the data block to generate a content fingerprint hash value, input the time period code, file format and structural characteristics of the code value into the parallel hash channel, and generate a non-overlapping bitmap coordinate set; Output a three-dimensional Bloom filter index matrix formed by the superposition of three independent bitmap spaces; Each dimension bitmap of the three-dimensional Bloom filter index matrix establishes a cross-mapping relationship with the metadata descriptor.

7. A distributed storage and indexing method according to claim 1, characterized in that: The association mechanism of the hierarchical index structure includes: Input B+ tree local index entries and inverted global index query requests; The index converter converts B+ tree sub-node entries into key-value pairs of the inverted global index, establishes a reverse mapping table to record the correspondence between the B+ tree local index and the inverted global index entries, and the dynamic weight allocator adjusts the index access weight according to the query request type: Exact match queries prioritize accessing the B+ tree local index; Fuzzy matching queries give priority to accessing the inverted global index; Output a joint index access instruction set with weight identification; The instruction set is fed back to the index update priority queue in real time.

8. The distributed storage and indexing method according to claim 1, wherein: The implementation of real-time incremental synchronization of the hotspot index through the heartbeat mechanism includes: The data is based on the index access frequency carried by the heartbeat packets between storage node groups and the network transmission delay data between nodes; The dynamic priority calculator calculates the index priority coefficient based on access frequency, data freshness, and node load, establishes a multi-level cache queue, and stores index entries with index priority coefficients higher than the threshold in the high-speed synchronization channel. When the heartbeat trigger module detects a difference in index versions between adjacent nodes, it extracts the difference entries and compares them with the index priority coefficient. The output generates a set of incremental synchronization instructions with priority tags; The incremental synchronization instruction set includes a target node group address, an index entry list, and a transmission order strategy, and mandates that the SSD storage node group give priority to responding to high-speed channel requests.

9. A distributed storage and indexing method according to claim 8, characterized in that: The cold data layer indexing process includes: Input the remaining capacity warning signal of the storage node group and the cold data layer access timestamp record; The cold data layer identifier marks indexes with no consecutive access records and HDD storage media as data to be migrated. The batch packager aggregates the index entries of the data to be migrated according to the rack topology, generates compressed data packets, and attaches topology routing information. Based on historical network traffic patterns, batch transfer is initiated during low-load periods. Output the migration task queue containing the target rack coordinates and storage media type requirements; When executing the migration task in the migration task queue, the source node index shadow copy is retained until the target node group confirms that the writing is completed.

10. A distributed storage and indexing method according to claim 1, characterized in that: The training process of the distributed query statistical probability table includes: Input historical query logs, node group storage media type, and network topology status records; Log feature extraction extracts query pattern features from historical query logs, including temporal correlation, data type preference, and path hop count. An environmental simulator constructs a virtual training scenario, injecting storage media type read / write delay parameters and network jitter variables. The model updater dynamically adjusts the storage media type weight coefficient and path selection strategy based on feedback rewards based on query success rate and response latency. Outputting a configuration file of a distributed query statistical probability table containing storage medium sensitivity parameters and network topology adaptation parameters; The path selection weight of the SSD storage node group in the configuration file is set to be greater than the path selection weight of the HDD storage node group, and the real-time response query is forcibly bound to the SSD storage node group path; The execution of the dynamic optimization query path includes: Probability distribution data and real-time node group status messages output based on distributed query statistical probability tables; The path generator generates three candidate query paths according to the probability weights, each candidate query path includes at least two storage medium type node groups, and the path evaluator calculates a composite evaluation value of each candidate query path; Output a query routing instruction packet containing the optimal path identifier and alternative path cache; The query routing instruction packet carries a storage medium type verification mark to ensure that the returned data matches the medium performance requirements of the query request.

Images