File storage management method and system for k8s distributed environment

Abstract

The application provides a file storage management method and system for a K8s distributed environment, relates to the technical field of K8s container cloud computing, and comprises the following steps: a topology mapping structure between a storage node and a container instance is constructed, the matching degree of a storage object and a node is calculated, metadata and data blocks are separately processed and a global index is established, a logical reorganization path is established between data shards, only incremental migration of affected data is realized when the container is migrated, and finally, a storage allocation strategy suitable for topology changes is generated. The application improves the storage resource utilization efficiency in a distributed environment and reduces the data migration cost.

Description

Technical Field

[0001] This invention relates to the field of K8s container cloud computing technology, and in particular to a file storage management method and system for K8s distributed environments. Background Technology

[0002] With the rapid development of cloud computing and container technologies, Kubernetes (K8s) has become the mainstream container orchestration platform, providing flexible deployment and management capabilities for distributed applications. In a K8s environment, applications are encapsulated as container instances, which dynamically migrate within the cluster due to resource scheduling, load balancing, or node failures. However, the migration of container instances poses a significant challenge to persistent data storage, especially in private cloud environments where customer-provided server resources often suffer from insufficient disk performance, and frequent I / O operations can severely impact the overall performance and stability of the storage system.

[0003] Existing storage systems lack a deep understanding of Kubernetes resource affinity constraints and cannot intelligently allocate storage based on the topology between container instances and storage nodes. This results in multiple disk I / O operations during data access, creating severe performance bottlenecks in private cloud environments with limited disk performance. When container instances migrate within a cluster, traditional storage solutions typically require migrating the entire storage object instead of incrementally migrating based on actual access needs. This not only generates significant data transfer overhead but also triggers intensive disk read / write operations, causing I / O blocking on servers with poor disk performance and prolonging container migration completion time. Existing technologies struggle to effectively identify and respond to dynamic topology changes in Kubernetes environments and lack adaptive storage policy adjustment mechanisms based on container migration events. They cannot optimize storage allocation strategies based on real-time access patterns and topology changes to reduce unnecessary I / O operations. Consequently, during long-term operation in private cloud environments, disk I / O load accumulates continuously, storage efficiency gradually decreases, and overall system performance deteriorates. Summary of the Invention

[0004] This invention provides a file storage management method and system for Kubernetes distributed environments, which can solve the problems in the prior art.

[0005] A first aspect of the present invention provides a file storage management method for a Kubernetes distributed environment, comprising:

[0006] Based on the resource affinity constraints in file storage requests for K8s container instances, a topology mapping structure between storage nodes and container instances in a distributed environment is constructed.

[0007] Based on the node positions and network distances in the topology mapping structure, and combined with access pattern characteristics, the matching degree between storage objects and storage nodes is calculated, and an initial storage allocation strategy is constructed based on the matching degree.

[0008] The metadata of the stored object is separated from the data block. A globally consistent index associated with the K8s resource object is built for the metadata and synchronized to all storage nodes in the distributed environment. The data block is sharded according to the initial storage allocation strategy, and a shard descriptor is generated for each data shard. The shard descriptor is used to establish a logical reorganization path for the data block between multiple storage nodes, and container migration events during the storage operation are collected.

[0009] The data shards affected by container migration are copied from the original storage node to the target storage node, the node identifier in the corresponding shard descriptor is updated and the logical reassembly path is regenerated, all data shards are read from the target storage node and other storage nodes, and spliced ​​in the reassembly order to build a complete storage object on the target storage node.

[0010] By utilizing the changes in the topological relationship between storage objects and container instances, the topology mapping structure is updated, and a final storage allocation strategy is generated based on the updated topology mapping structure and the access characteristics of the complete storage object.

[0011] Based on resource affinity constraints in file storage requests for Kubernetes container instances, the topology mapping structure between storage nodes and container instances in a distributed environment is constructed as follows:

[0012] Extract node selector tags and affinity rule descriptions from the K8s container instance file storage request to generate the resource affinity constraints;

[0013] Traverse all storage nodes in the distributed environment, obtain the node identifier, physical location identifier, and network topology hierarchy information of each storage node, and store them in the storage node information table;

[0014] Based on the resource affinity constraint, the storage nodes in the storage node information table are filtered by tag matching to obtain a set of candidate storage nodes with successful tag matching.

[0015] For each candidate storage node in the candidate storage node set, the network distance between the container instance and the candidate storage node is calculated based on the network topology layer information of the candidate storage node and the network topology layer information of the container instance. The rule conditions and rule priorities in the affinity rule description are parsed. For the candidate storage nodes that meet the rule conditions, a basic weight score is assigned according to the rule priority. The basic weight score is then adjusted in conjunction with the network distance to obtain the affinity weight value of the candidate storage node.

[0016] The node identifier, physical location identifier, network distance, and affinity weight value of each candidate storage node in the candidate storage node set are associated and bound to construct the topology mapping structure.

[0017] Based on the node locations and network distances in the topology mapping structure, and combined with access pattern characteristics, the matching degree between storage objects and storage nodes is calculated. An initial storage allocation strategy is then constructed based on this matching degree, including:

[0018] Obtain the node location and network distance of the candidate storage node from the topology mapping structure, calculate the location matching score based on the node location, and calculate the network performance score based on the network distance;

[0019] The spatial distribution vector and the position vector of the candidate storage node in the topological space are obtained from the access pattern features, and the spatial matching degree is obtained by calculating the vector similarity. The access load curve of the storage object is predicted based on the periodic pattern in the access pattern features and curve fitting analysis is performed with the resource availability curve of the candidate storage node to obtain the load matching degree. The spatial matching degree and the load matching degree are fused to obtain the access pattern adaptation score.

[0020] Based on the affinity weight value of the candidate storage node, the location matching score, the network performance score, and the access mode adaptation score are subjected to a nonlinear weighted transformation to obtain the comprehensive matching degree between the storage object and the candidate storage node.

[0021] All candidate storage nodes are sorted in descending order according to the overall matching degree, and the candidate storage node with the highest overall matching degree is selected as the target storage node. The mapping relationship between the storage object and the target storage node is recorded to obtain the initial storage allocation strategy.

[0022] The metadata of the stored object is separated from the data block. A globally consistent index is built on the metadata and associated with the Kubernetes resource object, and synchronized to all storage nodes in the distributed environment. The data block is sharded according to the initial storage allocation strategy, and a shard descriptor is generated for each data shard, including:

[0023] File attribute information, access permission information, and the associated K8s resource object identifier are extracted from the storage object as the metadata, and the remaining content is used as the data block;

[0024] Based on the K8s resource object identifier, establish an association mapping relationship between the metadata and the K8s namespace, container group instance, and persistent volume declaration. Organize the association mapping relationship and the metadata in a structured manner to generate a global consistency index entry. The global consistency index entry contains the version identifier and consistency verification identifier of the metadata.

[0025] The globally consistent index entries are synchronized to all storage nodes in the distributed environment through a distributed consistency protocol. A local copy of the globally consistent index entries is maintained on each storage node, and the data consistency between the local copies on all storage nodes and the source index entries is ensured based on the version identifier and the consistency verification identifier.

[0026] The shard size is determined based on the size of the data block and the storage capacity of the target storage node corresponding to the storage object. The data block is then divided into multiple data shards according to the shard size. For each data shard, a shard descriptor is generated based on the offset position of the data shard in the data block, the length of the data shard, and the node identifier of the target storage node.

[0027] Logical reorganization paths for data blocks are established across multiple storage nodes using shard descriptors, and container migration events during storage operations are collected, including:

[0028] Based on the offset position of the data fragment in the fragment descriptor, all data fragments are sorted in ascending order to obtain the reassembly order of the data fragments;

[0029] Based on the reassembly order and the node identifier of the target storage node, a cross-storage node data fragment access sequence is constructed; the data fragment access sequence is associated with the network topology information in the topology mapping structure, the network transmission delay between adjacent storage nodes in the data fragment access sequence is calculated, the network transmission delay is combined with the length of the data fragment to form a data transmission timing chain, and the set of storage nodes to be accessed, the storage node access order, and the expected transmission time are determined according to the data transmission timing chain, thereby generating the logical reassembly path of the data block;

[0030] During storage operations, the running status of all container group instances in the K8s cluster is periodically scanned, the current node position of each container group instance is recorded, the node position of the current scan cycle is compared with the node position of the previous scan cycle, and when a change in the node position of a container group instance is detected, the identifier of the container group instance whose position has changed, the node position before migration, and the node position after migration are extracted to generate the container migration event.

[0031] Based on the dependency chain in the shard descriptor, incremental migration is performed only on data shards affected by container migration. The construction of a complete storage object on the target storage node via the logical reorganization path includes:

[0032] Extract access association information between the data shard and the container group instance from the shard descriptor, and record the set of data shards that the container group instance needs to read and the reading order when accessing the storage object based on the access association information to obtain the data shard dependency chain.

[0033] The target storage node for migration is determined from the topology mapping structure based on the location of the migrated node;

[0034] Calculate the topological distance between the current storage node of each data shard in the data shard dependency chain and the target storage node for migration, and classify data shards whose topological distance exceeds a preset topological distance threshold as data shards affected by container migration;

[0035] The data shards affected by container migration are copied from the original storage node to the target storage node, and the node identifier in the shard descriptor corresponding to the data shards affected by container migration is updated.

[0036] The logical reorganization path is regenerated based on the updated shard descriptor. All data shards are read from the target storage node and other storage nodes according to the logical reorganization path. All data shards are then assembled according to the reorganization order of the data shards, and the complete storage object is constructed on the target storage node.

[0037] Based on the container migration event, the topology relationship change between the storage object and the container instance is identified, the topology mapping structure is updated using the topology relationship change, and the final storage allocation strategy is generated based on the updated topology mapping structure and the access characteristics of the complete storage object, including:

[0038] Based on the container group instance identifier, query the storage object identifier associated with the container group instance from the global consistency index;

[0039] Extract the set of storage nodes where all data fragments of the complete storage object corresponding to the storage object identifier are currently located. Calculate the topological distance between the set of storage nodes and the node positions before and after migration to obtain the topological relationship vectors before and after migration. Compare the differences to obtain the topological relationship change between the storage object and the container instance.

[0040] Based on the changes in the topology relationship, the topology distance information between the container group instance and the storage node is updated in the topology mapping structure. The number of times the complete storage object is accessed by the container group instance within a preset time window and the reading order of the data shards are counted to obtain the access characteristics of the complete storage object.

[0041] Based on the topological distance information between the container group instance and the storage node in the updated topological mapping structure, and combined with the access characteristics of the complete storage object, the storage benefit value of each data shard of the complete storage object on different storage nodes is calculated. The storage node with the highest storage benefit value is selected for file storage, and the final storage allocation strategy is generated.

[0042] A second aspect of the present invention provides a file storage management system for a Kubernetes distributed environment, comprising:

[0043] The first unit is used to construct a topology mapping structure between storage nodes and container instances in a distributed environment based on resource affinity constraints in file storage requests of K8s container instances;

[0044] The second unit is used to calculate the matching degree between storage objects and storage nodes based on the node positions and network distances in the topology mapping structure and the access mode characteristics, and to construct an initial storage allocation strategy based on the matching degree.

[0045] The third unit is used to separate the metadata of the storage object from the data block, build a globally consistent index associated with the K8s resource object for the metadata and synchronize it to all storage nodes in the distributed environment, divide the data block into fragments according to the initial storage allocation strategy, generate a fragment descriptor for each data fragment, use the fragment descriptor to establish a logical reorganization path for the data block between multiple storage nodes, and collect container migration events during the storage operation process.

[0046] The fourth unit is used to perform incremental migration only on data shards affected by container migration based on the dependency chain in the shard descriptor, and to build a complete storage object on the target storage node through the logical reorganization path.

[0047] The fifth unit is used to identify changes in the topological relationship between storage objects and container instances based on the container migration event, update the topological mapping structure using the changes in the topological relationship, and generate a final storage allocation strategy based on the updated topological mapping structure and the access characteristics of the complete storage object.

[0048] A third aspect of the present invention,

[0049] An electronic device is provided, comprising:

[0050] processor;

[0051] Memory used to store processor-executable instructions;

[0052] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0053] Fourth aspect of the embodiments of the present invention,

[0054] A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0055] The beneficial effects of this application are as follows:

[0056] By constructing a topology mapping structure between storage nodes and container instances, precise awareness of resource location relationships in a distributed environment is achieved, effectively improving the accuracy and rationality of storage resource allocation. The matching degree calculation method based on the topology mapping structure, network distance, and access pattern characteristics makes storage allocation decisions more intelligent, optimizing storage performance and improving data access efficiency. The metadata and data block separation processing mechanism, combined with the construction of a globally consistent index, significantly improves the retrieval efficiency and data consistency guarantee of the file storage system in a distributed environment. Data sharding storage and shard descriptor mechanisms enable flexible data organization and efficient management; by establishing logical reorganization paths, the integrity and availability of data across multiple nodes are guaranteed. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating the file storage management method for a K8s distributed environment according to an embodiment of the present invention.

[0058] Figure 2 This is a schematic diagram of the process for separating metadata and data blocks. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0061] Figure 1 This is a flowchart illustrating the file storage management method for a Kubernetes distributed environment according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0062] Based on the resource affinity constraints in file storage requests for K8s container instances, a topology mapping structure between storage nodes and container instances in a distributed environment is constructed.

[0063] Based on the node positions and network distances in the topology mapping structure, and combined with access pattern characteristics, the matching degree between storage objects and storage nodes is calculated, and an initial storage allocation strategy is constructed based on the matching degree.

[0064] The metadata of the stored object is separated from the data block. A globally consistent index associated with the K8s resource object is built for the metadata and synchronized to all storage nodes in the distributed environment. The data block is sharded according to the initial storage allocation strategy, and a shard descriptor is generated for each data shard. The shard descriptor is used to establish a logical reorganization path for the data block between multiple storage nodes, and container migration events during the storage operation are collected.

[0065] Based on the dependency chain in the shard descriptor, incremental migration is performed only on data shards affected by container migration, and a complete storage object is constructed on the target storage node through the logical reorganization path;

[0066] Based on the container migration event, the topology relationship change between the storage object and the container instance is identified, the topology mapping structure is updated using the topology relationship change, and the final storage allocation strategy is generated according to the updated topology mapping structure and the access characteristics of the complete storage object.

[0067] In one optional implementation, constructing a topology mapping structure between storage nodes and container instances in a distributed environment based on resource affinity constraints in Kubernetes container instance file storage requests includes:

[0068] Extract node selector tags and affinity rule descriptions from the K8s container instance file storage request to generate the resource affinity constraints;

[0069] Traverse all storage nodes in the distributed environment, obtain the node identifier, physical location identifier, and network topology hierarchy information of each storage node, and store them in the storage node information table;

[0070] Based on the resource affinity constraint, the storage nodes in the storage node information table are filtered by tag matching to obtain a set of candidate storage nodes with successful tag matching.

[0071] For each candidate storage node in the candidate storage node set, the network distance between the container instance and the candidate storage node is calculated based on the network topology layer information of the candidate storage node and the network topology layer information of the container instance. The rule conditions and rule priorities in the affinity rule description are parsed. For the candidate storage nodes that meet the rule conditions, a basic weight score is assigned according to the rule priority. The basic weight score is then adjusted in conjunction with the network distance to obtain the affinity weight value of the candidate storage node.

[0072] The node identifier, physical location identifier, network distance, and affinity weight value of each candidate storage node in the candidate storage node set are associated and bound to construct the topology mapping structure.

[0073] File storage requests from Kubernetes container instances typically use YAML or JSON format, which includes a `nodeSelector` field to specify the node selector tag and an `affinity` field to describe affinity rules. During extraction, the YAML or JSON file is parsed to locate the `nodeSelector` field, extracting the key-value pairs as the node selector tag. Simultaneously, the `nodeAffinity` or `storageAffinity` subfield within the `affinity` field is parsed to extract the `requiredDuringSchedulingIgnoredDuringExecution` and `preferredDuringSchedulingIgnoredDuringExecution` rules. For example, the node selector extracted from the request might contain the tag "zone=east", and the affinity rule might include the "preferredDuringSchedulingIgnoredDuringExecution" rule, requiring the storage node to be located at "rack=rack1" with a weight of 80.

[0074] The process iterates through all storage nodes in the distributed environment, retrieving the node identifier, physical location identifier, and network topology information for each node. The node identifier is typically a unique ID or name; the physical location identifier includes data center identifiers, rack identifiers, etc.; the network topology information describes the node's hierarchical structure within the network, such as in the format "datacenter / zone / rack / node". This information is obtained by calling the storage cluster management API or directly accessing the storage cluster's metadata service. The storage node information table can be stored using a hash table or a relational data structure for efficient querying and updating. Each record in the storage node information table contains the following fields: nodeID, dataCenter, zone, rack, hostName, labels (a set of labels), and topologyPath (e.g., "dc1 / zone-east / rack3 / node12").

[0075] Based on resource affinity constraints, storage nodes in the storage node information table are filtered by label matching. The matching process follows Kubernetes label matching rules: for labels in `nodeSelector`, the storage node must contain all specified label key-value pairs; for the `requiredDuringSchedulingIgnoredDuringExecution` rule in `nodeAffinity`, matching is performed according to the expression in the rule. For example, if `nodeSelector` requires "zone=east, diskType=ssd", then only storage nodes that possess both of these labels will be selected into the candidate set. In implementation, each record in the storage node information table is traversed, and its `labels` field is checked to see if it meets the requirements of `nodeSelector` and `requiredDuringSchedulingIgnoredDuringExecution`. Nodes that meet the conditions are added to the candidate storage node set.

[0076] For each candidate storage node in the candidate storage node set, calculate its network distance and affinity weight value with the container instance, and obtain the network topology information of the node where the container instance resides. Typically, this is done by retrieving the container instance's nodeName from the Kubernetes API and querying the node's network topology hierarchy. Based on the network topology path between the container instance and the candidate storage node, calculate the network hop count between them. For example, if the container instance is located at "dc1 / zone-east / rack1 / node5" and the storage node is located at "dc1 / zone-east / rack2 / node8", they share the same data center and availability zone but are located in different racks, resulting in a network hop count of 2 (requiring passage through an aggregation switch). The estimated network latency can be calculated based on the network hop count and preset network performance parameters, such as an additional 0.5 milliseconds of latency for each additional hop.

[0077] The process involves parsing the rule conditions and priorities in the affinity rule description, assigning basic weight scores to candidate storage nodes that meet the rule conditions, and extracting the weight value (typically ranging from 1 to 100) from the preferredDuringSchedulingIgnoredDuringExecution rule as the rule priority. If a candidate node satisfies the expression conditions in the rule, the weight value is used as the node's basic weight score. If multiple rules apply to a single node, the weight values ​​of these rules are summed to obtain the basic weight.

[0078] The final affinity weight value of candidate storage nodes is obtained by adjusting the base weight score based on network distance. The adjustment formula is: Final Weight = Base Weight × (1 - Network Distance Factor). Wherein, the network distance factor = min(Network Hop Count × Hop Count Weight Coefficient, 1.0), and the hop count weight coefficient can be set to 0.1, meaning that for each additional hop, the weight decreases by 10%, but the decrease does not exceed 100%. In this way, the farther the network distance, the lower the final weight value of the node, reflecting the advantage of proximity storage.

[0079] The node identifier, physical location identifier, network distance, and affinity weight value of each candidate storage node in the candidate storage node set are associated and bound to construct a topology mapping structure, represented by a graph structure. Container instances are the central nodes, candidate storage nodes are the edge nodes, and the edge weights are the affinity weight values. Each storage node entry in the topology mapping structure includes: nodeID, location (physical location), networkDistance (network distance), and affinityWeight (affinity weight value). This structure can be serialized into JSON format for subsequent storage volume creation and binding operations by the Kubernetes storage volume controller.

[0080] In one optional implementation, based on the node locations and network distances in the topology mapping structure, combined with access pattern characteristics, the matching degree between storage objects and storage nodes is calculated, and an initial storage allocation strategy is constructed based on the matching degree, including:

[0081] Obtain the node location and network distance of the candidate storage node from the topology mapping structure, calculate the location matching score based on the node location, and calculate the network performance score based on the network distance;

[0082] The spatial distribution vector and the position vector of the candidate storage node in the topological space are obtained from the access pattern features, and the spatial matching degree is obtained by calculating the vector similarity. The access load curve of the storage object is predicted based on the periodic pattern in the access pattern features and curve fitting analysis is performed with the resource availability curve of the candidate storage node to obtain the load matching degree. The spatial matching degree and the load matching degree are fused to obtain the access pattern adaptation score.

[0083] Based on the affinity weight value of the candidate storage node, the location matching score, the network performance score, and the access mode adaptation score are subjected to a nonlinear weighted transformation to obtain the comprehensive matching degree between the storage object and the candidate storage node.

[0084] All candidate storage nodes are sorted in descending order according to the overall matching degree, and the candidate storage node with the highest overall matching degree is selected as the target storage node. The mapping relationship between the storage object and the target storage node is recorded to obtain the initial storage allocation strategy.

[0085] The node locations and network distances of candidate storage nodes are obtained from the topology mapping structure. Node locations can be represented as coordinate vectors (x, y, z) to describe the location of the storage node in the topology space. Network distances can be obtained by measuring communication latency, bandwidth, or hop count between nodes, forming a distance matrix.

[0086] When calculating the location matching score based on node location, the historical access logs of the storage object are extracted, the node locations of each access source are counted, the centroid coordinates of the access source node locations are calculated as the access location center point of the storage object, the spatial distance between the candidate storage node location and the access location center point is calculated, the spatial distance is normalized, and the distance value is converted into a location matching score using a reverse mapping function. The smaller the distance value, the higher the location matching score. The specific conversion formula is: Location matching score = 1 / (1 + spatial distance / normalization coefficient).

[0087] When calculating network performance scores based on network distance, test data packets are sent to candidate storage nodes using network probing tools. The round-trip latency from the storage object visitor to the candidate storage node is measured as a network latency indicator, the amount of data that can be transmitted per unit time is measured as a bandwidth indicator, and the percentage of lost test data packets is calculated as a packet loss rate indicator. The reciprocal of the network latency is normalized to obtain the latency score, the bandwidth is normalized to obtain the bandwidth score, and the complement of the packet loss rate is normalized to obtain the reliability score. The latency score, bandwidth score, and reliability score are then weighted and summed according to preset weights to obtain the network performance score.

[0088] The spatial distribution vector and the position vectors of candidate storage nodes in the topological space are obtained from the access pattern features, and the spatial matching degree is obtained by calculating the vector similarity. The spatial distribution vector describes the spatial hotspot distribution of the accessed storage objects, which can be obtained by statistically analyzing the access frequency of different geographical locations or network regions in historical access logs. The similarity between this vector and the position vectors of candidate storage nodes can be calculated using methods such as cosine similarity or Pearson correlation coefficient. The cosine similarity calculation method is to divide the dot product of the two vectors by the product of their magnitudes, with a value range of [-1, 1], where the closer to 1, the higher the spatial matching degree.

[0089] Based on the periodic patterns in access pattern characteristics, the access load curve of storage objects is predicted. By analyzing historical access data, the periodic characteristics of access to storage objects are identified, such as daily, weekly, or monthly peak and trough changes. Time series analysis methods (such as moving average, exponential smoothing, or ARIMA models) are used to predict the access load curve for a future period. Simultaneously, the resource availability curves of candidate storage nodes are obtained, including changes in CPU utilization, memory usage, storage capacity, and I / O bandwidth over time. Curve fitting analysis is performed between the access load curve and the resource availability curve to calculate the complementarity of the two curves, obtaining the load matching degree. Specifically, this can be quantified by calculating the cross-correlation coefficient of the curves or the reciprocal of the mean square error. Ideally, a high load on a storage object should correspond to a high availability period on the storage node.

[0090] The access pattern suitability score is obtained by combining spatial matching and load matching. A weighted average method can be used, assigning different weights based on the characteristics of the stored objects. For example, spatial matching is given more weight to frequently accessed hot data, while load matching is given more weight to data with obvious access cycles.

[0091] Based on the affinity weights of candidate storage nodes, a nonlinear weighted transformation is applied to the location matching score, network performance score, and access mode adaptation score to obtain the overall matching degree between the storage object and the candidate storage node. The affinity weights reflect the storage node's processing preference or adaptability to a specific type of storage object and can be determined based on factors such as the node's hardware characteristics and historical processing efficiency. The nonlinear weighted transformation can use an exponential function or a sigmoid function, such as the sigmoid function, to highlight the influence of high-weight factors. The transformation formula can be expressed as: Overall Matching Degree = sigmoid(w1 × Location Matching Score + w2 × Network Performance Score + w3 × Access Mode Adaptation Score), where w1, w2, and w3 are affinity weights, and satisfy w1 + w2 + w3 = 1.

[0092] All candidate storage nodes are sorted in descending order of their overall matching score. The candidate storage node with the highest overall matching score is selected as the target storage node. The mapping relationship between storage objects and target storage nodes is recorded to obtain the initial storage allocation strategy. When multiple storage objects need to be allocated in the storage system, the above process is executed sequentially according to the priority of the objects. If multiple candidate nodes have similar overall matching scores, load balancing factors can be considered, and the node with the lightest current load can be selected as the target node.

[0093] In practical applications, the initial storage allocation strategy also needs to consider the overall balance and fault tolerance of the storage system. For important data, multiple replicas should be set up and allocated to different storage nodes to improve data availability and disaster recovery capabilities. In addition, a periodic evaluation mechanism can be set up to adjust the storage allocation strategy based on actual access conditions and system status, thereby achieving dynamic optimization of data storage.

[0094] In one optional implementation, the metadata of the stored object is processed separately from the data blocks. A globally consistent index is built on the metadata and associated with the K8s resource object, and synchronized to all storage nodes in the distributed environment. The data blocks are sharded according to the initial storage allocation strategy, and a shard descriptor is generated for each data shard, including:

[0095] File attribute information, access permission information, and the associated K8s resource object identifier are extracted from the storage object as the metadata, and the remaining content is used as the data block;

[0096] Based on the K8s resource object identifier, establish an association mapping relationship between the metadata and the K8s namespace, container group instance, and persistent volume declaration. Organize the association mapping relationship and the metadata in a structured manner to generate a global consistency index entry. The global consistency index entry contains the version identifier and consistency verification identifier of the metadata.

[0097] The globally consistent index entries are synchronized to all storage nodes in the distributed environment through a distributed consistency protocol. A local copy of the globally consistent index entries is maintained on each storage node, and the data consistency between the local copies on all storage nodes and the source index entries is ensured based on the version identifier and the consistency verification identifier.

[0098] The shard size is determined based on the size of the data block and the storage capacity of the target storage node corresponding to the storage object. The data block is then divided into multiple data shards according to the shard size. For each data shard, a shard descriptor is generated based on the offset position of the data shard in the data block, the length of the data shard, and the node identifier of the target storage node.

[0099] like Figure 2 As shown, the method includes:

[0100] Metadata is extracted from storage objects by parsing their file headers to obtain file attributes such as filename, creation time, modification time, and file size. Access control lists (ACLs) are also parsed to obtain read, write, and execute permissions. Furthermore, associated tags are parsed to obtain Kubernetes resource object identifiers, such as "app=nginx" or "environment=production". This information collectively constitutes metadata, while the remaining actual content within the storage object is treated as data blocks. For example, for a container image file, its file header information, permission settings, and associated tags are extracted as metadata, while the actual binary content of the image is treated as data blocks.

[0101] A mapping relationship between metadata and Kubernetes resources is established based on Kubernetes resource object identifiers. By querying the Kubernetes API server, the Kubernetes namespace (e.g., "default" or "kube-system"), container group instance identifier (e.g., Pod name "web-server-5d4d7"), and associated persistent volume claim (PVC) information (e.g., "data-volume-claim") of the storage object are obtained based on the extracted Kubernetes resource object identifier. These mapping relationships and metadata are organized into structured data to generate globally consistent index entries. These index entries are in JSON format and contain metadata content, mapping relationships, version identifiers (e.g., Unix timestamps), and consistency verification identifiers (e.g., SHA-256 hash values).

[0102] To ensure metadata consistency in a distributed environment, globally consistent index entries are synchronized to all storage nodes using a distributed consistency protocol. Specifically, the Raft protocol is used for index entry distribution. A master node is elected to be responsible for index updates. When metadata changes, the master node generates a new index entry, assigns a higher version number, and calculates a new consistency check value. The master node sends the update request to all slave nodes. The update is considered successfully committed only when a majority of nodes (N / 2+1) confirm receipt. Upon receiving the update, each storage node compares it with its local copy based on the version identifier. If the version is lower, the update is performed; otherwise, the consistency check identifier is used to verify data integrity, ensuring index consistency.

[0103] For data block processing, the fragment size is first determined based on the data block size and the target storage node capacity. For example, when the data block size is 1GB, there are 5 target storage nodes, and the remaining storage capacities of each node are 10GB, 8GB, 12GB, 6GB, and 9GB respectively, the fragment size can be set to 256MB, resulting in 4 fragments. For each fragment, its offset position in the original data block (e.g., offset 0 for the first fragment, 256MB for the second), fragment length (e.g., 256MB), and the target node identifier (e.g., "storage-node-1") are recorded to generate a fragment descriptor. The fragment descriptor uses a binary format, containing a 24-byte offset, an 8-byte length value, and a 16-byte node identifier, totaling 48 bytes. This ensures the integrity of the description information while minimizing metadata overhead.

[0104] In actual storage operations, when a user requests access to a specific storage object, the global consistency index is queried through the Kubernetes resource identifier to obtain the corresponding metadata information and shard descriptor. Based on the node identifier and offset information in the shard descriptor, data shards are obtained in parallel from each storage node, reassembled on the client, and the complete storage object is returned. This separate processing approach not only improves data access efficiency but also enhances the system's fault tolerance and scalability.

[0105] In one optional implementation, a logical reorganization path for data blocks is established across multiple storage nodes using fragment descriptors, and container migration events during storage operations are collected, including:

[0106] Based on the offset position of the data fragment in the fragment descriptor, all data fragments are sorted in ascending order to obtain the reassembly order of the data fragments;

[0107] Based on the reassembly order and the node identifier of the target storage node, a cross-storage node data fragment access sequence is constructed; the data fragment access sequence is associated with the network topology information in the topology mapping structure, the network transmission delay between adjacent storage nodes in the data fragment access sequence is calculated, the network transmission delay is combined with the length of the data fragment to form a data transmission timing chain, and the set of storage nodes to be accessed, the storage node access order, and the expected transmission time are determined according to the data transmission timing chain, thereby generating the logical reassembly path of the data block;

[0108] During storage operations, the running status of all container group instances in the K8s cluster is periodically scanned, the current node position of each container group instance is recorded, the node position of the current scan cycle is compared with the node position of the previous scan cycle, and when a change in the node position of a container group instance is detected, the identifier of the container group instance whose position has changed, the node position before migration, and the node position after migration are extracted to generate the container migration event.

[0109] Based on the offset positions of the data fragments in the fragment descriptors, all data fragments are sorted in ascending order to obtain the reassembly order. The fragment descriptors contain the offset position information of each data fragment, indicating its location within the original data block. By reading the offset positions of all data fragments, a quicksort algorithm is used to sort these data fragments in ascending order of offset position, forming an ordered sequence. This ordered sequence is the reassembly order of the data fragments, ensuring that in subsequent data reassembly processes, each data fragment can be assembled according to its correct position in the original data block.

[0110] Based on the reassembly order and the node identifier of the target storage node, a cross-storage node data fragment access sequence is constructed. In practice, the data fragment sequence, sorted by offset position, is traversed. For each data fragment, the storage node identifier is recorded, forming an access sequence containing "data fragment ID - storage node ID" pairs. This sequence describes all data fragments that need to be accessed during data block reassembly and their respective storage node information, providing a foundation for subsequent network transmission latency calculations.

[0111] The data shard access sequence is associated with the network topology information in the topology mapping structure. The network transmission latency between adjacent storage nodes in the data shard access sequence is calculated. The topology mapping structure stores the network topology information of the entire distributed storage system, including parameters such as network connection status, bandwidth, and latency between each storage node. By searching the network path between storage nodes containing adjacent data shards in the topology mapping structure, the network transmission latency values ​​between these nodes can be obtained. For example, if data shard 1 is located on storage node A and data shard 2 is located on storage node B, the network transmission latency value from A to B is found in the topology mapping structure.

[0112] The network transmission delay is then combined with the length of the data fragments to form a data transmission timing chain. Specifically, for each data fragment, the time required to transmit that fragment is calculated based on its size (byte length) and network bandwidth. This transmission time is added to the previously calculated network transmission delay to obtain the total time for transmitting the entire data fragment. These time values ​​are then concatenated according to the access order of the data fragments to form a timing chain, which represents the time sequence required to access all data fragments in a specific order.

[0113] Based on the data transmission timeline, the set of storage nodes required for reassembling the data block, the access order of these nodes, and the expected transmission time are determined. A logical reassembly path for the data block is generated. By analyzing the data transmission timeline, all unique storage nodes that need to be accessed are extracted, forming a storage node set. The access order of these storage nodes is determined according to the access order of the data fragments. Simultaneously, the time values ​​in the timeline are accumulated to obtain the total expected time for reassembling the entire data block. Combining this information generates a complete logical reassembly path, which contains all information about which nodes to retrieve and reassemble the target data block from, in what order, and with what expected time.

[0114] During storage operations, it is also necessary to periodically scan the running status of all container group instances in the Kubernetes cluster. A scheduled task is set up to retrieve the status information of all container groups (Pods) in the cluster via the Kubernetes API at regular intervals (e.g., every 30 seconds). For each container group instance, its current node location is recorded, i.e., the physical node identifier where the container group is running. This information is stored in a status table, with the container group instance identifier as the key and the node location as the value.

[0115] The node positions in the current scan cycle are compared with those in the previous scan cycle to detect whether a node migration has occurred between container group instances. For each container group instance, its node position value in the current state table and the previous cycle's state table is compared. If these two values ​​are different, it indicates that the container group instance has migrated between the two scan cycles.

[0116] When a change in the node location of a container group instance is detected, the identifier of the container group instance whose location changed, its node location before migration, and its node location after migration are extracted to generate a container migration event. The identifier of the detected container group instance, its original node location, and its new node location are organized into structured event data, including information such as event type (container migration), occurrence time, container identifier, source node, and target node. This migration event data can be used for subsequent storage optimization decisions, such as redistribution of data blocks or adjustment of caching strategies, to adapt to the changes in access patterns brought about by container migration.

[0117] Through the above steps, a logical reorganization path for data blocks is established across multiple storage nodes using shard descriptors, and container migration events during storage operations are collected in real time, providing important support for data access optimization in distributed storage systems.

[0118] In one optional implementation, based on the dependency chain in the shard descriptor, incremental migration is performed only on data shards affected by container migration, and the complete storage object is constructed on the target storage node through the logical reorganization path, including:

[0119] Extract access association information between the data shard and the container group instance from the shard descriptor, and record the set of data shards that the container group instance needs to read and the reading order when accessing the storage object based on the access association information to obtain the data shard dependency chain.

[0120] The target storage node for migration is determined from the topology mapping structure based on the location of the migrated node;

[0121] Calculate the topological distance between the current storage node of each data shard in the data shard dependency chain and the target storage node for migration, and classify data shards whose topological distance exceeds a preset topological distance threshold as data shards affected by container migration;

[0122] The data shards affected by container migration are copied from the original storage node to the target storage node, and the node identifier in the shard descriptor corresponding to the data shards affected by container migration is updated.

[0123] The logical reorganization path is regenerated based on the updated shard descriptor. All data shards are read from the target storage node and other storage nodes according to the logical reorganization path. All data shards are then assembled according to the reorganization order of the data shards, and the complete storage object is constructed on the target storage node.

[0124] Access association information between data shards and container group instances is extracted from shard descriptors. A shard descriptor is a metadata structure in a storage system used to describe the characteristics of data shards, containing fields such as shard identifier, storage location, size, access frequency, and association information with the container group. By parsing these fields, the complete set of data shards that a specific container group instance needs to read when accessing a storage object, as well as their reading order, can be obtained. For example, for a large file object composed of multiple shards, the container group accesses these shards in a specific order to reconstruct the complete file. This access pattern information is recorded and constructed into a data shard dependency chain, represented as a directed graph. Nodes represent data shards, edges represent access dependencies, and the weight of the edges can represent access frequency or priority.

[0125] Based on the migrated node location of the container group instance, the target storage node for migration is determined from the system's topology mapping structure. The topology mapping structure is a data structure describing the physical layout of the entire storage network, containing information such as node identifiers, inter-node connections, network bandwidth, and latency. When a container group instance migrates to a new compute node, based on the principle of proximity, the storage node with the closest network distance to the new compute node and sufficient storage capacity is queried from the topology mapping structure as the migration target storage node.

[0126] After determining the target storage node for migration, the topology distance between the current storage node of each data shard in the data shard dependency chain and the target storage node is calculated. Topology distance can be measured by network hop count, transmission latency, or a combined weighted metric. For example, in a data center environment, network hop count can be used as a basic metric; the more switches or routers between two nodes, the greater the topology distance. A preset topology distance threshold is set, typically determined based on the application's sensitivity to data access latency, and is usually 2 or 3 hops. Data shards with a topology distance exceeding this threshold are marked as data shards affected by container migration; these shards need to be migrated to ensure data access performance after container migration.

[0127] The affected data shards are replicated from the original storage node to the target storage node. During replication, incremental transfer technology is used, transferring only the data blocks themselves, rather than the entire storage object, thereby reducing network load. After replication is complete, the node identifier in the shard descriptor corresponding to these data shards is updated from the identifier of the original storage node to the identifier of the target storage node, while other metadata information remains unchanged. The update operation is completed through a distributed metadata service, ensuring that all components in the system are aware of the change in data location.

[0128] The updated shard descriptor regenerates the logical reassembly path, which is an execution plan guiding the system on how to reconstruct a complete storage object from data shards distributed across different nodes. It includes the location information of the data shards, the read order, and the reassembly logic. Following the new logical reassembly path, all necessary data shards are read in parallel from the target storage node and other storage nodes. After reading, the shards are assembled according to their reassembly order, typically based on their position within the original storage object or specific application logic. The assembly process involves data decryption, decompression, verification, and validation to ensure data integrity and correctness. Finally, a complete storage object is constructed on the target storage node for access by the migrated container group instances.

[0129] In a real-world application scenario, suppose a video processing container group migrates from data center A to data center B. This container group needs to access a video file in distributed storage. Using the method described above, only video fragments that are topologically far from the new location are migrated to the storage nodes in data center B, while other fragments remain in their original locations. When the container group runs in the new location, it can quickly access critical fragments in local storage while still being able to access remote non-critical fragments, ensuring performance while avoiding unnecessary data migration overhead.

[0130] In one optional implementation, based on the container migration event, identifying changes in the topology relationship between the storage object and the container instance, updating the topology mapping structure using the changes in the topology relationship, and generating a final storage allocation strategy based on the updated topology mapping structure and the access characteristics of the complete storage object, includes:

[0131] Based on the container group instance identifier, query the storage object identifier associated with the container group instance from the global consistency index;

[0132] Extract the set of storage nodes where all data fragments of the complete storage object corresponding to the storage object identifier are currently located. Calculate the topological distance between the set of storage nodes and the node positions before and after migration to obtain the topological relationship vectors before and after migration. Compare the differences to obtain the topological relationship change between the storage object and the container instance.

[0133] Based on the changes in the topology relationship, the topology distance information between the container group instance and the storage node is updated in the topology mapping structure. The number of times the complete storage object is accessed by the container group instance within a preset time window and the reading order of the data shards are counted to obtain the access characteristics of the complete storage object.

[0134] Based on the topological distance information between the container group instance and the storage node in the updated topological mapping structure, and combined with the access characteristics of the complete storage object, the storage benefit value of each data shard of the complete storage object on different storage nodes is calculated. The storage node with the highest storage benefit value is selected for file storage, and the final storage allocation strategy is generated.

[0135] When a container migration event is detected, the storage object identifier associated with the container group instance is retrieved from the global consistency index based on the container group instance identifier. The global consistency index is a mapping table structure that records the correspondence between container group instances and the storage objects they access. For example, for the container group instance "pod-abc123", the set of storage object identifiers associated with it ["vol-data001", "vol-config002"] can be retrieved from the global consistency index.

[0136] For each associated storage object identifier, extract the set of storage nodes where all data shards of the corresponding complete storage object are currently located. For example, the storage object "vol-data001" is divided into 5 data shards, stored on storage nodes "node-1", "node-3", "node-4", "node-5", and "node-7" respectively. After obtaining these storage node sets, calculate the topological distances between each shard and its node location before and after container migration.

[0137] Topological distance can be calculated using network hop count or physical distance as the metric. Taking network hop count as an example, if a container migrates from "host-A" to "host-B", and the network hop count from "host-A" to "node-1" is 2, while the network hop count from "host-B" to "node-1" is 1, then for the data shard stored on "node-1", the topological distance before migration is 2, and after migration it is 1. In this way, a topological relationship vector before and after migration can be constructed.

[0138] Suppose that for the 5 data shards of the storage object "vol-data001", the topology relationship vector before migration is [2, 3, 1, 4, 2], and the topology relationship vector after migration is [1, 2, 2, 3, 3]. By comparing the vector differences, the topology relationship change vector can be obtained as [-1, -1, +1, -1, +1]. Negative values ​​indicate a decrease in topology distance (faster access), and positive values ​​indicate an increase in topology distance (slower access).

[0139] Based on the calculated topology changes, the topology distance information between container group instances and storage nodes in the topology mapping structure is updated. The topology mapping structure can be a two-dimensional matrix, where rows represent container group instances, columns represent storage nodes, and matrix elements represent the topology distances between them. For the migrated container group instance, its topology distance values ​​with all relevant storage nodes are updated.

[0140] The access characteristics of a complete storage object are obtained by statistically analyzing the number of times it is accessed by the container group instance within a preset time window and the read order of its data shards. The preset time window can be set to the last 30 minutes or the last hour, and can be flexibly adjusted according to actual system load and response requirements. Access characteristics include access frequency and access pattern (sequential or random read). For example, it was found that data shards 1 and 3 are accessed three times more frequently than other shards, and are typically read in the order 1-3-2-4-5.

[0141] Based on the topological distance information between container group instances and storage nodes in the updated topology mapping structure, and combined with the access characteristics of the complete storage object, the storage revenue value of each data shard of the complete storage object on different storage nodes is calculated. The calculation of the storage revenue value can comprehensively consider the following factors: topological distance: the closer the distance, the higher the revenue value; access frequency: the more frequently accessed the data shard, the greater the weight of the distance factor; read order: placing sequentially adjacent data shards on the same node can obtain additional revenue; storage node load: nodes with lighter loads can obtain higher revenue.

[0142] Storage revenue can be calculated using a weighted formula. The revenue of a data shard on a specific storage node can be expressed as the product of the inverse of the topological distance and the access frequency, plus sequential adjacency and load factor adjustments.

[0143] The storage node with the highest storage benefit value is selected for file storage, generating the final storage allocation strategy. The storage allocation strategy is represented in key-value pairs, where the key is the data shard identifier and the value is the target storage node identifier. For example, for the five data shards of the storage object "vol-data001", the final storage allocation strategy is: {"shard1":"node-2", "shard2":"node-2", "shard3":"node-1", "shard4":"node-5", "shard5":"node-5"}.

[0144] Through the above steps, the entire process of identifying changes in the topology relationship between storage objects and container instances based on container migration events, updating the topology mapping structure, and generating the final storage allocation strategy by combining access characteristics is realized, effectively improving the data access efficiency in the container environment.

[0145] This invention relates to a file storage management system for a Kubernetes distributed environment, the system comprising:

[0146] The first unit is used to construct a topology mapping structure between storage nodes and container instances in a distributed environment based on resource affinity constraints in file storage requests of K8s container instances;

[0147] The second unit is used to calculate the matching degree between storage objects and storage nodes based on the node positions and network distances in the topology mapping structure and the access mode characteristics, and to construct an initial storage allocation strategy based on the matching degree.

[0148] The third unit is used to separate the metadata of the storage object from the data block, build a globally consistent index associated with the K8s resource object for the metadata and synchronize it to all storage nodes in the distributed environment, divide the data block into fragments according to the initial storage allocation strategy, generate a fragment descriptor for each data fragment, use the fragment descriptor to establish a logical reorganization path for the data block between multiple storage nodes, and collect container migration events during the storage operation process.

[0149] The fourth unit is used to perform incremental migration only on data shards affected by container migration based on the dependency chain in the shard descriptor, and to build a complete storage object on the target storage node through the logical reorganization path.

[0150] The fifth unit is used to identify changes in the topological relationship between storage objects and container instances based on the container migration event, update the topological mapping structure using the changes in the topological relationship, and generate a final storage allocation strategy based on the updated topological mapping structure and the access characteristics of the complete storage object.

[0151] A third aspect of the present invention provides an electronic device, comprising:

[0152] processor;

[0153] Memory used to store processor-executable instructions;

[0154] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0155] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0156] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A file storage management method for a Kubernetes distributed environment, characterized in that, include: Based on the resource affinity constraints in file storage requests for K8s container instances, a topology mapping structure between storage nodes and container instances in a distributed environment is constructed. Based on the node positions and network distances in the topology mapping structure, and combined with access pattern characteristics, the matching degree between storage objects and storage nodes is calculated, and an initial storage allocation strategy is constructed based on the matching degree. The metadata of the stored object is separated from the data block. A globally consistent index associated with the K8s resource object is built for the metadata and synchronized to all storage nodes in the distributed environment. The data block is sharded according to the initial storage allocation strategy, and a shard descriptor is generated for each data shard. The shard descriptor is used to establish a logical reorganization path for the data block between multiple storage nodes, and container migration events during the storage operation are collected. The data shards affected by container migration are copied from the original storage node to the target storage node, the node identifier in the corresponding shard descriptor is updated and the logical reassembly path is regenerated, all data shards are read from the target storage node and other storage nodes, and spliced ​​in the reassembly order to build a complete storage object on the target storage node. By utilizing the changes in the topological relationship between storage objects and container instances, the topology mapping structure is updated, and a final storage allocation strategy is generated based on the updated topology mapping structure and the access characteristics of the complete storage object.

2. The method according to claim 1, characterized in that, Based on resource affinity constraints in file storage requests for Kubernetes container instances, the topology mapping structure between storage nodes and container instances in a distributed environment is constructed as follows: Extract node selector tags and affinity rule descriptions from the K8s container instance file storage request to generate the resource affinity constraints; Traverse all storage nodes in the distributed environment, obtain the node identifier, physical location identifier, and network topology hierarchy information of each storage node, and store them in the storage node information table; Based on the resource affinity constraint, the storage nodes in the storage node information table are filtered by tag matching to obtain a set of candidate storage nodes with successful tag matching. For each candidate storage node in the candidate storage node set, the network distance between the container instance and the candidate storage node is calculated based on the network topology layer information of the candidate storage node and the network topology layer information of the container instance. The rule conditions and rule priorities in the affinity rule description are parsed. For the candidate storage nodes that meet the rule conditions, a basic weight score is assigned according to the rule priority. The basic weight score is then adjusted in conjunction with the network distance to obtain the affinity weight value of the candidate storage node. The node identifier, physical location identifier, network distance, and affinity weight value of each candidate storage node in the candidate storage node set are associated and bound to construct the topology mapping structure.

3. The method according to claim 1, characterized in that, Based on the node locations and network distances in the topology mapping structure, and combined with access pattern characteristics, the matching degree between storage objects and storage nodes is calculated. An initial storage allocation strategy is then constructed based on this matching degree, including: Obtain the node location and network distance of the candidate storage node from the topology mapping structure, calculate the location matching score based on the node location, and calculate the network performance score based on the network distance; The spatial distribution vector and the position vector of the candidate storage node in the topological space are obtained from the access pattern features, and the spatial matching degree is obtained by calculating the vector similarity. The access load curve of the storage object is predicted based on the periodic pattern in the access pattern features and curve fitting analysis is performed with the resource availability curve of the candidate storage node to obtain the load matching degree. The spatial matching degree and the load matching degree are fused to obtain the access pattern adaptation score. Based on the affinity weight value of the candidate storage node, the location matching score, the network performance score, and the access mode adaptation score are subjected to a nonlinear weighted transformation to obtain the comprehensive matching degree between the storage object and the candidate storage node. All candidate storage nodes are sorted in descending order according to the overall matching degree, and the candidate storage node with the highest overall matching degree is selected as the target storage node. The mapping relationship between the storage object and the target storage node is recorded to obtain the initial storage allocation strategy.

4. The method according to claim 1, characterized in that, The metadata of the stored object is separated from the data block. A globally consistent index is built on the metadata and associated with the Kubernetes resource object, and synchronized to all storage nodes in the distributed environment. The data block is sharded according to the initial storage allocation strategy, and a shard descriptor is generated for each data shard, including: File attribute information, access permission information, and the associated K8s resource object identifier are extracted from the storage object as the metadata, and the remaining content is used as the data block; Based on the K8s resource object identifier, establish an association mapping relationship between the metadata and the K8s namespace, container group instance, and persistent volume declaration. Organize the association mapping relationship and the metadata in a structured manner to generate a global consistency index entry. The global consistency index entry contains the version identifier and consistency verification identifier of the metadata. The globally consistent index entries are synchronized to all storage nodes in the distributed environment through a distributed consistency protocol. A local copy of the globally consistent index entries is maintained on each storage node, and the data consistency between the local copies on all storage nodes and the source index entries is ensured based on the version identifier and the consistency verification identifier. The shard size is determined based on the size of the data block and the storage capacity of the target storage node corresponding to the storage object. The data block is then divided into multiple data shards according to the shard size. For each data shard, a shard descriptor is generated based on the offset position of the data shard in the data block, the length of the data shard, and the node identifier of the target storage node.

5. The method according to claim 1, characterized in that, Logical reorganization paths for data blocks are established across multiple storage nodes using shard descriptors, and container migration events during storage operations are collected, including: Based on the offset position of the data fragment in the fragment descriptor, all data fragments are sorted in ascending order to obtain the reassembly order of the data fragments; Based on the reassembly order and the node identifier of the target storage node, a cross-storage node data fragment access sequence is constructed; the data fragment access sequence is associated with the network topology information in the topology mapping structure, the network transmission delay between adjacent storage nodes in the data fragment access sequence is calculated, the network transmission delay is combined with the length of the data fragment to form a data transmission timing chain, and the set of storage nodes to be accessed, the storage node access order, and the expected transmission time are determined according to the data transmission timing chain, thereby generating the logical reassembly path of the data block; During storage operations, the running status of all container group instances in the K8s cluster is periodically scanned, the current node position of each container group instance is recorded, the node position of the current scan cycle is compared with the node position of the previous scan cycle, and when a change in the node position of a container group instance is detected, the identifier of the container group instance whose position has changed, the node position before migration, and the node position after migration are extracted to generate the container migration event.

6. The method according to claim 5, characterized in that, The process involves copying data shards affected by container migration from the original storage node to the target storage node, updating the node identifier in the corresponding shard descriptor, regenerating the logical reassembly path, reading all data shards from the target storage node and other storage nodes, assembling them in reassembly order, and constructing a complete storage object on the target storage node, including: Extract access association information between the data shard and the container group instance from the shard descriptor, and record the set of data shards that the container group instance needs to read and the reading order when accessing the storage object based on the access association information to obtain the data shard dependency chain. The target storage node for migration is determined from the topology mapping structure based on the location of the migrated node; Calculate the topological distance between the current storage node of each data shard in the data shard dependency chain and the target storage node for migration, and classify data shards whose topological distance exceeds a preset topological distance threshold as data shards affected by container migration; The data shards affected by container migration are copied from the original storage node to the target storage node, and the node identifier in the shard descriptor corresponding to the data shards affected by container migration is updated. The logical reorganization path is regenerated based on the updated shard descriptor. All data shards are read from the target storage node and other storage nodes according to the logical reorganization path. All data shards are then assembled according to the reorganization order of the data shards, and the complete storage object is constructed on the target storage node.

7. The method according to claim 5, characterized in that, By leveraging changes in the topological relationship between storage objects and container instances, the topology mapping structure is updated. Based on the updated topology mapping structure and the access characteristics of the complete storage object, a final storage allocation strategy is generated, including: Based on the container group instance identifier, query the storage object identifier associated with the container group instance from the global consistency index; Extract the set of storage nodes where all data fragments of the complete storage object corresponding to the storage object identifier are currently located. Calculate the topological distance between the set of storage nodes and the node positions before and after migration to obtain the topological relationship vectors before and after migration. Compare the differences to obtain the topological relationship change between the storage object and the container instance. Based on the changes in the topology relationship, the topology distance information between the container group instance and the storage node is updated in the topology mapping structure. The number of times the complete storage object is accessed by the container group instance within a preset time window and the reading order of the data shards are counted to obtain the access characteristics of the complete storage object. Based on the topological distance information between the container group instance and the storage node in the updated topological mapping structure, and combined with the access characteristics of the complete storage object, the storage benefit value of each data shard of the complete storage object on different storage nodes is calculated. The storage node with the highest storage benefit value is selected for file storage, and the final storage allocation strategy is generated.

8. A file storage management system for a Kubernetes distributed environment, used to implement the method as described in any one of claims 1-7, characterized in that, include: The first unit is used to construct a topology mapping structure between storage nodes and container instances in a distributed environment based on resource affinity constraints in file storage requests of K8s container instances; The second unit is used to calculate the matching degree between storage objects and storage nodes based on the node positions and network distances in the topology mapping structure and the access mode characteristics, and to construct an initial storage allocation strategy based on the matching degree. The third unit is used to separate the metadata of the storage object from the data block, build a globally consistent index associated with the K8s resource object for the metadata and synchronize it to all storage nodes in the distributed environment, divide the data block into fragments according to the initial storage allocation strategy, generate a fragment descriptor for each data fragment, use the fragment descriptor to establish a logical reorganization path for the data block between multiple storage nodes, and collect container migration events during the storage operation process. The fourth unit copies the data fragments affected by container migration from the original storage node to the target storage node, updates the node identifier in the corresponding fragment descriptor and regenerates the logical reassembly path, reads all data fragments from the target storage node and other storage nodes, splices them in the reassembly order, and constructs a complete storage object on the target storage node. The fifth unit is used to update the topology mapping structure by utilizing the changes in the topology relationship between the storage object and the container instance, and to generate the final storage allocation strategy based on the updated topology mapping structure and the access characteristics of the complete storage object.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images