Computing power center capacity expansion method based on distributed storage and mixed data archiving

Abstract

The invention discloses a computing power center capacity expansion method based on distributed storage and mixed data archiving, and relates to the technical field of computing power capacity expansion. Data can be subjected to fragmentation management according to the difference of service loads through establishment and updating of an access frequency label, and an archiving snapshot mechanism is combined, so that the capacity expansion efficiency is improved. Rapid data migration and continuous storage and writing under the condition that the storage nodes are abnormal are realized, so that the fault-tolerant capability and the system stability in the capacity expansion process are effectively improved; meanwhile, by means of logical sequence splicing and archiving index construction of the low-access-frequency fragmentation units, the storage pressure brought by redundant data is reduced, and the efficiency of archiving retrieval and calling is improved; in addition, when the access frequency of the fragmentation units changes, an intelligent relocation strategy based on delay, load and topological distance can be realized, and quick response and balanced distribution of high access data are ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computing power expansion, and particularly relates to a computing power center expansion method based on distributed storage and mixed data archiving. BACKGROUND

[0002] With the rapid development of computing power centers in applications such as big data, artificial intelligence and high-performance computing, storage systems and data archiving capabilities have become important factors restricting the expansion and stable operation of computing power centers. In the expansion process of traditional computing power centers, linear superposition of computing resources or hardware upgrade is usually relied on to meet the growing business needs, but there are obvious deficiencies in data archiving processing, storage node migration and access efficiency maintenance. Especially in a distributed storage environment, the access frequency of archived data varies greatly, and how to efficiently manage and dynamically expand the archived data has become a technical problem to be solved.

[0003] CN119376932A discloses a method and system for expanding computing power resources in an intelligent computing center, mainly relying on proxy service nodes to perform hash mapping on DiskANN service node clusters, and determining target nodes for processing requests through a consistent hash mechanism. This method can expand computing power resources within a distributed file system, but the focus is mainly on request distribution and computing power node positioning. In terms of archiving data storage and write persistence, there is a lack of mechanism for differentiated management of access frequency, and dynamic migration of archived shards in abnormal situations is not considered, which can easily lead to a decrease in storage efficiency and stability during the expansion process. Therefore, when dealing with large-scale expansion of archived data in a computing power center, its adaptability is insufficient.

[0004] CN114503052A discloses a data center and an expansion method, by setting plug-in nodes and idle electrical connection ends on the bus, directly adding power equipment and separating bus power supply during expansion, thereby reducing the problem of dense wiring during expansion. The advantage lies in the convenience of power system expansion, but the expansion object is mainly hardware power equipment, and the management and storage optimization of archived data in the computing power center are not involved. For the core problems of data shard access, cross-node migration and index updating under distributed storage, this scheme cannot provide effective solutions, so there are still great limitations.

[0005] In summary, the existing technology in the expansion of computing power centers either focuses on computing power node distribution or emphasizes power supply expansion, and neither can solve the problem of differentiated access frequency of archived data and abnormal migration processing in a distributed storage environment. The computing power center expansion method based on distributed storage and mixed data archiving proposed by the present application effectively realizes data safety, efficient access and dynamic management during the expansion process through access frequency labeling, shard migration driven by archiving snapshots and multi-level archiving index mechanisms. SUMMARY

[0006] This section is intended to summarize some aspects of the embodiments of the present application and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section as well as in the Abstract and the Title of the application in order to avoid obscuring the subject matter of the present section, the Abstract and the Title. Such simplifications or omissions are not to be interpreted as limiting the scope of the present application in any way.

[0007] In view of the above existing problems, the present application is proposed.

[0008] Therefore, the technical problem solved by the present application is: how to realize dynamic management and efficient scheduling of archived data.

[0009] To solve the above technical problems, the present application provides the following technical solutions: when the expansion of the computing center begins, the access frequency of the archived data unit is measured, the access frequency label is generated, and the archived data unit is divided into a plurality of archived slice units according to the access frequency label, and a slice identification table is established in the distributed storage node; after the slice identification table is established, an archived snapshot is generated for the write process of the archived slice unit, when the distributed storage node appears abnormal, the archived slice unit with incomplete write is migrated to the normal distributed storage node according to the archived snapshot, and the archived write is continued; after the continued write under the archived snapshot is completed, the archived slice unit with low access frequency is logically sequentially spliced according to the distribution information of the archived slice unit in the slice identification table, and an archived index is generated in the distributed storage node; after the archived index is generated, the access frequency label of the archived slice unit is monitored, and the access frequency label and the storage location are updated in the slice identification table.

[0010] As a preferred scheme of the present application, wherein: the division of the archived data unit into a plurality of archived slice units according to the access frequency label comprises: collecting the archived data unit to be processed at the beginning of the expansion of the computing center, and generating an initial grouping according to the business source of the archived data unit; the number of calls of each archived data unit in a preset observation period is counted, the access frequency label is formed according to the statistical value and the historical access reference, and the access frequency label is divided into high access frequency and low access frequency; the access fluctuation rate of the archived data unit in the preset observation period is calculated, and when the access fluctuation rate exceeds the fluctuation threshold, the corresponding access frequency label is corrected to a fluctuation type label; after the access frequency label is corrected, the archived data unit is divided into a plurality of archived slice units according to the access frequency label; the archived data unit corresponding to the high access frequency and the fluctuation type label is divided into independent archived slice units; the archived data unit corresponding to the low access frequency label is divided in batches.

[0011] In a preferred embodiment of the present invention, the division of high access frequency and low access frequency includes: calculating a standardized access value for each archived data unit according to the following formula. : in, The number of times each archived data unit is accessed within a preset observation period; This represents the average or median number of accesses for all archived data units; if the access value is standardized... If the access frequency is greater than or equal to the preset access frequency threshold, the archived data unit will be marked as high access frequency or fluctuating type; otherwise, it will be marked as low access frequency.

[0012] As a preferred embodiment of the present invention, generating an archive snapshot during the writing process of an archive shard unit includes: after the shard identification table is established, tracking the writing progress of each archive shard unit on the distributed storage node and generating a write location index; after the archive shard write tracking is completed, writing the write location index and access frequency tag together into the archive snapshot, and encapsulating the archive snapshot in stages according to time segments.

[0013] As a preferred embodiment of the present invention, the step of migrating the archive shard unit that has not been written to the normal distributed storage node according to the archive snapshot includes: when the distributed storage node is abnormal, triggering a fault marker, and locating the archive shard unit and its current encapsulation write status corresponding to the abnormal distributed storage node in the archive snapshot; after the snapshot location is completed, transferring the archive shard unit from the abnormal distributed storage node to the normal distributed storage node, and continuing to write the archive shard unit according to the stage encapsulation point recorded in the archive snapshot, while updating the storage location in the shard identification table.

[0014] In a preferred embodiment of the present invention, the step of logically concatenating archive shard units with low access frequency tags includes: after completing the continued write under the archive snapshot driver, calling the distribution information of archive shard units in the shard identifier table, filtering archive shard units with low access frequency tags, and generating a sorting sequence based on the physical storage location; after the archive shards are sorted, logically concatenating the low access frequency archive shard units according to the sorting sequence, assigning segment numbers to each group of consecutively concatenated archive shard units, and generating an archive index within the distributed storage node; the archive index records the relationship between segment numbers and concatenation order.

[0015] In a preferred embodiment of the present invention, the monitoring of the access frequency tag of the archive shard unit includes: periodically detecting the access frequency tag of each archive shard unit; when the access frequency tag changes from a low access frequency state to a high access frequency state, a secondary determination is made as to whether the actual number of accesses exceeds a preset frequency threshold; if it exceeds, the archive shard unit is marked as a unit to be migrated; based on the characteristics of the unit to be migrated, a low-latency distributed storage node with low-latency access capability is selected from the candidate node set as the target node, and storage space is reserved for the target node before migration; the unit to be migrated is completely copied from the original distributed storage node to the target node, and data consistency verification is performed to ensure that the data after migration is error-free.

[0016] In a preferred embodiment of the present invention, the selection of the target node includes: after the archive shard unit to be migrated is marked as having a high access frequency, selecting nodes from the candidate node set that simultaneously meet the following conditions: the node's current storage load does not exceed a preset load limit; the node's average access latency is lower than a preset latency threshold; the network topology distance between the node and the nearest access source of the archive shard unit is less than a preset distance threshold; in the candidate node set, determining the final target node according to the following logical order: prioritizing the node with the shortest network topology distance; if the topology distances are the same, selecting the node with the lowest average access latency; if the latency is also the same, selecting the node with the lowest storage load; the finally determined node is used as the target node of the archive shard unit.

[0017] The beneficial effects of this invention are as follows: By establishing and updating access frequency tags, this invention can manage data in shards according to the differences in business load. Combined with the archiving snapshot mechanism, it enables rapid data migration and continued writing in the event of storage node anomalies, thereby effectively improving the fault tolerance and system stability during the expansion process. At the same time, by logically concatenating low-access-frequency shard units and constructing archiving indexes, it not only reduces the storage pressure caused by redundant data but also improves the efficiency of archive retrieval and retrieval. In addition, when the access frequency of shard units changes, an intelligent migration strategy based on latency, load, and topology distance can be implemented to ensure rapid response and balanced distribution of high-access data.

[0018] Overall, this invention ensures the smoothness of computing center expansion and data integrity while taking into account resource utilization and access performance, significantly improving the scalability and reliability of computing centers in complex business scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating the computing center expansion method based on distributed storage and hybrid data archiving as shown in this invention. Detailed Implementation

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0021] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort should fall within the scope of protection of this invention.

[0022] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0023] According to an embodiment of the present invention, in combination Figure 1 The flowchart shown illustrates a method for expanding computing centers based on distributed storage and hybrid data archiving, comprising: S1: When the computing center expands, the access frequency of the archived data unit is measured, an access frequency tag is generated, and the archived data unit is divided into several archived fragment units according to the access frequency tag. At the same time, a fragment identification table is established in the distributed storage node.

[0024] S1.1: When the expansion of the computing center begins, collect the archived data units to be processed and generate initial groups according to the business source of the archived data units.

[0025] S1.2: Statistically count the number of times each archived data unit is accessed within a preset observation period, form an access frequency tag based on the statistical value and historical access benchmark, and distinguish the access frequency tag into high access frequency and low access frequency.

[0026] The division between high access frequency and low access frequency includes: For each archived data unit, calculate the normalized access value according to the following formula. : in, The number of times each archived data unit is accessed within a preset observation period; The average or median number of accesses for all archived data units; If standardized access values If the access frequency is greater than or equal to the preset access frequency threshold, the archived data unit will be marked as high access frequency or fluctuating type; otherwise, it will be marked as low access frequency.

[0027] S1.3: Calculate the access volatility of the archived data unit within the preset observation period. When the access volatility exceeds the volatility threshold, correct the corresponding access frequency label to a volatility type label.

[0028] The access volatility is calculated using standard deviation or mean deviation, reflecting the degree of fluctuation in the number of accesses to archived data units within a period.

[0029] It should be noted that in conventional operations, most systems ignore access volatility and only determine high and low access frequencies based on the average number of accesses. This can lead to short-term, high-frequency data being classified as low-frequency data, making it impossible to migrate or optimize storage in a timely manner during capacity expansion. To solve this problem, this invention corrects the access frequency labels of archived data units through access volatility calculation. Data units with access volatility exceeding the volatility threshold are labeled as volatility-type data, thus distinguishing them from long-term high-frequency data.

[0030] S1.4: After the access frequency label is corrected, the archived data unit is divided into several archived fragment units according to the access frequency label; the archived data units corresponding to high access frequency and fluctuating labels are divided into independent archived fragment units; the archived data units corresponding to low access frequency labels are divided in batches.

[0031] Specifically, high-access-frequency and fluctuating units are independently sharded: All archived data units are traversed, and units with access frequency tags of "high access frequency" or "fluctuating" are individually sharded, with each shard assigned a unique shard number. The shard identifier table records each shard unit's number, original archived data unit identifier, access frequency tag, pre-allocated physical storage node location, and its business origin. This operation ensures that hot data is independently managed in distributed storage, facilitating rapid migration and high-frequency access optimization.

[0032] The low-access-frequency unit batch partitioning involves generating batch fragments for archived data units tagged with low access frequency according to the following rules: First, archived data units are initially grouped by their business origin to ensure that data from the same business origin is placed in the same logical fragment as much as possible. Within the initial grouping, units are sorted based on data timestamps or other business-related fields to ensure that data with high time or business relevance is arranged consecutively. A batch fragment unit is generated for each group of K consecutively arranged archived data units, where K is a preset batch threshold (adjustable based on storage node capacity or system performance). Data units with fewer than K units can be merged with the next group or generated as fragments independently. A unique number is assigned to each batch fragment unit, and the number, the included archived data unit identifier, access frequency tag, business origin, and pre-allocated physical storage node location are recorded in the fragment identification table.

[0033] It should be noted that this invention uses different partitioning strategies for data with different access frequencies to ensure that high-frequency data can be migrated quickly and low-frequency data is logically continuous, thereby improving storage utilization. Low-access-frequency data is not only grouped according to access frequency, but also sorted according to business relevance and time series, optimizing the efficiency of logical order splicing and index generation.

[0034] S2: After the fragment identification table is established, an archive snapshot is generated for the writing process of the archive fragment unit. When the distributed storage node is abnormal, the archive fragment unit that has not been written is migrated to the normal distributed storage node according to the archive snapshot, and the archiving writing continues.

[0035] S2.1: After the shard identification table is established, the write progress of each archive shard unit on the distributed storage node is tracked, and a write location index is generated.

[0036] The tracking information includes the data block number written to each shard unit, the write completion timestamp, the target storage node identifier, and the current write status flag. Through this tracking, the computing center can record the write progress of the archive shard unit on the physical node, forming a complete data block write path.

[0037] During operation, the write progress of each archive shard unit is polled, and the write position index table is updated in real time with each round of polling. The position index table is not only used to generate archive snapshots later, but also provides a precise location basis for data recovery in case of anomalies. At the same time, the write position index is associated with the shard identifier table to ensure that each index record corresponds to a unique archive shard unit and storage node, enabling efficient tracking and management of the write status of each shard unit in a distributed environment.

[0038] S2.2: After the archive fragment write tracing is completed, the write location index and access frequency label are written to the archive snapshot together, and the archive snapshot is encapsulated in stages according to time segments so that the write can be recovered from the most recent encapsulation point in case of storage node failure.

[0039] The archive snapshot includes the archive shard unit identifier, the storage node to which the shard unit belongs, the range of data blocks written, the write timestamp, the access frequency label, and the stage encapsulation identifier.

[0040] To facilitate recovery in the event of storage node failure, archive snapshots are encapsulated in stages according to preset time segments. Each stage of encapsulation records the write completion status and index position of the archive shard unit within that time segment, thereby ensuring that writing can continue from the most recent encapsulation point during recovery and avoiding duplicate writes or data loss.

[0041] The phased encapsulation can be divided by minutes, hours, or preset data volume, with each encapsulation point forming an independent record that can be used for rapid recovery.

[0042] S2.3: When a distributed storage node malfunctions, a fault flag is triggered, and the archive shard unit and the current encapsulated write status of the corresponding malfunctioning distributed storage node are located within the archive snapshot.

[0043] Specifically, when a distributed storage node malfunctions, a fault marking mechanism is triggered, recording the identifier of the malfunctioning node, the time of the malfunction, and the range of affected archive shard units. By traversing the phased encapsulation records of the archive snapshots, the write position index of the corresponding shard unit is matched according to the malfunctioning node identifier, generating an malfunction migration task table from the list of data blocks that have not been written. The malfunction migration task table records the unique identifier of each shard unit to be migrated, the range of data blocks that have not been written, the original storage node, and its access frequency tag.

[0044] S2.4: After the snapshot is located, the archive shard unit is transferred from the abnormal distributed storage node to the normal distributed storage node, and the archive shard unit is continued to be written according to the stage encapsulation points recorded in the archive snapshot. At the same time, the storage location in the shard identifier table is updated.

[0045] During the migration process, the starting point for continued writing is determined based on the staged encapsulation points in the archive snapshots, ensuring that data writing continues from the previous encapsulation point without repeatedly writing completed data blocks. Simultaneously, the current storage location, target node identifier, and migration timestamp of the archive shard unit are updated in the shard identifier table to ensure consistency between the shard identifier table and the actual data state.

[0046] During the write process, the archive shards are written block by block according to the order of their phased encapsulation, and the write position index and archive snapshot are updated in real time so that recovery can still be made from the most recent encapsulation point in the event of a secondary failure. Through the above mechanism, this invention ensures that the archive shards have high availability and continuous write capability in a distributed environment, while reducing the risk of access latency or data loss caused by node failure.

[0047] Optionally, after the shard migration is complete, in addition to updating the shard identifier table, it is also necessary to automatically trigger the generation of lightweight redundant replicas on the target node, as follows: For archive shard units, only differential check blocks (such as XOR parity or hash segments) are generated, instead of full copies; these differential check blocks are distributed across two adjacent storage nodes; and in the event of another node failure, redundant check blocks are used to quickly refill the data with the master data. These operations balance space utilization and reliability improvement, unlike conventional multi-replica storage.

[0048] S3: After the continued write driven by the archive snapshot is completed, the archive shard units with low access frequency tags are logically concatenated according to the distribution information of the archive shard units in the shard identifier table, and an archive index is generated in the distributed storage node.

[0049] S3.1: After completing the continued write under the archive snapshot driver, call the distribution information of the archive fragment units in the fragment identifier table, filter the archive fragment units with the access frequency tag as low access frequency, and generate a sorting sequence according to the physical storage location.

[0050] Specifically, by traversing the shard identifier table, archive shard units with low access frequency tags are selected, and their physical storage location and shard size information are recorded to provide an accurate basis for subsequent logical splicing; a sorting sequence of low access frequency shard units is generated based on the physical storage node location, the block number of the shard within the storage node, and the historical write order.

[0051] The sorting sequence is based on the physical storage location to ensure that the logical splicing order and the actual storage continuity of data on the nodes are consistent. At the same time, the differences in the size of the archive fragment units are taken into account to avoid discontinuity across nodes or blocks during the splicing process.

[0052] The sorting strategy of this invention optimizes the efficiency of archive index generation and data retrieval performance while maintaining the logical order of data access.

[0053] S3.2: After the archive fragments are sorted, archive fragment units with low access frequency are logically concatenated according to the sorting sequence. At the same time, segment numbers are assigned to each group of consecutively concatenated archive fragment units, and an archive index is generated in the distributed storage node. The archive index records the relationship between segment numbers and concatenation order.

[0054] Ideally, the splicing operation combines consecutive fragmented units into logical segments. Each logical segment retains physical block mapping information and fragmented unit identifiers during the splicing process, so that it can be mapped back to the actual storage location during subsequent access or index queries.

[0055] During the splicing process, a unique segment number is assigned to each group of consecutive archived fragments, and the splicing order of each fragment within a segment is recorded to ensure that the logical segment can completely and continuously correspond to the logical order of low-access-frequency archived data. Through this logical splicing, the system achieves batch data management while maintaining the traceability and continuity of data access.

[0056] Furthermore, the archive index records the segment number of each logical segment, the identifier of the archive fragment unit it contains, the splicing order, and the distribution node information.

[0057] The index structure uses a traversable table or linked list format to ensure that logical segments can be quickly mapped to actual storage nodes and block addresses, enabling efficient execution of data retrieval and migration operations. Furthermore, it is associated with archive snapshots and shard identification tables, providing cross-module consistency management support to ensure efficient processing of low-access-frequency data in both logical management and physical storage after the computing center is expanded.

[0058] The method of combining logical sequence concatenation with archive index generation breaks through the limitations of traditional low-access-frequency data management that relies solely on batch partitioning, achieving a dual improvement in data access efficiency and storage management efficiency.

[0059] Optionally, when logically assembling low-frequency segments, a hot-cold hybrid alignment strategy is used, that is, a small portion (e.g., 5%) of the most recently degraded high-frequency units are embedded at the end of each assembled logical segment to improve retrieval efficiency during batch scanning. The specific operation is as follows: Traverse the shard units that have changed from high frequency to low frequency in the most recent migration, collect timestamps and access frequency decay curves; append data units with stable decay curves to the end of adjacent low-frequency logical segments in chronological order; add a flag bit to these embedded units in the archive index for easy differentiation, which can prevent the loss of access efficiency immediately when hotspots turn cold, and improve the query friendliness of archived data.

[0060] S4: After the archive index is generated, monitor the access frequency label of the archive shard unit and update the access frequency label and storage location in the shard identifier table.

[0061] S4.1: Periodically detect the access frequency tag of each archive fragment unit. When the access frequency tag changes from a low access frequency state to a high access frequency state, make a second judgment on whether the actual number of accesses exceeds the preset frequency threshold. If it does, mark the archive fragment unit as a unit to be relocated.

[0062] S4.2: Based on the characteristics of the unit to be relocated, select a low-latency distributed storage node with low-latency access capability from the candidate node set as the target node, and reserve storage space for the target node before relocation.

[0063] The selection of the target node includes the following steps: After the archive shard unit to be migrated is marked as having high access frequency, nodes that meet the following conditions are selected from the candidate node set: the current storage load of the node does not exceed the preset load limit; the average access latency of the node is lower than the preset latency threshold; and the network topology distance between the node and the nearest access source of the archive shard unit is less than the preset distance threshold.

[0064] Among the candidate node set, the final target node is determined in the following logical order: the node with the shortest network topology distance is selected first; if the topology distance is the same, the node with the lowest average access latency is selected; if the latency is also the same, the node with the lowest storage load is selected; the finally determined node is used as the target node for the archive shard unit.

[0065] S4.3: Copy the unit to be migrated completely from the original distributed storage node to the target node, and perform data consistency verification to ensure that the data after migration is error-free.

[0066] The copying process strictly follows the shard unit identifier and physical block mapping to ensure that each data block is copied to the designated storage location on the target node in its original logical order. Simultaneously, the access frequency tags and archive index information of the archive shard units are preserved during the copying process so that the original logical management structure can be maintained on the target node after the migration is complete.

[0067] After the copy is completed, a data consistency check is performed, including checksum calculation of archive fragment units, block-level comparison, and access frequency tag matching verification.

[0068] After verification, update the storage location and access frequency tags of the archived shard units in the shard identifier table to ensure that the migrated data remains consistent in logical management, index mapping, and physical storage.

[0069] Through the above operations, the computing center can dynamically adjust the data location while ensuring the integrity and access efficiency of the archive shard units, avoiding data loss or logical confusion, and improving the system's high-frequency data processing capabilities and storage resource utilization.

[0070] The method also includes one or more processors and memory.

[0071] The memory is used to store operable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, including the flow of the computing center expansion method based on distributed storage and hybrid data archiving described in the foregoing embodiments, especially... Figure 1 The flowchart of the method is shown.

[0072] Other aspects disclosed in the embodiments of the present invention also propose a computer-readable medium for storing software, the software including instructions executable by one or more computers, the instructions causing the one or more computers to perform operations, including the flow of the computing center expansion method based on distributed storage and hybrid data archiving of the foregoing embodiments, particularly... Figure 1 The flowchart of the method is shown.

[0073] It should be recognized that embodiments of the present invention may be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium.

[0074] The method can be implemented using standard programming techniques, including a non-transitory computer-readable storage medium configured with a computer program in the computer program, wherein the storage medium is configured such that the computer operates in a specific and predefined manner.

[0075] Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system; however, if required, the program can be implemented in assembly or machine language.

[0076] In any case, the language can be either compiled or interpreted.

[0077] Furthermore, for this purpose, the program can run on programmed application-specific integrated circuits.

[0078] The processes described herein (or variations and / or combinations thereof) can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The computer program includes a plurality of instructions executable by one or more processors.

[0079] Furthermore, the method can be implemented in any suitable computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices.

[0080] Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether portable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein.

[0081] Furthermore, machine-readable code, or parts thereof, can be transmitted via wired or wireless networks.

[0082] When such media includes instructions or programs that combine with a microprocessor or other data processor to implement the steps described above, the invention described herein includes these and other different types of non-transitory computer-readable storage media.

[0083] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for expanding computing centers based on distributed storage and hybrid data archiving, characterized in that, include: When the computing center expansion begins, the access frequency of the archived data units is measured, access frequency tags are generated, and the archived data units are divided into several archived fragment units based on the access frequency tags. At the same time, a fragment identification table is established in the distributed storage nodes. After the fragment identification table is established, an archive snapshot is generated for the writing process of the archive fragment unit. When a distributed storage node fails, the archive fragment unit that has not been written is migrated to a normal distributed storage node according to the archive snapshot, and the archiving writing continues. After the continued write driven by archive snapshot is completed, archive shard units with low access frequency are logically concatenated according to the distribution information of archive shard units in the shard identifier table, and an archive index is generated in the distributed storage node. After the archive index is generated, the access frequency label of the archive shard unit is monitored, and the access frequency label and storage location are updated in the shard identifier table.

2. The computing center expansion method based on distributed storage and hybrid data archiving as described in claim 1, characterized in that: The method of dividing the archived data unit into several archived fragment units based on the access frequency tag includes: When the expansion of the computing center begins, archived data units to be processed are collected, and initial groups are generated according to the business source of the archived data units; The number of times each archived data unit is accessed within a preset observation period is statistically analyzed, and an access frequency label is formed based on the statistical value and historical access benchmark. The access frequency label is then divided into high access frequency and low access frequency. Calculate the access volatility of the archived data unit within the preset observation period. When the access volatility exceeds the volatility threshold, correct the corresponding access frequency label to a volatility type label. After the access frequency labels are corrected, the archived data units are divided into several archived fragment units according to the access frequency labels; the archived data units corresponding to high access frequency and fluctuating labels are divided into independent archived fragment units; and the archived data units corresponding to low access frequency labels are divided in batches.

3. The computing center expansion method based on distributed storage and hybrid data archiving as described in claim 2, characterized in that: The division between high access frequency and low access frequency includes: For each archived data unit, calculate the normalized access value according to the following formula. : in, The number of times each archived data unit is accessed within a preset observation period; The average or median number of accesses for all archived data units; If standardized access values If the access frequency is greater than or equal to the preset access frequency threshold, the archived data unit will be marked as high access frequency or fluctuating type; otherwise, it will be marked as low access frequency.

4. The computing center expansion method based on distributed storage and hybrid data archiving as described in claim 3, characterized in that: The process of writing to an archive fragment unit to generate an archive snapshot includes: After the shard identification table is established, the write progress of each archive shard unit on the distributed storage node is tracked, and a write location index is generated. After the archive fragment write tracing is completed, the write location index and access frequency label are written to the archive snapshot, and the archive snapshot is encapsulated in stages according to time segments.

5. The computing center expansion method based on distributed storage and hybrid data archiving as described in claim 4, characterized in that: The process of migrating archive shard units that have not yet been written to to normal distributed storage nodes based on archive snapshots includes: When a distributed storage node malfunctions, a fault flag is triggered, and the archive shard unit and the current encapsulated write status of the corresponding malfunctioning distributed storage node are located within the archive snapshot. After the snapshot is located, the archive shard unit is transferred from the abnormal distributed storage node to the normal distributed storage node, and the archive shard unit is continued to be written according to the stage encapsulation points recorded in the archive snapshot. At the same time, the storage location in the shard identifier table is updated.

6. The computing center expansion method based on distributed storage and hybrid data archiving as described in claim 5, characterized in that: The logical sequence concatenation of archive fragment units with low access frequency tags includes: After completing the continued write under the archive snapshot driver, the distribution information of the archive shard units in the shard identifier table is called to filter the archive shard units with the access frequency tag of low access frequency, and a sorting sequence is generated according to the physical storage location. After the archive fragments are sorted, the archive fragment units with low access frequency are logically concatenated according to the sorting sequence. At the same time, a segment number is assigned to each group of consecutively concatenated archive fragment units, and an archive index is generated in the distributed storage node. The archive index records the relationship between segment numbers and concatenation order.

7. The computing center expansion method based on distributed storage and hybrid data archiving as described in claim 6, characterized in that: The access frequency tags of the monitoring archive fragment unit include: The access frequency tag of each archive fragment unit is periodically detected. When the access frequency tag changes from a low access frequency state to a high access frequency state, a second determination is made as to whether the actual number of accesses exceeds a preset frequency threshold. If it does, the archive fragment unit is marked as a unit to be relocated. Based on the characteristics of the unit to be relocated, a low-latency distributed storage node with low-latency access capability is selected from the candidate node set as the target node, and storage space for the target node is reserved before relocation. The unit to be migrated is completely copied from the original distributed storage node to the target node, and data consistency verification is performed to ensure that the data after migration is error-free.

8. The computing center expansion method based on distributed storage and hybrid data archiving as described in claim 7, characterized in that: The selection of the target node includes: After the archive shard unit to be migrated is marked as high access frequency, nodes that meet the following conditions are selected from the candidate node set: the current storage load of the node does not exceed the preset load limit; the average access latency of the node is lower than the preset latency threshold; and the network topology distance between the node and the nearest access source of the archive shard unit is less than the preset distance threshold. Among the candidate node set, the final target node is determined in the following logical order: the node with the shortest network topology distance is selected first; if the topology distance is the same, the node with the lowest average access latency is selected; if the latency is also the same, the node with the lowest storage load is selected; the finally determined node is used as the target node for the archive shard unit.

9. One or more processors; The memory stores operable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, including the flow of the computing center expansion method based on distributed storage and hybrid data archiving as described in any one of claims 1 to 8.

10. A computer-readable medium for storing software, characterized in that: The software includes instructions executable by one or more computers, which cause the one or more computers to perform operations, including the flow of the computing center expansion method based on distributed storage and hybrid data archiving as described in any one of claims 1 to 8.

Images