Hybrid load oriented storage data reduction method and system

Abstract

The application provides a mixed load-oriented storage data reduction method, comprising: type identification on an input data block; different types of data blocks are processed in a corresponding manner; wherein the type identification on the data block comprises sampling to obtain a sample set representing the characteristics of the data block, analyzing the numerical characteristics, and judging the data type based on a preset threshold rule; for model data blocks, the redundant high-order indices are separated and compressed; for non-model data blocks, the redundant area and the non-redundant area are identified through fine-grained feature comparison, and are respectively processed by reference or compression. The application solves the technical problems of invalidity and low efficiency of the existing technology in dealing with AI model data and traditional file mixed load scenarios.

Description

Technical Field

[0001] This invention relates to the field of data storage, and more specifically, to a method and system for reducing stored data for mixed workloads. Background Technology

[0002] In the current cloud computing environment, with the acceleration of enterprise digital transformation, cloud service providers face enormous cost pressures in data storage and backup. Disk space, as a core resource, directly impacts service costs and system sustainability through its utilization efficiency. Especially driven by emerging applications such as artificial intelligence training and big data analytics, the volume of unstructured data is exponentially increasing, leading to an exponential rise in demand for cloud storage capacity.

[0003] To alleviate storage resource pressure and improve disk utilization efficiency, existing cloud storage systems commonly employ data deduplication and compression technologies. These technologies primarily reduce physical storage space consumption by identifying and eliminating redundant data. A typical data deduplication and compression process includes data segmentation, fingerprint calculation, and duplicate block identification and compression based on a global index. Specifically, the system divides the data stream into fixed-size data blocks (e.g., 4KB), calculates a hash fingerprint (e.g., SHA-1) for each data block, and if the fingerprint already exists in the index, it is stored as a pointer instead of the original data block; if it is a new fingerprint, the data block is compressed before storage. This mechanism can effectively reduce storage costs in application scenarios where text-based data is predominant.

[0004] However, with AI models and ordinary business files coexisting in the cloud, the structure of storage load has fundamentally changed, significantly reducing the effectiveness of the aforementioned traditional methods. Firstly, in terms of data composition, AI model data primarily consists of highly dynamic floating-point tensors, lacking block-level repeatability. This renders deduplication mechanisms based on block fingerprinting largely ineffective, and compression algorithms struggle to achieve significant compression rates on such data. Secondly, for performance optimization purposes, the system widely adopts megabyte-level large-block partitioning, which, while increasing throughput, further reduces the deduplication matching rate.

[0005] Furthermore, although existing technologies have attempted to optimize for specific workload types—for example, Chinese patent application CN202211033099.6 discloses a strategy combining hardware and software compression that can select the optimal compression path based on data type to improve processing efficiency—most of these methods are geared towards single-load scenarios. In current hybrid cloud storage workloads, ordinary files and model files coexist. The differences between different data types mean that a single algorithm may perform well on one type of data but experience performance degradation or even "negative compression" on another, leading to an overall decrease in efficiency.

[0006] It is evident that existing deduplication and compression technologies have significant limitations in dealing with diverse data types, especially mixed workloads of AI model data and traditional files. There is an urgent need for a new compression and deduplication mechanism that can adapt to the characteristics of mixed data in order to meet the comprehensive requirements of modern cloud storage systems for efficiency, versatility, and accuracy. Summary of the Invention

[0007] In view of the shortcomings of the prior art, the purpose of this invention is to provide a storage data reduction method and system for mixed workloads.

[0008] According to one aspect of the present invention, a storage data reduction method for mixed workloads includes: Step 1: identifying the type of input data blocks, wherein the data block type includes model data and non-model data; Step 2: processing data blocks of different types using appropriate methods.

[0009] Preferably, step one, identifying the type of the input data block, includes: sampling the data in the input data block to obtain a sample set that can represent the global features of the data block; dividing the data in the sample set into multiple groups; detecting the similarity between the contents of each group of data; dividing each group of data into a high similarity group and a low similarity group based on the comparison of the similarity with a preset threshold; and determining the type of the data block according to the ratio of the number of high similarity groups and low similarity groups.

[0010] Preferably, dividing the data in the sample set into multiple groups includes: sequentially traversing each byte of data in the sample set; obtaining the number N of data groups; performing a modulo operation on the index value i of the currently traversed byte data based on the number N; determining the group to which the current byte data belongs based on the operation result, wherein the number n of each group is the value of the modulo operation; performing an integer division operation on the index value of the currently traversed byte data; determining the position of the current byte data in its group based on the operation result; and storing the byte data in the corresponding group storage unit based on the group and its position in its group.

[0011] Preferably, detecting the similarity between the contents of each group of data includes: traversing each byte of data in the group and identifying the value of each byte; counting the number of times each value appears; counting the number of types of the appearing values; and calculating the similarity based on the number of types, wherein the similarity is calculated using the following formula:

[0012] Similarity=1-

[0013] in, The number of different types of values ​​that appear. This represents the total amount of data in the grouped bytes.

[0014] Preferably, in step two, different processing methods are adopted for different types of data blocks, including: for model data blocks, splitting the model data blocks into different byte sequences by bytes, and identifying the index values ​​of redundant data in the model data blocks based on high similarity groups, wherein the index values ​​of redundant data in the model data blocks satisfy the following formula:

[0015]

[0016] Where I is the index value of redundant data in the model data block, N is the number of data groups in the sample set, n is the number of highly similar groups in the sample set, a≥0, and a is a natural number;

[0017] Re-encode the byte data, extract redundant data, and place the redundant data into a contiguous redundant storage area; compress the redundant storage area and store it.

[0018] Preferably, in step two, different processing methods are adopted for different types of data blocks, including: for non-model data blocks, the non-model data blocks are divided into multiple sub-data segments, and fine-grained features are calculated for each sub-data segment; based on the fine-grained features, each sub-data segment is compared with existing data segments globally; if the fine-grained features are the same, the match is successful; otherwise, the match fails; for non-model data blocks where fine-grained features are successfully matched, all existing sub-data segments with the same fine-grained features are identified, the sub-data segments where fine-grained features are successfully matched are classified as redundant areas, and the data segments where fine-grained features fail to match are classified as non-redundant areas.

[0019] For sub-segments of the redundant area, reference information for data reconstruction is generated and stored, wherein the reference information includes a block identifier pointing to an existing data segment in the system, the starting offset of the redundant concentrated area in the existing data segment, and the length of the redundant segment; for sub-segments of the non-redundant area, compression and storage are performed.

[0020] According to another aspect of the present invention, a storage data reduction system for mixed workloads includes:

[0021] Data type identification module: Identifies the type of the input data block, including model data and non-model data;

[0022] Data processing module: Processes different types of data blocks using appropriate methods.

[0023] Preferably, the data type identification module includes: a feature sampling submodule: sampling data in the input data block to obtain a sample set that represents the global features of the data block; a pattern analysis submodule: submodule M1: dividing the data in the sample set into multiple groups; submodule M2: detecting the similarity between the contents of each group of data; submodule M3: dividing each group of data into a high similarity group and a low similarity group based on the comparison of similarity with a preset threshold; and a decision output submodule: determining the type of the data block based on the ratio of the number of high similarity groups and low similarity groups.

[0024] Preferably, the pattern analysis submodule M1 includes: submodule M1.1: sequentially traversing each byte of data in the sample set; submodule M1.2: obtaining the group number N of the data group, performing a modulo operation on the index value i of the currently traversed byte data based on the group number N, and determining the group to which the current byte data belongs based on the operation result, wherein the number n of each group is the value of the modulo operation; submodule M1.3: performing an integer division operation on the index value of the currently traversed byte data, and determining the position of the current byte data in its group based on the operation result; submodule M1.4: storing the byte data into the corresponding group storage unit according to the group of the current byte data and its position in its group.

[0025] Preferably, the submodule M3 includes: submodule M2.1: traversing each byte of data in the group and identifying the value of each byte of data; submodule M2.2: counting the number of times each value appears; submodule M2.3: counting the number of types of the appearing values; submodule M2.4: calculating the similarity based on the number of types, wherein the similarity is calculated using the following formula:

[0026] Similarity=1-

[0027] in, The number of different types of values ​​that appear. This represents the total amount of data in the grouped bytes.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. This invention achieves high-precision automatic identification and differential processing of data types. Through lightweight sampling and numerical feature analysis, the system can automatically distinguish between model data and non-model data based on the internal patterns of data blocks, thereby importing the data into corresponding dedicated processing paths. This mechanism effectively avoids the problem of uneven processing effects for different data types in traditional uniform compression algorithms, laying the foundation for storage optimization in mixed workload environments.

[0030] 2. This invention designs optimized compression and deduplication techniques for different types of data. For model data blocks, the system improves the compressibility of model parameter data by extracting structurally redundant parts such as the high-order bits of floating-point exponents, performing aggregation and dedicated encoding. For non-model data blocks, the system identifies redundant regions within and across blocks based on a fine-grained feature matching mechanism, and achieves efficient removal of local redundancy by replacing duplicate storage with reference information.

[0031] 3. This invention significantly improves storage efficiency and system performance while ensuring data integrity. Through a collaborative architecture of "type identification - branch processing - logical reference", this method improves the overall data reduction rate by approximately 20% to 30% compared to traditional methods in mixed load scenarios, and its advantages are further enhanced as the proportion of model data increases, demonstrating good scalability, compatibility, and engineering application value. Attached Figure Description

[0032] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0033] Figure 1 This is a flowchart illustrating the method of an embodiment of the invention.

[0034] Figure 2 This is a system block diagram of an embodiment of the present invention;

[0035] Figure 3 This is a flowchart of the data type identification module in an embodiment of the present invention.

[0036] Figure 4 This is a flowchart of the model data processing module in an embodiment of the present invention.

[0037] Figure 5 This is a flowchart of the non-model data processing module in an embodiment of the present invention. Detailed Implementation

[0038] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0039] For ease of understanding, the terms or concepts involved in this application are explained below:

[0040] (1) Data block: The basic processing unit of system input, consisting of a fixed length of continuous bytes, and is the object of data reduction operation.

[0041] (2) Model data: derived from model files or floating-point data containing numerical structures, characterized by concentrated exponent bits and dispersed mantissa bits.

[0042] (3) Non-model data: general data that does not have a fixed positional structure, such as text, images, logs, etc., with high randomness.

[0043] (4) Sampling: Select a portion of bytes from the data block to form a sample set to represent the overall characteristics and reduce the amount of computation.

[0044] (5) Grouping: The sample byte index is moduloed and divided according to the preset number of groups N to divide the data into several step subsequences.

[0045] (6) Similarity: An indicator that measures the degree of concentration of values ​​within a group, with the formula Similarity=1 u / SUM, where u is the number of different numerical categories and SUM is the total number of bytes in the group.

[0046] (7) High similarity group: Groups with similarity higher than the threshold indicate that the byte distribution is concentrated and often correspond to the exponent of the model data.

[0047] (8) Low similarity group: Groups with similarity below the threshold, with diverse byte values, often corresponding to the last digit or non-model data.

[0048] (9) Threshold: The boundary value that distinguishes between high similarity groups and low similarity groups, set based on experience or statistics.

[0049] (10) Redundant data: The parts of the model data with low information content due to repeated exponents or concentrated values ​​can be recoded and compressed.

[0050] (11) Re-encoding: Based on the redundancy detection results, the byte data is rearranged and the redundant parts are stored in a centralized manner to improve compression efficiency.

[0051] (12) Redundant storage area: A contiguous storage area specifically for storing redundant bytes, which facilitates compression and indexing.

[0052] (13) Fine-grained features: Identification information used to represent the characteristics of sub-data fragment content, such as hash values ​​or local fingerprints, for global comparison.

[0053] (14) Reference information: A structure describing the relationship between the redundant fragment and the original data, including the block identifier, starting offset and length.

[0054] (15) Data reconstruction: When decoding or reading, the compressed or re-encoded data is restored to the original data block according to the reference information and encoding rules.

[0055] This application proposes a storage data reduction method for mixed workloads. The method first identifies the type of the input data blocks and then proceeds to the corresponding processing flow based on the data block type. This overall process ensures differentiated processing of model data and non-model data within the same mixed workload, achieving a reduction effect that combines compatibility and gain aggregation within the same data channel. In terms of the process, after basic block segmentation and routine deduplication, the data blocks enter the type identification stage. Subsequently, based on the identification result, they are routed to either the model data processing or non-model data processing branch. The outputs of both branches together constitute the final reduced storage data.

[0056] Combination Figure 1 The following is a detailed explanation of each step:

[0057] Step 1, as follows Figure 3 As shown, when performing type identification on the input data block, the data block is first sampled to obtain a sample set that represents the global characteristics of the data block. The purpose of sampling is to approximate the statistical distribution and structural pattern of the original data block with low computational overhead, so that subsequent grouping and similarity detection on the subset yield judgment results that are approximately the same as those on the full dataset. Sampling can be performed by selecting bytes at equal intervals or with fixed spans, or by selecting according to a content-sensitive strategy, provided that the sample set is representative of the overall distribution.

[0058] The data in the sample set is divided into multiple groups to establish straddle alignment. Specifically, each byte of data in the sample set is traversed sequentially, the preset number of groups N is obtained, and a modulo operation i mod N is performed on the index value i of the currently traversed byte. The modulo result n is the group number to which the byte belongs. An integer division operation is then performed on the index value i. i / N The result gives the sequential position of the byte within its group. Through the above "modulo grouping and integer division positioning", the sample set is split into N subsequences extracted at fixed spans, and each group is stored in a corresponding group storage unit. This grouping method establishes a periodic mapping consistent with the underlying data structure (such as the byte bits of a fixed-length value), which is beneficial for amplifying statistical differences at the stride position.

[0059] For each group, content similarity is checked to statistically measure the concentration of numerical values ​​within the group. Specifically, each byte in the group is traversed and its value is identified, the frequency of each value is counted, and the number of different value types, *u*, is also counted. Let the total number of bytes in the group be *SUM*, and the similarity be defined as *Similarity = 1*. u / SUM. This definition reflects that if there are only a very few distinct values ​​in the group (u is much smaller than SUM), then the Similarity is close to 1, indicating high concentration; if the values ​​are widely distributed (u is close to SUM), then the Similarity is close to 0, indicating low concentration. This similarity can effectively characterize the low-entropy characteristics of the model data exponential bits at fixed byte / bit positions.

[0060] The similarity of each group is compared with a preset threshold to classify them into high-similarity groups and low-similarity groups. The threshold is set based on experience or calibration data and is used to distinguish between "centralized distribution" (low-entropy regions such as the exponent or sign bit) and "discrete distribution" (high-entropy regions such as the mantissa). The number of high-similarity groups and their step positions constitute the core basis for determining the data block type.

[0061] The data block type is determined based on the ratio of high-similarity groups to low-similarity groups. For floating-point model data, the exponent portion is typically highly concentrated within a specific width and exhibits high similarity in several fixed groups after step grouping. For example, with a data block length of 160, group number N=4, and a similarity threshold of 60%: if one group is detected as high-similarity and three as low-similarity, it can be identified as 32-bit floating-point model data; if two groups are detected as high-similarity and two as low-similarity, it can be identified as 16-bit floating-point model data. This determination is based on the fact that high-similarity groups originate from the exponent portion. The position of the exponent group is recorded, and the data block is identified as model data accordingly. If the above ratio is not met, it is identified as non-model data. This determination rule relies on the periodicity of the floating-point encoding structure at the byte level: after step grouping alignment, the exponent group exhibits stable low-entropy characteristics, while the mantissa group exhibits high-entropy characteristics due to data changes.

[0062] Step two, for the data blocks determined to be model data, such as Figure 4 As shown, an exponential bit-guided redundancy extraction and recoding compression strategy is adopted. First, the model data block is split into different byte sequences, and the global index I of redundant data is identified based on the number of the highly similar group. The index of redundant data in the model data block satisfies I = a×N + n, where N is the number of sample set groups, n is the number of the highly similar group, and a is a natural number ≥ 0. This formula originates from the inverse mapping of step grouping: elements within the highly similar group n appear at equal intervals with a step size N in the original linear index space, thus allowing for precise location of all suspected redundant bytes through an arithmetic sequence.

[0063] After identifying redundant locations, the byte data is re-encoded. The data identified as redundant is extracted and written into a contiguous redundant storage area according to its original order or the order within the block. This step achieves structured replacement: it gathers weak information bytes that were originally scattered throughout the block and were difficult to utilize into a contiguous area, significantly improving the efficiency and compression ratio of subsequent compression algorithms (such as entropy coding, dictionary coding, or bit-plane coding), while retaining the location information or rearrangement rules required for reconstruction.

[0064] Redundant storage areas are subjected to specialized compression and storage. Due to the significant low entropy and repetitiveness of the content in these areas, the compression algorithm can achieve a higher compression ratio. Correspondingly, non-redundant areas can be stored in their original order or using adaptive coding strategies as needed, to achieve efficient data reduction of the model data overall. During reconstruction, the compressed data is restored to its original location based on the record group number n and the arithmetic progression index relationship I = a×N + n.

[0065] For data blocks that are determined to be non-model data, such as Figure 5 As shown, a fine-grained feature-driven global matching and referencing strategy is employed. First, the non-model data block is divided into multiple sub-data segments, segmented with fixed lengths or content-sensitive boundaries. Fine-grained features are then calculated for each sub-data segment. These fine-grained features can be collision-resistant summaries (e.g., segmented hashing, rolling hashing, local fingerprinting) or statistical / structural features describing the segment content, used for deduplication matching across data blocks and across time dimensions.

[0066] Based on fine-grained features, each sub-data segment is compared with existing data segments globally. If the fine-grained features are the same, the match is considered successful, indicating that the sub-segment already has equivalent content in the system and can be avoided by referencing it; if the features are different, the match is considered unsuccessful. To reduce the risk of false matches, necessary byte-level verification can be performed after a successful feature match.

[0067] For non-model data blocks where fine-grained feature matching is successful, all sub-segments with identical features are identified and classified as redundant regions; the remaining sub-segments that fail to match are classified as non-redundant regions. This classification decouples global deduplication from local compression: redundant regions use references instead of entity storage, while non-redundant regions retain the entities to be compressed.

[0068] For sub-segments of the redundant area, reference information for data reconstruction is generated and stored. This reference information includes at least: a block identifier pointing to an existing data segment in the system, the starting offset of the redundant segment within that existing data segment, and the length of the redundant segment. This ternary information ensures that the corresponding content can be located and backfilled without loss during the reconstruction phase, achieving precise reuse across data blocks and versions.

[0069] For sub-segments of non-redundant regions, compression is performed and the data is stored. Non-redundant regions typically have higher entropy or lack reusable equivalent segments. Encoding them using general or content-adaptive compression algorithms can reduce storage overhead while ensuring lossless decoding. Together with the reference information of the redundant regions, they constitute a complete reduced data representation.

[0070] Key parameters such as similarity threshold, number of groups N, and fine-grained feature granularity can be set through offline calibration or online adaptive strategies. Taking a data block length of 160 bytes and N=4 as an example, each group has a capacity of 40 bytes after step-by-step grouping. In the model data scenario, the exponent bit produces a low-entropy distribution on a fixed group, causing the u of this group to be much less than 40, thus the Similarity is significantly higher than 0.6 and it is judged as a high-similarity group; the mantissa and some sign / alignment bytes have u close to 40 due to the variety of values, corresponding to a Similarity lower than the threshold, and are judged as low-similarity groups. Based on the structural characteristics that the number of high-similarity groups is 1 or 2, corresponding to 32-bit or 16-bit floating-point respectively, further subdivision and positioning of the model data type can be achieved. The entire method achieves targeted data reduction for mixed loads through the chain of "statistical separability → structural mapping → position indexing → structured recoding / reference".

[0071] The invention also provides a storage data reduction system for mixed workloads, such as Figure 2 As shown, the storage data reduction system for mixed workloads can be implemented by executing the process steps of the storage data reduction method for mixed workloads. That is, those skilled in the art can understand the storage data reduction method for mixed workloads as a preferred embodiment of the storage data reduction system for mixed workloads.

[0072] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0073] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A storage data reduction method for mixed workloads, characterized in that, include: Step 1: Identify the type of the input data blocks, where the data block types include model data and non-model data; Step 2: Process different types of data blocks using appropriate methods; Step one involves identifying the type of the input data block, including: The data in the input data block is sampled to obtain a sample set that can represent the global features of the data block; Divide the data in the sample set into multiple groups; Detect the similarity between the contents of each set of data; Based on the comparison of similarity with a preset threshold, each group of data is divided into a high similarity group and a low similarity group; The type of data block is determined based on the ratio of highly similar groups to low similar groups; In step two, different types of data blocks are processed in different ways, including: For the model data block, the model data block is split into different byte sequences. Based on the high similarity group, the index value of redundant data in the model data block is identified. The index value of redundant data in the model data block satisfies the following formula: Where I is the index value of redundant data in the model data block, N is the number of data groups in the sample set, n is the number of highly similar groups in the sample set, a≥0, and a is a natural number; Re-encode the byte data, extract redundant data, and place the redundant data into a contiguous redundant storage area; Compress and store redundant storage areas; In step two, different types of data blocks are processed in different ways, including: For non-model data blocks, the non-model data blocks are divided into multiple sub-data segments, and fine-grained features are calculated for each sub-data segment; Based on fine-grained features, each sub-data segment is compared with existing data segments in a global scope. If the fine-grained features are the same, the match is successful; otherwise, the match fails. For non-model data blocks where fine-grained features are successfully matched, identify all sub-data segments with the same fine-grained features, divide the sub-data segments where fine-grained features are successfully matched into redundant areas, and divide the data segments where fine-grained features are not matched into non-redundant areas. For the sub-segments of the redundant region, reference information for data reconstruction is generated and stored, wherein the reference information includes a block identifier pointing to an existing data segment in the system, the starting offset of the redundant region in the existing data segment, and the length of the sub-segment of the redundant region. For sub-segments in non-redundant areas, compression and storage are performed.

2. The method according to claim 1, characterized in that, The process of dividing the data in the sample set into multiple groups includes: Iterate through each byte of data in the sample set in sequence; Get the number of data groups N. Based on the number of groups N, perform a modulo operation on the index value i of the currently traversed byte data. Based on the operation result, determine the group to which the current byte data belongs, where the number n of each group is the value of the modulo operation. Perform an integer division operation on the index value of the currently traversed byte data, and determine the position of the current byte data in its group based on the result; Based on the current byte data's grouping and its position within the group, the byte data is stored in the corresponding group storage unit.

3. The method according to claim 2, characterized in that, The detection of similarity between the contents of each set of data includes: Iterate through each byte of data in the group and identify the value of each byte; Count the number of times each value appears; Count the number of different types of values ​​that appear; Similarity is calculated based on the number of categories, using the following formula: Similarity=1- in, The number of different types of values ​​that appear. This represents the total amount of data in the grouped bytes.

4. A storage data reduction system for mixed workloads, applied to the method described in any one of claims 1-3, characterized in that, include: Data type identification module: Identifies the type of the input data block, including model data and non-model data; Data processing module: Processes different types of data blocks using appropriate methods.

5. The system according to claim 4, characterized in that, The data type identification module includes: Feature sampling submodule: Samples data from the input data block to obtain a sample set that represents the global features of the data block; Pattern Analysis Submodule: Submodule M1: Divides the data in the sample set into multiple groups; Submodule M2: Detects the similarity between the contents of each set of data; Submodule M3: Based on the comparison of similarity with a preset threshold, it divides each group of data into a high similarity group and a low similarity group; Decision output submodule: Determines the type of data block based on the ratio of high similarity groups to low similarity groups.

6. The system according to claim 5, characterized in that, The submodule M1 of the pattern analysis submodule includes: Submodule M1.1: Iterates through each byte of data in the sample set sequentially; Submodule M1.2: Obtain the number of data groups N, and based on the number of groups N, perform a modulo operation on the index value i of the currently traversed byte data. Based on the operation result, determine the group to which the current byte data belongs, where the number n of each group is the value of the modulo operation. Submodule M1.3: Performs an integer division operation on the index value of the currently traversed byte data, and determines the position of the current byte data in its group based on the operation result; Submodule M1.4: Based on the current byte data's grouping and its position within the group, store the byte data into the corresponding group storage unit.

7. The system according to claim 5, characterized in that, The submodule M2 includes: Submodule M2.1: Traverses each byte of data in the group and identifies the value of each byte; Submodule M2.2: Counts the number of times each value appears; Submodule M2.3: Counts the number of different types of values ​​that appear; Submodule M2.4: Calculates similarity based on the number of categories, where similarity is calculated using the following formula: Similarity=1- in, The number of different types of values ​​that appear. This represents the total amount of data in the grouped bytes.

Images