Data loading and conversion processing method under lake-warehouse fusion architecture

Abstract

The invention belongs to the technical field of data management and intelligent analysis, and particularly relates to a data loading and conversion processing method under a lake-warehouse fusion architecture, which comprises the following steps of: after an externally uploaded file is received, performing compatibility verification and content verification on the externally uploaded file in sequence; identifying that the tables and the partitions which are influenced by data loading and need to be synchronously updated are added, executing atomic data loading operation, synchronously updating the tables and the partitions in the table and the partitions, and generating corresponding version numbers for each table and each partition at the same time; for a data block set under the lake-warehouse fusion architecture, performing hierarchical adjustment according to the access popularity of the data block set; income-cost evaluation is carried out on each data block, and the data blocks needing to be subjected to Z-sequence clustering rearrangement are recognized and added into a requeuing column; and performing Z-sequence clustering rearrangement on each data block in the requeuing column.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data management and intelligent analysis, and particularly relates to a data loading and conversion processing method under a lake warehouse integration architecture. BACKGROUND

[0002] With the development of cloud computing and big data technology, enterprise-level data systems gradually shift from a single data warehouse to a lake warehouse integrated architecture. This architecture uses object storage as a unified base, supports structured and unstructured data, and achieves high flexibility and low-cost storage capabilities. However, under this architecture, there are still the following outstanding problems in data loading and conversion processing: First, high import delay and uncontrollable transactions: the traditional ETL process needs to go through multiple layers of copying and format conversion during the import stage, lacks file-level transaction control mechanisms, and cannot guarantee the consistency of concurrent imports, which can easily cause data loss or duplication.

[0003] Second, inefficient data conversion process: data conversion usually uses full-scan and sequential processing methods, without using statistical information and data distribution characteristics, resulting in a large number of irrelevant I / O operations and repeated calculations.

[0004] Third, lack of traceability of historical versions: existing systems often cannot trace back or differentially restore historical data states, and need to reload complete data sets after data pollution, which seriously affects system usability.

[0005] Therefore, there is an urgent need for a general algorithm system that can achieve fast import, transaction consistency, and efficient conversion under a lake warehouse integration architecture to meet the performance requirements of enterprises in real-time analysis and large-scale data governance scenarios. SUMMARY

[0006] Therefore, the purpose of the present application is to provide a data loading and conversion processing method under a lake warehouse integration architecture, which realizes second-level import and low-delay conversion of large-scale heterogeneous data, and ensures data consistency and traceability.

[0007] The technical solutions of the present application are as follows: In the first aspect, the present application provides a data loading and conversion processing method under a lake warehouse integration architecture, and the specific process is as follows: Enhanced transactional data loading: when receiving an externally uploaded file, perform compatibility verification and content verification on it in sequence; for the file set that passes the verification , according to the global dependency graph of metadata under the lake warehouse integration architecture, identify the tables and partitions that need to be synchronously updated due to data loading ; perform atomic data loading operations and synchronously update the tables and partitions in , and generate corresponding version numbers for each table and partition; Data transformation processing based on access heat and Z-sequence clustering: for a data block set under a lake warehouse fusion architecture, hierarchical adjustment is performed according to the access heat thereof; for each data block, benefit-cost evaluation is performed, data blocks that need to be rearranged by Z-sequence clustering are identified and added to a rearrangement queue The method further comprises the following steps of: performing Z-sequence clustering rearrangement on each data block in the rearrangement queue. The method further comprises the following steps of: performing Z-sequence clustering rearrangement on each data block in the rearrangement queue.

[0008] Optionally, when receiving an externally uploaded file, the application performs transaction registration on the file, assigns a unique transaction number to each file, and writes the transaction number, the file path, the receiving time and the initial state into a transaction log.

[0009] Optionally, the specific process of the compatibility check is as follows: performing structural analysis on the uploaded file, generating a unified structure signature according to three components of a field statistical abstract, a path structure abstract and a sample abstract, comparing the structure signature with a signature of a target table mode pre-stored in the system, and if the similarity is lower than a set threshold, the compatibility check fails and the content check is not performed.

[0010] Optionally, the structure signature is as follows:

[0011] wherein, is a field statistical abstract, an abstract vector obtained by concatenating the data type, the nullability and the rough base estimate result of each field; is a path structure abstract, obtained by encoding a path sequence obtained by performing pre-order traversal on semi-structured data; is a sample abstract, an abstract vector obtained by extracting multi-field quantile numbers from sample records and performing hash compression, denotes hash conversion.

[0012] Optionally, the content check is performed in parallel for each field and record in the file, and the check rules include: Field type matching: verifying whether the data type of the field in the file is consistent with the target table mode; Primary key integrity check: checking whether there is a primary key conflict in the file; Data constraint check: ensuring that the data satisfies the predefined constraint.

[0013] Optionally, when performing the content check, a load-aware parallel scheduling mechanism is adopted, that is, the check parallelism is determined by the system runtime load.

[0014] wherein, for maximum parallelism, for CPU utilization, for I / O latency, for queue length to be checked, for weight parameter.

[0015] Optionally, the specific process of the atomic data loading operation of the present application is as follows: First, determine the target table and partition that each file needs to be written into in the collection , and generate a file for the file, which records all successfully submitted file paths and their structure signatures; Second, perform the atomic data writing operation of the file in the target table and partition, and synchronously update each table and partition in to generate the corresponding version number ; if an error occurs during writing, the system will restore all tables to the previous version through the rollback mechanism; Finally, after the data writing operation is completed, the system generates a snapshot file , which records the version information of this transaction and its related files.

[0016] Optionally, the specific process of adjusting the level according to the access heat is as follows: First, calculate the access heat of the data block at time :

[0017] wherein, is the decay coefficient; Second, set double thresholds and to adjust the storage level of the data block; when the access heat of the data block is higher than the threshold , its level is promoted from the original level to a faster storage medium; when the heat of the block is lower than the threshold for a long time, the block is downgraded to a lower speed medium; if the heat of the block is between the two thresholds, the original level is maintained.

[0018] Optionally, the specific process of the benefit-cost evaluation is as follows: First, for the data block , estimate the filtering performance improvement that may be brought by the Z-order clustering operation on the field set according to the field distribution characteristics of the block to obtain the benefit value : Secondly, according to the storage hierarchy of the data block , the cost of performing Z-order rearrangement is estimated : Finally, the system establishes a trigger condition between the benefits and the cost, and only when the rearrangement benefit is greater than the rearrangement cost, that is , and the expected amount of data rewriting of the block is lower than the limit, the data block is added to the rearrangement queue.

[0019] Optionally, the specific process of Z-order cluster rearrangement according to the application is as follows: For the data block in the reordering queue , the system generates Z-order encoding for its internal records according to the key field set , and performs local rearrangement of the records in the data block according to the encoding as the sorting basis. After sorting, the system writes the rearranged data block to a new storage location and generates new metadata for it, and the rearranged block is denoted as .

[0020] Advantages: First, the application adopts a transaction registration and structure signature generation mechanism in the file import stage, which can filter structure incompatible files before import, avoid invalid I / O and subsequent rollback operation, and significantly reduce import delay. Second, the application improves the verification throughput and resource utilization in high-concurrency scenarios by parallel verification and adaptive parallelism adjustment based on system load, while ensuring the integrity of the primary key and the completeness of the constraint verification. Third, the application constructs a transaction submission plan based on a global dependency graph, uses to realize consistent update across tables (and partitions), and supports multi-version backtracking combined with version number and snapshot file management mechanism. When writing abnormally, it can automatically roll back to the previous valid version, thereby ensuring data transaction consistency and traceability in lake warehouse environment.

[0021] Fourth, the application selectively performs Z-order cluster rearrangement on high-benefit data blocks by using access heat-driven data block layering and benefit-cost evaluation model, so that related data is continuously arranged on physical storage, effectively improving I / O locality and subsequent analytical query execution performance.

[0022] In summary, the application realizes second-level import and efficient conversion in lake warehouse integrated scenarios while ensuring consistency and traceability, and can be widely applied to enterprise-level data lake, data warehouse, AI data middle platform and high-concurrency data fusion scenarios. BRIEF DESCRIPTION OF DRAWINGS​​

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the overall system architecture of the present invention; Figure 2 Flowchart for loading transactional data; Figure 3 This is a schematic diagram of multi-level cache scheduling based on access frequency; Figure 4 Flowchart for data transformation and Z-order cluster layout optimization. Detailed Implementation The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0025] It should be noted that, in the absence of conflict, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0026] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0027] like Figure 1 As shown in the figure, an embodiment of this application provides a data loading and transformation processing method under a lakeware fusion architecture, which includes the following process: Enhanced transactional data loading: Upon receiving an externally uploaded file, perform compatibility and content checks sequentially; then, for the set of files that pass the checks... Based on the global dependency graph of metadata under the lakeware fusion architecture, tables and partitions that need to be synchronously updated due to data loading are identified and added to the database. In the middle; perform atomic data loading operations, and on Tables and partitions in the table are synchronously updated, and a corresponding version number is generated for each table and partition; Data transformation processing based on access heat and Z-sequence clustering: For a data block set under a lake warehouse fusion architecture, the access heat is adjusted according to the level; For each data block, the benefit-cost evaluation is performed, the data block that needs to be rearranged by Z-sequence clustering is identified and added to the rearrangement queue In the rearrangement queue Z-sequence clustering rearrangement is performed on each data block in the rearrangement queue.

[0028] The data processing processes of the above two stages are described in detail as follows: As shown in Figure 2 , the embodiment proposes a transactional data loading algorithm for a lake warehouse fusion architecture, which includes four stages of transaction registration, structure signature generation, parallel verification, transaction submission plan construction and data writing. The following contents respectively make specific limitations and list necessary formulas for each step.

[0029] 1. Transaction registration When an external file upload triggers a system event, the system first registers the transaction of the file. The registration operation assigns a unique transaction number to each file and writes it to the transaction log together with the file path , the receiving time and the initial state.

[0030] The main output of the transaction registration stage: : used to identify the import transaction and as the associated primary key for the subsequent process; : used to read the file content in the structure signature generation stage.

[0031] 2. Structure signature generation and structure compatibility determination The structure signature generation stage obtains and from the transaction registration, and performs structure analysis on the file. The embodiment adopts multi-dimensional structure information verification, and generates a unified structure signature through field statistical summary, path structure summary and sample summary three components:

[0032] Among them, is the field statistical summary, which is obtained by concatenating the data type, nullability and rough cardinality estimation result of each field (i.e. the concatenation summary of field statistical information); is the path structure summary, which is obtained by encoding the path sequence obtained by performing pre-order traversal on semi-structured data (i.e. the path sequence summary of structure information); For sample abstract, extract multi-field quantile from sample record and get abstract vector (i.e. compressed abstract of sample information) by hash compression, Indicates hash conversion.

[0033] The obtained structure signature will be compared with the signature of the target table pattern in the system. If the similarity is lower than the threshold , the file will be marked as structure incompatible, so as to avoid invalid verification from entering the execution of the next step.

[0034] The main output of this phase is: Structure signature : the rule selection basis relied on by verification; 3. Parallel verification execution The parallel verification phase relies on structure signature , transaction number and . The system selects the applicable verification rule according to the content of ; at the same time, it associates all files and rule execution results in this parallel verification process to the same transaction by using transaction number , which is used for subsequent commit or rollback decision. The verification rules mainly include: Field type matching: verifying whether the data type of the field in the file is consistent with the target table pattern; Primary key integrity verification: checking whether there is a primary key conflict in the file; Data constraint verification: ensuring that the data meets the predefined constraints (such as non-empty, uniqueness, etc.).

[0035] The system reads the file content according to , processes each data record of the file body (i.e. each business record in the file) in parallel at the record granularity, and checks in sequence according to the three types of verification rules based on the field definition recorded in the structure signature. In the rule execution process, the system writes the verification result of each record and the current transaction number into the transaction log for the construction of the subsequent transaction commit plan. To adapt to the high concurrency scenario, the verification process is executed in parallel. At the same time, this embodiment adopts a load-aware parallel scheduling mechanism to determine the verification parallelism :

[0036] Among them, is the CPU utilization rate, is the I / O delay, is the length of the to-be-verified queue, is the weight parameter. This mechanism enables the verification phase to reduce concurrency in congestion situations and increase processing rate when idle, thereby maintaining overall low latency.

[0037] After the verification process is completed, the system records the results of this parallel verification in the transaction log as the key . If a file passes all verifications, it will be added to the set of successfully verified files ; if the verification fails, the file will be marked as failed, and further processing of the current transaction will be terminated. The main output of this phase is the set of successfully verified files .

[0038] 4. Transaction commit plan construction and data writing In the process of executing data loading, first, the system constructs a transaction commit plan and prepares to execute the actual writing operation of data based on the set of successfully verified files and the structure signature generated in the parallel verification execution phase. This phase not only completes data writing, but also implements atomic update across tables through , and finally generates a version snapshot.

[0039] First, the system identifies the target table and target partition corresponding to each file using and . In this invention, "partition" refers to data partitioning or storage sharding on the lake warehouse storage layer, which can correspond to part of the data of a table, or can be used as a shared storage area for a group of related tables as a transaction commit and dependency analysis unit at the same level as "table".

[0040] Second, the system identifies which tables and partitions are mutually dependent on the target table / partition and are affected by this data loading according to the global dependency graph in the metadata, and adds the identified tables or partitions to .

[0041] Global dependency graph is represented as:

[0042] wherein is the set of all related tables or partitions, is the dependency relationship between tables. The generation of dependency relationships is based on the structure of the table, data flow, and business logic (for example, some tables may depend on the data update of other tables). Based on these dependency relationships, the system generates , which represents the tables and partitions that need to be updated synchronously in this load transaction.

[0043]

[0044] wherein each represents a table or partition affected by this data loading operation. is an important mechanism to ensure data consistency and atomicity, which defines which tables / partitions must be updated simultaneously in the same transaction, thereby avoiding partial commit situations.

[0045] After generating , the system begins to perform data write operations. First, the system determines which files need to be written to which target tables or partitions according to and , and generates files for these files, which record all successfully committed file paths and their structure signatures.

[0046] The file is represented as:

[0047] where each pair ( , ) represents a file and its corresponding structure information.

[0048] Subsequently, the system performs atomic data commit operations through the table / partition information in . For each table or partition in , the system generates a version number , and all related tables are updated synchronously within the same transaction, ensuring data consistency.

[0049] If any error occurs during the write process, the system will restore all tables to the previous version through the rollback mechanism, avoiding the situation where some tables or partitions are successfully committed and some tables / partitions are not updated.

[0050] After the data write operation is completed, the system generates a snapshot file , which records the version information of this transaction and its related files.

[0051] This snapshot file includes the following contents:

[0052] where : the newly generated version number, : the unique identifier of this transaction, : the file path and structure signature committed by this transaction, : the timestamp of this snapshot generation.

[0053] Through the dependent execution of the above several stages, this embodiment realizes the structured verification of heterogeneous files, load adaptive verification, cross-table atomic commit, and version-level snapshot management. The final output of the new snapshot All the files, structure abstracts and version states of this submission are recorded, and the strong consistency, multi-version and traceable transactional data loading in the lake warehouse fusion architecture is realized. Thus, the transactional data loading algorithm is completely executed.

[0054] As Figures 3-4 shown, the embodiment provides a data transformation method for a lake warehouse fusion architecture. The method takes data blocks as the basic processing unit, identifies data regions worthy of optimization by analyzing access behavior, and selectively performs Z-sequence cluster rearrangement combined with a benefit-cost model to improve data locality and structure optimization. The entire method is executed in the order of "access heat analysis and level preparation → benefit-cost judgment → cluster rearrangement", thereby realizing the rapid transformation of large-scale data.

[0055] 1. Data block storage level adjustment based on access heat In this step, the system first analyzes the historical access records of each data block in the data block set . For block , its access timestamp sequence is denoted as . The system uses a time decay model to assign different weights to access behavior, so that recent access has a greater impact on heat. The access heat at time is denoted as

[0056] where is the decay coefficient, and is dynamically adjusted according to the deviation between the current cache hit rate and the target hit rate, so that the heat calculation can adapt to system load changes.

[0057] After obtaining the access heat, the system further adjusts the storage level of the block according to the double thresholds and . When the heat of the block is higher than the threshold , its level is promoted from the original level to a faster storage medium; when the block heat is lower than the threshold for a long time, the block is downgraded to a lower speed medium; if the block heat is between the two thresholds, the original level is maintained. The adjusted level is denoted as

[0058] After the above calculation and adjustment, the output of step 1 is:

[0059] which contains the access heat and updated storage level information of each data block.

[0060] 2. Z-sequence rearrangement decision based on benefit-cost evaluation First, this step takes as input and performs a benefit-cost evaluation on the data block to determine whether to perform Z-sequence cluster rearrangement on the block. For block , the system first estimates the filter performance improvement that can be achieved by performing Z-sequence clustering on the field set , obtaining the benefit value:

[0061] where is the benefit evaluation function, which estimates the query performance improvement that can be achieved by performing Z-sequence clustering on the target field set based on the value distribution characteristics of the data block and the typical query filter conditions, and its output represents the expected benefit of performing Z-sequence rearrangement on the data block .

[0062] Second, the system estimates the cost of performing Z-sequence rearrangement in combination with the storage hierarchy calculated in step 1:

[0063] where is the cost evaluation function, which estimates the resource overhead required to perform Z-sequence cluster rearrangement on the block once based on the size of the data block , the estimated amount of rewritten data, and the media characteristics (such as read-write bandwidth, I / O delay, and single I / O cost) of the storage hierarchy where the block is located, and its output can be represented as the expected rearrangement time or equivalent I / O cost; blocks on different storage hierarchies will have different cost evaluation values, and blocks on high-speed media have lower read and rewrite costs.

[0064] Finally, the system establishes a trigger condition between the benefit and the cost, and only when the rearrangement benefit is greater than the rearrangement cost, i.e. and the estimated amount of rewritten data for the block is lower than the limit, i.e. , the block is added to the rearrangement queue. Thus, the output of step 2 is:

[0065] where represents the set of data blocks that need to perform Z-sequence cluster rearrangement.

[0066] 3. Z-sequence cluster rearrangement and transformation result generation In this step, the system performs Z-ordered cluster rearrangement on each block in the rearrangement queue obtained in step 2 . For a block , the system generates Z-ordered encoding for its internal records according to the key field set :

[0067] and performs local rearrangement on the records in the block according to the encoding. After sorting, the system writes the rearranged block to a new storage location and generates new metadata for it, and the rearranged block is denoted as .

[0068] Finally, the system organizes all the rearranged blocks and the original blocks that do not participate in rearrangement into a new data version structure. The version can be formally represented as

[0069] where the Z-ordered optimized blocks obtain better local order, and the blocks that are not rearranged maintain their original structure, and the two together constitute a complete and consistent new data version.

[0070] That is, the data transformation result of the embodiment can be directly used for subsequent analytical query tasks.

[0071] To sum up, the above is only a preferred embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for data loading and conversion processing under a lake warehouse fusion architecture, characterized in that, The specific process is as follows: Enhanced transactional data loading: After receiving an external uploaded file, compatibility check and content check are sequentially performed on it; for the file set that passes the check , according to the global dependency graph of metadata under the lake warehouse integration architecture, the tables and partitions affected by the data loading and needing to be synchronously updated are identified ; the atomic data loading operation is performed, and the tables and partitions in are synchronously updated, and the corresponding version number is generated for each table and partition; Data transformation processing based on access heat and Z-sequence clustering: for a data block set under a lake warehouse fusion architecture, a hierarchical adjustment is performed according to access heat thereof; a benefit-cost evaluation is performed for each data block, and a data block needing Z-sequence clustering rearrangement is identified and added to a rearrangement queue Z-sequence clustering rearrangement is performed on each data block in the rearrangement queue ​ 2. The method of claim 1, wherein the data loading and conversion processing method under the lake warehouse fusion architecture is characterized by, When receiving the externally uploaded file, the file is transacted, a unique transaction number is assigned to each file, and the file path, receiving time and initial state are written into the transaction log.

3. The method of claim 1, wherein the data loading and conversion processing method is performed in the lake-house architecture. The specific process of the compatibility check is as follows: the uploaded file is analyzed in structure, a unified structure signature is generated according to the field statistical abstract, path structure abstract and sample abstract, the structure signature is compared with the signature of the target table mode pre-stored in the system, and if the similarity is lower than the set threshold, the compatibility check fails and the content check is not performed.

4. The method of claim 3, wherein the data loading and conversion processing is performed in the lakehouse architecture. The structural signature Is: wherein, is a field statistics summary, a summary vector obtained by concatenating the data type, nullability and rough cardinality estimation result of each field; is a path structure summary, obtained by encoding the path sequence obtained by performing pre-order traversal on semi-structured data; is a sample summary, a summary vector obtained by extracting multi-field quantiles from sample records and performing hash compression, denotes a hash conversion.

5. The method of claim 3, wherein the data loading and conversion processing method under the lake warehouse fusion architecture is characterized by, The content check is performed in parallel for each field and record in the file, and the check rules include: Field type matching: verifying whether the data type of the field in the file is consistent with the target table mode; Primary key integrity check: checking whether there is a primary key conflict in the file; Data constraint check: ensuring that the data meets the predefined constraints.

6. The method of claim 5, wherein the data loading and conversion processing is performed in the lakehouse architecture. In the content checking, a load-aware parallel scheduling mechanism is adopted, that is, the checking parallelism is determined by the system runtime load : wherein, is the maximum parallelism, is the CPU utilization, is the I / O latency, is the queue length to be checked, is the weight parameter.

7. The method of claim 1, wherein the data loading and conversion process is performed in the lake-house architecture. The specific process of the atomic data loading operation is as follows: First, determine the target table and partition each file needs to be written to, and generate a file record for each file file, which records all successfully committed file paths and their structure signatures; Secondly, perform atomic data write operations on the files in the target table and partition, and then... Each table and partition in the database is synchronized and updated to generate a corresponding version number. If an error occurs during the write process, the system will use a rollback mechanism to restore all tables to the previous version. ; Finally, after the data write operation is completed, the system generates a snapshot file , recording the version information of this transaction and its related files.

8. The method of claim 1, wherein the data loading and conversion processing is performed in the lakehouse architecture. The specific process of adjusting the level according to the access heat is as follows: First, the data block At time The access heat of the data block wherein is the attenuation coefficient; Secondly, double thresholds are set With The storage level of a data block is adjusted; when the access hotness of the data block is higher than a threshold , its level is promoted from the original level to a faster storage medium; when the hotness of the data block is lower than a threshold for a long time , the block is degraded to a slower medium; if the hotness of the block is between the two thresholds, the original level is maintained.

9. The method of claim 1 or 8, wherein the data loading and conversion process is performed in the lake-house architecture. The specific process of the benefit-cost evaluation is as follows: First, for a data block , according to the field distribution characteristics, the filtering performance improvement that may be brought by performing Z-order clustering operation on the field set is estimated to obtain a benefit value : Second, the cost of performing Z-sequential reordering is estimated according to the storage hierarchy of the data blocks : Finally, the system establishes a trigger condition between the benefit and cost of the rearrangement, and only when the benefit of rearrangement is greater than the cost of rearrangement, i.e. and the expected amount of data to be rewritten for the block is below the limit, the data block is added to the rearrangement queue.

10. The method of claim 9, wherein the data loading and conversion processing is performed in the lakehouse architecture. The specific process of the Z sequence clustering rearrangement is as follows: For data blocks in the rearranged sequence The system is based on the set of key fields. Generate Z-order codes for its internal records , to encode To determine the sorting criteria, the records within a data block are partially rearranged. After sorting, the system writes the rearranged data block to a new storage location and generates new metadata for it. The rearranged block is denoted as [insert block name here]. .

Images