Data skew adaptive processing method, device, equipment and medium

Abstract

The present application relates to the technical field of cloud computing, and can be applied to business scenarios such as financial technology, and discloses a data skew adaptive processing method, device, equipment and medium, comprising: performing non-iterative sampling on a target data set through a metadata management interface, obtaining key value distribution information and identifying a target hot key set; combining the operation operator type of the data processing task and the data magnitude characteristics of the hot key to match an optimized execution strategy from a preset optimization strategy library, performing logical shunting processing on the target data set and completing result merging; collecting resource consumption indicators and data transmission indicators during task execution, comparing with historical benchmarks and updating the priority parameters of the optimized execution strategy. The present application realizes dynamic identification of data skew and adaptive adjustment of strategies by combining metadata analysis and running feedback, reduces resource consumption caused by skew, and improves the execution efficiency and stability of data processing tasks.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, and in particular to a data skew adaptive processing method, apparatus, device, and medium. Background Technology

[0002] In financial data processing scenarios, enterprises typically need to perform calculations such as grouping and summarizing, period carry-forward, reconciliation and verification, and multi-dimensional correlation analysis on data such as accounting entries, payment and receipt records, and general ledger details. Because financial data naturally exhibits a non-uniform distribution across dimensions such as accounts, periods, and categories, data skew is a common problem when performing batch processing based on distributed computing frameworks. This can easily lead to abnormally concentrated loads on some computing nodes, resulting in significantly longer task execution times, increased resource consumption fluctuations, and decreased computational stability.

[0003] In existing technologies, financial data skew commonly manifests as a large number of transaction records being carried by a few high-frequency transaction accounts or core accounts. For example, when grouping and summarizing by account code or account identifier as the key, the data volume corresponding to some key values ​​is much higher than the overall average, resulting in concentrated computational pressure at single points during the grouping and aggregation stage. Similarly, during concentrated operation windows such as financial closing and period carry-over, data for specific accounting periods is intensively accessed and repeatedly calculated. If the period dimension is used as the partition or processing key value, it is easy to form local data density and uneven distribution of computing resources in a short period of time, thereby amplifying the impact of skew. In addition, when performing correlation analysis between financial factual data and master data dimension tables such as accounts, customers, and suppliers, due to the skewed distribution of key values ​​and the difference in the scale of correlation inputs, conventional correlation execution methods are prone to data accumulation in some partitions, resulting in network transmission and memory consumption during the correlation stage being significantly higher than other partitions, affecting the overall operation throughput.

[0004] To address the aforementioned issues, existing technologies typically employ data shuffling by adding random prefixes or suffixes to key-value pairs to alleviate single-point pressure during aggregation or association phases. However, this approach often alters the original key-value representation, necessitating additional restoration and re-aggregation processes in subsequent processing. This increases the complexity of the process and may introduce additional intermediate data and data transfer overhead, resulting in a longer and more difficult-to-maintain overall processing flow. Meanwhile, some solutions attempt to distribute computational pressure by increasing parallelism, adding partitions, or upgrading resource specifications. However, when hot key-value pairs constitute a very high proportion or exhibit significant skewness, hot partitions can still become long-tail bottlenecks. Resource expansion offers limited improvement in reducing overall execution time and significantly increases resource consumption. Summary of the Invention

[0005] The main objective of this invention is to provide a data skew adaptive processing method, apparatus, device, and storage medium, aiming to solve the technical problem that existing technologies are unable to automatically identify hot key values ​​and adaptively select and evolve processing strategies based on the dynamic changes in data distribution without destroying the original business semantics of financial data. This results in data skew processing relying on manual experience or static rules, with unstable optimization effects and difficulty in long-term reuse.

[0006] To achieve the above objectives, the present invention provides a data skew adaptive processing method, comprising: Before starting the data processing task, the target dataset is sampled non-traversally by accessing the metadata management interface of the storage system to obtain metadata distribution information containing key-value distribution characteristics. The statistical distribution parameters of the target dataset are calculated based on the metadata distribution information, and the distribution characteristics of each key value in the metadata distribution information are compared with the statistical distribution parameters to identify the target hot key set; The operation operator type of the data processing task is parsed, and the data volume characteristics of the target hot spot key set are analyzed. Based on the operation operator type and the data volume characteristics, the optimization execution strategy corresponding to the target hot spot key set is matched from the preset optimization strategy library. The target dataset is logically split, and the optimized execution strategy is applied to the data corresponding to the target hot key set to generate hot data processing results. The default strategy is applied to the data corresponding to the key values ​​in the target dataset other than the target hot key set to generate non-hot data processing results. The results of processing the hot data are merged with the results of processing the non-hot data to complete the data processing task; The resource consumption and data transmission metrics of the data processing task during execution are collected, and the resource consumption and data transmission metrics are compared with historical benchmarks. Based on the comparison results, the priority parameters of the optimization execution strategies in the preset optimization strategy library are updated.

[0007] Furthermore, to achieve the above objectives, the present invention provides a data skew adaptive processing apparatus, comprising: The metadata sampling and analysis module is used to perform non-traversal sampling of the target dataset by accessing the metadata management interface of the storage system before starting the data processing task, and to obtain metadata distribution information containing key-value distribution characteristics. The hotspot key identification module is used to calculate the statistical distribution parameters of the target dataset based on the metadata distribution information, and compare the distribution characteristics of each key value in the metadata distribution information with the statistical distribution parameters to identify the target hotspot key set; The strategy matching and decision module is used to parse the operation operator type of the data processing task, analyze the data volume characteristics of the target hot spot key set, and match the optimization execution strategy corresponding to the target hot spot key set from the preset optimization strategy library according to the operation operator type and the data volume characteristics. The data splitting execution module is used to logically split the target dataset, apply the optimized execution strategy to process the data corresponding to the target hot key set, generate hot data processing results, and apply the default strategy to process the data corresponding to key values ​​in the target dataset other than the target hot key set, generate non-hot data processing results. The result merging and output module is used to merge the hot data processing results with the non-hot data processing results to complete the data processing task. The strategy feedback update module is used to collect resource consumption indicators and data transmission indicators during the execution of the data processing task, compare the resource consumption indicators and data transmission indicators with historical benchmarks, and update the priority parameters of the optimized execution strategies in the preset optimization strategy library based on the comparison results.

[0008] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a data skew adaptive processing program stored in the memory and executable on the processor, wherein when the data skew adaptive processing program is executed by the processor, it implements the steps of the data skew adaptive processing method as described above.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a data skew adaptive processing program, which, when executed by a processor, implements the steps of the data skew adaptive processing method as described above.

[0010] Beneficial Effects: This invention relates to the field of cloud computing technology and can be applied to business scenarios such as fintech. It discloses a data skew adaptive processing method, apparatus, device, and medium, comprising: performing non-traversal sampling of the target dataset through a metadata management interface to obtain key-value distribution information and identify the target hot key set; combining the operation operator type of the data processing task with the data volume characteristics of the hot keys, matching an optimized execution strategy from a preset optimization strategy library, performing logical splitting processing on the target dataset, and merging the results; collecting resource consumption indicators and data transmission indicators during task execution, comparing them with historical benchmarks, and updating the priority parameters of the optimized execution strategy. This invention achieves adaptive identification and strategy adjustment of data skew through metadata-level analysis and operational feedback mechanisms, reducing reliance on manual configuration, lowering performance losses caused by skew, and improving the overall execution efficiency and stability of data processing tasks. Attached Figure Description

[0011] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for the data skew adaptive processing method in one embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the data tilt adaptive processing method of the present invention; Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the data tilt adaptive processing device of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0013] The data skew adaptive processing method provided in this invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The server can use the client to perform non-traversal sampling of the target dataset based on the metadata management interface, obtain key-value distribution information, and identify the target hot key set. Combining the operation operator type of the data processing task with the data volume characteristics of the hot keys, it matches an optimization execution strategy from a preset optimization strategy library, performs logical splitting processing on the target dataset, and completes result merging. During task execution, it collects resource consumption indicators and data transmission indicators, compares them with historical benchmarks, and updates the priority parameters of the optimization execution strategy. This invention achieves adaptive identification and strategy adjustment for data skew through metadata-level analysis and operational feedback mechanisms, reducing reliance on manual configuration, minimizing performance losses caused by skew, and improving the overall execution efficiency and stability of data processing tasks. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster composed of multiple servers. The invention will be described in detail below through specific embodiments.

[0014] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the data skew adaptive processing method provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0015] like Figure 2 As shown, the data skew adaptive processing method proposed in this invention includes the following steps: S10, Before starting the data processing task, the target dataset is sampled non-traversally by accessing the metadata management interface of the storage system to obtain metadata distribution information containing key value distribution characteristics. In this embodiment, during the period before the data processing task is initiated, the execution object is limited to the target dataset, which corresponds to the boundary of the data set to be processed. The boundary source can be an input path list, table registration information, or partition selection conditions. By accessing the storage system's metadata management interface, a metadata query request is initiated to the target dataset. The content output by the metadata management interface is limited to metadata rather than data row content. The metadata can include a partition list, file entries, file block entries, and corresponding statistical fields.

[0016] Non-traversal sampling of the target dataset does not involve performing a full read and parsing of all data rows in the target dataset. Instead, it constructs a sampling view based on the metadata information returned by the metadata management interface. The sampling view can select subsets at the partition level, or at the file entry or file block entry level. The selection rules can include uniform sampling, stratified sampling by size, or sampling by time window. The output of the sampling view remains a collection of metadata.

[0017] Obtaining metadata distribution information containing key-value distribution features involves mapping statistical fields in the sampling view to the key-value dimension and performing aggregation. A key value corresponds to a set of values ​​in the target dataset used to distinguish data ownership; the key value can originate from partition definitions or other key fields that can be used for aggregation. Key-value distribution features correspond to the scale representation at the key-value level, which can be represented by row count statistics or storage size statistics. The structure of the metadata distribution information can include a summary of the row count statistics corresponding to the key value and a summary of the storage size statistics, allowing the metadata distribution information to directly express the distribution status of the key-value dimension.

[0018] In one implementation, the metadata management interface returns a list of partitions and file entries within each partition. Non-traversal sampling evenly extracts file entries by partition at the file entry level. The row count and storage size statistics of the extracted file entries are mapped to key values ​​according to the partition definition and aggregated to generate metadata distribution information. In another implementation, the metadata management interface returns file block entries and block-level statistical fields. Non-traversal sampling hierarchically extracts file block entries by block size at the file block entry level. The statistical fields of the file block entries are aggregated using key values ​​to generate metadata distribution information.

[0019] This embodiment accesses the storage system's metadata management interface and performs non-traversal sampling before starting the data processing task. Based on the metadata, it constructs metadata distribution information and forms key-value distribution features in the key-value dimension. This reduces the need to read and parse the target dataset's data rows, lowers the input / output burden during the startup phase, and shortens the path for obtaining distribution information.

[0020] S20, calculate the statistical distribution parameters of the target dataset based on the metadata distribution information, and compare the distribution characteristics of each key value in the metadata distribution information with the statistical distribution parameters to identify the target hotspot key set; In this embodiment, after obtaining the metadata distribution information, statistical quantitative analysis is performed on the target dataset. The statistical distribution parameters correspond to the mathematical characterization of the overall distribution state and are used to form a unified reference benchmark. The statistical distribution parameters are derived from the aggregated set of key-value distribution features in the metadata distribution information, and are obtained by performing statistical operations on this set. The statistical operations may include a measure of central tendency and a measure of dispersion of the key-value distribution features. The measure of central tendency is used to reflect the overall distribution level, and the measure of dispersion is used to reflect the deviation of the key-value distribution features from the overall level.

[0021] The distribution characteristics of each key value in the metadata distribution information are compared with statistical distribution parameters. The purpose of this comparison is to map the actual scale and state of a single key value onto the overall distribution coordinate system for measurement. The comparison process does not alter the key value distribution characteristics themselves; it only constructs a judgment reference based on the statistical distribution parameters. The comparison results are used to determine whether the distribution characteristics of a particular key value significantly deviate from the overall distribution level, thereby forming an abnormal cluster in the overall dataset.

[0022] The process involves identifying and summarizing the comparison results corresponding to the target hotspot key set. When the distribution characteristics of a key value meet preset deviation conditions in the comparison results, that key value is marked as a candidate hotspot key. All key values ​​that meet the deviation conditions are summarized to form the target hotspot key set. The target hotspot key set is based on key value identifiers and does not contain the original data content; it only retains key value range information for subsequent processing.

[0023] In one implementation, the arithmetic mean and standard deviation of the key value distribution features are calculated as statistical distribution parameters. Hotspot key identification is achieved by comparing the difference between the distribution features of individual key values ​​and the arithmetic mean, and by combining this with the standard deviation to construct deviation judgment conditions. In another implementation, the median and quantile intervals of the key value distribution features are calculated as statistical distribution parameters. Hotspot key identification is achieved by comparing whether the key value distribution features fall outside the quantile intervals, thus adapting to datasets with asymmetrical distributions.

[0024] This embodiment calculates statistical distribution parameters based on metadata distribution information and performs an overall comparison of key value distribution characteristics, so that the identification of hot keys is based on the global distribution, avoiding misjudgments caused by relying solely on absolute thresholds, and improving the stability and adaptability of key value anomaly identification.

[0025] S30, parse the operation operator type of the data processing task, analyze the data volume characteristics of the target hot spot key set, and match the optimization execution strategy corresponding to the target hot spot key set from the preset optimization strategy library according to the operation operator type and the data volume characteristics; In this embodiment, after obtaining the target hotspot key set, the data processing task is parsed at the structural level. The operation operator type corresponds to the form of operation applied to the data during the task's execution, reflecting the data access pattern and computational behavior characteristics. The process of parsing the operation operator type is based on the task's logical execution description information. It parses the computational nodes expressed in the execution description and extracts the operational structure features related to key values, such as whether there are key-value-based aggregation or association operations. This parsing process does not depend on the specific data content but on the task's own operational expression structure, thus ensuring that the parsing result can be determined before the task is executed.

[0026] After determining the type of operation operator, the data magnitude characteristics of the target hotspot key set are analyzed. Data magnitude characteristics characterize the actual size of the hotspot key set within the target dataset and originate from the key value distribution feature set obtained in the preceding identification stage. The analysis of data magnitude characteristics is based on the distribution characteristics of the hotspot keys themselves, quantifying the data scale of the hotspot keys in the overall dataset, thereby providing refined input for subsequent strategy matching.

[0027] Given that the types of operators and the data scale characteristics are clearly defined, the preset optimization strategy library is accessed. This library maintains the mapping relationships between processing strategies under different computational scenarios and data scales. Each strategy mapping rule uses computational characteristics and data scale characteristics as matching conditions. By using the operator type and data scale characteristics as joint matching inputs, and performing matching calculations with the strategy mapping rules, optimized execution strategies corresponding to the target hotspot key set are selected. The matching process focuses on feature conformity rather than fixed rule triggers, thus ensuring flexibility in strategy selection.

[0028] In one implementation, the operation operator type is determined to be an aggregation operation type by parsing the grouping and aggregation nodes expressed in the task logic execution structure, and the distribution characteristics corresponding to the target hotspot key set are used as data volume features for strategy matching. In another implementation, the operation operator type is determined to be an association operation type by parsing the association expression nodes in the execution structure, and a composite data volume feature is formed by combining the hotspot key distribution characteristics and the data storage scale of the associated objects, which is used to match an optimized execution strategy suitable for large-scale association scenarios.

[0029] This embodiment uses both the type of operation operator and the data volume characteristics of the hotspot key set as the basis for strategy matching, so that the selection of optimization execution strategy can simultaneously reflect the characteristics of the operation structure and the actual data scale status, reducing the risk of mismatch between strategy selection and running scenario.

[0030] S40, Logically split the target dataset, apply the optimized execution strategy to process the data corresponding to the target hot key set, generate hot data processing results, and apply the default strategy to process the data corresponding to key values ​​in the target dataset other than the target hot key set, generate non-hot data processing results; In this embodiment, after completing the optimization execution strategy matching, logical data splitting is performed on the target dataset. Logical splitting is used to differentiate data along computational paths without changing the original physical storage structure of the data. Logical splitting uses the target hotspot key set as the criterion, matching and judging the key values ​​associated with each data record in the target dataset, thereby forming different data processing flows at the computational level. This splitting process establishes mapping rules based on key-value matching relationships, rather than based on data rewriting or data splitting operations, thus enabling differentiated processing while maintaining data integrity.

[0031] For data matching the target hotspot key set, an optimized execution strategy is loaded and applied for processing. This strategy adapts to the characteristics of hotspot data in terms of data size, computational density, or resource consumption. By adjusting the computation method or execution path, hotspot data can be scheduled and computed independently during processing, thus avoiding resource contention caused by mixing it with non-hotspot data. The optimized execution strategy only applies to data records corresponding to the target hotspot key set. After processing, a hotspot data processing result is generated, maintaining an output format consistent with the original data structure.

[0032] For data in the target dataset corresponding to key-value pairs that do not belong to the target hot key set, a default strategy is loaded and applied for processing. The default strategy maintains the original data processing behavior, ensuring that non-hot data completes the processing flow according to the established computational logic, thereby guaranteeing the integrity and consistency of the overall processing results. After processing with the default strategy, a non-hot data processing result is generated. This result is structurally compatible with the hot data processing result, providing conditions for subsequent integration.

[0033] In one implementation, a key-value mapping relationship is built within the computing node, mapping the target hot key set to a fast decision structure in memory. Logical routing is achieved through key-value matching, and optimized execution strategies and default strategies are scheduled and processed separately within the same execution environment. In another implementation, independent computing channels are built for hot and non-hot data at the task execution description level, allowing different strategies to be executed in parallel on different computing channels, thereby reducing mutual interference and improving processing efficiency.

[0034] This embodiment performs logical splitting of the target dataset at the computational level and applies different processing strategies to isolate hot data from non-hot data during execution, reducing the interference of hot data on the overall processing process and improving the stability of the data processing process and the rationality of resource utilization.

[0035] S50, the hot data processing results are merged with the non-hot data processing results to complete the data processing task; In this embodiment, after completing different strategy processing, the results of hot data processing and non-hot data processing need to be merged at the computation result level to form a unified data set that directly corresponds to the output requirements of the data processing task. This merging behavior occurs after the computation logic is completed, does not participate in the preceding computation scheduling, and does not affect the execution path in each processing process. The merging process is based on the premise of result structure consistency. By verifying the data field composition of the two processing results, it ensures that the field names, field types, and field order are consistent, thereby avoiding the introduction of structural conflicts during the merging process.

[0036] Provided structural consistency is satisfied, a result-level integration operation is performed on the processing results of hot data and non-hot data. This integration operation only focuses on the data concatenation relationship between the result sets, without recalculating or rearranging the result content, so that the two results logically form a complete data output set. The integrated data set remains consistent with the original output definition of the data processing task, thus it can be directly used in subsequent data storage or consumption stages.

[0037] After the merge is complete, the termination condition of the data processing task is triggered, and the integrated result is written or committed, while the execution status of the task is updated synchronously. This process is used to confirm that the current data processing task has completed all calculations and result generation, and at the same time releases the computing resources occupied during task execution to avoid long-term resource occupation.

[0038] This embodiment unifies the data generated by different processing paths in the result stage, so that while maintaining differentiated execution, the data processing tasks can output processing results with complete structure and consistent semantics, avoiding the additional integration costs caused by multiple result outputs.

[0039] S60, collect the resource consumption indicators and data transmission indicators during the execution of the data processing task, compare the resource consumption indicators and the data transmission indicators with historical benchmarks, and update the priority parameters of the optimization execution strategies in the preset optimization strategy library according to the comparison results.

[0040] In this embodiment, the running status is quantitatively perceived during the execution of data processing tasks to reflect the performance of different execution strategies under real load. Resource consumption indicators are used to characterize the load intensity at the computing level. Their sources may include the cumulative execution time of processing units, the time occupied by parallel tasks, or the intensity of memory usage, reflecting the consumption level of computing resources during task execution. Data transmission indicators are used to characterize the flow of data between computing nodes or between storage and computing. Their sources may include the amount of intermediate result exchange, the scale of data transmitted across nodes, reflecting the transmission overhead caused by data distribution and scheduling.

[0041] After collecting resource consumption and data transmission metrics, the metric values ​​corresponding to the current task are compared with historical benchmarks formed during past operations. Historical benchmarks are derived from statistical results of similar data processing tasks in previous executions, reflecting the normal performance level under similar data scales and execution environments. By comparing the current metrics with historical benchmarks, a comparison result reflecting the degree of performance change is obtained. This comparison result can show whether resource consumption has decreased, and whether data transmission load has improved or worsened.

[0042] After obtaining the comparison results, these results are mapped to the priority parameters of the optimized execution strategy, and the corresponding strategies in the preset optimization strategy library are adjusted. The priority parameters reflect the relative order or weight of the strategies selected in subsequent tasks. When the comparison results show that the current strategy performs better in terms of resource consumption or data transmission, the corresponding priority parameter is increased; when the comparison results show that the performance does not meet expectations, the corresponding priority parameter is decreased, thereby enabling the strategy library to form a dynamically evolving ranking relationship during continuous operation.

[0043] This embodiment continuously collects resource consumption and data transmission metrics and compares them with historical benchmarks, enabling the priority of optimized execution strategies to be dynamically adjusted based on actual operational results. This avoids long-term reliance on static experience for strategy selection and improves the rationality of strategy matching in subsequent tasks.

[0044] In one embodiment, step S10 above includes: S101, Establish a communication connection with the metadata management interface of the storage system; S102, query the underlying file block list and block-level statistical metadata of the target dataset through the communication connection; S103, parse the block-level statistical metadata and extract the line count statistics and storage size statistics of each underlying file block in the underlying file block list; S104, based on the partition definition of the target dataset, map the row count statistics and the storage size statistics to the key value dimension; S105, aggregate and calculate the row count statistics and storage size statistics under the key value dimension to generate key value distribution features; S106, construct metadata distribution information based on the key value distribution characteristics.

[0045] In this embodiment, establishing a communication connection with the storage system's metadata management interface is used to obtain a usable metadata entry point without reading the detailed data of the target dataset. The communication connection can originate from a remote procedure call channel exposed by the storage system, an interface access channel based on session tokens, or a request-response channel based on message queues. Establishing the communication connection includes interface address resolution, access credential loading, session persistence parameter settings, and timeout retry parameter settings. This ensures that subsequent queries to the metadata management interface can be completed within the same connection context, thereby avoiding latency fluctuations caused by multiple handshakes and providing a stable transmission path for the integrity of subsequent query results.

[0046] Querying the underlying file block list and block-level statistical metadata of the target dataset via a communication connection is used to map the target dataset to directly measurable data carrier units. The underlying file block list corresponds to the physical fragments of the target dataset on the storage medium, and the block-level statistical metadata corresponds to the set of statistical information generated during the writing or compression phase of each underlying file block. Its source can be the file end statistics area, an independent metadata table, or an index structure maintained by the storage system. The query action includes determining the unique identifier of the target dataset, limiting the set of returned fields, and limiting the returned range conditions. The returned results must at least cover the resolvable carriers of the underlying file block identifier, block boundary information, and block-level statistical metadata, providing input for subsequent parsing actions.

[0047] Parsing block-level statistical metadata and extracting the row count and storage size statistics of each underlying file block in the underlying file block list is used to convert metadata carriers that cannot be directly used in calculations into aggregable numerical items. Parsing block-level statistical metadata can perform field decoding, field validation, and missing field completion based on the metadata encoding format. The row count statistics come from counter snapshots during the write phase or row group statistics in columnar storage, while the storage size statistics come from the logical size of the block, its compressed size, or the size occupied by file system records. The extraction process requires establishing a one-to-one correspondence between each underlying file block and its row count and storage size statistics, and processing outliers into calculable alternative values ​​according to preset rules. For example, missing row count statistics are estimated based on the number of records within the block, and missing storage size statistics are estimated based on block boundaries and encoding overhead, thereby maintaining the continuity of subsequent mapping and aggregation links.

[0048] Based on the partition definition of the target dataset, row count statistics and storage size statistics are mapped to the key-value dimension to migrate the statistics of physical shards to the statistics of business key values. The partition definition of the target dataset comes from the partition rules maintained by the metadata management interface, and may include the set of partition fields, the correspondence between partition fields and directory levels, and the encoding method of partition field values. The mapping process involves parsing partition field values ​​from the underlying file block identifier or path information, combining the parsed partition field values ​​to form the key value of the key-value dimension, and associating this key value with the corresponding row count statistics and storage size statistics. The meaning of the key-value dimension can cover single-field key values ​​or multi-field combined key values, thereby adapting to different granularity of tilt recognition needs such as accounting period, account code, and account identifier.

[0049] The algorithm aggregates and calculates row count statistics and storage size statistics at the key-value dimension to generate key-value distribution features. This is used to converge statistics scattered across multiple underlying file blocks into a total quantity at the key-value level. The aggregation calculation includes grouping by key-value dimension, performing summation or weighted summation on row count statistics, and performing summation or weighted summation on storage size statistics. When necessary, it truncates or corrects the contributions of outlier blocks to reduce the perturbation of statistics by extreme values ​​in a single block. The key-value distribution features are derived from the aggregated numerical pairs and can simultaneously express the data row size and storage usage size corresponding to a certain key-value. It can also be extended to generate proportional features, such as the proportion of key-value storage size in the total storage size of the target dataset or the proportion of key-value row count in the total row count of the target dataset. However, the extended generation is still completed based on row count statistics and storage size statistics as input.

[0050] Metadata distribution information is constructed based on key-value distribution characteristics to form structured results that can be directly consumed by subsequent identification processes. The construction of metadata distribution information includes structured encapsulation of keys and key-value distribution characteristics, solidification of field naming and field types, application of missing key-value imputation rules, and application of result deduplication rules, ensuring a consistent data structure across different batches of execution. The carrier of metadata distribution information can be an in-memory table, a key-value index structure, or an external persistent file; the choice depends on the access method and capacity requirements of subsequent processing links. Furthermore, the construction process maintains zero-read access to the detailed data of the target dataset, thereby achieving non-traversal sampling.

[0051] This embodiment establishes a communication connection with the metadata management interface and queries the underlying file block list and block-level statistical metadata, switching the acquisition of statistical information for the target dataset from detailed reading to metadata reading, thus reducing the consumption of storage and computing resources. By parsing the block-level statistical metadata, it extracts row count statistics and storage size statistics, maps them to the key-value dimension according to the partition definition, and then performs aggregation calculations on the key-value dimension to generate key-value distribution features. This allows the metadata distribution information to characterize the data scale distribution at the key-value granularity, providing a stable, structured, and comparable distribution basis for subsequent processing, while maintaining the non-traversal sampling characteristics of the target dataset.

[0052] In one embodiment, step S20 above includes: S201, Extract the distribution characteristics of each key value from the metadata distribution information; S202, Perform statistical operations on the distribution characteristics of each key value to calculate the arithmetic mean and standard deviation; S203, Calculate the adaptive judgment threshold based on the arithmetic mean and the standard deviation; S204, calculate the sum of the distribution characteristics of each key value to obtain the total data volume of the target dataset; S205, calculate the proportion of the distribution characteristics of each key value in the total data volume of the target dataset; S206, Calculate the standard deviation multiple of the distribution characteristics of each key value based on the arithmetic mean and the standard deviation; S207, compare the standard deviation multiple with the adaptive judgment threshold, and compare the proportion value with the preset proportion threshold; S208, filter out key values ​​whose standard deviation multiple is greater than the adaptive judgment threshold and whose proportion value is greater than the preset proportion threshold; S209, summarize the filtered key values ​​and generate the target hotspot key set.

[0053] In this embodiment, the distribution features of each key value are extracted from the metadata distribution information to limit the input of subsequent statistical operations to a set of key-value granularity metrics. The metadata distribution information originates from the structured results constructed in the preceding steps, and it contains at least key-value identifiers and distribution feature fields. The extraction process includes locating the distribution feature fields, establishing one-to-one correspondences between key-value identifiers and feature entries, and performing consistency processing on missing or abnormal feature entries to ensure that the data entering the statistical operations has a unified numerical type and comparability. The distribution features of each key value can come from row count statistics, storage size statistics, or composite metrics derived from both. During extraction, the meaning of the distribution feature fields is kept unchanged to avoid mistakenly replacing key-value granularity metrics with global metrics.

[0054] Statistical operations are performed on the distribution characteristics of each key value to calculate the arithmetic mean and standard deviation, forming a global statistical characterization of the target dataset along the key value dimension. The arithmetic mean represents the centrality of the distribution characteristics of each key value in the overall dataset, while the standard deviation represents the dispersion of the distribution characteristics of each key value relative to the centrality. The implementation of statistical operations includes summing and counting the distribution characteristic sequence to obtain the arithmetic mean, and then performing square root extraction on the distribution characteristic sequence based on the arithmetic mean after summing the squared deviations and normalizing to obtain the standard deviation. The handling of outliers during statistical operations remains consistent with that in the extraction stage to avoid threshold drift caused by changes in statistical methods.

[0055] The adaptive threshold, calculated based on the arithmetic mean and standard deviation, binds hotspot detection criteria to the current distribution of the target dataset, causing the threshold to change with the distribution. The adaptive threshold is calculated using the arithmetic mean and standard deviation as inputs. It can be calculated using a linear combination, where the standard deviation is superimposed on the arithmetic mean with a preset coefficient, or a piecewise approach, where different coefficients are selected for different standard deviation intervals. The adaptive threshold is derived from the arithmetic mean and standard deviation output by statistical operations. The calculation process must fix the source of the coefficients and the coefficient update criteria to prevent the threshold from being unreproducible in different execution batches for the same target dataset.

[0056] The sum of the distribution characteristics of each key value is calculated to obtain the total data volume of the target dataset, which is used to construct a unified denominator for the percentage values. The total data volume of the target dataset is obtained by summing the distribution characteristics of each key value, implying that the distribution characteristics are additive in the key-value dimension. The calculation involves accumulating the distribution characteristic sequence, with the accumulation method consistent with the meaning of the distribution characteristic fields. If the distribution characteristic uses a row count statistic, the total data volume of the target dataset reflects the total number of rows; if the distribution characteristic uses a storage size statistic, the total data volume of the target dataset reflects the overall storage usage.

[0057] The proportion of each key value's distribution characteristics within the total data volume of the target dataset is calculated to form a key value contribution metric, introducing scale constraints beyond dispersion assessment. The proportion is calculated by dividing the distribution characteristics of each individual key value by the total data volume of the target dataset. The proportion is derived from both the distribution characteristics and the total data volume of the target dataset. The calculation process allows for precision control and boundary clipping of the proportion, ensuring that extremely small denominators or anomalous distribution characteristics do not render the proportion unusable. In subsequent comparisons, the proportion is combined with a preset proportion threshold to ensure that hotspot identification simultaneously satisfies both deviation and scale contribution dimensions.

[0058] The standard deviation multiple, calculated based on the arithmetic mean and standard deviation, normalizes the deviation of key values ​​into a dimensionless indicator. The standard deviation multiple takes the distribution characteristics, arithmetic mean, and standard deviation of each key value as input, and is obtained by calculating the difference between the distribution characteristics and the arithmetic mean and dividing by the standard deviation. The standard deviation multiple is derived from the arithmetic mean, standard deviation, and distribution characteristics of each key value. For cases where the standard deviation is zero or close to zero, a computable alternative rule needs to be provided, such as limiting the standard deviation multiple to zero or a preset upper bound, to ensure the subsequent comparison logic remains executable. The standard deviation multiple incorporates the differences in dispersion among different target datasets into the standard deviation term, allowing the same decision framework to adapt to datasets of different sizes and distribution patterns.

[0059] A two-condition decision chain is formed by comparing the standard deviation multiple with an adaptive decision threshold and the percentage value with a preset percentage threshold. The comparison process includes determining the relationship between the standard deviation multiple and the adaptive decision threshold, and between the percentage value and the preset percentage threshold, retaining the Boolean result or equivalent marker result for each key value in both judgments. The preset percentage threshold can be sourced from system configuration items, task-level parameters, or threshold entries in the policy library, and its numerical scope is consistent with that of the percentage value, avoiding misjudgments caused by inconsistent denominator scope. This two-comparison structure ensures that hotspot determination does not rely on a single statistic, reducing the possibility of small-scale abnormal key values ​​entering the hotspot set based solely on deviation.

[0060] Keys with a standard deviation multiple greater than an adaptive threshold and a percentage value greater than a preset percentage threshold are selected to map the dual-condition judgment results to a candidate hotspot key value set. The filtering process involves simultaneously checking the standard deviation multiple and percentage value comparison results for each key value; only when both conditions are met is the key value written into the filtering result. The filtering process maintains the original form of the key value identifier, without recoding or hashing the key values, to avoid the problem of irreversible key values ​​in subsequent matching stages. The filtering result can include the standard deviation multiple and percentage value corresponding to the key value as tracking fields, but the core output of the candidate hotspot key values ​​remains the key value body set.

[0061] The selected key values ​​are aggregated and a target hotspot key set is generated to form a hotspot set output that can be directly referenced by subsequent processing steps. The aggregation action includes deduplication of the filtered results, sorting or binning, and quantity capping or range control, thus ensuring that the target hotspot key set has stable size boundaries across different batches. Quantity capping can be determined based on the capacity threshold of the target hotspot key set or based on the top few key values ​​after sorting by percentage value, while range control can be further pruned based on the minimum threshold of the percentage value. The target hotspot key set originates from the filtered key values; after the generation action is completed, the target hotspot key set is output, allowing it to serve as a matching basis when performing differential processing on the data corresponding to the target hotspot key set.

[0062] This embodiment extracts the distribution characteristics of each key value from the metadata distribution information and calculates the arithmetic mean and standard deviation to form a global statistical characterization of the key value dimension. Then, an adaptive judgment threshold is generated based on the arithmetic mean and standard deviation, allowing the hotspot judgment conditions to adjust synchronously with changes in the dispersion of the target dataset. By calculating the total data volume of the target dataset and generating a percentage value, the degree of key value deviation and the contribution of key value size are simultaneously included in the comparison judgment. A target hotspot key set is generated using a dual-condition screening process: a standard deviation multiple and an adaptive judgment threshold, and a percentage value and a preset percentage threshold. This ensures that the output target hotspot key set simultaneously satisfies the constraints of significant deviation and significant proportion, reducing the probability of small-scale abnormal key values ​​entering the set and maintaining a controllable size boundary for the target hotspot key set. This provides a stable and reproducible input for subsequent processing steps based on the target hotspot key set.

[0063] In one embodiment, step S30 above includes: S301, Scan the logical execution plan of the data processing task and identify the operation operator types involved in the target hotspot key set, wherein the operation operator types include at least grouping aggregation type or table association type; S302, when the operation operator type is the grouping and aggregation type, the distribution characteristics of the target hot spot key set in the target dataset are used as the data volume characteristics; S303, when the operation operator type is the table association type, identify the dimension data table that establishes an association relationship with the target hotspot key set, and take the distribution characteristics of the target hotspot key set and the data storage capacity of the dimension data table as the data volume characteristics. S304, Access the preset optimization strategy library and read the pre-configured strategy mapping rules in the preset optimization strategy library; S305, perform matching calculations on the operation operator type, the data volume characteristics, and the strategy mapping rules, and select an optimized execution strategy from the preset optimization strategy library based on the matching results.

[0064] In this embodiment, the source of the logical execution plan can be an operator tree generated by the execution engine, a relational algebra expression, a physical plan text, or a binary description object. The scanning action includes reading the logical execution plan, locating the operator node containing the target hotspot key set, and extracting the type identifier and input-output relationship of the operator node. When identifying the operation operator types involved in the target hotspot key set, it is necessary to establish an association path between the target hotspot key set and the operator node. The construction of the association path can be based on the field lineage of key-value fields in the operator input, based on key-value field references in the filtering conditions, or based on key-value field references in the group key list or association key list, thereby limiting the operation operator types to a set of operator types directly related to the target hotspot key set. The operation operator types include at least group aggregation types or table association types. The constraints included at least allow for finer-grained operator type extensions in the logical execution plan. For example, window aggregation is merged into group aggregation types, and broadcast association is merged into table association types. The merging rules are constrained by the operator type mapping table to maintain the stability of the classification caliber.

[0065] When the operation operator type is grouping and aggregation, the distribution characteristics of the target hotspot key set in the target dataset are used as data volume features to characterize the scale of hotspots and bind it to the actual workload at the key-value granularity. The distribution features originate from previously generated metadata distribution information or equivalent statistical results. Distribution features can be row count statistics, storage size statistics, or a composite measure derived from both. When using distribution features as data volume features, they need to satisfy comparability and mappability. Comparability is reflected in the ability to determine the size relationship between different target hotspot key sets, and mappability is reflected in their ability to be referenced by policy mapping rules and participate in matching calculations. To make matching under grouping and aggregation more stable, the data volume features can be further derived into auxiliary quantities such as quantile intervals, the ratio of maximum to mean, and head concentration, while maintaining the unified entry point of the data volume features externally to avoid field meaning drift in policy mapping rules across different tasks.

[0066] When the operation operator type is a table join type, the dimension data table that establishes a relationship with the target hotspot key set is used to characterize the scale and distribution boundaries of the data on the join side, thus enabling the data volume characteristics to simultaneously cover both the hotspot side and the dimension table side. The identification of the dimension data table originates from table references in the logical execution plan, join key mappings, projection field source table annotations, or table lineage information output by intermediate operators. The identification actions include extracting the input table set from the table join type operator node, determining whether the input table set contains a set of dimension attribute fields, and distinguishing between the fact table and the dimension data table by combining field cardinality and update frequency characteristics. When using the distribution characteristics of the target hotspot key set and the data storage capacity of the dimension data table together as data volume characteristics, it is necessary to place the distribution characteristics and data storage capacity in the same feature structure while preserving their respective meaning boundaries to avoid mistakenly replacing data storage capacity with distribution characteristics or vice versa. The data storage capacity can be sourced from metadata statistics of dimension data tables, aggregation results of storage layer directories, or summary values ​​of table-level block statistics. The combined methods can be tuples, vectors, or bucket label sets. The combined data volume characteristics provide multi-dimensional decision-making basis for strategy mapping rules, enabling the strategy selection of table association types to simultaneously reflect the hot spot side tilt intensity and the dimension table side volume difference.

[0067] Accessing the preset optimization strategy library and reading the pre-configured strategy mapping rules within it is used to switch strategy selection from hard-coded decision-making to configuration-driven decision-making. The preset optimization strategy library can be implemented using key-value storage, relational tables, object storage files, or memory-mapped structures. The access action includes locating the set of strategy entries, loading the version of the strategy mapping rules, and verifying the completeness and scope of the strategy mapping rules. The structure of the strategy mapping rules can include a mapping table from operation operator types to candidate optimization execution strategies, segmentation conditions from data volume features to the strategy candidate set, conflict resolution priorities, disabling conditions, and fallback conditions. The read action must ensure that the field definitions of the strategy mapping rules are consistent with those of the operation operator types and data volume features. For example, the distribution feature intervals corresponding to grouping and aggregation types and the joint intervals of distribution features and data storage capacity corresponding to table association types should be defined separately at the rule level to avoid reusing the same threshold across scenarios and causing incorrect strategy selection.

[0068] Matching operation operator types, data volume characteristics, and policy mapping rules is used to determine the optimal execution strategy corresponding to the target hotspot key set in the candidate policy space. The input set for the matching calculation consists of operation operator types, data volume characteristics, and policy mapping rules. The matching calculation process may include rule branch selection for operation operator types, interval hit judgment for data volume characteristics, logical combination of joint conditions, and conflict resolution. Conflict resolution can determine the final hit rule based on priority fields within the policy mapping rules, specificity scores based on rule matching, and the principle of minimum coverage. When selecting an optimal execution strategy from the preset optimization policy library based on the matching results, the selection action includes selecting an optimal execution strategy from the set of policy entries associated with the hit rule, performing secondary verification on the applicability conditions of the optimal execution strategy, and outputting the optimal execution strategy identifier and its associated parameter set, so that subsequent processing stages can directly load the optimal execution strategy and apply it to the data corresponding to the target hotspot key set. The selected optimal execution strategies maintain a one-to-one correspondence with the policy entries in the preset optimization policy library, facilitating the adjustment of priority parameters for the optimal execution strategies in subsequent feedback stages.

[0069] This embodiment scans the logical execution plan and identifies the operation operator type, ensuring that the strategy selection for data processing tasks is based on reproducible structured criteria, reducing classification instability caused by runtime fluctuations. By using the distribution characteristics of the target hotspot key set within the target dataset as a data magnitude feature under the grouping and aggregation type, the hotspot scale characterization maintains consistency at the key-value granularity, enabling the strategy mapping rules to perform matching calculations around the actual load. By identifying dimensional data tables under the table association type and using both distribution characteristics and data storage capacity as data magnitude features, the matching calculation incorporates the scale constraints of both the hotspot side and the dimension table side, reducing strategy bias caused by insufficient information on one side. By accessing a preset optimization strategy library and reading strategy mapping rules, and then performing matching calculations and selecting optimized execution strategies based on the operation operator type, data magnitude features, and strategy mapping rules, the process of determining optimized execution strategies possesses configurability controllability and decision consistency, providing stable strategy input for subsequent execution stages.

[0070] In one embodiment, step S40 above includes: S401, construct the target hotspot key set into a key-value index table, and distribute the key-value index table to each computing node that processes the target dataset; S402, On the computing node, the key-value index table is used to perform key-value matching on the target dataset to perform logical splitting, and the data records that are successfully matched are divided into hot data shards, and the data records that are not matched are divided into non-hot data shards. S403, construct an independent computing channel for the hot data shards, load and execute the optimized execution strategy in the independent computing channel, and output the hot data processing results; S404, construct a regular computing channel for the non-hotspot data shards, load and execute the default strategy in the regular computing channel, and output the processing results of the non-hotspot data.

[0071] In this embodiment, logical splitting means dividing data in the same target dataset into two non-confusing data streams based on whether the key value belongs to the target hot key set. This ensures that the optimization execution strategy only applies to the data corresponding to the target hot key set, while the default strategy only applies to the data corresponding to key values ​​in the target dataset other than those in the target hot key set. Implementing logical splitting requires transforming the target hot key set into a lookup structure that can be directly used by the computing nodes for judgment. Therefore, the target hot key set is constructed as a key-value index table. The construction of the key-value index table includes at least three actions: key-value deduplication, key-value encoding normalization, and key-value index structure generation. Key-value deduplication eliminates duplicate key-value entries within the target hot key set, preventing multiple hit paths in the key-value index table. Key-value encoding normalization standardizes the expression of key values ​​in terms of serialization, case sensitivity, character set, or numerical precision, ensuring that subsequent key-value matching at the computing node does not result in misjudgments due to representation differences. Key-value index structure generation is used to form a fast-hitting index structure within the key-value index table. The index structure can be a hash index, a sorted array with binary search positioning, a prefix tree, or a bitmap mapping. The choice of index structure is related to the size of the target hot key set, the key value type, and the memory budget of the computing node.

[0072] The key-value index table is distributed to each compute node processing the target dataset, enabling each compute node to perform key-value matching locally and avoiding additional cross-node transmissions to determine hot and non-hot topics. Distribution can be manifested as writing the key-value index table to a distributed cache and having the compute nodes retrieve it, or as the scheduler sending the key-value index table to the compute nodes along with the task startup parameters. The distribution process must include version identification and verification information so that compute nodes can confirm the integrity of the key-value index table and prevent truncation or mismatches during distribution. After distribution, each compute node holds a semantic copy of the same key-value index table, ensuring consistent judgment criteria across different compute nodes for logical routing.

[0073] On the compute nodes, key-value matching is performed on the target dataset using a key-value index table to execute logical traffic splitting. The key-value matching targets the key-value fields corresponding to each data record in the target dataset. The extraction of key-value fields must be consistent with the key-value criteria used when forming the target hot key set. Key-value extraction can come from the primary key field, grouping key field, or association key field of the data record. The key-value matching process includes three stages: key-value extraction, key-value normalization, and index hit determination. Key-value normalization maintains the same rule set as the key-value encoding normalization during the key-value index table construction stage to reduce missed detections caused by different values. The index hit determination outputs two results: successful match or failed match. Successfully matched data records are divided into hot data shards, and failed match data records are divided into non-hot data shards. The division of hot and non-hot data shards is reflected not only in logical labels but also in physical partitions. For example, independent partition identifiers or independent partition paths can be set for hot data shards, allowing subsequent compute channels to directly subscribe to the corresponding partitions and avoid cross-reading.

[0074] Constructing independent computing channels for hot data shards provides independent execution boundaries for optimized execution strategies. The construction of these independent computing channels involves at least three actions: input binding, operator chain assembly, and resource configuration. Input binding ensures that hot data shards are the sole input source for the independent computing channel, preventing non-hot data shards from entering the optimized execution strategy path. Operator chain assembly connects the processing operators corresponding to the optimized execution strategy into an executable processing pipeline based on dependencies. This can be manifested as generating independent operator subgraphs within the execution engine, or as generating independent job branches for hot data shards and binding them with dedicated execution parameters. Resource configuration allocates thread quotas, memory quotas, parallelism parameters, or caching strategies to the independent computing channel, ensuring that the optimized execution strategy maintains a controllable operational boundary under hot loads. When the independent computing channel loads and executes the optimized execution strategy, the loading action includes reading the strategy identifier, parameter set, and applicable conditions of the optimized execution strategy; mapping the strategy identifier to specific processing operator combinations; injecting the parameter set into the operator runtime configuration; and triggering a strategy degradation to the default strategy or triggering a fallback path when the applicable conditions are not met. After execution, the hot data processing results are output. The hot data processing results need to retain the same field scope or output scope constraints as the target dataset so that the subsequent merging process can perform structural consistency verification and merging operations.

[0075] A regular computing channel is constructed for non-hotspot data shards to carry the processing chain of the default strategy. This regular computing channel exists in parallel with the independent computing channel, but their inputs are mutually exclusive. The construction of the regular computing channel also includes input binding, operator chain assembly, and resource configuration. Input binding restricts non-hotspot data shards to the sole input source. Operator chain assembly binds the set of processing operators corresponding to the default strategy. Resource configuration prioritizes throughput stability and partition balancing. When the regular computing channel loads and executes the default strategy, the loading action includes loading the default strategy processing parameters and initializing the execution environment. The execution action includes partition reading of the non-hotspot data shards, operator pipeline execution, and result writing, ultimately outputting the non-hotspot data processing results. The non-hotspot data processing results need to be consistent with the hotspot data processing results in terms of field sets, field types, field order, or encoding rules to facilitate subsequent merging of the hotspot and non-hotspot data processing results to complete the data processing task.

[0076] This embodiment constructs the target hot key set into a key-value index table and distributes it to each computing node, enabling key-value matching to be completed locally on the computing node. Logical splitting does not rely on additional cross-node decision links, thus reducing the data transmission overhead introduced by the splitting action. By dividing the target dataset into hot data shards and non-hot data shards using key-value matching on the computing node side, the optimized execution strategy and the default strategy are applied to mutually exclusive data ranges, reducing the interference of hot load on non-hot processing links. By constructing independent computing channels for hot data shards and loading optimized execution strategies, while constructing regular computing channels for non-hot data shards and loading default strategies, the two types of data are processed in parallel within different processing boundaries. The generation paths of hot data processing results and non-hot data processing results are clear and controllable, providing a structurally consistent input foundation for subsequent merging stages.

[0077] In one embodiment, step S50 above includes: S501, compare the data field definitions of the hot data processing result and the non-hot data processing result to verify the consistency of the data structure between the hot data processing result and the non-hot data processing result; S502, after the data structure consistency verification is passed, a merging operation is performed on the hot data processing results and the non-hot data processing results to generate a unified result set; S503, parse the output configuration of the data processing task and write the unified result set into the specified target storage path; S504, after confirming that the unified result set has been successfully written, mark the data processing task as completed and release computing resources.

[0078] In this embodiment, a comparison of data field definitions is required before the merging operation. The meaning of data field definitions includes a set of field names, a set of field types, field order constraints, field nullability constraints, field encoding rule constraints, and field default value rule constraints. When comparing data field definitions, hot data processing results and non-hot data processing results each provide their own field definition descriptions. These descriptions can come from the result metadata maintained by the execution engine, or from the header metadata area of ​​the result data or an external metadata directory. Field name set comparison is used to confirm the coverage relationship of fields with the same name; field type set comparison is used to confirm the type compatibility relationship of fields with the same name; field order constraint comparison is used to confirm the consistency of field arrangement during serialization output; field nullability constraints and field default value rule constraints are used to confirm that missing values ​​will not trigger write failures or read ambiguities after merging; and field encoding rule constraints are used to confirm that string, numeric, and time fields can be directly concatenated under the same encoding caliber.

[0079] Data structure consistency verification is based on the comparison results of data field definitions to form a verification conclusion. Data structure consistency means that the results of hot data processing and non-hot data processing meet the prerequisite of direct merging under the target structure of a unified result set. Verification actions can include field alignment mapping generation, which establishes a one-to-one correspondence between fields in hot data processing results and fields in non-hot data processing results, used for outputting according to the same field sequence in subsequent merging operations. Verification actions can also include type compatibility judgment, which identifies cases where field types are not completely identical but can be safely converted, and provides the conversion direction and rules. Conversion rules can be reflected in numerical precision expansion, integer to floating-point improvement, unified timestamp precision, or unified character set. Verification actions can also include field missing judgment, which identifies cases where one side lacks a target field and generates completion rules when a field is missing. Completion rules can use default value rules or null value filling rules. These verification sub-actions provide executable judgment criteria for passing data structure consistency verification, rather than merely forming abstract judgments.

[0080] After the data structure consistency verification passes, a merge operation is performed on the hot data processing results and the non-hot data processing results. The merge operation combines the two types of results into a unified result set under a unified field sequence and a unified encoding standard. The merge operation can be manifested as rearranging the fields of the two types of results based on field alignment mapping, ensuring that the output column order of the hot data processing results and the non-hot data processing results is consistent. Alternatively, it can be manifested as performing type conversion on inconsistent fields based on type compatibility judgment, ensuring that the two types of results can be uniformly serialized under the same field type set. The merge operation can also include result partitioning and arrangement. Result partitioning and arrangement means assigning partition identifiers and partition boundaries to the unified result set, creating a controllable file number and file size structure when writing to the target storage path. Result partitioning and arrangement can merge or repartition based on the number of partitions in the hot data processing results and the non-hot data processing results, and use partition boundary control to avoid commit timeouts caused by excessively large single partitions during the write phase. After merging, the unified result set has a structure that can be directly consumed by the write logic. The metadata of the unified result set should include field definition descriptions and partitioning and arrangement descriptions to support the parameter assembly of subsequent write actions.

[0081] The output configuration of the parsing data processing task is used to extract write-related constraints and parameters from the data processing task. The output configuration includes the target storage path location information, write mode parameters, partition write parameters, file format parameters, compression parameters, and commit semantic parameters. The target storage path location information determines the write endpoint of the unified result set and can be represented as a directory path, table path, or object storage prefix. Write mode parameters describe the choice between overwrite, append write, or partition-based overwrite. Partition write parameters describe the partition column set and partition layout rules. File format parameters describe the columnar or row-based serialization method. Compression parameters describe the compression algorithm and compression level. Commit semantic parameters describe the switching rules between temporary path writes and final path commits. After parsing the output configuration, the unified result set is written to the specified target storage path. The write action can be broken down into write task generation, write buffer generation, data serialization output, temporary artifact placement, and commit artifact switching. Write task generation maps the partition arrangement description of the unified result set to several write units, each write unit corresponding to a set of partition ranges or file output ranges. Data serialization output is used to convert a unified result set into a persistent byte stream according to the data field definitions and file format parameters. Temporary artifact disposal is used to generate commitable file artifacts in a temporary area of ​​the target storage path. Commit artifact switching is used to atomically map the temporary artifacts to the final location of the target storage path and update the directory metadata or table metadata related to the target storage path, making the unified result set visible to external readers.

[0082] After confirming the successful writing of the unified result set, the data processing task is marked as complete and computing resources are released. Confirming a successful write means performing a consistency check on the final visible artifacts in the target storage path. This consistency check can include file list verification, partition list verification, file size and quantity verification, and write commit flag verification. Marking the completion status means writing a completion status identifier to the task state storage of the data processing task, and recording the target storage path identifier and write version identifier related to the unified result set, so that subsequent processes can use the completion status as a trigger condition. Releasing computing resources means reclaiming the resources occupied by the computing nodes executing the data processing task, including releasing the execution container, releasing memory quotas, releasing intermediate data caches, and releasing the temporary storage space related to hot and non-hot data processing results, returning the computing nodes to a schedulable state. The completion status and computing resource release logically follow the confirmation of a successful write, avoiding premature release before the write is committed, which could lead to incomplete write artifacts or broken commit semantics.

[0083] This embodiment compares and verifies the data structure consistency between the data field definitions of hot data processing results and non-hot data processing results. This ensures that the merging operation is based on the constraint of consistent field, type, and encoding, reducing the probability of write failures due to structural inconsistencies when writing the unified result set to the target storage path. By performing the merging operation and generating the unified result set after the data structure consistency verification is passed, the output side only faces a single data entity of the unified result set, reducing processing forks caused by managing multiple result sets in the final stage of the data processing task. By parsing the output configuration of the data processing task and writing the unified result set to the specified target storage path, the write behavior is constrained by the output configuration, forming a controllable write mode and commit semantics. The visibility and consistency of the unified result set have a clear parameter source. By marking the completion status and releasing computing resources after confirming the successful writing of the unified result set, a verifiable causal relationship is established between the completion status and the write result, reducing downstream read anomalies caused by premature completion status generation, and decoupling the computing resource reclamation action from the write commit action under the same verification condition.

[0084] In one embodiment, step S60 above includes: S601, by using monitoring probes deployed on computing nodes, the cumulative execution time of the processing unit of the data processing task is recorded as a resource consumption indicator, and the amount of data transmitted through the shuffle is recorded as a data transmission indicator. S602, retrieve the historical average processing unit execution time and historical average shuffle data transmission volume of similar tasks from the task history log library, and combine the historical average processing unit execution time and the historical average shuffle data transmission volume to construct a historical benchmark. S603, calculate the performance comparison ratio of the resource consumption index and the data transmission index relative to the historical benchmark, and generate the comparison result; S604, Based on the performance comparison ratio in the comparison results, adjust the priority parameter value of the optimization execution strategy in the preset optimization strategy library.

[0085] In this embodiment, resource consumption and data transmission metrics during the execution of a data processing task are collected, compared with historical benchmarks, and the priority parameters of the optimized execution strategies in the preset optimization strategy library are updated accordingly. This processing chain means transforming the execution cost of a data processing task into a measurable set of metrics, aligning this set with historical benchmarks for similar tasks to form a comparison result, and finally mapping the comparison result to adjustments in priority parameter values. This ensures that the preset optimization strategy library has an updatable sorting basis when accessed subsequently. A data processing task corresponds to a schedulable and executable computational job, which can originate from the task definition of the scheduling system, the job description of the execution engine, or the task instance identifier generated by the orchestration platform. The execution process corresponds to the continuous interval of the data processing task from resource request, operator execution, data exchange to result submission. The collection action needs to be bound to this continuous interval to avoid mixing noise metrics from the task initialization or cleanup phases into the comparison criteria.

[0086] Resource consumption metrics characterize the resource occupancy intensity on the computing side. Cumulative execution time per processing unit depicts the cumulative computation time of a processing unit within the task's lifecycle. The term "processing unit" encompasses computing cores, execution threads, or allocable computing power units within the execution engine. Cumulative execution time covers the total time accumulated by time slices, events, or samples. The sources of cumulative execution time per processing unit can be runtime counters of computing nodes, performance measurement interfaces of execution containers, or task measurement streams exposed by the execution engine. Data transmission metrics characterize the transmission cost on the data exchange side. Shuffling data transmission volume depicts the scale of data transmission during partition redistribution, cross-node exchange, or intermediate result retrieval. The sources of shuffling data transmission volume can be exchange operator metrics of the execution engine, network stack metrics of computing nodes, or read / write statistics of intermediate storage files. Resource consumption metrics and data transmission metrics need to be collected using the same time window caliber so that both metrics can be aligned with the same historical benchmark. The time window can be bound to the task start time and task end time, or to the execution interval before and after writing a unified result set.

[0087] Monitoring probes are deployed on compute nodes to bring data collection down to the execution site. The meaning of a monitoring probe includes metric collection logic, metric aggregation logic, and metric reporting logic. Computation nodes include physical nodes, virtual nodes, or container nodes that carry out task execution. Deployment can be manifested as injection upon task startup or as a resident daemon process subscribing to and binding to the task measurement channel. Recording the cumulative execution time of processing units as a resource consumption indicator requires clearly defined start and end points for timing. The start point can be bound to the event identifier when the data processing task enters the running state, and the end point can be bound to the event identifier when the data processing task enters the completed state. Timing can be achieved by accumulating the active intervals of processing units, avoiding including idle waiting time in the cumulative execution time. Recording the amount of data transferred during shuffling as a data transfer indicator requires clearly defined statistical boundaries. Statistical boundaries can be bound to the input / output ports during the exchange phase or to the total read / write volume of intermediate data files. The statistical caliber can be accumulated by bytes or aggregated at the partition level before being aggregated into a task-level indicator. The two types of indicators output by the monitoring probes need to be accompanied by a data processing task identifier and a timestamp, providing a traceable key for subsequent retrieval of similar tasks and the construction of historical benchmarks.

[0088] The task history log repository stores historical execution data. The source of this repository can be an execution log storage system, a time-series index repository, or a task audit repository. Determining similar tasks requires a reusable matching key. This key can be generated by combining the task name, logical execution plan fingerprint, target dataset identifier, set of operation operator types, or output configuration summary, ensuring that similar tasks are comparable not only in name but also in execution structure. Retrieving historical average processing unit execution time and historical average shuffle data transmission volume requires selecting historical samples from the task history log repository that meet the criteria for similar tasks. The cumulative execution time of processing units and the shuffle data transmission volume within each sample are then aggregated to form an average value. This average value represents the arithmetic aggregation of multiple historical samples; alternatively, an average after removing outliers can be used to maintain stability in the presence of outliers. When the historical average processing unit execution time and the historical average shuffle data transmission volume are combined to construct a historical benchmark, the historical benchmark means using the resources and transmission costs of similar tasks in historical samples as a reference system. The combination means encapsulating the two types of average indicators into the same comparison object. The encapsulation form can be a tuple, a set of structured fields, or a vector representation that can be consumed by the comparison function, thereby supporting the subsequent joint comparison of resource consumption indicators and data transmission indicators.

[0089] The performance comparison ratio of computational resource consumption indicators and data transmission indicators relative to historical benchmarks is used to convert current operating costs into a relative scale. The performance comparison ratio represents the normalized expression of the proportional or difference relationship between the current indicator and the historical benchmark. "Relative" means using the historical benchmark as the denominator or reference term to form a horizontally comparable dimension. The performance comparison ratio can be calculated separately for the cumulative execution time of the processing unit and the amount of data transmitted during shuffling, and then combined through weighted summation to obtain a joint ratio. The weighting can be based on preset weights, resource constraints related to the data processing task, or weights derived from historical volatility. Generating comparison results means solidifying the performance comparison ratio along with its computational context. The comparison results can include the performance comparison ratio value, the resource consumption indicator values ​​participating in the comparison, the data transmission indicator values ​​participating in the comparison, the historical benchmark value, the similar task judgment key, and the collection window identifier, making the adjustment of priority parameter values ​​recalculated.

[0090] Adjusting the priority parameter values ​​of optimization strategies in the preset optimization strategy library based on the performance comparison ratio in the comparison results requires mapping the performance comparison ratio to an increment or decrement of the priority parameter value. The priority parameter is a parameterized field expressing the selection or recommended order of optimization strategies in the preset optimization strategy library; higher or lower values ​​correspond to order rules determined by the reading and filtering logic of the preset optimization strategy library. Adjustments can be made using segmented mapping, where the performance comparison ratio falls into different intervals and corresponds to different adjustment magnitudes, or continuous mapping, where the performance comparison ratio is converted into an adjustment magnitude using a function. To maintain the controllability of the sorting within the library, adjustments can include numerical boundary constraints, limiting the priority parameter values ​​to between upper and lower limits to prevent excessive jumps in priority parameter values ​​from a single comparison result. To maintain the traceability of the update process, adjustments can record the comparison results and priority parameter value adjustments in the task history log library or strategy update log, providing auditable clues for changes in the priority parameter values ​​of the same optimization strategy at different points in time. The update granularity of the preset optimization strategy library can be bound to the optimization execution strategy identifier, or it can be bound to the strategy mapping rule entries related to the target hotspot key set, so that the priority parameter values ​​of the same optimization execution strategy under different data volume characteristics or different operation operator types do not interfere with each other.

[0091] Example Explanation: In practical applications, data processing tasks correspond to the accounting entry summarization and analysis work in an enterprise financial system. This work involves grouping, aggregating, and performing correlation calculations on large-scale detailed data based on key values ​​such as accounting subjects, customer accounts, and accounting periods. Due to the highly uneven distribution of historical transactions, a few core subjects or accounts have significantly higher data volumes than other key values. If conventional calculations are performed directly, it is very easy to create obvious data skew during the grouping or correlation stages.

[0092] Before a task is scheduled for execution, the processing system first accesses the metadata management interface provided by the storage system. Through this interface, it directly reads the statistical information of the underlying file blocks, rather than scanning the data content itself. By establishing a communication connection with the metadata management interface, the system obtains the list of underlying file blocks corresponding to the target dataset, as well as the row count and storage size statistics maintained for each file block. Subsequently, based on the partitioning rules defined at the business level for the target dataset, these block-level statistics are mapped to specific key-value dimensions, such as accounting subject codes or account identifiers. Statistics under the same key-value are then aggregated and calculated to form a set of key-value distribution characteristics that reflects the actual data scale of each key-value pair, and metadata distribution information is constructed accordingly.

[0093] After obtaining the metadata distribution information, the system extracts the distribution features corresponding to each key value and performs statistical operations on these features to calculate the overall arithmetic mean and standard deviation, which characterize the overall distribution pattern of the target dataset in the key value dimension. Based on the mean and standard deviation, the system calculates an adaptive judgment threshold to describe the judgment boundary of significant deviations from the normal distribution range. Simultaneously, the system sums the distribution features of all key values ​​to obtain the total data volume of the target dataset and calculates the proportion of each key value in the total data volume. Combining the standard deviation multiple and the proportion, the system compares and judges each key value. Only when a key value exceeds the preset conditions in both deviation degree and overall proportion is it identified and summarized into the target hotspot key set, thereby avoiding misjudging short-term fluctuations or marginal anomalies as hotspots.

[0094] After identifying the target hotspot key set, the system further analyzes the logical execution plan corresponding to the data processing tasks to identify the types of operation operators involving the hotspot keys. For example, in a financial summary scenario, some tasks perform grouping aggregation using accounting subjects as grouping keys, while others require linking accounting entries with subject tables or customer tables. Based on the identified operation operator types and the distribution characteristics of the target hotspot key set in the target dataset, the system performs quantitative analysis on the hotspot data volume. When the operation operator type corresponds to grouping aggregation, the distribution characteristics of the hotspot key itself are directly used as the data volume characteristic; when the operation operator type corresponds to table association, the system further identifies the dimension data tables involved in the association and uses the distribution characteristics of the hotspot keys and the data storage capacity of the dimension data table as input for the data volume characteristic. Subsequently, the system accesses the preset optimization strategy library, reads the strategy mapping rules maintained therein, and performs matching calculations between the operation operator type, data volume characteristics, and strategy mapping rules to select the optimization execution strategy that best matches the current hotspot key set.

[0095] During the execution phase, the system logically distributes the target dataset based on the target hotspot key set. The system constructs a key-value index table from the target hotspot key set and distributes this index table to each participating computing node. On each computing node, the key-value index table is used to match and determine the input data records. Successfully matched data records are divided into hotspot data shards, while unmatched data records are divided into non-hotspot data shards. For hotspot data shards, the system constructs independent computing channels and loads the optimized execution strategies obtained from the matching process. For example, in grouping and aggregation scenarios, hotspot keys are segmented and aggregated; in association scenarios, the data slices corresponding to hotspot keys are processed in parallel. Simultaneously, for non-hotspot data shards, the system constructs regular computing channels and loads default strategies for execution, thereby achieving differentiated parallel computing within the same task.

[0096] After each computing channel completes its processing, the system merges the results of hot data processing with those of non-hot data processing. Before merging, the system first compares the data field definitions of the two types of results to verify data structure consistency, ensuring that the results can be processed uniformly at the semantic and structural levels. After the consistency verification is passed, the system performs the merging operation to generate a unified result set, parses the output configuration of the data processing task, and writes the unified result set to the specified target storage path. Once the write is confirmed to be successful, the system marks the current data processing task as complete and releases the corresponding computing resources.

[0097] During task execution, the system continuously collects resource consumption and data transmission metrics. For example, monitoring probes deployed on computing nodes record the cumulative execution time of processing units as a resource consumption metric, and simultaneously record the amount of data transmitted during shuffling as a data transmission metric. After the task is completed, the system retrieves the historical average execution time of processing units and the historical average amount of data transmitted during shuffling from the task history log library for similar tasks, and combines these historical metrics to construct a historical benchmark. By calculating the performance comparison ratio of the current task metrics relative to the historical benchmark, the system generates a comparison result and adjusts the priority parameters of relevant optimization strategies in the preset optimization strategy library accordingly. In this way, the system can dynamically strengthen effective strategies and weaken or replace ineffective strategies in subsequent tasks, forming a continuously evolving adaptive optimization closed loop.

[0098] This embodiment collects the cumulative execution time and shuffling data transmission volume of the processing unit using monitoring probes on the computing node, and constructs a historical benchmark by retrieving the historical average processing unit execution time and historical average shuffling data transmission volume from the task history log library. Then, it calculates the performance comparison ratio of resource consumption indicators and data transmission indicators relative to the historical benchmark to form a comparison result, and adjusts the priority parameter values ​​of the optimized execution strategies in the preset optimization strategy library accordingly. This ensures that the priority parameter values ​​are based on a measurable comparison of resource and transmission costs. The sorting criteria of the preset optimization strategy library change synchronously with the update of the comparison result, so that the selection of optimized execution strategies in the subsequent matching stage is closer to the actual running performance of similar tasks.

[0099] In one embodiment, a data skew adaptive processing apparatus is provided, which corresponds one-to-one with the data skew adaptive processing method described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the data skew adaptive processing device of the present invention. The modules include: metadata sampling and analysis module 10, hotspot key identification module 20, strategy matching and decision-making module 30, data splitting execution module 40, result merging and output module 50, and strategy feedback and update module 60. Detailed descriptions of each functional module are as follows: The metadata sampling and analysis module 10 is used to perform non-traversal sampling of the target dataset by accessing the metadata management interface of the storage system before starting the data processing task, and to obtain metadata distribution information containing key value distribution characteristics. Hotspot key identification module 20 is used to calculate the statistical distribution parameters of the target dataset based on the metadata distribution information, and compare the distribution characteristics of each key value in the metadata distribution information with the statistical distribution parameters to identify the target hotspot key set; The strategy matching decision module 30 is used to parse the operation operator type of the data processing task, analyze the data volume characteristics of the target hot spot key set, and match the optimization execution strategy corresponding to the target hot spot key set from the preset optimization strategy library according to the operation operator type and the data volume characteristics. The data splitting execution module 40 is used to logically split the target dataset, apply the optimized execution strategy to process the data corresponding to the target hot key set, generate hot data processing results, and apply the default strategy to process the data corresponding to the key values ​​in the target dataset other than the target hot key set, generate non-hot data processing results. The result merging and output module 50 is used to merge the hot data processing results with the non-hot data processing results to complete the data processing task. The strategy feedback update module 60 is used to collect resource consumption indicators and data transmission indicators during the execution of the data processing task, compare the resource consumption indicators and data transmission indicators with historical benchmarks, and update the priority parameters of the optimization execution strategy in the preset optimization strategy library according to the comparison results.

[0100] Specific limitations regarding the data skew adaptive processing device can be found in the foregoing limitations of the data skew adaptive processing method, and will not be repeated here. Each module in the aforementioned data skew adaptive processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0101] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides deterministic and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a data skew adaptive processing method on the server side.

[0102] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a data skew adaptive processing method.

[0103] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Before starting the data processing task, the target dataset is sampled non-traversally by accessing the metadata management interface of the storage system to obtain metadata distribution information containing key-value distribution characteristics. The statistical distribution parameters of the target dataset are calculated based on the metadata distribution information, and the distribution characteristics of each key value in the metadata distribution information are compared with the statistical distribution parameters to identify the target hot key set; The operation operator type of the data processing task is parsed, and the data volume characteristics of the target hot spot key set are analyzed. Based on the operation operator type and the data volume characteristics, the optimization execution strategy corresponding to the target hot spot key set is matched from the preset optimization strategy library. The target dataset is logically split, and the optimized execution strategy is applied to the data corresponding to the target hot key set to generate hot data processing results. The default strategy is applied to the data corresponding to the key values ​​in the target dataset other than the target hot key set to generate non-hot data processing results. The results of processing the hot data are merged with the results of processing the non-hot data to complete the data processing task; The resource consumption and data transmission metrics of the data processing task during execution are collected, and the resource consumption and data transmission metrics are compared with historical benchmarks. Based on the comparison results, the priority parameters of the optimization execution strategies in the preset optimization strategy library are updated.

[0104] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, and a computer program is stored thereon, which, when executed by a processor, performs the following steps: Before starting the data processing task, the target dataset is sampled non-traversally by accessing the metadata management interface of the storage system to obtain metadata distribution information containing key-value distribution characteristics. The statistical distribution parameters of the target dataset are calculated based on the metadata distribution information, and the distribution characteristics of each key value in the metadata distribution information are compared with the statistical distribution parameters to identify the target hot key set; The operation operator type of the data processing task is parsed, and the data volume characteristics of the target hot spot key set are analyzed. Based on the operation operator type and the data volume characteristics, the optimization execution strategy corresponding to the target hot spot key set is matched from the preset optimization strategy library. The target dataset is logically split, and the optimized execution strategy is applied to the data corresponding to the target hot key set to generate hot data processing results. The default strategy is applied to the data corresponding to the key values ​​in the target dataset other than the target hot key set to generate non-hot data processing results. The results of processing the hot data are merged with the results of processing the non-hot data to complete the data processing task; The resource consumption and data transmission metrics of the data processing task during execution are collected, and the resource consumption and data transmission metrics are compared with historical benchmarks. Based on the comparison results, the priority parameters of the optimization execution strategies in the preset optimization strategy library are updated.

[0105] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0106] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0107] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

[0108] The user personal information involved in this application embodiment is all authorized (knowing and consenting) by the relevant parties or fully authorized by all parties, and the executing entity can obtain it through various open, legal and compliant means. The collection, storage, use, processing, transmission, provision and disclosure of the information, data and signals involved all comply with the relevant laws and regulations of the relevant countries and regions, and do not violate public order and good morals.

Claims

1. A data skew adaptive processing method, characterized in that, Includes the following steps: Before starting the data processing task, the target dataset is sampled non-traversally by accessing the metadata management interface of the storage system to obtain metadata distribution information containing key-value distribution characteristics. The statistical distribution parameters of the target dataset are calculated based on the metadata distribution information, and the distribution characteristics of each key value in the metadata distribution information are compared with the statistical distribution parameters to identify the target hot key set; The operation operator type of the data processing task is parsed, and the data volume characteristics of the target hot spot key set are analyzed. Based on the operation operator type and the data volume characteristics, the optimization execution strategy corresponding to the target hot spot key set is matched from the preset optimization strategy library. The target dataset is logically split, and the optimized execution strategy is applied to the data corresponding to the target hot key set to generate hot data processing results. The default strategy is applied to the data corresponding to the key values ​​in the target dataset other than the target hot key set to generate non-hot data processing results. The results of processing the hot data are merged with the results of processing the non-hot data to complete the data processing task; The resource consumption and data transmission metrics of the data processing task during execution are collected, and the resource consumption and data transmission metrics are compared with historical benchmarks. Based on the comparison results, the priority parameters of the optimization execution strategies in the preset optimization strategy library are updated.

2. The data skew adaptive processing method as described in claim 1, characterized in that, Before initiating the data processing task, the target dataset is sampled non-traversally by accessing the storage system's metadata management interface to obtain metadata distribution information containing key-value distribution characteristics, including: Establish a communication connection with the storage system's metadata management interface; The underlying file block list and block-level statistical metadata of the target dataset can be queried through the communication connection. Parse the block-level statistical metadata and extract the line count and storage size statistics of each underlying file block in the underlying file block list; Based on the partition definition of the target dataset, the row count statistics and the storage size statistics are mapped to the key-value dimension; The row count statistics and storage size statistics under the key value dimension are aggregated and calculated to generate key value distribution features; Metadata distribution information is constructed based on the key-value distribution characteristics.

3. The data skew adaptive processing method as described in claim 1, characterized in that, Calculate the statistical distribution parameters of the target dataset based on the metadata distribution information, and compare the distribution characteristics of each key value in the metadata distribution information with the statistical distribution parameters to identify the target hotspot key set, including: Extract the distribution characteristics of each key value from the metadata distribution information; Statistical operations are performed on the distribution characteristics of each key value to calculate the arithmetic mean and standard deviation. Calculate the adaptive decision threshold based on the arithmetic mean and the standard deviation; Calculate the sum of the distribution characteristics of each key value to obtain the total data volume of the target dataset; Calculate the proportion of the distribution characteristics of each key value in the total data volume of the target dataset; Based on the arithmetic mean and the standard deviation, calculate the standard deviation multiple of the distribution characteristics of each key value; The standard deviation multiple is compared with the adaptive judgment threshold, and the proportion value is compared with the preset proportion threshold; Filter out key values ​​whose standard deviation multiple is greater than the adaptive judgment threshold and whose proportion value is greater than the preset proportion threshold; The selected key values ​​are aggregated to generate a target hotspot key set.

4. The data skew adaptive processing method as described in claim 1, characterized in that, The data processing task's operation operator type is parsed, and the data volume characteristics of the target hotspot key set are analyzed. Based on the operation operator type and the data volume characteristics, an optimization execution strategy corresponding to the target hotspot key set is matched from a preset optimization strategy library, including: Scan the logical execution plan of the data processing task to identify the types of operation operators involved in the target hotspot key set, wherein the types of operation operators include at least grouping aggregation type or table association type; When the operation operator type is the grouping and aggregation type, the distribution characteristics of the target hotspot key set in the target dataset are used as the data volume characteristics; When the operation operator type is the table association type, the dimension data table that establishes an association with the target hotspot key set is identified, and the distribution characteristics of the target hotspot key set and the data storage capacity of the dimension data table are used together as data volume characteristics. Access the preset optimization strategy library and read the pre-configured strategy mapping rules in the preset optimization strategy library; The operation operator type, the data volume characteristics, and the strategy mapping rules are matched and calculated, and the optimized execution strategy is selected from the preset optimization strategy library based on the matching results.

5. The data skew adaptive processing method as described in claim 1, characterized in that, The target dataset is logically split, and the optimized execution strategy is applied to the data corresponding to the target hot key set to generate hot data processing results. The default strategy is applied to the data corresponding to key values ​​in the target dataset other than the target hot key set to generate non-hot data processing results, including: The target hotspot key set is constructed into a key-value index table, and the key-value index table is distributed to each computing node that processes the target dataset; On the computing node, the key-value index table is used to perform key-value matching on the target dataset to execute logical splitting, dividing the successfully matched data records into hot data shards and the unmatched data records into non-hot data shards; Construct an independent computing channel for the hot data shards, load and execute the optimized execution strategy in the independent computing channel, and output the hot data processing results; A regular computing channel is constructed for the non-hotspot data shards. The default strategy is loaded and executed in the regular computing channel, and the processing results of the non-hotspot data are output.

6. The data skew adaptive processing method as described in claim 1, characterized in that, The data processing task is completed by merging the results of the hot data processing with the results of the non-hot data processing, including: Compare the data field definitions of the hot data processing results with those of the non-hot data processing results to verify the consistency of the data structure between the hot data processing results and the non-hot data processing results; Once the data structure consistency verification is passed, a merging operation is performed on the hot data processing results and the non-hot data processing results to generate a unified result set; Parse the output configuration of the data processing task and write the unified result set to the specified target storage path; After confirming that the unified result set has been successfully written, the data processing task is marked as completed and computing resources are released.

7. The data skew adaptive processing method as described in claim 1, characterized in that, Collect resource consumption and data transmission metrics during the execution of the data processing task, compare these metrics with historical benchmarks, and update the priority parameters of the optimization execution strategies in the preset optimization strategy library based on the comparison results, including: By using monitoring probes deployed on computing nodes, the cumulative execution time of the processing units of the data processing task is recorded as a resource consumption indicator, and the amount of data transmitted through the shuffle is recorded as a data transmission indicator. Retrieve the historical average processing unit execution time and historical average shuffle data transmission volume of similar tasks from the task history log library, and combine the historical average processing unit execution time and the historical average shuffle data transmission volume to construct a historical benchmark; Calculate the performance comparison ratio of the resource consumption index and the data transmission index relative to the historical benchmark, and generate comparison results; Based on the performance comparison ratio in the comparison results, the priority parameter values ​​of the optimization execution strategies in the preset optimization strategy library are adjusted.

8. A data skew adaptive processing device, characterized in that, The data tilt adaptive processing device includes: The metadata sampling and analysis module is used to perform non-traversal sampling of the target dataset by accessing the metadata management interface of the storage system before starting the data processing task, and to obtain metadata distribution information containing key-value distribution characteristics. The hotspot key identification module is used to calculate the statistical distribution parameters of the target dataset based on the metadata distribution information, and compare the distribution characteristics of each key value in the metadata distribution information with the statistical distribution parameters to identify the target hotspot key set; The strategy matching and decision module is used to parse the operation operator type of the data processing task, analyze the data volume characteristics of the target hot spot key set, and match the optimization execution strategy corresponding to the target hot spot key set from the preset optimization strategy library according to the operation operator type and the data volume characteristics. The data splitting execution module is used to logically split the target dataset, apply the optimized execution strategy to process the data corresponding to the target hot key set, generate hot data processing results, and apply the default strategy to process the data corresponding to key values ​​in the target dataset other than the target hot key set, generate non-hot data processing results. The result merging and output module is used to merge the hot data processing results with the non-hot data processing results to complete the data processing task. The strategy feedback update module is used to collect resource consumption indicators and data transmission indicators during the execution of the data processing task, compare the resource consumption indicators and data transmission indicators with historical benchmarks, and update the priority parameters of the optimized execution strategies in the preset optimization strategy library based on the comparison results.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a data skew adaptive processing program stored in the memory and executable on the processor, wherein the data skew adaptive processing program, when executed by the processor, implements the steps of the data skew adaptive processing method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a data skew adaptive processing program, which, when executed by a processor, implements the steps of the data skew adaptive processing method as described in any one of claims 1-7.

Images