物理计划生成方法、装置、设备及存储介质
By constructing dataset dependencies and generating full-order physical plans using full-order code optimization algorithms, replacing traditional caching methods and applying lazy execution mode, the problem of slow execution and long resource consumption time of batch processing jobs in cloud Spark clusters is solved, achieving efficient execution of SQL jobs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA MERCHANTS BANK
- Filing Date
- 2023-11-14
- Publication Date
- 2026-07-17
AI Technical Summary
Cloud-based Spark clusters experience slow batch processing jobs with long resource consumption times. This is especially true for systems with long chains, large and complex relationships, where resource consumption is high, running time is long, and trial-and-error costs are high, affecting the timely completion of important downstream systems.
By obtaining a set of SQL statements, the dataset dependencies are constructed based on preset subquery instructions, and a full-order physical plan is generated using a pre-built full-order code optimization algorithm to replace the Cache Table command. The caching method of select + temporary table/view name is adopted, the show function is canceled, the Catalog API is called to register cache information, the cache level and method are configured, and the lazy execution mode is applied to build the full-process execution plan.
It enables batch execution of SQL jobs, reducing time consumption and resource usage, solving the problem of slow batch processing jobs in cloud Spark clusters, and improving resource utilization efficiency.
Smart Images
Figure CN117591533B_ABST