Commodity summarizing and counting method based on cost model

Abstract

The invention discloses a commodity summary statistics method based on a cost model. The method comprises the following steps: collecting table management information; analyzing the SQL statement to obtain an execution plan, calculating the cost of the execution plan according to the SQL statement, and rating the calculated cost; aiming at the cost level after the SQL execution plan, distributingthe SQL statement to different resources for execution; and storing the execution log for evaluation and analysis, dividing a cost cluster level, and carrying out continuous learning and adjustment byusing an algorithm according to a relationship among the cost value, the cost cluster level and a cost calculation result. According to the method, the SQL statement is subjected to cost calculationand analysis, the resource consumption conditions caused by the SQL statements are evaluated, summarized and counted, the SQL statements are issued to different resources for task scheduling executionaccording to the cost rating result, the resource isolation and protection effects are achieved, and the service of a user is also perfected.

Description

technical field [0001] The invention relates to the field of commodity data summary, in particular to a commodity summary statistics method based on a cost (cost) model. Background technique [0002] At present, the horizontal expansion of commodity data has led to a sharp increase in the amount of data, and it is necessary to collect data for various commodities. Generally, distributed systems use scheduling algorithms such as FIFO (first in first out) and FAIR (fair scheduling) for OLAP (Online Analytical Processing) queries to allocate resources. For example, Spark configures multiple resource pools. By default, the scheduling mode of Spark is FIFO ( First-in, first-out), whoever submits first executes first, and the subsequent tasks need to wait for the previous tasks to be executed, while the FAIR (fair scheduling) mode supports grouping tasks in the scheduling pool. Different scheduling pools have different weights, and tasks can be assigned according to the weight To...

AI Technical Summary

Problems solved by technology

In a multi-user environment, resource allocation and isolation can be performed for users, but for each user resource, scheduling algorithms such as FIFO (first in first out) and FAIR (fair scheduling) are still used, which cannot be based on the cost (cost) brought by the query. ) for resource isolation

[0003] In the query summary of general product data, the consumption of resources is particularly large, and the execution time will be very long

If several such queries are executed concurrently under the same user at the same time, it is easy to cause the cluster resources to be fully occupied, making it impossible for other queries to obtain enough resources (even if this query occupies a small amount of resources and executes quickly), in this way, commodity data The summary statistics performance is low, which affects user experience and decision-making efficiency. In severe cases, it may cause application OOM or cluster crash

Images