Unlock instant, AI-driven research and patent intelligence for your innovation.

A Feature Determination Method and Device for Predicting Query Resource Occupancy

A determination method and technology of occupancy, applied in the field of data processing, can solve the problems affecting the execution of query tasks, inaccurate memory footprint, and few input feature values, so as to optimize resource allocation, improve utilization, and rationalize query tasks. Effect

Active Publication Date: 2022-06-21
BEIJING GRIDSUM TECH CO LTD
View PDF14 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] For an Impala query, a supervised learning algorithm is used to predict the memory space used by the query. When using a supervised learning algorithm for memory space prediction, the feature used is the total number of files to be processed, but only the files to be processed are The total number is used as the feature input of the supervised learning algorithm. Due to the small number of input feature values, the predicted memory footprint will be inaccurate, which will affect the execution of subsequent query tasks.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A Feature Determination Method and Device for Predicting Query Resource Occupancy
  • A Feature Determination Method and Device for Predicting Query Resource Occupancy
  • A Feature Determination Method and Device for Predicting Query Resource Occupancy

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0053] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be more thoroughly understood, and will fully convey the scope of the present disclosure to those skilled in the art.

[0054] The embodiment of the present invention provides a feature determination method for predicting the occupancy of query resources, where the occupancy of query resources may be memory occupied space.

[0055] refer to figure 1 , which can include:

[0056] S11. Obtain the data to be queried;

[0057] Wherein, the data to be queried is the data input when performing the query operation;

[0058] Specifically, the dat...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The present invention discloses a feature determination method and device for predicting query resource occupancy. Data to be queried is obtained, query plan data corresponding to the data to be queried is determined, and a tree structure corresponding to the query plan data is generated. Based on the tree structure, determine the feature dimension value corresponding to the preset feature dimension; wherein the preset feature dimension is used to determine the resources occupied by the query operation, and the number of the preset feature dimensions is multiple. Through the embodiments of the present invention, multiple feature dimension values ​​can be determined. Compared with only one feature dimension value, inputting multiple feature dimension values ​​into a supervised learning algorithm can predict more accurate query resources.

Description

technical field [0001] The present invention relates to the field of data processing, and more particularly, to a feature determination method and device for predicting the occupancy of query resources. Background technique [0002] Impala is a new query system that provides structured query language SQL semantics and can query petabytes of petabyte-level big data stored in Hydup Hadoop's distributed file system HDFS (Hadoop Distributed File System) and HBase. [0003] For an Impala query, the supervised learning algorithm is used to predict the memory space used in the query. When using the supervised learning algorithm to predict the memory space, the feature used is the total number of files to be processed, but only the files to be processed are The total number is used as the feature input of the supervised learning algorithm. Due to the small number of input feature values, the predicted memory footprint will be inaccurate and the execution of subsequent query tasks wi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/2453G06F9/50G06F11/34
CPCG06F9/5016G06F11/3442G06F2201/80
Inventor 张双燕
Owner BEIJING GRIDSUM TECH CO LTD