Unlock instant, AI-driven research and patent intelligence for your innovation.

Computing resource determination method and device based on Spark operation

A technology for computing resources and determining methods, applied in computing, computer components, neural learning methods, etc., can solve problems such as long processing time, slow equipment operation, and long data processing time, and achieve the effect of reducing the dependence of artificial experience

Active Publication Date: 2020-01-14
NAT UNIV OF DEFENSE TECH
View PDF12 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] If fewer computing resources are allocated to the data, less data will be processed in parallel, and the processing time will be longer; if more computing resources are allocated to the data, other resource consumption of the device, such as network communication, disk I / O (Input / Output, input / output) is more, resulting in slower operation of the device, and longer data processing time

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Computing resource determination method and device based on Spark operation
  • Computing resource determination method and device based on Spark operation
  • Computing resource determination method and device based on Spark operation

Examples

Experimental program
Comparison scheme
Effect test

Embodiment approach

[0078] In the first aspect, assuming that there are n samples of data, the amount of training data in the previous embodiment is n, and the amount of training data in this embodiment can reach C n 2 , the amount of training data in this embodiment is large, and the prediction model obtained by training is more accurate; secondly, in the previous embodiment, the output of the prediction model is time-consuming, and the output results need to be compared later, while this implementation In the method, the output of the prediction model is the time-consuming difference, that is to say, the output result directly represents the comparison result, and compared with the previous embodiment, the comparison steps are reduced.

[0079] S102: Among the predicted time consumption, determine the computing resource corresponding to the shortest time consumption as the computing resource corresponding to the data to be processed.

[0080] Assume that in S101, {to-be-processed data A, 5 CPU...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention discloses a computing resource determination method and device based on Spark jobs, and the method comprises the steps: taking data sets with different data sizes and various computing resources distributed for the data sets as the input, taking the processing time as the supervision, carrying out the training of a neural network of a preset structure, and obtaininga prediction model; utilizing the prediction model to respectively predict the time consumed for processing the to-be-processed data by utilizing each computing resource; in the predicted consumed time, determining a computing resource corresponding to the shortest consumed time as a computing resource corresponding to the to-be-processed data. Visibly, according to the scheme, the computing resources required for processing the data are determined through the prediction model, and dependence on artificial experience is reduced.

Description

technical field [0001] The present invention relates to the technical field of parallel computing, in particular to a method and device for determining computing resources based on Spark jobs. Background technique [0002] In some scenarios, parallel processing of data is usually required. For example, if face recognition needs to be performed on 100 images, the 100 images can be allocated to multiple processing units, and these processing units perform face recognition on these images in parallel. A processing unit may be understood as a computing resource in a device. For example, a processing unit may be a CPU (Central Processing Unit, central processing unit), or a GPU (Graphics Processing Unit, graphics processing unit), or other processing chips. [0003] If fewer computing resources are allocated to the data, less data will be processed in parallel, and the processing time will be longer; if more computing resources are allocated to the data, other resource consumpti...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06K9/00G06T1/20G06N3/08
CPCG06T1/20G06N3/08G06V40/172
Inventor 郭得科胡智尧
Owner NAT UNIV OF DEFENSE TECH