Data processing task load processing method and device
A technology of data processing and processing methods, applied in the field of data processing, can solve problems such as high labor cost, rigidity, and poor flexibility of processing methods
Pending Publication Date: 2019-11-29
THE FOURTH PARADIGM BEIJING TECH CO LTD
View PDF0 Cites 3 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0003] At present, in the process of processing big data tasks, it is necessary to manually select the appropriate processing method for the task. For different tasks at different stages, when there are a large number of tasks to be processed in the system, the manual operation method will consume a lot of resources. At the same time, based on the differences in processing methods for different tasks at different stages, the process of manually setting the processing methods is often rigid and mechanical. For example, in the scenario of data compression, when processing tasks at different stages, It is necessary to manually set the compression method, and the compression methods required by the tasks in different stages are likely to be different, which makes the flexibility of the manual setting method difficult to meet the actual needs, and when there are many compression tasks to be processed, manual operation Apparently a lot of manpower is consumed when setting up compression for big data tasks
Therefore, in the load processing process of the existing data processing tasks, the flexibility of the processing method is poor, and there is a problem of manpower consumption
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0080] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the invention may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.
[0081] Embodiments of the present invention provide a load processing method and device for data processing tasks, the main purpose of which is to reduce manpower consumption and improve the flexibility of processing methods in the process of processing large data tasks.
[0082] Specifically, the method involves specific execution steps that can be as follows figure 1 shown, including:
[0083] 101. Perform feature extraction on data processi...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a data processing task load processing method and device, and relates to the technical field of data processing. The method comprises the steps that feature extraction is carried out on a data processing task at the current stage to obtain task features of the data processing task, and the task features comprise at least one of task related information features, preheatingprocessing features, historical processing features and system state features; based on the task characteristics, automatically determining a load processing mode corresponding to the data processingtask; and executing load processing corresponding to the data processing task according to the determined load processing mode. The method and the device are used in the load processing process of thedata processing task.
Description
technical field [0001] The present invention relates to the technical field of data processing, in particular to a load processing method and device for data processing tasks. Background technique [0002] With the advent of the era of big data, the amount of data in the network has increased significantly. In the process of data processing, the processing system will consume a large amount of resources due to the amount of data. In this case, how to maximize the use of resources while effectively balancing the load in the system has become a problem for people when processing data. Frequently asked questions. For example, in the process of processing big data tasks, after a task is processed in a certain stage, the generated data often needs to be used in the next stage, and the data needs to be stored before use. During storage, the data is usually compressed based on the quantity, but some resources of the system need to be occupied during the compression process, so wh...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/50
CPCG06F9/505
Inventor 包新启王太泽陈靓范晓亮陈迪豪
Owner THE FOURTH PARADIGM BEIJING TECH CO LTD