System and method for adapting flink task to hadoop ecology

A task status and task technology, applied in the field of big data services, can solve the problems that flink tasks cannot be fully controlled and cannot be accurately obtained, so as to reduce user labor intensity, good scalability and scalability, and improve operability and controllability sexual effect

Active Publication Date: 2022-08-09
中电云计算技术有限公司 +1
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0012] In view of this, the present invention aims to solve the defect that the submitted flink task cannot be fully controlled by submitting the flink job to the yarn cluster in the application program in the prior art, and the problem that the task log, status, and performance monitoring cannot be accurately obtained

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • System and method for adapting flink task to hadoop ecology
  • System and method for adapting flink task to hadoop ecology
  • System and method for adapting flink task to hadoop ecology

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0065] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0066] It should be noted that the following embodiments and the features in the embodiments can be combined with each other without conflict; and, based on the embodiments in the present disclosure, those of ordinary skill in the art can obtain the results obtained without creative work. All other embodiments fall within the protection scope of the present disclosure.

[0067] It is noted that various aspects of embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms and that any specific structure and / or function described herein is illustrative only. Based on this disclosure, those skilled in the art should appreciate that an aspect described herein may be implemented independently of any other aspects and that two or more of these ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a system and a method for adapting an flink task to hadoop ecology, and the system comprises an application layer which is used for constructing the flink task and submitting the flink task to an agency service layer, and is used for transmitting the information of task stop, task continuous running or task trial running to the agency service layer; the proxy service layer is used for task submitting and publishing, task stopping, task state checking, task continuous running and task trial running requests; the Hadoop layer is used for executing an flink task, generating a task execution log, generating a monitoring index and pushing the monitoring index to a Prometheus module of the monitoring layer; the monitoring service layer is used for sending the running state and the performance index of the flink task to a task checking module of the proxy service layer; and the storage layer comprises a Redis, a kafka, an elastsearch and a temporary data storage service module. According to the system and method for adapting the flink task to the hadoop ecology, the operability and controllability of the system to the flink task can be improved, system resources are saved, and the system and method are suitable for various data management production scenes.

Description

technical field [0001] The invention relates to the field of big data services, in particular to a system and method for adapting a flink task to a hadoop ecosystem. Background technique [0002] Flink technology is a real-time computing task based on the big data ecology, and its core is a distributed stream data stream engine written in Java and Scala. Flink executes arbitrary streaming data programs in a data-parallel and pipelined manner, and Flink's pipelined runtime system can execute batch and stream processing programs. In addition, Flink's runtime itself supports the execution of iterative algorithms. After the flink task runs to a big data computing engine such as yarn, it will reside in the computing engine in real time and complete the required computing tasks in real time. [0003] The existing implementation of flink submission or simply submitting flink tasks to the yarn cluster in the application only solves the deployment problem of Hadoop configuration fi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/48G06F16/11G06F16/178G06F16/18G06F16/182
CPCG06F9/485G06F9/4881G06F16/182G06F16/128G06F16/1815G06F16/178Y02D10/00
Inventor 汪昱帅冯治赵晨曦刘峰余任杰
Owner 中电云计算技术有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap