Service data processing method and device based on Flink engine

A technology of business data and processing methods, applied in the field of big data processing, can solve problems such as high development cost and maintenance cost

Active Publication Date: 2020-01-17
SUNING CLOUD COMPUTING CO LTD
View PDF4 Cites 12 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The existing conventional real-time task development requires customized development for each real-time task. When a real-time task is developed, the possibility that the subseque

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Service data processing method and device based on Flink engine
  • Service data processing method and device based on Flink engine
  • Service data processing method and device based on Flink engine

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0050] see figure 1 and figure 2 , this embodiment provides a Flink engine-based business data processing method, including:

[0051] Receive the JSON file, which is generated by the real-time business data based on the preset standard JSON data protocol; verify and parse the JSON file, and map it to the entity class information corresponding to the JSON file; use the Flink engine to build the Flink task according to the entity class information, and Submit Flink tasks to the cluster for processing.

[0052]In the business data processing method based on the Flink engine provided in this embodiment, by converting real-time business data into JSON files based on the standard JSON data protocol, the data access method becomes more flexible, whether it is visual drag and drop, code construction, or even JSON files generated by manual editing and other methods can be connected to the Flink engine, which is beneficial to reduce the development threshold and improve development e...

Embodiment 2

[0069] This embodiment provides a Flink engine-based business data processing device, including:

[0070] The conversion unit is used to receive the JSON file, and the JSON file is generated by the real-time business data based on the preset standard JSON data protocol;

[0071] The parsing unit is used to verify and parse the JSON file, which is mapped to entity class information corresponding to the JSON file;

[0072] The task construction unit is used to use the Flink engine to construct a Flink task according to the entity class information, and submit the Flink task to the cluster for processing.

[0073] Preferably, the conversion unit includes:

[0074] The standard protocol preset module is used for abstract modeling of data structure and operator features based on Flink tasks, and the standard JSON data protocol is preset;

[0075] The data conversion module is used to convert real-time business data into JSON files through the standard JSON data protocol;

[0076...

Embodiment 3

[0087] This embodiment provides a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is run by a processor, the steps of the above-mentioned business data processing method based on the Flink engine are executed.

[0088] Compared with the prior art, the beneficial effect of the computer-readable storage medium provided by this embodiment is the same as that of the Flink engine-based business data processing method provided by the above technical solution, and details are not repeated here.

[0089] Those of ordinary skill in the art can understand that all or part of the steps in the above-mentioned inventive method can be completed by instructing related hardware through a program. The above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, it includes: For each step of the method in the foregoing embodiments, the foregoing storage medium may be: R...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a service data processing method and device based on an Flink engine, relates to the technical field of big data processing, and can effectively reduce the development cost andmaintenance cost of real-time tasks. The method comprises the steps of receiving a JSON file, wherein the JSON file is generated by real-time service data based on a preset standard JSON data protocol; verifying and analyzing the JSON file, and mapping the JSON file into entity class information corresponding to the JSON file; and constructing an Flink task by utilizing an Flink engine accordingto the entity class information, and submitting the Flink task to a cluster for processing. The device applies the method provided by the scheme.

Description

technical field [0001] The present invention relates to the technical field of big data processing, in particular to a business data processing method and device based on a Flink engine. Background technique [0002] With the rapid development of the business of Internet companies, the resulting data has also shown a blowout growth, followed by requiring the real-time platform department to provide easy-to-use, stable, and efficient real-time data services for the business department. Therefore, data-based real-time Computing business operations have begun to emerge, and more and more are being put into online use, such as real-time recommendation, real-time large-screen statistics on Double Eleven, real-time anti-fraud, etc. [0003] The existing conventional real-time task development requires customized development for each real-time task. When a real-time task is developed, the possibility that the subsequent real-time task to be developed can reuse the previous real-tim...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/25G06F16/2458
CPCG06F16/25G06F16/2462
Inventor 王军孙迁张毅王广邦陈晖
Owner SUNING CLOUD COMPUTING CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap