Real-time and offline big data processing system and method, storage medium and terminal
A big data processing and data technology, applied in the field of big data processing, can solve the problems of long landing period of big data projects, inability to achieve configuration and monitoring, and high labor costs, so as to achieve full-process configuration management and reduce learning costs and development costs, the effect of ensuring accuracy and timeliness
Inactive Publication Date: 2020-01-07
卓尔购信息科技(武汉)有限公司
View PDF5 Cites 3 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
The above-mentioned problems are the main problems faced by big data processing at present. These problems also lead to the relatively long landing period of big data projects and relatively high labor costs. Although some products on the market have realized configuration management in the acquisition module, data conversion and The output stage still needs to be developed on demand, and the configuration and monitoring of the whole process cannot be achieved
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0035] The principles and features of the present invention are described below in conjunction with the accompanying drawings, and the examples given are only used to explain the present invention, and are not intended to limit the scope of the present invention.
[0036] Such as figure 1 As shown, a real-time and off-line big data processing system includes a data configuration module, a data acquisition module, a data conversion module and a data output module;
[0037] The data configuration module is configured to read configuration file information, and configure the data acquisition module, data conversion module and data output module respectively according to the configuration file information;
[0038]The data acquisition module is used to call a corresponding plug-in according to the configuration file information to extract data sources from offline or real-time data streams, and convert them into table format data information;
[0039] The data conversion module i...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention relates to a real-time and offline big data processing system and a method, a storage medium and a terminal. The system comprises a data configuration module, a data acquisition module,a data conversion module and a data output module, the data configuration module reads configuration file information and configures the data acquisition module, the data conversion module and the data output module respectively; the data acquisition module extracts a data source from an offline or real-time data stream and converts the data source into tabular data information; the data conversion module converts the data information; and the data output module outputs the converted data information. Whole-course configuration management of big data processing is realized through the data configuration module, and acquisition of multi-source multi-format data, conversion of mass data and output of different formats and different data storage formats can be processed. The enterprise learning cost and development cost are reduced, the process of deploying big data processing flows by enterprises is accelerated, modular and plug-in design is adopted, configuration is flexible, expansionis simple, and maintenance is convenient.
Description
technical field [0001] The invention relates to the technical field of big data processing, in particular to a real-time and off-line big data processing system, method, storage medium and terminal. Background technique [0002] The complete processing flow of big data mainly includes data collection, data processing, and data output. These three steps are executed sequentially and depend on each other. Errors in one step will affect the execution of the next step, especially in scenarios with complex business. At present, these three stages on the market have not achieved complete configuration management. It takes a long time from development and testing to project packaging, compilation and deployment. The process is relatively cumbersome, and the intermediate links are prone to errors, which will bring a lot of learning. cost and time cost. The current mainstream practices in these three stages are as follows: In the data collection stage, there are many data sources, s...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/25
CPCG06F16/258
Inventor 朱水斌罗稳刘建
Owner 卓尔购信息科技(武汉)有限公司