A general big data real-time processing development platform and a data processing method thereof
A technology of real-time processing and development platform, applied in the field of data processing, can solve problems such as difficulty in screening and waste of company resources, and achieve the effect of improving development efficiency and reducing development costs
Inactive Publication Date: 2019-04-26
SHENZHEN THINKIVE INFORMATION TECH CO LTD
View PDF6 Cites 7 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
For small and medium-sized technology companies, data processing is an indispensable part of their own business. If they adopt the self-developed route, they may need to repeatedly develop these data processing systems from beginning to end for each business, and the workload accounts for more than half of the project. Development time, and after the completion of the system, due to the coupling of business characteristics, it is necessary to arrange special personnel to maintain different business systems, which is a serious waste of company resources
If an open source or hybrid solution is adopted, it is necessary to select technology types for different businesses among many open source components, and there is a certain degree of difficulty in technology selection.
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.
[0032] It should be noted that the descriptions involving "first", "second" and so on in the present invention are only for the purpose of description, and should not be understood as indicating or implying their relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In addition, the technical ...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a general big data real-time processing development platform and a data processing method thereof, and the platform comprises a data collection module which is used for obtaining a plurality of heterogeneous data sources from a database; The data transmission module is used for publishing the plurality of heterogeneous data sources; The data processing module is used for respectively processing the batch data and the stream data in the plurality of heterogeneous data sources and calling a corresponding preset database so as to construct a service application; The storage module is used for storing the processed batch data and stream data; The output query module is used for querying the stored batch data and stream data; And the application program coordination service module is used for monitoring processing events of all the modules so as to call corresponding services to process corresponding data when a certain processing event is abnormal. According to thetechnical scheme, the development efficiency of the data processing system can be improved, and the development cost is reduced.
Description
technical field [0001] The invention relates to the technical field of data processing, in particular to a general big data real-time processing development platform and a data processing method thereof. Background technique [0002] At present, there are many different but essentially similar solutions in the field of data processing. The data processing process of these solutions usually includes four steps: data acquisition, data transmission, data processing, and data storage. Each step can be used as A separate module, when designing a data processing system, the above-mentioned separate modules can be self-developed or arranged and combined with open source components. For small and medium-sized technology companies, data processing is an indispensable part of their own business. If they adopt the self-developed route, they may need to repeatedly develop these data processing systems from beginning to end for each business, and the workload accounts for more than half ...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/2458G06F16/2453
Inventor 赵晓炳
Owner SHENZHEN THINKIVE INFORMATION TECH CO LTD