Data access unified management platform for big data

A data access and management platform technology, applied in the field of big data, can solve problems such as abnormal data collection, difficult maintenance, and difficult to find and troubleshoot data problems.

Pending Publication Date: 2021-05-18
辽宁长江智能科技股份有限公司
View PDF0 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] 1) Data input and output are mostly local disks or RDBMS, and the method is relatively simple
As the business becomes more and more complex, it is necessary to continuously develop processing functions that adapt to different input and output methods, and the difficulty of development and maintenance continues to increase
[0005] 2) Under the existing architecture, the components are relatively independent and the structure is loose
Each component needs to be maintained separately, and it is difficult to maintain the association and dependency between components. In complex business scenarios, it is even more difficult to maintain and easy to misuse
[0006] 3) In the process of data collection, there is a lack of necessary audit statistics
Lack of management of data assets, and it is not easy to find and troubleshoot data problems
[0007] 4) Poor fault tolerance
When network fluctuations, interruptions, or other circumstances lead to abnormal data collection, data is prone to missing or dirty data, and data quality is degraded

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data access unified management platform for big data
  • Data access unified management platform for big data
  • Data access unified management platform for big data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0036] see figure 1 , figure 1 It is a schematic structural diagram of a big data data access unified management platform disclosed in the embodiment of this application. Such as figure 1 As shown, a big data data access unified management platform in the embodiment of the present application includes a high availability + load balancing module, a distributed collaboration module, a data collection cluster module, a data computing cluster module, and a WEB unified management and scheduling platform;

[0037] Wherein, the high availability + load balancing module is connected to the data collection cluster module, the data collection cluster module is connected to the data computing cluster module, and the output terminal of the distributed collaboration module is connected to the data collection cluster module and the A data computing cluster module, the WEB unified management and scheduling platform is connected to the high availability + load balancing module, the data col...

Embodiment 2

[0046] see figure 2 , figure 2 It is a schematic structural diagram of another big data data access unified management platform disclosed in the embodiment of this application. The second embodiment is a further improvement on the basis of the first embodiment, and its difference from the first embodiment is:

[0047] The data collection cluster module includes several streaming data collectors and several batch processing data collectors.

[0048] Optionally, the data collection cluster module is configured to start one or more collection services according to configuration and business requirements, and perform data distribution according to the configuration and registered computing services in the distributed collaboration module; send log data To the message queue; accept the retransmission message in the message queue, generate a retransmission task, and retransmit the data as a batch task.

[0049] Optionally, the data computing cluster module is used to start corr...

Embodiment 3

[0051] see image 3 , image 3 It is a schematic structural diagram of another big data data access unified management platform disclosed in the embodiment of this application. This embodiment three is a further improvement on the basis of embodiment two, and its difference with embodiment two is:

[0052] The platform also includes a fault-tolerant identification module, which is used to retrieve log data from the message queue for analysis and statistics.

[0053] Data collection is often abnormal due to network fluctuations, interruptions, or other conditions. The specific performance is that data is prone to missing or dirty data, and the resulting decline in data quality will seriously affect the reliability of the data. Therefore, this application sets up a fault-tolerant identification module to retrieve log data from the message queue for analysis and statistics, so as to find abnormal data in time. In addition, the processing of abnormal data can include deletion o...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a data access unified management platform for big data. The data access unified management platform is characterized by comprising a high availability + load balancing module, a distributed collaboration module, a data acquisition cluster module, a data calculation cluster module and a WEB unified management scheduling platform. By setting the unified access management platform, unified management can be carried out on processing functions with different input and output modes and corresponding data, independent programs or scripts do not need to be written in each data acquisition link, the development and maintenance difficulty is effectively reduced, and the performance and stability of the platform are also remarkably improved.

Description

technical field [0001] The present application relates to the technical field of big data, and specifically relates to a unified management platform for data access of big data. Background technique [0002] With the gradual popularization of big data technology and applications, more and more companies choose to embrace big data in the face of increasing business and data growth. However, with the development of the company, the business continues to expand, and the data shows an explosive growth trend. As the basis of big data work, data collection has become particularly important, and it is also facing more problems and challenges. [0003] Although the distributed cluster solution has been adopted for file storage and computing, most of the data collection links are written independent programs or scripts, or even BS-based collection tools. As a result, a series of problems have arisen, such as single point of failure, average performance and stability, unreasonable al...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/25G06F9/50G06F16/18G06F16/2458
CPCG06F16/25G06F9/5083G06F9/5072G06F16/2462G06F16/1815
Inventor 丁武胡泉李林陈学志于洋
Owner 辽宁长江智能科技股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap