Data access unified management platform for big data
A data access and management platform technology, applied in the field of big data, can solve problems such as abnormal data collection, difficult maintenance, and difficult to find and troubleshoot data problems.
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Embodiment 1
[0036] see figure 1 , figure 1 It is a schematic structural diagram of a big data data access unified management platform disclosed in the embodiment of this application. Such as figure 1 As shown, a big data data access unified management platform in the embodiment of the present application includes a high availability + load balancing module, a distributed collaboration module, a data collection cluster module, a data computing cluster module, and a WEB unified management and scheduling platform;
[0037] Wherein, the high availability + load balancing module is connected to the data collection cluster module, the data collection cluster module is connected to the data computing cluster module, and the output terminal of the distributed collaboration module is connected to the data collection cluster module and the A data computing cluster module, the WEB unified management and scheduling platform is connected to the high availability + load balancing module, the data col...
Embodiment 2
[0046] see figure 2 , figure 2 It is a schematic structural diagram of another big data data access unified management platform disclosed in the embodiment of this application. The second embodiment is a further improvement on the basis of the first embodiment, and its difference from the first embodiment is:
[0047] The data collection cluster module includes several streaming data collectors and several batch processing data collectors.
[0048] Optionally, the data collection cluster module is configured to start one or more collection services according to configuration and business requirements, and perform data distribution according to the configuration and registered computing services in the distributed collaboration module; send log data To the message queue; accept the retransmission message in the message queue, generate a retransmission task, and retransmit the data as a batch task.
[0049] Optionally, the data computing cluster module is used to start corr...
Embodiment 3
[0051] see image 3 , image 3 It is a schematic structural diagram of another big data data access unified management platform disclosed in the embodiment of this application. This embodiment three is a further improvement on the basis of embodiment two, and its difference with embodiment two is:
[0052] The platform also includes a fault-tolerant identification module, which is used to retrieve log data from the message queue for analysis and statistics.
[0053] Data collection is often abnormal due to network fluctuations, interruptions, or other conditions. The specific performance is that data is prone to missing or dirty data, and the resulting decline in data quality will seriously affect the reliability of the data. Therefore, this application sets up a fault-tolerant identification module to retrieve log data from the message queue for analysis and statistics, so as to find abnormal data in time. In addition, the processing of abnormal data can include deletion o...
PUM
Abstract
Description
Claims
Application Information
- R&D Engineer
- R&D Manager
- IP Professional
- Industry Leading Data Capabilities
- Powerful AI technology
- Patent DNA Extraction
Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.
© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com