Time-limited automatic processing method for multi-source heterogeneous mass data

A technology of massive data and processing methods, applied in the field of data processing, can solve problems such as lack of effective control, difficulty in rationally designing resource allocation strategies, time-limited processing of massive heterogeneous public credit information and difficulty in achieving expected results, and improve processing efficiency , Improve the efficiency of concurrent access to read and write, and avoid the waste of data computing resources

Pending Publication Date: 2020-05-08
NANJING LES INFORMATION TECH
View PDF4 Cites 15 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, due to the lack of effective control of the process and the difficulty in rationally designing resource allocation strategies, these mainstream methods are difficult to achieve the expected results in solving the problem of time-limited processing of massive heterogeneous public credit information.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Time-limited automatic processing method for multi-source heterogeneous mass data
  • Time-limited automatic processing method for multi-source heterogeneous mass data
  • Time-limited automatic processing method for multi-source heterogeneous mass data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0068] In order to facilitate the understanding of those skilled in the art, the present invention will be further described below in conjunction with the embodiments and accompanying drawings, and the contents mentioned in the embodiments are not intended to limit the present invention.

[0069] refer to figure 1 As shown, a time-limited processing method for multi-source heterogeneous massive data of the present invention comprises the following steps:

[0070] 1) Build a data processing operating environment based on Docker;

[0071] Run the data processing management program in a virtualized container, use the sandbox mechanism to completely virtualize the complete program running environment, and there will be no interface between the containers, so that the isolation between the container and the host, and between the container and the container more thorough. Each container has its own authority management, independent network and storage stack, and resource managemen...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a time-limited automatic processing method for multi-source heterogeneous mass data. The method comprises the following steps of building a data processing operation environment based on a container technology; establishing data acquisition task scheduling management; analyzing and optimizing a design data file; performing distributed parallel processing on data processing;enabling the modular design processing flow arrangement to be controllable; automatically optimizing data processing and monitoring; designing events and messages; optimizing storage and data accessdesign; optimizing data acquisition management. Through the automatic optimization design of the data processing flow and the optimization design that the process can be flexibly controlled, the analysis, processing and processing efficiency of the semi-structured data file is greatly improved, and the association and fusion efficiency of massive historical data and real-time data is greatly improved.

Description

technical field [0001] The invention belongs to the technical field of data processing, and specifically refers to a method for time-limited automatic processing of multi-source heterogeneous massive data. Background technique [0002] The social public credit information basic database platform is different from general government affairs systems. It needs to serve governments at all levels and the public at the same time. It has the characteristics of high complexity, high management standards, wide coverage, high performance requirements, and huge data volume. Among them, the most typical features are high performance requirements and huge data volume. Taking the bidding index announced by a national platform as an example, under the condition of normal and reasonable resource investment, the platform needs to connect with 120 central departments, 32 provinces, 43 pilot cities, relevant financial institutions, relevant Internet institutions, Nearly 250 heterogeneous syst...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/50G06F16/27G06F16/25G06F16/21G06F9/455
CPCG06F9/505G06F16/273G06F16/25G06F16/21G06F9/45558G06F2009/4557
Inventor 高翔李琬琰陈明
Owner NANJING LES INFORMATION TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap