Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Based on distributed computing service cluster system

A technology of computing services and cluster systems, applied in transmission systems, data exchange through path configuration, electrical components, etc., can solve the problems of high delay in processing data, poor memory control, high delay, etc., to achieve easy understanding and mastery, Efficient computing power and the effect of improving utilization

Active Publication Date: 2022-04-08
成都网阔信息技术股份有限公司
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Provides Map and Reduce operations on the surface layer. The processing logic is hidden in the code. The overall logic is not clear enough, and the data processing delay is high. It is only suitable for batch processing data, and it is difficult to process real-time data. The disadvantage is that the delay is high and the processing flow is fixed.
[0008] The stream processing framework Storm is very suitable for processing pure stream processing loads with high latency requirements, but it cannot meet the batch processing requirements. If you need batch processing capabilities, you need to cooperate with other software
The disadvantage is that there are many dependencies on other components, the memory control is not good, and batch data cannot be processed separately
[0009] The hybrid processing framework Spark is based on the caching mechanism for input data, which requires a large amount of memory, and at the same time, the incremental update efficiency of data is poor.
[0010] Summarize the technical shortcomings of the existing distributed computing frameworks on the market are all based on the master-slave structure, and the efficiency of the framework needs to be improved

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Based on distributed computing service cluster system
  • Based on distributed computing service cluster system
  • Based on distributed computing service cluster system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0075] Such as figure 1 As shown, the distributed-based computing service cluster system includes at least two physical machines on which binary programs are deployed, and an operation and maintenance node, an allocation node, a working node and at least one working program are also deployed on the physical machines; The operation and maintenance nodes on all physical machines form a Hash ring. Preconditions for this architecture include:

[0076] Only when input tasks have clear and divisible characteristics can the advantages of distributed systems be fully demonstrated, but non-dividable tasks are still applicable in this framework; a set of general exception data processing mechanisms is available; each level of data flow has a corresponding mark Track task completion; this framework is a computing-intensive architecture, and there should be no high-speed transmission of a large amount of data between each node or node and the client, and the data channel and control cha...

Embodiment 2

[0088] Such as figure 2As shown, the difference between this embodiment and Embodiment 1 is that it includes the above computing service cluster framework, and also includes a logical structure composed of a custom module layer, a framework module layer and a network layer;

[0089] Described network layer comprises at least one in network library C++ interface, network library Python interface, network library C interface and Python common library;

[0090] The framework module layer includes an operation and maintenance node module, an allocation node module, and a work node module, and also includes a data structure module, a thread function module, a process module, and a protocol module connected to the operation and maintenance node module, the allocation node module, and the work node module and the Util module;

[0091] Data structure module: responsible for custom data structure and data processing;

[0092] Thread function module: responsible for implementing vari...

Embodiment 3

[0101] Such as Figure 3-5 As shown, the task processing method based on the distributed computing service cluster system includes the above-mentioned computing service cluster framework, and also includes a client equipped with a client program matching the computing service cluster framework, and also includes the following steps:

[0102] S1. The client applies to the cluster maintainer for calling the account and password; the client program initiates a login request to the operation and maintenance node through the account and password;

[0103] S2. The operation and maintenance node verifies the account number and password sent by the client's program. After the verification is successful, the operation and maintenance node obtains the allocation node information and returns the allocation information to the client;

[0104] S3. The client connects to the allocation node according to the allocation information returned by the operation and maintenance node;

[0105] S4....

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a distributed-based computing service cluster framework. The framework structure is designed as a ring to skillfully solve the problem of active and standby nodes, so that each node is in an equivalent position, and the efficiency and robustness of the framework are improved; completely independent development, Does not rely on any third-party programs, less code, easy to understand and master; adding task request tracking mode solves the problem of low real-time performance of the existing framework; operation and maintenance nodes, distribution nodes, and work node processing logic are not related to specific businesses Instead, the specific business processing is placed in the work program independently developed by the customer, which greatly improves the adaptability of the same computing framework to data types. This computing framework can effectively integrate idle and inefficient computing resources, output efficient computing capabilities, improve the utilization of existing computing resources, and contribute to the construction of a green, environmentally friendly and energy-saving society.

Description

technical field [0001] The invention relates to the field of distributed computing, in particular to a distributed computing service cluster system. Background technique [0002] At present, there are a large number of dedicated computing service frameworks on the market, such as the batch processing framework Hadoop, the stream processing framework Storm, and the hybrid processing framework Spark. [0003] Hadoop mainly relies on the MapReduce engine to implement batch processing, uses the distributed file system HDFS, and relies on multiple copies of data blocks to back up data blocks to solve server downtime and disk damage problems. [0004] Strom performs calculations on data coming into the system in real-time, making it ideal for data that must respond to fluctuations or spikes and needs to focus on trends over time. The Storm stream processing framework can orchestrate the DAG called Topology, which can specify the different steps that each incoming fragment needs t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): H04L67/10H04L12/42
CPCH04L67/10H04L12/42
Inventor 刘宏基
Owner 成都网阔信息技术股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com