Unlock instant, AI-driven research and patent intelligence for your innovation.

Message data service cluster

A technology for message data and service clusters, applied in data exchange networks, digital transmission systems, electrical components, etc., can solve problems such as inconvenient storage, inability to be consumed in a timely manner, and outdated data that cannot be eliminated in a timely manner

Active Publication Date: 2020-08-14
杭州多麦电子商务股份有限公司
View PDF11 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This type of topic message data is not convenient to use files for storage
First of all, file storage is designed for messages with a relatively long life cycle, and expired data cannot be eliminated in a timely manner; secondly, for file storage, the data that can be consumed must have landed (that is, persisted to the hard disk, has valid Logical version number), and in the file storage mode, there is a time difference between the topic message data being received and landing, and cannot be consumed in time

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Message data service cluster
  • Message data service cluster
  • Message data service cluster

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0022] Such as image 3 Shown is the framework of the distributed message system of the present invention, including a production cluster for generating topic message data, a service cluster for storing topic message data, a consumption cluster for consuming topic message data, and a management cluster for managing consumption clusters and service clusters . The service cluster includes a topic partition for storing topic message data. The topic partition includes a master copy (Leader) and a slave copy (Follower) distributed in different servers of the service cluster, and the slave copy is a redundant backup of the master copy. The service cluster consists of one or more Agent servers (servers), and each server is responsible for external services or data redundancy of several topic partitions. The server provides message submission / consumption / partition addressing functions. One of the servers (Agents) in the same topic partition is the aforementioned master copy, while t...

Embodiment 2

[0068] For the sake of brevity, the parts of this embodiment that are the same as those of the first embodiment will not be repeated here, and only the different parts of the second embodiment and the first embodiment will be described.

[0069] In this embodiment, there are two storage modes for topic message data on the server. One is the same file mode as the first embodiment, and the other is the memory mode. The file mode is suitable for storing message data with a long expiration time and a large volume, including logs, mysql binlog, and simple activity information of applications, etc.

[0070]The memory mode is suitable for simple fast message delivery. These messages usually have a short life cycle (maybe only a few minutes), require small transmission delays, and need to reach the consumer relatively quickly. It is inconvenient to store this type of topic message data in a file. First of all, file storage is designed for messages with a relatively long life cycle, ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to the field of message processing, and particularly relates to a message data service cluster. The cluster includes topic partitions for storing topic message data, and the topic partitions comprise different servers distributed over the service cluster; and the cluster is characterized in that the topic message data is stored in the service cluster in a memory mode or a file mode; in the memory mode, the topic partitions receive the topic message data generated by production ends and map the topic message data into the memories of the servers; and in the file mode, the topic partitions receive the topic message data generated by the production ends and store the topic message data in the hard disks of the servers.

Description

technical field [0001] The invention relates to the field of message processing, in particular to a message data service cluster. Background technique [0002] There are many message systems currently used for message (such as log) processing, and the more popular one is the distributed message system. [0003] Distributed message system framework such as figure 1 shown. Including Producer (message producer, referred to as PD), Agent (message cacher), Consumer (message processor, referred to as CS) and third-party management clusters, each role can have multiple. The Producer sends a message to the Agent, the message is persistently stored on the Agent, and the Consumer obtains the message from the Agent for processing. The third-party management cluster is used to store some status information of Producer, Consumer and Agent. [0004] The distributed message system performs message management based on the topic of the message (Topic). The message storage device is also...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): H04L29/08H04L12/58
CPCH04L51/00H04L67/10H04L67/14
Inventor 胡悦吴文龙
Owner 杭州多麦电子商务股份有限公司