Remote data transmission method based on Netty

Abstract

The invention discloses a remote data transmission method based on Netty. The method comprises the following steps: initiating an RPC request; in the process of establishing the connection, initializing a thread group; assigning an NIO thread, and establishing a thread pool; starting a data transponder, and processing an I / O event; processing packet boundaries; constructing a connection server channel; putting the connected channels into a queue; sending the data to a buffer area and then refreshing and outputting; when receiving the data, accessing the data into a kafka message queue; packaging the data into Producer Record of kafka; judging whether a specified partition exists in the topic or not; if it is determined that the specific partition does not exist in the top, judging whethera specific key exists or not; if it is determined that the specified key exists, specifying a partition for the top by adopting a corresponding hash algorithm according to the value of the key; and accessing to the partitions corresponding to the topics, and sequentially storing the data into the buffer pools corresponding to the partitions. By means of the mode, high-throughput data transmissioncan be achieved, the I / O event processing efficiency is effectively improved, and the phenomenon of server downtime caused by thread stack overflow is avoided.

Description

technical field [0001] The invention relates to the technical field of the Internet, in particular to a Netty-based remote data transmission method. Background technique [0002] With the rapid development of the mobile Internet, the scale of the website is expanding day by day. For example, communication, business, logistics, and game system visits are all rapidly increasing numbers. It is said that they all face the test of big data carrying capacity. [0003] The remote procedure call of the traditional RPC framework or the remote service based on the remote method call in the form of RMI all use synchronous blocking I / O (input / output) events, and synchronous blocking I / O events are easy to increase with the increase in client concurrency and the network Latency increases and frequent waiting causes I / O threads to easily block. If the thread cannot be released in time, the I / O event processing efficiency will drop sharply, and even the thread stack overflow will cause t...

AI Technical Summary

Problems solved by technology

[0003] The remote procedure call of the traditional RPC framework or the remote service based on the remote method call in the form of RMI all use synchronous blocking I / O (input / output) events, and synchronous blocking I / O events are easy to increase with the increase in client concurrency and the network Frequent waiting for delay growth causes I / O threads to easily block

If the thread cannot be released in time, the I / O event processing efficiency will drop sharply, and even the thread stack overflow will cause the server to go down.

Images