Data processing method, device, equipment and storage medium based on Netty framework
By using multi-threaded model and SBE protocol in the Netty framework, the data is encoded and decoded, which solves the problems of high and unreliable data transmission delay in the network environment, and achieves more efficient and reliable data transmission.
Patent Information
- Application Number
- CN202210280862.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-22
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-03-22
AI Technical Summary
The prior art has problems of high delay and unreliable data transmission in the data transmission process in the network environment, especially when facing events such as disconnection and reconnection, network congestion and abnormal code streams.
Using the Netty framework-based data processing method, the data to be sent by a server and distributed to multiple threads for execution are subjected to preset SBE protocol encoding processing to reduce delay and improve transmission efficiency and reliability. Specific steps include data distribution, encoding, sending and receiving processing, and encoding and decoding using a multi-threaded model and SBE protocol.
It effectively reduces the delay in data transmission process, improves the efficiency and reliability of data transmission, and improves the speed and reliability of data processing through multi-threading and the use of SBE protocols.
Smart Images

Figure CN114640719B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, equipment and storage medium based on a Netty framework. Background Art
[0002] Data processing technology includes technologies for processing data sending and receiving, especially data distribution in a network environment and NIO processing technology for data. Existing data processing solutions are mainly implemented through traditional Java built-in socket network programming, which has problems such as poor performance and cumbersome class libraries. The client designed on this basis will face disconnection and reconnection, network flash disconnection, half-packet reading and writing, heartbeat, failed cache, network congestion and abnormal code stream processing. Due to poor unpacking performance, data transmission will be unreliable during the data transmission process; due to cumbersome class libraries and low transmission efficiency, there will be high latency during the data transmission process.
[0003] Netty is the most popular NIO framework, and its robustness, functionality, performance, customizability, and scalability are second to none among similar frameworks. However, the transmission efficiency of the transmission protocols adopted by the existing data processing methods based on the Netty framework is relatively low, such as xml, json, protobufer, thrift, and avro, which still leads to high latency problems in data transmission.
[0004] In summary, existing data processing technologies have the problems of high delay and unreliable data transmission during data transmission. Summary of the invention
[0005] The main purpose of this application is to provide a data processing method, device, equipment and storage medium based on the Netty framework to solve the problems of high data transmission delay and unreliable data transmission in the data transmission and processing process in the network environment in the prior art.
[0006] A first aspect of the present invention provides a data processing method based on a Netty framework, wherein the Netty framework is provided with a plurality of parallel execution threads, and the data processing method comprises: obtaining data to be sent on a server, and distributing the data to be sent to a plurality of threads for execution; encoding and processing the corresponding data to be sent according to a preset SBE protocol by the threads to obtain an SBE message; and sending the SBE message to a client.
[0007] Optionally, in a first implementation method of the first aspect of the present invention, the data to be sent on the client is obtained and the data to be sent is distributed to multiple threads for execution, including: obtaining data with a sending mark on the server to obtain the data to be sent; dividing the data to be sent into multiple fragments according to the storage rules of the message queue to form a queue of messages to be processed; obtaining at least two threads with idle resources on the server, and sorting the at least two threads according to priority to obtain a thread sequence; polling the queue of messages to be processed in turn by each thread in the thread sequence to obtain the resource requirements in the queue of messages to be processed, and determining whether the idle resources on the corresponding thread meet the resource requirements; if so, distributing the fragments to the corresponding threads for execution until all fragments are distributed.
[0008] Optionally, in a second implementation of the first aspect of the present invention, the encoding and processing of the data to be sent according to a preset SBE protocol to obtain an SBE message includes: after the data to be sent is divided into fragments and distributed to corresponding threads for execution, constructing a channel through the thread and building a pipeline corresponding to the channel; encoding and processing the fragments on the Handler chain on the pipeline according to the preset SBE protocol to obtain multiple SBE message fragments, wherein the Handler chain includes at least one Handler; merging the multiple SBE message fragments to obtain an SBE message.
[0009] Optionally, in a third implementation of the first aspect of the present invention, sending the SBE message to the client includes: after the data to be sent is divided into fragments and distributed to corresponding threads for execution, creating a sub-channel for processing write events through the thread; calling a call chain on the sub-channel to send the SBE message to the server, wherein the call chain includes at least one Handler.
[0010] Optionally, in a fourth implementation of the first aspect of the present invention, each thread in the thread sequence polls the pending message queue in turn to obtain the resource requirements in the pending message queue, and determines whether the idle resources on the corresponding thread meet the resource requirements, including: polling the pending message queue to determine whether there are the pending messages in the message queue; if there are the pending messages in the message queue, obtaining the event type corresponding to the pending message, wherein different event types correspond to different resource requirements; calling the selectNow() method of Selector to trigger the selection operation of Selector, and if there is a ready Channel, returning a ready channel set, and if there is no ready channel, returning 0; according to the ready channel set, reading the SelectionKey values of the ready channels in turn to obtain the event types corresponding to the ready channels; if the event type corresponding to the ready channel contains the event type corresponding to the pending message, determining whether the idle resources on the corresponding thread meet the resource requirements, otherwise determining that they do not meet the resource requirements.
[0011] Optionally, in a fifth implementation of the first aspect of the present invention, the data processing method further includes: monitoring data sent from multiple clients; distributing the monitored data to multiple threads for execution; processing and receiving the monitored data on corresponding threads; decoding the SBE message in the received data according to a preset SBE protocol to obtain decoded data; and sending the decoded data to a processing module on the server.
[0012] Optionally, in a sixth implementation of the first aspect of the present invention, the processing and receiving operations on the monitored data on the corresponding thread include: after the monitored data is distributed to the threads in the main thread pool for execution, the threads in the main thread pool encapsulate the monitored data into events to be processed; the threads in the main thread pool determine whether the events to be processed are connection events; if so, process the connection event on the thread; if not, pass the event to the slave thread pool.
[0013] The second aspect of the present invention provides a data processing device based on the Netty framework, including: a first distribution module, used to obtain the data to be sent on the server, and distribute the data to be sent to multiple threads for execution; an encoding module, used to encode the data to be sent according to a preset SBE protocol to obtain an SBE message; a first sending module, used to send the SBE message to the client.
[0014] Optionally, in a first implementation method of the second aspect of the present invention, the first distribution module includes: an acquisition unit, used to acquire data with a sending mark on the server to obtain data to be sent; a queue generation unit, used to divide the data to be sent into multiple fragments according to the storage rules of the message queue to form a queue of messages to be processed; a thread sorting unit, used to acquire at least two threads with idle resources on the server, and sort the at least two threads according to priority to obtain a thread sequence; a polling judgment unit, used to poll the queue of messages to be processed in turn through each thread in the thread sequence to obtain the resource requirements in the queue of messages to be processed, and judge whether the idle resources on the corresponding thread meet the resource requirements; a first distribution unit, used to distribute the fragments to the corresponding threads for execution until all fragments are distributed.
[0015] Optionally, in a second implementation of the second aspect of the present invention, the encoding module includes: a first construction unit, used to construct a channel through the thread and build a pipeline corresponding to the channel after the fragments into which the data to be sent are divided are distributed to the corresponding threads for execution; an encoding unit, used to encode the fragments on the Handler chain on the pipeline according to a preset SBE protocol to obtain multiple SBE message fragments, and merge the multiple SBE message fragments to obtain an SBE message.
[0016] Optionally, in a third implementation of the second aspect of the present invention, the first sending module includes: a second construction unit, used to create a sub-channel for processing write events through the thread after the fragments into which the data to be sent are divided are distributed to the corresponding threads for execution; a sending unit, used to call the call chain on the sub-channel to send the SBE message to the server, wherein the call chain includes at least one Handler.
[0017] Optionally, in a fourth implementation manner of the second aspect of the present invention, the polling judgment unit is specifically used to: poll the pending message queue to determine whether there are pending messages in the message queue; if there are pending messages in the message queue, obtain the event type corresponding to the pending message, wherein different event types correspond to different resource requirements; call the selectNow() method of Selector to trigger the selection operation of Selector, and if there is a ready Channel, return a ready channel set, and if there is no ready channel, return 0; according to the ready channel set, read the SelectionKey values of the ready channels in turn to obtain the event types corresponding to the ready channels; if the event type corresponding to the ready channel contains the event type corresponding to the pending message, determine whether the free resources on the corresponding thread meet the resource requirements, otherwise determine that they do not meet the requirements.
[0018] Optionally, in a fifth implementation of the second aspect of the present invention, the data processing device based on the Netty framework also includes: a monitoring module, used to monitor data sent by multiple clients; a second distribution module, used to distribute the monitored data to multiple threads for execution; a receiving and processing module, used to process and receive the monitored data on the corresponding threads; a decoding module, used to decode the SBE message in the received data according to a preset SBE protocol to obtain decoded data; and a second sending module, used to send the decoded data to the server.
[0019] Optionally, in a sixth implementation of the second aspect of the present invention, the receiving and processing module includes: an encapsulation unit, which is used to encapsulate the monitored data into events to be processed after the monitored data is distributed to the threads in the main thread pool for execution; a judgment unit, which is used to judge whether the event to be processed is a connection event through the threads in the main thread pool; a connection processing unit, which is used to process the connection event on the thread; and an event delivery unit, which is used to deliver the event to the slave thread pool.
[0020] The third aspect of the present invention provides a computer device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the computer device executes the above-mentioned steps of data processing based on the Netty framework.
[0021] A fourth aspect of the present invention provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes each step of the above-mentioned data processing method based on the Netty framework.
[0022] In the technical solution of the present invention, the method specifically obtains the data to be sent on the server, distributes the data to be sent to multiple threads for execution, encodes the data to be sent according to a preset SBE protocol, obtains an SBE message, and sends the SBE message to the client; monitors the data and distributes it to multiple threads for execution, performs processing and receiving operations, and decodes the SBE message in the received data according to the preset SBE protocol to obtain decoded data, and sends the decoded data to the server; in the above, based on the Netty framework, the sending and receiving processing of data is distributed to multiple threads for execution, thereby reducing the delay in the data transmission process; in the data transmission process, the preset SBE protocol is used for encoding and decoding, thereby improving the efficiency and reliability of data transmission. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 This is a schematic diagram of a first embodiment of a data processing method based on the Netty framework in an embodiment of the present invention;
[0024] Figure 2 Schematic diagram of a second embodiment of a data processing method based on a Netty framework in an embodiment of the present invention;
[0025] Figure 3 Schematic diagram of a third embodiment of a data processing method based on the Netty framework in an embodiment of the present invention;
[0026] Figure 4 A schematic diagram of an embodiment of a data processing device based on the Netty framework in an embodiment of the present invention;
[0027] Figure 5 It is a schematic diagram of another embodiment of a data processing device based on the Netty framework in an embodiment of the present invention;
[0028] Figure 6 FIG. 1 is a schematic diagram of an embodiment of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION
[0029] In order to solve the problems of high data transmission delay and unreliable data transmission in the data transmission and processing process of the prior art in the network environment. The present application provides a data processing method based on the Netty framework, which obtains the data to be sent on the server, distributes the data to be sent to multiple threads for execution, encodes the data to be sent according to the preset SBE protocol, obtains the SBE message, and sends the SBE message to the client; in the above, based on the Netty framework, the sending and receiving processing of data is distributed to multiple threads for execution, which reduces the delay in the data transmission process; in the data transmission process, the preset SBE protocol is used for encoding and decoding, which improves the efficiency and reliability of data transmission.
[0030] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0031] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , the first embodiment of the data processing method based on the Netty framework in the embodiment of the present invention, the implementation steps of the method are as follows:
[0032] 101. Obtain the data to be sent on the server, and distribute the data to be sent to multiple threads for execution;
[0033] In this step, the distribution of the data to be sent is implemented through NioEventLoopGroup, which is composed of multiple NioEventLoops, and each NioEventLoop is bound to a thread, that is, a thread pool is scheduled by NioEventLoopGroup for distribution.
[0034] In this embodiment, the process of distributing data to multiple threads for execution includes:
[0035] The data to be sent enters the message queue, and the NioEventLoop in the NioEventLoopGroup polls the message queue and determines whether the thread bound to it has a ready channel to process the data;
[0036] If the thread meets the requirements, the data is passed to the serial Handler chain for processing.
[0037] In actual applications, if the data to be sent on the client is to be obtained and distributed to multiple threads for execution, after the data to be sent enters the message queue in the client, the data is distributed to the multiple threads in the above manner.
[0038] 102. Encode the data to be sent according to a preset SBE protocol to obtain an SBE message;
[0039] In this step, the process of encoding the data to be sent according to the preset SBE protocol includes: after the data to be sent is distributed to the thread for execution, the serial Handler chain corresponding to the thread processes the data, wherein the Handler with an encoder based on the SBE protocol encodes the data to obtain an SBE message.
[0040] In this embodiment, the encoder based on the SBE protocol adopts the encoder preset in the SBE library, and the SbeTool related to the encoder based on the SBE protocol also adopts the encoder preset in the SBE library.
[0041] In practical applications, in order to achieve the technical effect of simplifying the structural design so as to obtain the data interaction process, the Handler with the encoder based on the SBE protocol is defined, recorded as SBEHandler, and the construction method of the SBEHandler includes:
[0042] Implement a custom initialization class, namely Initializer, on the server side, and add the SBE protocol-based encoder to the pipeline in the initialization class;
[0043] SBEHandler acts as a custom ChannelHandler in the pipeline (ChannelPipeline). When it is added, ChannelPipeline will create a corresponding ChannelHandlerContext.
[0044] In actual applications, when the fragment is encoded according to the preset SBE protocol on the Handler chain on the pipeline, the data fragment will be passed to the Handler with an encoder based on the SBE protocol for processing, for example, to the SBEHandler. At this time, ChannelHandlerContext is used to assist in fast transmission, including:
[0045] When a fragment is passed into the pipeline (ChannelPipeline), an event is generated;
[0046] The event is passed to the first handler in the pipeline (ChannelHandler);
[0047] If the first handler is the SBEHandler, then the SBEHandler handles the event;
[0048] The encoder based on the SBE protocol of the SBEHandler encodes the data corresponding to the event to obtain an SBE message fragment;
[0049] If the first processor is not the SBEHandler, the processor passes the event to the next processor in the pipeline by using the ChannelHandlerContext associated therewith until it is passed to the SBEHandler, which performs encoding processing to obtain an SBE message.
[0050] 103. Send the SBE message to the client;
[0051] In this step, the sending operation of the SBE message is still implemented through the thread to which the corresponding data is originally distributed. Specifically, a channel architecture for processing write events is created through the thread, and executed in the call chain of the corresponding channel to send the SBE message to the client;
[0052] Construct a DownstreamMessageEvent event and place it at the end of the Pipeline queue (downstream), and execute it in the call chain of the child channel (child Channel), where the downstream event is executed starting from the end of the Pipeline queue;
[0053] Put the DownstreamMessageEvent event in the write buffer queue (writeBufferQueue);
[0054] Call the poll() method in the write buffer queue to obtain the DownstreamMessageEvent event;
[0055] Apply for allocation of available free storage area in the ring buffer and output log records of related data operations;
[0056] Use the database operation tool fire to configure the WriteComplete event, including placing it at the head of the Pipeline team (upstream);
[0057] Execute the WriteCompletion event in the child pipeline call chain and send the SBE message to the client;
[0058] After all the SBE messages to be sent are sent to the client, call the closeAll() method to traverse all Channels, and call their Unsafe.close() methods to close all links and release resources.
[0059] Furthermore, the process of applying for allocation of available free storage area in the ring buffer (RingBuffer) and outputting log records of related data operations is based on the Disruptor queue framework.
[0060] The Disruptor queue framework has a ring array RingBuffer and an AvailableBuffer array structure of the same size, wherein the RingBuffer is a ring storage unit, and the queue is written by overwriting storage, and the read and write status marking is realized by the AvailableBuffer structure, including:
[0061] During the writing process, when a certain position of RingBuffer is written successfully, the corresponding position of AvailableBuffer is set through the setAvailable() method to mark it as written successfully;
[0062] During the reading process, first traverse the AvailableBuffer, use the isAvailable() method to determine whether the element is ready, and then read the element corresponding to the ready position on the RingBuffer.
[0063] Through the implementation of the above method, by obtaining the data to be sent on the server, the data to be sent is distributed to multiple threads for execution, the data to be sent is encoded and processed according to the preset SBE protocol to obtain an SBE message, and the SBE message is sent to the client; in the above, based on the thread model under the Netty framework, the sending and receiving processing of data is distributed to multiple threads for execution, thereby reducing the delay in the data transmission process; in the data transmission process, the preset SBE protocol is used for encoding and decoding, thereby improving the efficiency and reliability of data transmission; in the process of encoding data, based on the serial Handler chain and channel mode of the Netty framework, the Handler context (ChannelHandlerContext) is used to support fast encoding, thereby improving the speed and reliability of encoding.
[0064] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 2 , the second embodiment of the data processing method based on the Netty framework in the embodiment of the present invention, the implementation steps of the method are as follows:
[0065] 201. Obtain the data to be sent on the server, and distribute the data to be sent to multiple threads for execution;
[0066] In this embodiment, this step includes:
[0067] Polling the pending message queue to determine whether there is the pending message in the message queue;
[0068] If there is the to-be-processed message in the message queue, obtaining the event type corresponding to the to-be-processed message, wherein different event types correspond to different resource requirements;
[0069] Call the selectNow() method of Selector to trigger the selection operation of Selector. If there is a ready channel, it returns the ready channel set. If there is no ready channel, it returns 0.
[0070] According to the ready channel set, the SelectionKey values of the ready channels are read in sequence to obtain the event types corresponding to the ready channels;
[0071] If the event type corresponding to the ready channel (Channel) includes the event type corresponding to the message to be processed, it is determined whether the free resources on the corresponding thread meet the resource requirements, otherwise it is determined that they are not satisfied. Among them, the ready channel (Channel) has four SelectionKey values: "SelectionKey.OP_CONNECT", "SelectionKey.OP_ACCEPT", "SelectionKey.OP_READ" and "SelectionKey.OP_WRITE", which correspond to four event types, namely Connect, Accept, Read and Write event types.
[0072] 202. Encode the data to be sent according to a preset SBE protocol to obtain an SBE message;
[0073] In this step, the process of encoding the data to be sent according to the preset SBE protocol includes: after the data to be sent is distributed to the thread for execution, the serial Handler chain corresponding to the thread processes the data, wherein the Handler with an encoder based on the SBE protocol encodes the data to obtain an SBE message.
[0074] Furthermore, the Handler having an encoder using the SBE protocol encodes the fragment to obtain a plurality of SBE message fragments, including:
[0075] Input the fragment into SbeTool in the Handler for compilation to obtain a stub of the fragment;
[0076] Based on the encoding method of the data determined in the stub;
[0077] According to the encoding method, converting the fragment encoding into assembly code;
[0078] The assembly code is converted into an SBE binary format to generate a plurality of SBE message fragments.
[0079] 203. Send the SBE message to the client;
[0080] In this step, if the SBE message is stored in an array, all bytes in the array buff are written to the output stream and sent to the client by calling the void write(byte[]buf) method;
[0081] If the array contains other data, the offset off and length len corresponding to the SBE message stored in the array are obtained, and the byte data corresponding to the SBE message is written to the output stream by calling the void write(byte[]b, int off, int len) method and sent to the client.
[0082] In actual applications, the byte data corresponding to the SBE message may also be written into the output stream by calling the void write(int c) method through the OutputStream class and sent to the client.
[0083] In actual applications, if the SBE message is to be sent to the server, the client sends data according to the above corresponding methods according to different application scenarios.
[0084] 204. Monitor data sent by multiple clients, and distribute the monitored data to multiple threads for execution;
[0085] In this step, the process of distributing the monitored data to multiple threads for execution includes: using two NioEventLoopGroups to schedule two thread pools to implement this process. The thread bound to BossEventLoop in the main thread pool judges the various types of channels (for example, socket channels SocketChannel) that are ready and performs corresponding processing, wherein the processing strategy is: if it is a connection event, the connection event is processed on the thread; if it is not a connection event, it is transferred to the slave thread pool for processing.
[0086] Specifically, the process of processing the connection event on the thread includes:
[0087] Determine the attachment type of the channel (SocketChannel);
[0088] Get the inner class Unsafe from the channel to determine whether the current selection key is available;
[0089] If it is not available, call the close method of Unsafe to release the connection resources;
[0090] If the selection key is available, continue to judge the network operation bit;
[0091] If it is a read or connect operation, call the Unsafe read method;
[0092] If the network operation bit is in the connection state, the connection result needs to be interpreted;
[0093] Modify the network operation bit, cancel SelectionKey.OP_CONNECT, and perform finishConnect judgment;
[0094] If it is a NioTask, call the processSelectedKey() method to convert its type.
[0095] 205. Decode the received SBE message according to the preset SBE protocol to obtain decoded data, and send the decoded data to the server;
[0096] In this step, the process of decoding the received data according to the preset SBE protocol includes: after the data to be sent is distributed to the thread for execution, the serial Handler chain corresponding to the thread processes the data, wherein the Handler with a decoder based on the SBE protocol decodes the data to obtain decoded data, and sends the decoded data to the server through the channel constructed by the thread.
[0097] Specifically, the process of decoding data by the Handler having a decoder based on the SBE protocol includes:
[0098] Input the message pattern to be processed into SbeTool for compilation, and use the compiler in the SBE library for compilation;
[0099] If it is not an XML message mode, a corresponding stub is generated, in which a pair of encoding and decoding methods are generated for each field;
[0100] Decode the data through the decode method generated by the stub;
[0101] converting the data pattern into assembly code according to a decoding method generated by the stub;
[0102] Convert assembly code into SBE binary format and generate SBE messages;
[0103] If it is an XML message schema, generate an intermediate representation (IR);
[0104] The intermediate representation (IR) is serialized into the SBE binary format to generate an SBE message.
[0105] Through the implementation of the above method, by obtaining the data to be sent on the server, distributing the data to be sent to multiple threads for execution, encoding the data to be sent according to the preset SBE protocol, obtaining an SBE message, and sending the SBE message to the client; by monitoring the data and distributing it to multiple threads for execution, processing and receiving operations are performed, and the SBE message in the received data is decoded according to the preset SBE protocol to obtain decoded data, and the decoded data is sent to the server; in the above, based on the Netty framework, the sending and receiving processing of data is distributed to multiple threads for execution, which reduces the delay in the data transmission process; in the data transmission process, the preset SBE protocol is used for encoding and decoding, which improves the efficiency and reliability of data transmission; the design of two NioEventLoopGroup scheduling master and slave thread pools is adopted, and the events are distributed and processed according to the processing strategy of "if it is a connection event, the connection event is processed on this thread; if it is not a connection event, it is transferred to the slave thread pool for processing", which improves the processing efficiency and reduces the blocking probability, thereby reducing the delay in the data transmission process.
[0106] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 3 , the third embodiment of the data processing method based on the Netty framework in the embodiment of the present invention, the implementation steps of the method are as follows:
[0107] 301. Obtain the data to be sent on the server, and distribute the data to be sent to multiple threads for execution;
[0108] In this step, the process of distributing the data to be sent to multiple threads for execution is implemented through the Reactor thread model under the Netty framework, including polling the message queue and determining whether the bound thread has a ready channel to process the data; if satisfied, the data is passed to the serial Handler chain for processing.
[0109] In actual applications, if the data to be sent on the client is to be obtained and distributed to multiple threads for execution, after the data to be sent enters the message queue in the client, the data is distributed to the multiple threads in the above manner.
[0110] 302. Encode the data to be sent according to a preset SBE protocol to obtain an SBE message;
[0111] In this step, the process of encoding the data to be sent according to the preset SBE protocol includes: after the data to be sent is distributed to the thread for execution, the serial Handler chain corresponding to the thread processes the data, wherein the Handler with an encoder based on the SBE protocol encodes the data to obtain an SBE message.
[0112] In this embodiment, the decoder based on the SBE protocol adopts the decoder preset in the SBE library, and the SbeTool related to the decoder based on the SBE protocol also adopts the one preset in the SBE library.
[0113] 303. Send the SBE message to the client;
[0114] In this embodiment, this step includes:
[0115] The thread at the sending end polls. If the thread polls a socket channel (SocketChannel) in the ready state, it means that there is an event to be processed, that is, there is an SBE message to be sent;
[0116] Call the connect() method to initiate an asynchronous connection;
[0117] Call parentChannel.socket.accept() to create a child channel (childChannel) for handling write events, where the child channel is ready for write events (Selectionkey.OP_WRITE);
[0118] After the connection is successful, the constructor of NioSocketChannel is called to encapsulate the socket channel (SocketChannel) as a client channel (NioSocketChannel). In this process, a corresponding pipeline (Pipeline) component is created, wherein a corresponding handler (Handler) is provided on the pipeline (Pipeline).
[0119] Data is written through the channel and the ready write event, and the SBE message is sent to the client.
[0120] 304. Monitor data sent by multiple clients, and distribute the monitored data to multiple threads for execution;
[0121] In this step, the distribution of the monitored data is achieved by scheduling two thread pools through two NioEventLoopGroups, wherein the main thread pool is used to process connection events, and the corresponding NioEventLoopGroup is recorded as BossEventLoopGroup, and the slave thread pool is used to process read and write events, and the corresponding NioEventLoopGroup is recorded as WorkerEventLoopGroup. The NioEventLoopGroup is composed of multiple NioEventLoops, and each NioEventLoop is bound to a thread. The NioEventLoop in the BossEventLoopGroup is recorded as BossEventLoop, and the NioEventLoop in the WorkerEventLoopGroup is recorded as WorkerEventLoop.
[0122] In this embodiment, the process of distributing the monitored data to multiple threads for execution includes:
[0123] Monitor the data sent by the client. If data is detected, the message queue is polled through BossEventLoop to determine whether the bound thread has a ready channel to process the data.
[0124] If the thread meets the requirements, the data is passed to the serial Handler chain for processing;
[0125] Determine whether the event to be processed is a connection event, and if it is a connection event, process the connection event on the thread;
[0126] If it is not a connection event, the event is passed to the corresponding thread in WorkerEventLoopGroup for execution.
[0127] 305. Receive and process the monitored data and output log records;
[0128] For this step, in this embodiment, the Disruptor queue framework is used to complete the process of applying for allocation of available free storage area in the ring buffer (RingBuffer) and outputting log records of related data operations, wherein the structure designed based on the producer-consumer model includes:
[0129] Data storage and provider (dataProvide), implemented by RingBuffer and AvailableBuffer;
[0130] The producer (Sequencer) is the same instance for consumers and multi-producer sequences. It contains a cursor sequence number Sequence, which is used to control the writing process of multi-producer sequences. Among them, multi-producer sequences refer to producer sequences used to process data from the SBE decoder, unencoded data from the business handler, and encoded data from the business handler.
[0131] Sequence Barrier, or simply Barrier, is the same instance for both consumer and multi-producer sequences. It contains a wait strategy (such as Timeout strategy), a RingBuffer reference, a cursor sequence number, and a dependency sequence number.
[0132] The consumer (BatchEventProcessor) contains a RingBuffer reference and a sequence number barrier, which is used to read and process the data produced by the producer. The RingBufferLogEvent.FACTORY method is used to generate a log object instance. The barrier uses the cursor sequence number Sequence of the RingBuffer's MultiProducerSequencer.
[0133] Event handler (EventHandler), specifically, RingBufferLogEventHandler;
[0134] Sequencer, used to publish serial numbers;
[0135] Event thread pool (EventHandlerGroup), a thread pool used to process producer-consumer model data processing events.
[0136] In this embodiment, the process of receiving and processing the monitored data and outputting the log record includes:
[0137] Traverse the EventHandler list and encapsulate it as BatchEventProcessor, and register it with the original eventHandler and barrier to the consumer repository consumerRepository.
[0138] Get the batchEventProcessor sequence number, which defaults to -1, and cache it to processorSequences to indicate that it is being processed. Bind processorSequences, disruptor, and consumerRepository to EventHandlerGroup.
[0139] Disruptor starts traversing the consumer resource library and starts the consumer BatchEventProcessor;
[0140] The consumer increments the local sequence number (i.e. -1+1=0) and applies to the sequence number barrier for consumption of the sequence number. The default policy is Timeout.
[0141] The barrier receives the request for waitFor sequence number. If the current barrier cursor sequence number is less than the requested consumer sequence number, it waits for the producer to produce to the current sequence number. If it times out, an exception is thrown (the local sequence number is not updated and retries are continued); if there is no timeout, the barrier's dependentSequence sequence number is assigned to availableSequence and returned;
[0142] If the valid sequence number of availableSequence (i.e., the cursor sequence number of the barrier) is less than the sequence number requested for consumption, then availableSequence is returned directly (i.e., the consumption exceeds the production speed, and the sequence number requested by the consumer moves back to the valid sequence number);
[0143] If the requested sequence number is less than or equal to the valid sequence number, the consumer reads the message event corresponding to the consumption sequence number and updates the sequence number of the local BatchEventProcessor. It extracts the data at the corresponding position from dataProvide(RingBuffer.entries) according to the subscript, and generates a log object instance using the RingBufferLogEvent.FACTORY method.
[0144] If the requested sequence number is greater than the valid sequence number, the consumer's local sequence number is set to the valid sequence number (i.e., if consumption exceeds the production speed, the consumption sequence number is moved back);
[0145] If any abnormality occurs during the process, the current index will be skipped. The index and corresponding events when the exception occurs will be handled by exceptionHandler (default is AsyncLoggerDefaultExceptionHandler asynchronous processing), and the exception event will be output to the system's standard error channel;
[0146] Apply for RingBuffer storage space, the producer calls the method containing the CAS algorithm (for example: MultiProducerSequencer) and tries to publish a message;
[0147] The producer cursor sequence number tries to increment itself to determine whether there is enough space at present and whether the current cursor + step - bufferSize is greater than the smallest gate sequence number gatingSequences (that is, the smallest sequence number in the list of processorSequences of all consumers' local cursor sequence numbers). At the same time, the smallest sequence number is cached in the local gatingSequenceCache for the next judgment to reduce the number of traversals of all gate sequence numbers.
[0148] If the current cursor + step - bufferSize is greater than the smallest gate sequence number, return false and enter the next round of production;
[0149] If the current cursor + step - bufferSize is less than the smallest gate sequence number, after the self-increment is successful, the message is converted into event data corresponding to the sequence number subscript position;
[0150] The Sequencer publishes the sequence number, sets the current sequence number to valid (availableBuffer), and wakes up the waiting consumer according to the waiting strategy. The awakened consumer obtains the event data at the corresponding index according to the published sequence number for processing to complete the data reception and processing operations.
[0151] The log object instance generated by the consumer using the RingBufferLogEvent.FACTORY method is output to obtain a log record.
[0152] 306. Decode the received SBE message according to the preset SBE protocol to obtain decoded data, and send the decoded data to the server;
[0153] In this step, the process of decoding the data to be sent according to the preset SBE protocol includes: after the data to be sent is distributed to the thread for execution, the serial Handler chain corresponding to the thread processes the data, wherein the Handler with a decoder based on the SBE protocol decodes the data, obtains the decoded data and sends the decoded data to the server.
[0154] In this embodiment, the decoder based on the SBE protocol adopts the decoder preset in the SBE library, and the SbeTool related to the decoder based on the SBE protocol also adopts the one preset in the SBE library.
[0155] In practical applications, in order to achieve the technical effect of simplifying the structural design so as to obtain the data interaction process, the Handler with the decoder based on the SBE protocol is defined as SBEHandler. The construction method of the SBEHandler includes:
[0156] Implement a custom initialization class, namely Initializer, on the server side, and add the SBE protocol-based decoder to the pipeline in the initialization class;
[0157] SBEHandler acts as a custom ChannelHandler in the pipeline (ChannelPipeline). When it is added, ChannelPipeline will create a corresponding ChannelHandlerContext.
[0158] In actual applications, when the fragment is decoded according to the preset SBE protocol on the Handler chain on the pipeline, the data fragment will be passed to the Handler with a decoder based on the SBE protocol for processing, for example, to the SBEHandler. At this time, ChannelHandlerContext is used to assist in fast transmission, including:
[0159] When a fragment is passed into the pipeline (ChannelPipeline), an event is generated;
[0160] The event is passed to the first handler in the pipeline (ChannelHandler);
[0161] If the first handler is the SBEHandler, then the SBEHandler handles the event;
[0162] The SBEHandler has a decoder based on the SBE protocol to decode the data corresponding to the event to obtain an SBE message fragment;
[0163] If the first processor is not the SBEHandler, the processor passes the event to the next processor in the pipeline by using the ChannelHandlerContext associated with it, until it is passed to the SBEHandler for processing and obtains the SBE message fragment.
[0164] Through the implementation of the above method, by obtaining the data to be sent on the server, distributing the data to be sent to multiple threads for execution, encoding the data to be sent according to the preset SBE protocol, obtaining an SBE message, and sending the SBE message to the client; by monitoring the data and distributing it to multiple threads for execution, processing and receiving operations are performed, and the SBE message in the received data is decoded according to the preset SBE protocol to obtain decoded data, and the decoded data is sent to the server; in the above, based on the Netty framework, the sending and receiving processing of data is distributed to multiple threads for execution, thereby reducing the delay in the data transmission process; in the data transmission process, the preset SBE protocol is used for encoding and decoding, thereby improving the efficiency and reliability of data transmission, and adopting the producer-consumer model design to complete the process of applying for allocation of available free storage areas in the ring buffer (RingBuffer) and outputting log records of related data operations, thereby improving the speed of data processing and the efficiency of output log records.
[0165] The above describes the data processing method based on the Netty framework in the embodiment of the present invention. The following describes the data processing device based on the Netty framework in the embodiment of the present invention. Please refer to Figure 4 , an embodiment of a data processing device based on the Netty framework in an embodiment of the present invention includes:
[0166] The first distribution module 401 is used to obtain the data to be sent on the server, and distribute the data to be sent to multiple threads for execution;
[0167] The encoding module 402 is used to encode the data to be sent according to a preset SBE protocol to obtain an SBE message;
[0168] The first sending module 403 is configured to send the SBE message to the client.
[0169] Through the implementation of the above-mentioned device, by obtaining the data to be sent on the server, distributing the data to be sent to multiple threads for execution, encoding the data to be sent according to the preset SBE protocol, obtaining an SBE message, and sending the SBE message to the client; by monitoring the data and distributing it to multiple threads for execution, processing and receiving operations are performed, and the SBE message in the received data is decoded according to the preset SBE protocol to obtain decoded data, and the decoded data is sent to the server; in the above, based on the Netty framework, the sending and receiving processing of data is distributed to multiple threads for execution, thereby reducing the delay in the data transmission process; in the data transmission process, the preset SBE protocol is used for encoding and decoding, thereby improving the efficiency and reliability of data transmission.
[0170] See also Figure 5 Another embodiment of the data processing device based on the Netty framework in the embodiment of the present invention includes:
[0171] The first distribution module 401 is used to obtain the data to be sent on the server, and distribute the data to be sent to multiple threads for execution;
[0172] The encoding module 402 is used to encode the data to be sent according to a preset SBE protocol to obtain an SBE message;
[0173] A first sending module 403, configured to send the SBE message to the client;
[0174] A monitoring module 404 is used to monitor data sent by multiple clients;
[0175] The second distribution module 405 is used to distribute the monitored data to multiple threads for execution;
[0176] The receiving and processing module 406 is used to process and receive the monitored data on the corresponding thread;
[0177] The decoding module 407 is used to decode the SBE message in the received data according to the preset SBE protocol to obtain decoded data;
[0178] The second sending module 408 is used to send the decoded data to the server.
[0179] Wherein, the first distribution module 401 includes:
[0180] The acquisition unit 4011 is used to acquire data with a sending mark on the server to obtain data to be sent;
[0181] The queue generating unit 4012 is used to divide the data to be sent into multiple segments according to the storage rule of the message queue to form a queue of messages to be processed;
[0182] The thread sorting unit 4013 is used to obtain at least two threads with idle resources on the server, and sort the at least two threads according to priority to obtain a thread sequence;
[0183] The polling judgment unit 4014 is used to poll the pending message queue in sequence through each thread in the thread sequence to obtain the resource requirements in the pending message queue, and judge whether the idle resources on the corresponding thread meet the resource requirements;
[0184] In this embodiment, the polling determination unit 4014 is specifically used for:
[0185] Poll the pending message queue to determine whether there is the pending message in the message queue; if there is the pending message in the message queue, obtain the event type corresponding to the pending message, where different event types correspond to different resource requirements; call the selectNow() method of Selector to trigger the selection operation of Selector, and return the ready channel set if there is a ready channel, and return 0 if there is no ready channel; according to the ready channel set, read the SelectionKey value of the ready channel in turn to obtain the event type corresponding to the ready channel; if the event type corresponding to the ready channel contains the event type corresponding to the pending message, determine whether the idle resources on the corresponding thread meet the resource requirements, otherwise determine that they do not meet;
[0186] A first distribution unit 4015 is used to distribute the fragments to corresponding threads for execution until all fragments are distributed;
[0187] In this embodiment, the encoding module 402 includes:
[0188] A first construction unit 4021 is used for constructing a channel through the thread and building a pipeline corresponding to the channel after the segments into which the to-be-sent data are divided are distributed to corresponding threads for execution;
[0189] The encoding unit 4022 is used to encode the fragment according to the preset SBE protocol on the Handler chain on the pipeline to obtain multiple SBE message fragments, and merge the multiple SBE message fragments to obtain an SBE message;
[0190] In this embodiment, the first sending module 403 includes:
[0191] The second construction unit 4031 is used for creating a subchannel for processing a write event through the thread after the segments into which the to-be-sent data is divided are distributed to corresponding threads for execution;
[0192] The sending unit 4032 is used to call the call chain on the subchannel to send the SBE message to the server, wherein the call chain includes at least one Handler.
[0193] In this embodiment, the receiving processing module 406 includes:
[0194] The encapsulation unit 4061 is used for encapsulating the monitored data into a to-be-processed event after the monitored data is distributed to the threads in the main thread pool for execution;
[0195] A judging unit 4062 is used to judge whether the event to be processed is a connection event through threads in the main thread pool;
[0196] A connection processing unit 4063, used for processing connection events on the thread;
[0197] The event delivery unit 4064 is used to deliver the event to the slave thread pool.
[0198] Through the implementation of the above-mentioned device, by obtaining the data to be sent on the server, distributing the data to be sent to multiple threads for execution, encoding the data to be sent according to the preset SBE protocol, obtaining an SBE message, and sending the SBE message to the client; by monitoring the data and distributing it to multiple threads for execution, processing and receiving operations are performed, and the SBE message in the received data is decoded according to the preset SBE protocol to obtain decoded data, and the decoded data is sent to the server; in the above, based on the Netty framework, the sending and receiving processing of data is distributed to multiple threads for execution, thereby reducing the delay in the data transmission process; in the data transmission process, the preset SBE protocol is used for encoding and decoding, thereby improving the efficiency and reliability of data transmission.
[0199] See also Figure 6 , an embodiment of the computer device in the embodiment of the present invention is described in detail below from the perspective of hardware processing.
[0200] Figure 6 6 is a schematic diagram of the structure of a computer device provided by an embodiment of the present invention. The computer device 600 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 610 (for example, one or more processors) and a memory 620, and one or more storage media 630 (for example, one or more mass storage devices) storing application programs 633 or data 632. Among them, the memory 620 and the storage medium 630 can be short-term storage or permanent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the computer device 600. Furthermore, the processor 610 can be configured to communicate with the storage medium 630 to execute a series of instruction operations in the storage medium 630 on the computer device 600.
[0201] The computer device 600 may also include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input and output interfaces 660, and / or one or more operating systems 631, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 6 The computer device structure shown does not constitute a limitation on the computer device provided in the present application, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0202] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the above-mentioned data processing method based on the Netty framework.
[0203] In practical applications, the above-mentioned methods can be implemented based on artificial intelligence technology, where artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. It can be specifically executed based on a server, which can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.
[0204] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0205] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.
[0206] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A data processing method based on a Netty framework, wherein the Netty framework is provided with a plurality of threads executed in parallel, characterized in that: The data processing method comprises: Obtaining data to be sent on the server, and distributing the data to be sent to multiple threads for execution; The thread encodes the corresponding data to be sent according to a preset SBE protocol to obtain an SBE message; Sending the SBE message to the client; The encoding process of the corresponding data to be sent according to the preset SBE protocol through the thread to obtain an SBE message includes: After the data to be sent is divided into segments and distributed to corresponding threads for execution, a channel is constructed through the threads, and a pipeline corresponding to the channel is constructed; The fragments are encoded according to a preset SBE protocol on a Handler chain on the pipeline to obtain a plurality of SBE message fragments, wherein the Handler chain includes at least one Handler; The multiple SBE message fragments are merged to obtain an SBE message.
2. The data processing method based on the Netty framework according to claim 1 is characterized in that: The obtaining of the data to be sent on the server and distributing the data to be sent to multiple threads for execution includes: Get the data marked as sent on the server to get the data to be sent; According to the storage rules of the message queue, the data to be sent is divided into multiple segments to form a queue of messages to be processed; Acquire at least two threads with idle resources on the server, and sort the at least two threads according to priority to obtain a thread sequence; By polling the to-be-processed message queue in sequence through each thread in the thread sequence, resource requirements in the to-be-processed message queue are obtained, and it is determined whether the idle resources on the corresponding thread meet the resource requirements; If so, the fragments are distributed to corresponding threads for execution until all fragments are distributed.
3. The data processing method based on the Netty framework according to claim 1 is characterized in that: The sending of the SBE message to the client includes: After the fragments into which the data to be sent are divided are distributed to corresponding threads for execution, a subchannel for processing write events is created through the threads; The call chain on the subchannel is called to send the SBE message to the client, wherein the call chain includes at least one Handler.
4. The data processing method based on the Netty framework according to claim 2 is characterized in that: The method of sequentially polling the to-be-processed message queue through each thread in the thread sequence to obtain resource requirements in the to-be-processed message queue, and determining whether idle resources on the corresponding thread meet the resource requirements, includes: Polling the pending message queue to determine whether there is the pending message in the message queue; If there is the to-be-processed message in the message queue, obtaining the event type corresponding to the to-be-processed message, wherein different event types correspond to different resource requirements; Call the selectNow() method of Selector to trigger the selection operation of Selector. If there is a ready channel, it returns the ready channel set. If there is no ready channel, it returns 0. According to the ready channel set, the SelectionKey values of the ready channels are read in sequence to obtain the event types corresponding to the ready channels; If the event type corresponding to the ready channel includes the event type corresponding to the to-be-processed message, it is determined whether the idle resources on the corresponding thread meet the resource requirement; otherwise, it is determined that the resource requirement is not met.
5. The data processing method based on the Netty framework according to any one of claims 1 to 4, characterized in that: The data processing method further includes: Monitor the data sent by multiple clients; Distribute the monitored data to multiple threads for execution; Processing and receiving the monitored data on the corresponding thread; Decoding the SBE message in the received data according to a preset SBE protocol to obtain decoded data; The decoded data is sent to the server.
6. The data processing method based on the Netty framework according to claim 5 is characterized in that: The processing and receiving operation of the monitored data on the corresponding thread includes: After the monitored data is distributed to the threads in the main thread pool for execution, the threads in the main thread pool encapsulate the monitored data into events to be processed; Determine whether the event to be processed is a connection event by using a thread in the main thread pool; If so, the connection event is processed on this thread; If not, the event is passed to the slave thread pool.
7. A data processing device based on the Netty framework, characterized in that: The data processing device based on the Netty framework includes: A first distribution module is used to obtain the data to be sent on the server, and distribute the data to be sent to multiple threads for execution; The encoding module is used to encode the data to be sent according to a preset SBE protocol to obtain an SBE message; A first sending module, used to send the SBE message to the client; The encoding module includes: a first construction unit, which is used to construct a channel through the thread and build a pipeline corresponding to the channel after the fragments into which the data to be sent are divided are distributed to the corresponding threads for execution; an encoding unit, which is used to encode the fragments according to the preset SBE protocol on the Handler chain on the pipeline to obtain multiple SBE message fragments, and merge the multiple SBE message fragments to obtain an SBE message.
8. The data processing device based on the Netty framework according to claim 7, characterized in that: The first distribution module includes: an acquisition unit, which is used to acquire data with a sending mark on the server to obtain data to be sent; a queue generation unit, which is used to divide the data to be sent into multiple fragments according to the storage rules of the message queue to form a queue of messages to be processed; a thread sorting unit, which is used to acquire at least two threads with idle resources on the server, and sort the at least two threads according to priority to obtain a thread sequence; a polling judgment unit, which is used to poll the queue of messages to be processed in turn through each thread in the thread sequence to obtain the resource requirements in the queue of messages to be processed, and judge whether the idle resources on the corresponding thread meet the resource requirements; a first distribution unit, which is used to distribute the fragments to the corresponding threads for execution until all fragments are distributed.
9. A computer device, characterized in that: include: A memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected via a line; The at least one processor calls the instructions in the memory so that the computer device executes each step of the data processing method based on the Netty framework as described in any one of claims 1 to 6.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data processing method based on the Netty framework as claimed in any one of claims 1 to 6 are implemented.
Citation Information
Patent Citations
Customized communication protocol and service method based on netty framework
CN111683069A
Remote data transmission method based on Netty
CN112148500A
