Real-time centerless heterogeneous communication method based on embedded linux

Abstract

The invention provides a real-time centerless heterogeneous communication method based on embedded linux. The real-time centerless heterogeneous communication method comprises the following steps: S1, setting related information of a known message consumer module in a producer module; setting a required message theme in the consumer module, and setting related information of the known message producer module; the producer module is started, the producer module notifies the consumer modules according to the configuration file and the state protocol after being started, and the consumer modules obtain the message production capacity, the deployment position, the support communication mode and the message length content of the producer module according to the state notification protocol. According to the embedded linux-based real-time centerless heterogeneous communication method provided by the invention, a heterogeneous communication mode is adopted in the aspect of real-time performance, a communication mode can be selected more flexibly according to an application mode, the minimum transmission delay of data is ensured, and zero copy and other processing modes closer to bottom hardware are adopted in a data forwarding processing scheme.

Description

Technical Field

[0001] The invention relates to the field of centerless heterogeneous communication, and in particular to a real-time centerless heterogeneous communication method based on embedded Linux. Background Art

[0002] With the rapid development of edge computing, software atomic development and effective decoupling of software, a real-time, decentralized, multi-channel message publishing technology is becoming increasingly important. From device interconnection to communication between atomic component information sharing, real-time message publishing technology plays a key role. However, for some specific application scenarios, such as implementing real-time, decentralized, heterogeneous communication message publishing technology on the armlinux platform, there are still challenges.

[0003] Currently, the message sending technology available on the ARM-based Linux system generally adopts a service architecture, such as Kafka technology, which is implemented by building a broker service on the central device to associate producer and consumer data.

[0004] Broker service is mainly used for message storage and message distribution:

[0005] Message storage: Broker stores received messages persistently on disk to ensure data reliability and the ability to process large amounts of data;

[0006] Message distribution: It is responsible for delivering messages to consumers who have subscribed to the corresponding topics, maintaining partition copies of messages, and distributing messages to corresponding partitions according to rules.

[0007] Ka'fka uses a TCP-based communication protocol for communication between producers, consumers, and brokers.

[0008] Since Kafka relies on Zookeeper services, the entire system is huge, which makes deployment and management more complicated, and has high requirements on hardware performance. It is not friendly to Linux systems under the arm architecture and may be limited by hardware performance in embedded arm Linux scenarios. The Kafka broker mechanism needs to determine a central service. When deploying at the edge, an additional central service needs to be specified. If the central service fails, the entire system may fail.

[0009] Therefore, it is necessary to provide a real-time centerless heterogeneous communication method based on embedded Linux to solve the above technical problems. Summary of the invention

[0010] The present invention provides a real-time centerless heterogeneous communication method based on embedded Linux, which solves the problem that a kafkabroker mechanism needs to determine a central service and an additional central service needs to be specified when edge deployment is performed. If an abnormality occurs in the central service, the entire system may also become abnormal.

[0011] In order to solve the above technical problems, the present invention provides a real-time centerless heterogeneous communication method based on embedded Linux, comprising the following steps:

[0012] S1. Set the relevant information of its known message consumer module in the producer module;

[0013] S2. Set the required message topic in the consumer module and set the known message producer module related information;

[0014] S3. Start the producer module. After the producer module is started, it notifies each consumer module according to the configuration file and the status protocol. The consumer module obtains the producer module's message production capacity, deployment location, supported communication mode, and message length content according to the status notification protocol.

[0015] S4. Start the consumer module. After the consumer module is started, it obtains the message production capacity, deployment location, supported communication mode, message length, etc. of each producer module according to the configuration file and the capability query protocol;

[0016] S5. After the consumer module is started, the consumer module determines whether to use socket communication based on the acquired producer capability information and the producer consumer deployment location;

[0017] S6. After the producer and consumer modules establish the data channel, the message management module distributes and forwards the data.

[0018] Preferably, S3 and S4 are intended to solve the problem of the startup sequence of producers and consumers.

[0019] Preferably, S5 comprises the following steps:

[0020] S51, if producers and consumers are on different platforms, socket communication is required;

[0021] S52. If the producer and consumer are on the same platform, they need to determine the amount of data to be transmitted.

[0022] Preferably, the S52 comprises the following steps:

[0023] S521, when the amount of data to be transmitted is large, a shared memory method is used for communication;

[0024] S522. When the amount of data to be transmitted is small, a pipeline method is used for communication.

[0025] Preferably, the implementation of the message management module in S6 includes the following steps:

[0026] S61, the message management module is deployed together with the producer module, and after the message communication channel is established, the zero copy technology is adopted;

[0027] S62, using a data memory cache mechanism to cache data that needs to be cached using a memory database mechanism to avoid delays caused by disk data reading and writing;

[0028] S63. Keep the established data channel connected in real time, and directly map the data to the data sending channel through zero copy after the data is generated, so as to achieve zero message response.

[0029] Preferably, the production data in S61 is then processed using C++ zero-copy technology.

[0030] Preferably, zero copy, memory buffer, and message zero response mechanism are adopted in S61, S62, and S63 to ensure low message latency.

[0031] Preferably, the S1 and the S2 flexibly adapt to local communication and cross-device communication through heterogeneous communication to ensure minimum delay in data transmission.

[0032] Compared with the related art, the real-time centerless heterogeneous communication method based on embedded Linux provided by the present invention has the following beneficial effects:

[0033] The present invention provides a real-time centerless heterogeneous communication method based on embedded Linux. Through the operation steps of S1, S2, S3, S4, S5 and S6, a heterogeneous communication method can be adopted in terms of real-time performance, and a communication method can be more flexibly selected according to the application mode, thereby ensuring the minimum data transmission delay. In the data forwarding processing scheme, a processing method closer to the underlying hardware such as zero copy is adopted. In terms of processing mechanism, the data transmission delay is guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A structural schematic diagram of a preferred embodiment of a real-time centerless heterogeneous communication method based on embedded Linux provided by the present invention;

[0035] Figure 2 A schematic diagram of the data generation capabilities for publishing production modules;

[0036] Figure 3 To obtain a schematic diagram of the data generation capabilities of the production module;

[0037] Figure 4 A schematic diagram established for the communication method. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0039] Please refer to Figure 1 , Figure 2 , Figure 3 and Figure 4 ,in, Figure 1 A structural schematic diagram of a preferred embodiment of a real-time centerless heterogeneous communication method based on embedded Linux provided by the present invention; Figure 2 A schematic diagram of the data generation capabilities for publishing production modules; Figure 3 To obtain a schematic diagram of the data generation capabilities of the production module; Figure 4 A real-time centerless heterogeneous communication method based on embedded Linux includes the following steps:

[0040] S1. Set the relevant information of its known message consumer module in the producer module;

[0041] S2. Set the required message topic in the consumer module and set the known message producer module related information;

[0042] S3. Start the producer module. After the producer module is started, it notifies each consumer module according to the configuration file and the status protocol. The consumer module obtains the producer module's message production capacity, deployment location, supported communication mode, and message length content according to the status notification protocol.

[0043] S4. Start the consumer module. After the consumer module is started, it obtains the message production capacity, deployment location, supported communication mode, message length, etc. of each producer module according to the configuration file and the capability query protocol;

[0044] S5. After the consumer module is started, the consumer module determines whether to use socket communication based on the acquired producer capability information and the producer consumer deployment location;

[0045] S6. After the producer and consumer modules establish the data channel, the message management module distributes and forwards the data.

[0046] The S3 and S4 are intended to solve the problem of the startup sequence of producers and consumers.

[0047] The S5 comprises the following steps:

[0048] S51, if producers and consumers are on different platforms, socket communication is required;

[0049] S52. If the producer and consumer are on the same platform, they need to determine the amount of data to be transmitted.

[0050] The S52 comprises the following steps:

[0051] S521, when the amount of data to be transmitted is large, a shared memory method is used for communication;

[0052] S522. When the amount of data to be transmitted is small, a pipeline method is used for communication.

[0053] The implementation of the message management module in S6 includes the following steps:

[0054] S61, the message management module is deployed together with the producer module, and after the message communication channel is established, the zero copy technology is adopted;

[0055] S62, using a data memory cache mechanism to cache data using a memory database mechanism to avoid delays caused by disk data reading and writing;

[0056] S63. Keep the established data channel connected in real time, and directly map the data to the data sending channel through zero copy after the data is generated, so as to achieve zero message response.

[0057] The production data in S61 is then processed using C++ zero-copy technology.

[0058] Zero-copy technology in C++ is a technology that prevents the CPU from copying data from one storage unit to another. The main idea is to prevent the CPU from being fully responsible for writing data in memory to other components, but only plays a management role. There are many specific ways to implement zero-copy technology, such as sendfile, mmap, splice, direct I / O, etc. Different zero-copy technologies are suitable for different application scenarios.

[0059] In C++, some specific functions or methods can be used to implement zero copy technology, such as mmap and sendfile. mmap is a memory mapping file mechanism provided by Linux. It realizes the mapping of the read buffer address in the kernel with the user space buffer address, thereby realizing the sharing of the kernel buffer and the user buffer, reducing the CPU copy between user state and kernel state. The sendfile system call was introduced in Linux kernel version 2.1. It establishes a transmission channel between two files and can replace the read / write system call at one time. By using DMA technology and passing file descriptors, zero copy is realized.

[0060] Zero-copy technology can improve performance in scenarios such as network transmission and file reading and writing, and reduce CPU overhead and memory bandwidth usage. However, the specific usage and effects will vary depending on the specific application requirements and system environment. In actual applications, it is necessary to select the appropriate zero-copy technology according to the specific situation and make appropriate optimizations and adjustments.

[0061] The zero copy, memory buffer and message zero response mechanism are adopted in S61, S62 and S63 to ensure low message delay.

[0062] The S1 and the S2 flexibly adapt to local communication and cross-device communication through heterogeneous communication to ensure minimum delay in data transmission.

[0063] Establish a ring queue: divide the ring queue into a write queue and a read queue;

[0064] Receiving target data: receiving target data uploaded by heterogeneous systems in a multiplexing method through a selector;

[0065] Initialize event instance: Initialize the event instance written into the empty queue in the queue and convert it into a public action;

[0066] Write target data: write the target data into the empty queue, and create the corresponding actual event action in combination with the public action and the target data, and update the write queue and the read queue at the same time;

[0067] Processing actual event actions: reading the actual event actions in the queue by calling the idle thread in the thread pool, and controlling the event processor to process them. After processing, the corresponding actual event actions are restored to public actions;

[0068] The main purpose of the central heterogeneous communication method is to reduce the resource usage of the receiving end while reducing the probability of request failure, quickly process the received target data with a limited and controllable memory size, and improve the data processing speed.

[0069] When implementing the central heterogeneous communication method, you can set up the ring queue and selector by following the steps below:

[0070] Create a ring queue: Create a ring queue and divide it into a write queue and a read queue;

[0071] Set the head identifier and tail identifier: embed the head identifier and tail identifier set at intervals in the ring queue;

[0072] Adjusting the positions of the head identifier and the tail identifier: updating the write queue and the read queue in a circular manner by adjusting the positions of the head identifier and the tail identifier;

[0073] Use of selectors: Use selectors to receive target data uploaded by heterogeneous systems in a multiplexed manner.

[0074] The working principle of the real-time centerless heterogeneous communication method based on embedded Linux provided by the present invention is as follows:

[0075] When in use, set the relevant information of the known message consumer modules in the producer module, set the required message topic in the consumer module, and set the relevant information of the known message producer modules, start the producer module, and after the producer module is started, according to the configuration file, notify each consumer module according to the status protocol, the consumer module obtains the message production capacity, deployment location, supported communication mode, message length content of the producer module according to the status notification protocol, and starts the consumer module. After the consumer module is started, according to the configuration file, according to the capability query protocol, obtains the message production capacity, deployment location, supported communication mode, message length and other contents of each producer module. After the consumer module is started, the consumer module determines whether to use socket communication based on the obtained producer capability information and the producer consumer deployment location. After the producer and consumer modules establish a data channel, the message management module distributes and forwards data.

[0076] Compared with the related art, the real-time centerless heterogeneous communication method based on embedded Linux provided by the present invention has the following beneficial effects:

[0077] The present invention provides a real-time centerless heterogeneous communication method based on embedded Linux. Through the operation steps of S1, S2, S3, S4, S5 and S6, a heterogeneous communication method can be adopted in terms of real-time performance, and a communication method can be more flexibly selected according to the application mode, thereby ensuring the minimum data transmission delay. In the data forwarding processing scheme, a processing method closer to the underlying hardware such as zero copy is adopted. In terms of processing mechanism, the data transmission delay is guaranteed.

[0078] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A real-time centerless heterogeneous communication method based on embedded Linux, characterized in that: The following steps are involved: S1. Set the relevant information of its known message consumer module in the producer module; S2. Set the required message topic in the consumer module and set the known message producer module related information; S3. Start the producer module. After the producer module is started, it notifies each consumer module according to the configuration file and the status protocol. The consumer module obtains the producer module's message production capacity, deployment location, supported communication mode, and message length content according to the status notification protocol. S4. Start the consumer module. After the consumer module is started, it obtains the message production capacity, deployment location, supported communication mode, message length, etc. of each producer module according to the configuration file and the capability query protocol; S5. After the consumer module is started, the consumer module determines whether to use socket communication based on the acquired producer capability information and the producer consumer deployment location; S6. After the producer and consumer modules establish the data channel, the message management module distributes and forwards the data.

2. The method for real-time centerless heterogeneous communication based on embedded Linux according to claim 1 is characterized in that: The S3 and S4 are intended to solve the problem of the startup sequence of producers and consumers.

3. The method for real-time centerless heterogeneous communication based on embedded Linux according to claim 1 is characterized in that: The S5 comprises the following steps: S51, if producers and consumers are on different platforms, socket communication is required; S52. If the producer and consumer are on the same platform, they need to determine the amount of data to be transmitted.

4. The method for real-time centerless heterogeneous communication based on embedded Linux according to claim 3 is characterized in that: The S52 comprises the following steps: S521, when the amount of data to be transmitted is large, a shared memory method is used for communication; S522. When the amount of data to be transmitted is small, a pipeline method is used for communication.

5. The method for real-time centerless heterogeneous communication based on embedded Linux according to claim 1 is characterized in that: The implementation of the message management module in S6 includes the following steps: S61, the message management module is deployed together with the producer module, and after the message communication channel is established, the zero copy technology is adopted; S62, using a data memory cache mechanism to cache data using a memory database mechanism to avoid delays caused by disk data reading and writing; S63. Keep the established data channel connected in real time, and directly map the data to the data sending channel through zero copy after the data is generated, so as to achieve zero message response.

6. The method for real-time centerless heterogeneous communication based on embedded Linux according to claim 5 is characterized in that: The production data in S61 is then processed using C++ zero-copy technology.

7. The method for real-time centerless heterogeneous communication based on embedded Linux according to claim 5 is characterized in that: The zero copy, memory buffer and message zero response mechanism are adopted in S61, S62 and S63 to ensure low message delay.

8. The method for real-time centerless heterogeneous communication based on embedded Linux according to claim 1, characterized in that: The S1 and the S2 flexibly adapt to local communication and cross-device communication through heterogeneous communication to ensure minimum delay in data transmission.

Images