A communication method and system based on shared memory for communication middleware

Abstract

The present invention discloses a communication method and system based on shared memory for communication middleware. By utilizing shared memory, efficient end-to-end communication between publishers and subscribers is achieved. The method comprises the following steps: the publisher creates a shared memory area through a predefined channel identifier, writes data into the memory, and simultaneously sends notification information through a global notification queue; the subscriber locates the target memory through an index structure and reads the data. The present invention designs mechanisms such as memory state control, read-write mutual exclusion, and capacity rule optimization to ensure efficient use of shared memory and security of inter-process access. In addition, the receiving process implements real-time monitoring and data distribution through a distribution module, supporting complex communication scenarios with multiple publishers and multiple subscribers. Compared with the existing technology, the present invention has the advantages of high transmission efficiency, low latency, and good concurrency performance, and is suitable for the design and implementation of high-performance communication middleware.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a communication method and system based on shared memory and oriented to communication middleware. Background Art

[0002] With the development of modern computer technology and network communication technology, multi-process communication mechanisms have been widely used in distributed systems, real-time data processing, and high-performance computing. However, traditional multi-process communication methods (such as those based on message queues, sockets, or remote procedure calls) face many challenges in efficiency and performance. Specifically:

[0003] Bottlenecks of traditional communication methods: (1) Communication methods based on message queues require frequent data copying and context switching, which increases system overhead and makes it difficult to meet the requirements of high throughput and low latency; (2) Communication methods based on network protocols (such as TCP / IP) require data to be transmitted across hosts. Although suitable for distributed scenarios, they are too complex and inefficient for communication between multiple processes on a single host; (3) Although the file system's method of sharing data is simple, it is limited by the high latency and low concurrency of file IO operations and cannot meet the requirements of real-time communication.

[0004] Advantages and Challenges of Shared Memory: As a method of directly using memory for data exchange, shared memory has the characteristics of high data transmission efficiency and low latency, and is particularly suitable for multi-process communication within a single host. However, the communication mechanism based on shared memory also faces the following problems: (1) It is necessary to design an efficient memory management strategy to avoid memory fragmentation and resource waste; (2) Data consistency and concurrent access security are key to achieving efficient communication, and traditional methods lack a common design paradigm; (3) There is a lack of a unified notification mechanism to achieve real-time data distribution between publishers and subscribers.

[0005] Communication middleware requirements: Currently, communication middleware plays a crucial role in message routing, load balancing, and data transmission in distributed systems. The efficiency of its underlying communication mechanisms directly determines the overall performance of the system. Therefore, designing an efficient and reliable shared memory-based communication mechanism is crucial for optimizing the performance of communication middleware.

[0006] Based on the above background, the present invention proposes a communication method and system based on shared memory for communication middleware. Summary of the Invention

[0007] To address the above technical issues, the present invention provides a shared memory-based communication method and system for communication middleware to address the low efficiency, high latency, and insufficient concurrency performance of existing multi-process communication methods. This invention innovatively proposes a low-level communication mechanism based on shared memory. By designing memory management, data transfer, and notification mechanisms, it achieves efficient and reliable end-to-end data communication, meeting the requirements of high throughput, low latency, and multiple concurrent accesses.

[0008] The technical solution adopted by the present invention to solve the technical problem is:

[0009] A communication method based on shared memory for communication middleware, the method comprising the following steps:

[0010] S100: Bind a channel identifier channel_id to each publisher. Subscribers register their channel_id with Segmentmap. The publisher creates a shared memory area segment between the publisher and subscriber based on the channel_id. Design a dynamic allocation strategy based on capacity rules to dynamically adjust the structure and capacity of the memory area according to the message size.

[0011] S200: The publisher completes data writing by obtaining the available block and buffer indexes in the segment. The write operation process controls read-write mutual exclusion and uses a circular rotation index mechanism to manage the write position.

[0012] S300: Create a global shared memory area Indicator on the same host as a notification queue to store the publisher's data write notification information ReadableInfo. ReadableInfo contains the host identifier host_id, channel_id, and Block index, so that subscribers can quickly locate data.

[0013] S400: Design the distribution module of the receiving process, and monitor the ReadableInfo in the global shared memory area in real time through an independent thread. The subscriber locates the target Segment and Block in the SegmentMap according to the channel_id in the ReadableInfo to achieve efficient data reading and distribution.

[0014] Preferably, S100 includes:

[0015] S110: When a publisher sends data, it is bound to a specific channel_id. A shared memory area Segment is created based on this channel_id. Each memory area includes a State, multiple Blocks, and corresponding Buffers. The State is used to control the shared state between the current Segment processes, the Block is used to store the message body data, and the Buffer is used to store metadata information.

[0016] S120: The subscriber follows at least one channel_id. When the subscriber starts, it registers the channel_id it follows with SegmentMap. SegmentMap stores the index relationship between the channel_id and the corresponding shared memory segment.

[0017] S130: Design a dynamic allocation strategy based on capacity rules to dynamically adjust the structure and capacity of the segment according to the message size.

[0018] Preferably, the rules in S130 include:

[0019] For messages between 0 and 10 KB, the Segment structure and capacity are set as follows: the number of blocks is 512, and the size of each message is 16 KB.

[0020] For messages between 10 and 100 KB, the Segment structure and capacity are set as follows: the number of blocks is 128, and the size of each message is 128 KB.

[0021] For messages between 100kb and 1MB, the segment structure and capacity are set to: 64 blocks, and 1MB per message.

[0022] For messages between 1MB and 6MB, the segment structure and capacity are set as follows: the number of blocks is 32, and the size of each message is 6MB.

[0023] For messages between 6MB and 10MB in size, the Segment structure and capacity are set as follows: the number of blocks is 16, and the size of each message is 16MB.

[0024] For messages larger than 10MB, the Segment structure and capacity are set as follows: the number of blocks is 8 and the size of each message is 32MB.

[0025] Preferably, the Segment State includes four atomic variables: need_remap_ is used to control whether the current Segment needs to be remapped, ceiling_msg_size_ represents the size of the message data, that is, the number of bytes in a buffer, seq_ represents the index of the block currently being written. The sender writes a block and a buffer each time it sends a message. After each data is written, seq+1 is added. reference_count_ represents the number of users using this memory.

[0026] Block includes three atomic variables. lock_num_ is an atomic variable that controls the read and write mutual exclusion of the buffer corresponding to the Block, and is used to achieve safe reading and writing between processes. msg_size_ and msg_info_size_ are the lengths of the message.

[0027] Preferably, S200 includes:

[0028] S210: Before publishing data, the publisher creates a segment based on the type and size of the data structure to be published and the rules. In the Linux environment, shared memory can be created through System V IPC or POSIX IPC.

[0029] S220: Obtain a writable block index based on the state variable of the segment state. The initial value of the state variable is 0. Each time a writable block is obtained, the state variable is incremented by 1. If the state variable value exceeds the number of blocks in the current segment, the state variable is rotated through the modulo operation to limit the state variable value to between 0 and the number of blocks.

[0030] S230: Loop to obtain writable blocks until a block with lock_num_ equal to 0 is found, and then return the index of the block. lock_num_ is an atomic variable used to control the read-write mutual exclusion of the block. The initial value is 0, indicating that the block is not occupied.

[0031] S240: Write the data into the buffer corresponding to the writable block. After the writing is completed, release the block by setting lock_num_ to 0 to ensure that other processes can continue to use the block.

[0032] Preferably, S230 includes:

[0033] When a process attempts to write to a block, it atomically checks the value of lock_num_. If lock_num_ is 0, it sets it to a negative number, indicating that the block is occupied. If lock_num_ is negative, it means that the block is occupied by another process and the current process cannot write to it. It continues to search for other writable blocks until it finds a block with lock_num_ of 0 and returns the index of the block.

[0034] Preferably, S300 includes:

[0035] S310: When the first publisher process starts, a globally unique shared memory is created to store the indicator. The indicator is a queue-shaped data structure used to store ReadableInfo. Each ReadableInfo corresponds to an index stored in the seq array at the end of the indicator. The next_seq at the head of the indicator is an atomic variable that represents the index of the next writable ReadableInfo.

[0036] S320: The publisher writes data to the Block and Buffer in the Segment corresponding to a channel_id and requests a writable ReadableInfo index from the Indicator;

[0037] S330: Before writing ReadableInfo, the publisher atomically increases next_seq by 1. The publisher writes ReadableInfo to the position corresponding to next_seq in Indicator. The subscriber obtains the notification through Indicator and reads the data. Among them, ReadableInfo contains the host identifier, channel identifier and the Block index where the data is located.

[0038] Preferably, S400 includes:

[0039] S410: Design the dispatcher module of the receiving process, which is used to read the ReadableInfo on the indicator and distribute the data. Dispatch starts an independent thread inside. The thread body is an infinite loop function. The thread content stores a local seq index separately, with an initial value of 0.

[0040] S420: The thread loop function cyclically reads the next_seq in the indicator header and compares it with the local seq. If next_seq is not equal to the local seq, it means that another publisher has published data and written ReadableInfo. The seq array of the indicator is indexed according to the local seq, the new seq value is obtained, the local seq value is updated, and the corresponding ReadableInfo is read according to the seq.

[0041] S430: Compare channel_id to determine whether the current publisher process contains a subscriber of this channel_id. If so, index Segment according to channel_id, and then read the data written by the publisher according to block_index in ReadableInfo.

[0042] A communication system based on shared memory for communication middleware, including a shared memory area creation module, a data writing module, a global shared memory area creation module and a distribution module.

[0043] Shared memory area creation module: A channel identifier channel_id is bound to each publisher. Subscribers register their channel_id with Segmentmap. The publisher creates a shared memory area segment between the publisher and subscriber based on the channel_id. A dynamic allocation strategy based on capacity rules is designed to dynamically adjust the structure and capacity of the memory area according to the message size.

[0044] Data writing module: The publisher completes data writing by obtaining the available block and buffer indexes in the segment. The writing operation process controls the read-write mutual exclusion and uses a circular rotation index mechanism to manage the writing position.

[0045] Global shared memory area creation module: Creates a global shared memory area Indicator on the same host as a notification queue, stores publisher data and writes notification information ReadableInfo. ReadableInfo contains the host identifier host_id, channel_id, and Block index, allowing subscribers to quickly locate data.

[0046] Distribution module: The receiving process monitors the ReadableInfo in the global shared memory area in real time through an independent thread. The subscriber locates the target Segment and Block in the SegmentMap based on the channel_id in the ReadableInfo to achieve efficient data reading and distribution.

[0047] The above-mentioned shared memory-based communication method and system for communication middleware incorporates mechanisms such as memory state control, read-write mutual exclusion, and capacity rule optimization to ensure efficient shared memory utilization and secure inter-process access. Furthermore, the receiving process implements real-time monitoring and data distribution through the dispatcher module, supporting complex communication scenarios with multiple publishers and subscribers. Compared to existing technologies, this invention offers the advantages of high transmission efficiency, low latency, and excellent concurrency performance, making it suitable for the design and implementation of high-performance communication middleware. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flow chart of a communication method based on shared memory for communication middleware in one embodiment of the present invention;

[0049] Figure 2 A communication flow chart of a communication method based on shared memory for communication middleware in one embodiment of the present invention;

[0050] Figure 3 Schematic diagram of the memory structure of a Segment in one embodiment of the present invention;

[0051] Figure 4 A schematic diagram of the structure of State in one embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram of the structure of Block in one embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram of the structure of an indicator in one embodiment of the present invention;

[0054] Figure 7 Schematic diagram of the structure of ReadableInfo in one embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below with reference to the accompanying drawings.

[0056] In one embodiment, Figure 1 As shown, a communication method based on shared memory for communication middleware includes the following steps:

[0057] S100: Bind a channel identifier channel_id to each publisher. Subscribers register their channel_id with Segmentmap. The publisher creates a shared memory area segment between the publisher and subscriber based on the channel_id. Design a dynamic allocation strategy based on capacity rules to dynamically adjust the structure and capacity of the memory area according to the message size.

[0058] S200: The publisher completes data writing by obtaining the available block and buffer indexes in the segment. During the write operation, the lock_num_ field is used to control read-write mutual exclusion to ensure the security of concurrent access. The circular rotation index mechanism (seq_ variable) is used to manage the write position to avoid resource contention and deadlock.

[0059] S300: Create a global shared memory area Indicator on the same host as a notification queue to store the publisher's data write notification information ReadableInfo. ReadableInfo contains the host identifier host_id, channel_id, and Block index, so that subscribers can quickly locate data.

[0060] S400: Design the distribution module of the receiving process, and monitor the ReadableInfo in the global shared memory area in real time through an independent thread. The subscriber locates the target Segment and Block in the SegmentMap according to the channel_id in the ReadableInfo to achieve efficient data reading and distribution.

[0061] Specifically, in the communication middleware, the data publisher sends data to the data subscriber. In order to achieve end-to-end transmission, a topic or channel is usually set up for the identification and authentication of both parties. Based on this communication model, the present invention proposes an end-to-end communication mechanism based on shared memory. The communication process is as follows: Figure 2 shown.

[0062] In one embodiment, S100 includes:

[0063] S110: When a publisher sends data, it is bound to a specific channel_id. A shared memory area Segment is created based on this channel_id. Each memory area includes a State, multiple Blocks, and corresponding Buffers. The State is used to control the shared state between the current Segment processes, the Block is used to store the message body data, and the Buffer is used to store metadata information.

[0064] S120: The subscriber follows at least one channel_id. When the subscriber starts, it registers the channel_id it follows with SegmentMap. SegmentMap stores the index relationship between the channel_id and the corresponding shared memory segment.

[0065] S130: Design a dynamic allocation strategy based on capacity rules to dynamically adjust the structure and capacity of the segment according to the message size.

[0066] Specifically, end-to-end communication involves communication between a publisher in one process and a subscriber in another process. Because shared memory communication is used, both processes reside on the same host. When a publisher sends data, it binds it to a specific channel_id and creates a shared memory segment based on this channel_id. This segment is defined as a segment. Each subscriber has its own channel_id, and a SegmentMap is designed in the receiving process:<channel_id, Segment> The data structure of SegmentMap stores the index relationship between channel_id and the corresponding shared memory Segment. When the subscriber obtains the data on the corresponding channel_id, it will first register with SegmentMap. Subsequently, the subscriber obtains the data by finding the Segment according to the channel_id in SegmentMap.

[0067] Furthermore, in the communication process described in the present invention, the shared memory created according to channel_id is defined as Segment, and the memory structure of Segment is defined as follows Figure 3 As shown:

[0068] Each segment shares the following: a State, n Blocks, and n Buffers. The State controls the shared state between the current segment processes. The memory usage of a segment is closely related to the size of the message data to be transmitted. The specific rules are defined below.

[0069] In one embodiment, the rules in S130 include:

[0070] For messages between 0 and 10 KB, the Segment structure and capacity are set as follows: the number of blocks is 512, and the size of each message is 16 KB.

[0071] For messages between 10 and 100 KB, the Segment structure and capacity are set as follows: the number of blocks is 128, and the size of each message is 128 KB.

[0072] For messages between 100kb and 1MB, the segment structure and capacity are set to: 64 blocks, and 1MB per message.

[0073] For messages between 1MB and 6MB, the segment structure and capacity are set as follows: the number of blocks is 32, and the size of each message is 6MB.

[0074] For messages between 6MB and 10MB in size, the Segment structure and capacity are set as follows: the number of blocks is 16, and the size of each message is 16MB.

[0075] For messages larger than 10MB, the Segment structure and capacity are set as follows: the number of blocks is 8 and the size of each message is 32MB.

[0076] Specifically, for example, if the size of a transmitted message is between 10 KB and 100 KB, then rule 2 applies. The rule defines each block as 1024 bytes. A segment contains 128 blocks, which corresponds to 128 buffers. A buffer consists of two parts: the message and the additional MessageInfo. The size of the MessageInfo is fixed at 1024 bytes, so the buffer size is 128 KB + 1 KB = 129 KB.

[0077] The rule also specifies that the State size of the Segment header is 1024 bytes, and the unused portion of the Segment tail occupies a size of 1024*4=4kb.

[0078] In one embodiment, Figure 4 and 5 As shown, the Segment State includes four atomic variables: need_remap_ is used to control whether the current Segment needs to be remapped, ceiling_msg_size_ represents the size of the message data, that is, the number of bytes in a buffer, seq_ represents the index of the block currently being written, and the sender writes a block and a buffer each time it sends a message. After each data is written, seq+1 is added, and reference_count_ represents the number of users using this memory.

[0079] Block includes three atomic variables. lock_num_ is an atomic variable that controls the read and write mutual exclusion of the buffer corresponding to the Block, and is used to achieve safe reading and writing between processes. msg_size_ and msg_info_size_ are the lengths of the message.

[0080] In one embodiment, S200 includes:

[0081] S210: Before publishing data, the publisher creates a segment based on the type and size of the data structure to be published and the rules. In the Linux environment, shared memory can be created through System V IPC or POSIX IPC.

[0082] S220: Obtain a writable block index based on the state variable of the segment state. The initial value of the state variable is 0. Each time a writable block is obtained, the state variable is incremented by 1. If the state variable value exceeds the number of blocks in the current segment, the state variable is rotated through the modulo operation to limit the state variable value to between 0 and the number of blocks.

[0083] S230: Loop to obtain writable blocks until a block with lock_num_ equal to 0 is found, and then return the index of the block. lock_num_ is an atomic variable used to control the read-write mutual exclusion of the block. The initial value is 0, indicating that the block is not occupied.

[0084] S240: Write the data into the buffer corresponding to the writable block. After the writing is completed, release the block by setting lock_num_ to 0 to ensure that other processes can continue to use the block.

[0085] In one embodiment, S230 includes:

[0086] When a process attempts to write to a block, it atomically checks the value of lock_num_. If lock_num_ is 0, it sets it to a negative number, indicating that the block is occupied. If lock_num_ is negative, it means that the block is occupied by another process and the current process cannot write to it. It continues to search for other writable blocks until it finds a block with lock_num_ of 0 and returns the index of the block.

[0087] Specifically, according to the communication process, one of the publishers of the sending process will write data to a channel, and the type of data structure written is also determined in advance by the sender. Each channel corresponds to a segment, so when the sender publishes data, it will first create a segment according to the type and size of the data structure to be published according to the above rules. Segment is a global shared memory. In the Linux environment, there are two ways to create shared memory, based on System V IPC and POSIX IPC. Both methods can be used.

[0088] After the Segment is created, data can be written to it. The goal of data writing is to fill data into one of the Blocks in the Segment and the corresponding Buffer, so you need to get a Buffer index in advance, and the State's seq_ records the index. The initial value of seq_ in State is 0. Each time a writable Block index is obtained, the value of seq_ will be increased by 1. If the value of seq_ exceeds the number of Blocks in the current Segment, it will rotate from the beginning, that is, use the remainder operation to limit the value of seq_ to between 0 and the number of Blocks. At the same time, obtaining the writable Block index is a loop operation. Only when it is determined that the Block corresponding to the obtained Block index is writable will this index be returned to exit the loop operation.

[0089] Block is read and written mutually exclusive based on the lock_num_ in the header. lock_num_ is an atomic variable with an initial value of 0. Therefore, reading and writing this variable are process-safe. Assuming that the current Block is writable, it will atomically check whether the value of lock_num_ is 0. If it is 0, the value of lock_num_ will be set to a negative number, indicating that a process has occupied this Block. If another process also wants to write to this Block at this time, it will first check the value of lock_num_ and find that it is a negative number, which means that it cannot be written by this process.

[0090] Through the above operations, you can get a writable Block and the corresponding Buffer from the Segment, and then write data to this Block and Buffer. After writing the data, you need to release the exclusive ownership of this Block and Buffer. Here, the value of lock_num_ of this Block is reset to 0, indicating that no process is writing data to this Block.

[0091] In one embodiment, S300 includes:

[0092] S310: When the first publisher process starts, a globally unique shared memory is created to store the indicator. The indicator is a queue-shaped data structure used to store ReadableInfo. Each ReadableInfo corresponds to an index stored in the seq array at the end of the indicator. The next_seq at the head of the indicator is an atomic variable that represents the index of the next writable ReadableInfo.

[0093] S320: The publisher writes data to the Block and Buffer in the Segment corresponding to a channel_id and requests a writable ReadableInfo index from the Indicator;

[0094] S330: Before writing ReadableInfo, the publisher atomically increases next_seq by 1. The publisher writes ReadableInfo to the position corresponding to next_seq in Indicator. The subscriber obtains the notification through Indicator and reads the data. Among them, ReadableInfo contains the host identifier, channel identifier and the Block index where the data is located.

[0095] Specifically, after writing data to a Segment, the publisher will send a notification message to indicate that the data has been written, and the subscriber can get the data from this Segment.

[0096] In order to implement the message notification mechanism between publishers and subscribers in multiple processes, a special data structure Indicator is created. When the first publishing node is started, a globally unique shared memory will be allocated to store the indicator. This indicator is shared by all publishers and subscribers on the host. The data format of the indicator is as follows: Figure 6 shown.

[0097] Indicator can be understood as a queue-shaped data structure that can store 4096 ReadableInfos. Each ReadableInfo corresponds to an index, which is written to the seq array at the end of the Indicator. Therefore, the size of the seq array is also 4096. The next_seq at the head is an atomic variable that represents the index of the next writable ReadableInfo. The data format of ReadableInfo is as follows: Figure 7As shown, host_id is the unique identifier of each host, channel_id represents the segment on which channel_id the publisher writes data, and block_index represents the block index of the segment where the data is written.

[0098] After the publisher finishes writing data to a Block and Buffer in a Segment on a channel_id, it needs to fill in a ReadableInfo. Therefore, it first needs to request it from the Indicator. The next_seq in the global Indicator header is the index of the writable ReadableInfo. Before writing the ReadableInfo, the publisher atomically increments next_seq by 1 and then writes the notification information to the obtained index.

[0099] In one embodiment, S400 includes:

[0100] S410: Design the dispatcher module of the receiving process, which is used to read the ReadableInfo on the indicator and distribute the data. Dispatch starts an independent thread inside. The thread body is an infinite loop function. The thread content stores a local seq index separately, with an initial value of 0.

[0101] S420: The thread loop function cyclically reads the next_seq in the indicator header and compares it with the local seq. If next_seq is not equal to the local seq, it means that another publisher has published data and written ReadableInfo. The seq array of the indicator is indexed according to the local seq, the new seq value is obtained, the local seq value is updated, and the corresponding ReadableInfo is read according to the seq.

[0102] S430: Compare channel_id to determine whether the current publisher process contains a subscriber of this channel_id. If so, index Segment according to channel_id, and then read the data written by the publisher according to block_index in ReadableInfo.

[0103] Specifically, there is a Dispatcher module in the subscriber process. This module is used to monitor whether the data on the Segment has been updated and written, and read the ReadableInfo information on the Indicator to distribute the data. The Dispatcher will store a local seq index separately, with an initial value of 0. The thread loop function will cyclically read the value of next_seq in the Indicator header and compare it with the local seq. If the value of next_seq is not equal to the local seq, it means that other publishers have published data and written ReadableInfo. At this time, the Indicator index seq array can be used according to the locally saved seq, and then the local seq can be updated to the value in the seq array. Then, the ReadableInfo corresponding to the local seq can be obtained. Finally, the channel_id is compared to determine whether the current publisher process contains a subscriber with this channel_id. If so, the Segment is indexed according to the channel_id, and the data written by the publisher is read according to the block_index in the ReadableInfo.

[0104] A communication system based on shared memory for communication middleware, including a shared memory area creation module, a data writing module, a global shared memory area creation module and a distribution module.

[0105] Shared memory area creation module: A channel identifier channel_id is bound to each publisher. Subscribers register their channel_id with Segmentmap. The publisher creates a shared memory area segment between the publisher and subscriber based on the channel_id. A dynamic allocation strategy based on capacity rules is designed to dynamically adjust the structure and capacity of the memory area according to the message size.

[0106] Data writing module: The publisher completes data writing by obtaining the available block and buffer indexes in the segment. The writing operation process controls the read-write mutual exclusion and uses a circular rotation index mechanism to manage the writing position.

[0107] Global shared memory area creation module: Creates a global shared memory area Indicator on the same host as a notification queue, stores publisher data and writes notification information ReadableInfo. ReadableInfo contains the host identifier host_id, channel_id, and Block index, allowing subscribers to quickly locate data.

[0108] Distribution module: The receiving process monitors the ReadableInfo in the global shared memory area in real time through an independent thread. The subscriber locates the target Segment and Block in the SegmentMap based on the channel_id in the ReadableInfo to achieve efficient data reading and distribution.

[0109] Regarding the specific definition of a shared memory-based communication system for communication middleware, please refer to the definition of a shared memory-based communication method for communication middleware above, which will not be repeated here. Each module in the above-mentioned shared memory-based communication system for communication middleware can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0110] Compared with the prior art, the above-mentioned communication method and system based on shared memory for communication middleware has the following beneficial effects:

[0111] Direct data transfer based on shared memory avoids multiple data copies and context switches, significantly improving data transmission efficiency and reducing latency. The capacity-based dynamic allocation strategy and round-robin indexing mechanism efficiently adapt to varying message sizes and communication requirements. The design of atomic variables and mutexes ensures the security and stability of concurrent multi-process access. A unified notification mechanism and real-time distribution module meet the needs of high-real-time scenarios. This invention is suitable for the design and implementation of high-performance communication middleware and can be widely used in fields such as real-time data processing, distributed system communications, and the Industrial Internet of Things.

[0112] The above is a detailed introduction to a communication method and system based on shared memory for communication middleware provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A communication method based on shared memory for communication middleware, characterized in that: The method comprises the following steps: S100: Bind a channel identifier channel_id to each publisher. Subscribers register their channel_id with Segmentmap. The publisher creates a shared memory area segment between the publisher and subscriber based on the channel_id. Design a dynamic allocation strategy based on capacity rules to dynamically adjust the structure and capacity of the memory area according to the message size. S200: The publisher completes data writing by obtaining the available block and buffer indexes in the segment. The write operation process controls read-write mutual exclusion and uses a circular rotation index mechanism to manage the write position. S200 includes: S210: Before publishing data, the publisher creates a segment based on the type and size of the data structure to be published and the rules. In the Linux environment, shared memory can be created through System V IPC or POSIX IPC. S220: Obtain a writable block index based on the state variable of the segment state. The initial value of the state variable is 0. Each time a writable block is obtained, the state variable is incremented by 1. If the state variable value exceeds the number of blocks in the current segment, the state variable is rotated through the modulo operation to limit the state variable value to between 0 and the number of blocks. S230: Loop to obtain writable blocks until a block with lock_num_ equal to 0 is found, and then return the index of the block. lock_num_ is an atomic variable used to control the read-write mutual exclusion of the block. The initial value is 0, indicating that the block is not occupied. S240: Write the data into the buffer corresponding to the writable block. After the writing is completed, release the block by setting lock_num_ to 0 to ensure that other processes can continue to use the block. S300: Create a global shared memory area Indicator on the same host as a notification queue to store the publisher's data write notification information ReadableInfo. ReadableInfo contains the host identifier host_id, channel_id, and Block index, so that subscribers can quickly locate data. S400: Design the distribution module of the receiving process, and monitor the ReadableInfo in the global shared memory area in real time through an independent thread. The subscriber locates the target Segment and Block in the SegmentMap according to the channel_id in the ReadableInfo to achieve efficient data reading and distribution.

2. The method according to claim 1, characterized in that S100 includes: S110: When a publisher sends data, it is bound to a specific channel_id. A shared memory area Segment is created based on this channel_id. Each memory area includes a State, multiple Blocks, and corresponding Buffers. The State is used to control the shared state between the current Segment processes, the Block is used to store the message body data, and the Buffer is used to store metadata information. S120: The subscriber follows at least one channel_id. When the subscriber starts, it registers the channel_id it follows with SegmentMap. SegmentMap stores the index relationship between the channel_id and the corresponding shared memory segment. S130: Design a dynamic allocation strategy based on capacity rules to dynamically adjust the structure and capacity of the segment according to the message size.

3. The method according to claim 2, characterized in that The rules in S130 include: For messages between 0 and 10 KB, the Segment structure and capacity are set as follows: the number of blocks is 512, and the size of each message is 16 KB. For messages between 10 and 100 KB, the Segment structure and capacity are set as follows: the number of blocks is 128, and the size of each message is 128 KB. For messages between 100kb and 1MB, the segment structure and capacity are set to: 64 blocks, and 1MB per message. For messages between 1MB and 6MB, the segment structure and capacity are set as follows: the number of blocks is 32, and the size of each message is 6MB. For messages between 6MB and 10MB in size, the Segment structure and capacity are set as follows: the number of blocks is 16, and the size of each message is 16MB. For messages larger than 10MB, the Segment structure and capacity are set as follows: the number of blocks is 8 and the size of each message is 32MB.

4. The method according to claim 3, characterized in that The Segment State consists of four atomic variables: need_remap_ is used to control whether the current Segment needs to be remapped, ceiling_msg_size_ represents the size of the message data, that is, the number of bytes in a buffer, seq_ represents the index of the block currently being written. The sender writes a block and a buffer each time it sends a message. After each data is written, seq+1 is added. reference_count_ represents the number of users using this memory. Block includes three atomic variables. lock_num_ is an atomic variable that controls the read and write mutual exclusion of the buffer corresponding to the Block, and is used to achieve safe reading and writing between processes. msg_size_ and msg_info_size_ are the lengths of the message.

5. The method according to claim 4, characterized in that The S230 includes: When a process attempts to write to a block, it atomically checks the value of lock_num_. If lock_num_ is 0, it sets it to a negative number, indicating that the block is occupied. If lock_num_ is negative, it means that the block is occupied by another process and the current process cannot write to it. It continues to search for other writable blocks until it finds a block with lock_num_ of 0 and returns the index of the block.

6. The method according to claim 5, characterized in that S300 includes: S310: When the first publisher process starts, a globally unique shared memory is created to store the indicator. The indicator is a queue-shaped data structure used to store ReadableInfo. Each ReadableInfo corresponds to an index stored in the seq array at the end of the indicator. The next_seq at the head of the indicator is an atomic variable that represents the index of the next writable ReadableInfo. S320: The publisher writes data to the Block and Buffer in the Segment corresponding to a channel_id and requests a writable ReadableInfo index from the Indicator; S330: Before writing ReadableInfo, the publisher atomically increases next_seq by 1. The publisher writes ReadableInfo to the position corresponding to next_seq in Indicator. The subscriber obtains the notification through Indicator and reads the data. Among them, ReadableInfo contains the host identifier, channel identifier and the Block index where the data is located.

7. The method according to claim 6, characterized in that S400 includes: S410: Design the dispatcher module of the receiving process, which is used to read the ReadableInfo on the indicator and distribute the data. Dispatch starts an independent thread inside. The thread body is an infinite loop function. The thread content stores a local seq index separately, with an initial value of 0. S420: The thread loop function cyclically reads the next_seq in the indicator header and compares it with the local seq. If next_seq is not equal to the local seq, it means that another publisher has published data and written ReadableInfo. The seq array of the indicator is indexed according to the local seq, the new seq value is obtained, the local seq value is updated, and the corresponding ReadableInfo is read according to the seq. S430: Compare channel_id to determine whether the current publisher process contains a subscriber of this channel_id. If so, index Segment according to channel_id, and then read the data written by the publisher according to block_index in ReadableInfo.

8. A communication system based on shared memory for communication middleware, characterized in that: Including shared memory area creation module, data writing module, global shared memory area creation module and distribution module, Shared memory area creation module: A channel identifier channel_id is bound to each publisher. Subscribers register their channel_id with Segmentmap. The publisher creates a shared memory area segment between the publisher and subscriber based on the channel_id. A dynamic allocation strategy based on capacity rules is designed to dynamically adjust the structure and capacity of the memory area according to the message size. Data writing module: The publisher completes data writing by obtaining the available block and buffer indexes in the segment. The writing operation process controls the read-write mutual exclusion and uses a circular rotation index mechanism to manage the writing position. The data writing module includes: Before publishing data, the publisher creates a segment based on the type and size of the data structure to be published and the rules. In the Linux environment, shared memory can be created through System V IPC or POSIX IPC. Obtain the writable block index based on the state variable of the segment state. The initial value of the state variable is 0. Each time a writable block is obtained, the state variable is incremented by 1. If the state variable value exceeds the number of blocks in the current segment, the state variable value is rotated through the modulo operation to limit the state variable value to between 0 and the number of blocks. Loop to obtain writable blocks until a block with lock_num_ equal to 0 is found, and then return the block index. lock_num_ is an atomic variable used to control the read-write mutual exclusion of the block. Its initial value is 0, indicating that the block is not occupied. Write the data into the buffer corresponding to the writable block. After writing is completed, release the block by setting lock_num_ to 0 to ensure that other processes can continue to use the block. Global shared memory area creation module: Creates a global shared memory area Indicator on the same host as a notification queue, stores publisher data and writes notification information ReadableInfo. ReadableInfo contains the host identifier host_id, channel_id, and Block index, allowing subscribers to quickly locate data. Distribution module: The receiving process monitors the ReadableInfo in the global shared memory area in real time through an independent thread. The subscriber locates the target Segment and Block in the SegmentMap based on the channel_id in the ReadableInfo to achieve efficient data reading and distribution.

Images