Communication middleware-oriented communication method and system based on shared memory

Abstract

The invention discloses a communication middleware-oriented communication method and system based on a shared memory, efficient end-to-end communication between a publisher and a subscriber is realized by utilizing the shared memory, and the method comprises the following steps: the publisher creates a shared memory area through a predefined channel identifier, and writes data into the memory; meanwhile, notification information is sent through the global notification queue; and the subscriber locates the target memory through the index structure and reads the data. Mechanisms such as memory state control, read-write mutual exclusion, capacity rule optimization and the like are designed, and efficient utilization of the shared memory and security of inter-process access are ensured. In addition, the receiving process realizes real-time monitoring and data distribution through a distribution module, and supports a complex communication scene of multiple publishers and multiple subscribers. Compared with the prior art, the method has the advantages of high transmission efficiency, low delay and good concurrency performance, and is suitable for design and implementation of high-performance communication middleware.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and particularly relates to a communication method and system based on shared memory for communication middleware. Background Art

[0002] With the development of modern computer technology and network communication technology, multi-process communication mechanisms have been widely used in fields such as distributed systems, real-time data processing, and high-performance computing. However, traditional multi-process communication methods (such as communication based on message queues, sockets, or remote procedure calls) face many challenges in terms of efficiency and performance. Specifically: Bottlenecks of traditional communication methods: (1) The communication method based on message queues requires frequent data copying and context switching, increasing system overhead and making it difficult to meet the requirements of high throughput and low latency; (2) The communication method based on network protocols (such as TCP / IP) needs to transfer data across hosts. Although suitable for distributed scenarios, it is too complex and inefficient in communication between multiple processes on a single host; (3) The method of sharing data through the file system is simple, but limited by the high latency and low concurrency of file I / O operations, it cannot meet the requirements of real-time communication.

[0003] Advantages and challenges of shared memory: As a way to directly utilize memory for data exchange, shared memory has the characteristics of high data transfer efficiency and low latency, and is particularly suitable for multi-process communication within a single host. However, the communication mechanism based on shared memory also faces the following problems: (1) It is necessary to design an efficient memory management strategy to avoid memory fragmentation and resource waste; (2) Data consistency and security of concurrent access are the keys to achieving efficient communication, and traditional methods lack a general design paradigm; (3) There is a lack of a unified notification mechanism to achieve real-time data distribution between publishers and subscribers.

[0004] Requirements for communication middleware: Currently, communication middleware plays an important role in message routing, load balancing, and data transmission in distributed systems, and the efficiency of its underlying communication mechanism directly determines the overall performance of the system. Therefore, designing an efficient and reliable communication mechanism based on shared memory is of great significance for optimizing the performance of communication middleware.

[0005] Based on the above background, the present invention proposes a communication method and system based on shared memory for communication middleware. Summary of the Invention

[0006] In view of the above technical problems, the present invention provides a communication method and system based on shared memory for a communication middleware to solve the problems of low efficiency, high latency, and insufficient concurrency performance in existing multi-process communication methods. The present invention innovatively proposes a bottom-layer communication mechanism based on shared memory, and through the design of memory management, data transfer, and notification mechanisms, realizes efficient and reliable end-to-end data communication, meeting the requirements of high throughput, low latency, and multi-concurrent access.

[0007] The technical solution adopted by the present invention to solve its technical problems is as follows: A communication method based on shared memory for a communication middleware, the method comprising the following steps: S100: Bind a channel identifier channel_id to each publisher. The subscriber registers the channel_id it is interested in with the Segmentmap. The publisher creates a shared memory area Segment between the publisher and the subscriber according to the channel_id, designs a dynamic allocation policy based on capacity rules, and dynamically adjusts the structure and capacity of the memory area according to the message size; S200: The publisher completes data writing by obtaining available Block and Buffer indexes in the Segment. The writing operation process controls read-write mutual exclusion, and uses a round-robin index mechanism to manage the writing position; S300: Create a global shared memory area Indicator as a notification queue in the same host to store the data writing notification information ReadableInfo of the publisher. ReadableInfo includes the host identifier host_id, channel_id, and Block index, facilitating the subscriber to quickly locate the data; S400: Design a distribution module for the receiving process, and monitor the ReadableInfo in the global shared memory area in real time through an independent thread. The subscriber locates the target Segment and Block in the SegmentMap according to the channel_id in the ReadableInfo to achieve efficient data reading and distribution.

[0008] Preferably, S100 includes: S110: When a publisher sends data, it binds a definite channel_id, and creates a shared memory area Segment according to this channel_id. Each memory area includes a State, multiple Blocks, and corresponding Buffers. The State is used to control the shared state between current Segment processes. The Block is used to store the message body data, and the Buffer is used to store metadata information; S120: The subscriber subscribes to at least one channel_id. When the subscriber starts up, it registers the channel_ids it subscribes to with the SegmentMap, which stores the index relationship between the channel_ids and the corresponding shared memory Segments. S130: Design a dynamic allocation strategy based on capacity rules to dynamically adjust the structure and capacity of the Segment according to the message size.

[0009] Preferably, the rules in S130 include: For messages with a size between 0 - 10 kb, the structure and capacity of the Segment are set as follows: the number of Blocks is 512, and the size of each message is 16 kb; For messages with a size between 10 - 100 kb, the structure and capacity of the Segment are set as follows: the number of Blocks is 128, and the size of each message is 128 kb; For messages with a size between 100 kb - 1 MB, the structure and capacity of the Segment are set as follows: the number of Blocks is 64, and the size of each message is 1 MB; For messages with a size between 1 MB - 6 MB, the structure and capacity of the Segment are set as follows: the number of Blocks is 32, and the size of each message is 6 MB; For messages with a size between 6 MB - 10 MB, the structure and capacity of the Segment are set as follows: the number of Blocks is 16, and the size of each message is 16 MB; For messages with a size exceeding 10 MB, the structure and capacity of the Segment are set as follows: the number of Blocks is 8, and the size of each message is 32 MB.

[0010] Preferably, the State of the Segment includes four atomic variables. need_remap_ is used to control whether the current Segment needs to be remapped. ceiling_msg_size_ represents the size of the message data, i.e., the number of bytes in a buffer. seq_ represents the index of the block that is currently being written. Each time the sender sends a message, it will write to a Block and a buffer, and after writing the data each time, seq will be incremented by 1. reference_count_ represents the number of users using this piece of memory; The Block includes three atomic variables. lock_num_ is an atomic variable that is used to control the read-write mutual exclusion of the buffer corresponding to the Block, so as to achieve safe read and write between processes. msg_size_ and msg_info_size_ are the lengths of the messages.

[0011] Preferably, S200 includes: S210: Before publishing data, the publisher creates a Segment according to the type and size combination rule of the data structure to be published. Among them, in the Linux environment, shared memory can be created through two methods: System V IPC or POSIX IPC. S220: Obtain the writable Block index based on the state variable of the State of the Segment. The initial value of the state variable is 0. Each time a writable Block is obtained, the state variable is incremented by 1. If the value of the state variable exceeds the number of Blocks in the current Segment, a modulo operation is performed for rotation so that the value of the state variable is limited between 0 and the number of Blocks. S230: Continuously obtain the writable Block until a Block with lock_num_ equal to 0 is found, and then return the index of the Block. lock_num_ is an atomic variable used to control the read-write mutual exclusion of the Block, and its initial value is 0, indicating that the Block is not occupied. S240: Write the data into the Buffer corresponding to the writable Block. After writing is completed, release the Block by setting lock_num_ to 0 to ensure that other processes can continue to use this Block.

[0012] Preferably, S230 includes: When a process attempts to write to a Block, atomically check the value of lock_num_. If lock_num_ is 0, set it to a negative number, indicating that the Block is occupied. If lock_num_ is negative, it means that the Block is occupied by other processes and the current process cannot write. Continue to find other writable Blocks until a Block with lock_num_ equal to 0 is found and then return the index of the Block.

[0013] Preferably, S300 includes: S310: When the first publisher process starts, create a globally unique shared memory to store the Indicator. Here, the Indicator is a queue-shaped data structure used to store ReadableInfo. Each ReadableInfo corresponds to an index, and the index is stored in the seq array at the tail of the Indicator. The next_seq at the head of the Indicator is an atomic variable representing the index of the next writable ReadableInfo. S320: The publisher writes data to the Block and Buffer in the Segment corresponding to a certain channel_id, and requests a writable ReadableInfo index from the Indicator. S330: Before writing the ReadableInfo, the publisher atomically increments next_seq by 1, and writes the ReadableInfo to the position corresponding to next_seq in the Indicator. The subscriber obtains this notification through the Indicator and reads the data. Here, the ReadableInfo contains the host identifier, channel identifier, and the Block index where the data is located.

[0014] Preferably, S400 includes: S410: Design the Dispatcher, the distribution module of the receiving process, to read the ReadableInfo on the Indicator and perform data distribution. An independent thread is started inside the Dispatcher. The main body of the thread is a function with an infinite loop. A local seq index is separately saved inside the thread, and the initial value is 0. S420: The thread loop function repeatedly reads next_seq at the head of the Indicator and compares it with the local seq. If next_seq is not equal to the local seq, it means that other publishers have published data and written the ReadableInfo. According to the local seq index the seq array of the Indicator, obtain the new seq value, update the local seq value, and read the corresponding ReadableInfo according to seq. S430: Compare the channel_id to determine whether the current publisher process has subscribers for this channel_id. If so, index the Segment according to the channel_id, and then read the data written by the publisher according to the block_index in the ReadableInfo.

[0015] A communication system based on shared memory for communication middleware, comprising a shared memory area creation module, a data writing module, a global shared memory area creation module, and a distribution module. Shared memory area creation module: Bind a channel identifier channel_id to each publisher. Subscribers register the channel_ids they are interested in with Segmentmap. The publisher creates a shared memory area Segment between the publisher and the subscriber based on the channel_id, designs a dynamic allocation policy based on capacity rules, and dynamically adjusts the structure and capacity of the memory area according to the message size. Data writing module: The publisher completes data writing by obtaining available Block and Buffer indices in Segment. The write operation process controls read-write mutual exclusion and uses a circular rotation index mechanism to manage the write position. Global shared memory area creation module: Create a global shared memory area Indicator as a notification queue in the same host, store the data write notification information ReadableInfo of the publisher, and ReadableInfo includes the host identifier host_id, channel_id, and Block index, facilitating subscribers to quickly locate data. Distribution module: A receiving process, which monitors the ReadableInfo in the global shared memory area in real time through an independent thread. Subscribers locate the target Segment and Block in SegmentMap according to the channel_id in ReadableInfo to achieve efficient data reading and distribution.

[0016] The above communication method and system based on shared memory for communication middleware design mechanisms such as memory state control, read-write mutual exclusion, and capacity rule optimization to ensure the efficient utilization of shared memory and the security of inter-process access. In addition, the receiving process realizes real-time monitoring and data distribution through the distribution module Dispatcher, supporting complex communication scenarios with multiple publishers and multiple subscribers. Compared with the prior art, the present invention has the advantages of high transmission efficiency, low latency, and good concurrency performance, and is suitable for the design and implementation of high-performance communication middleware. Description of the Drawings

[0017] Figure 1 It is a flowchart of a communication method based on shared memory for communication middleware in an embodiment of the present invention. Figure 2 It is a communication flowchart of a communication method based on shared memory for communication middleware in an embodiment of the present invention. Figure 3 It is a schematic diagram of the memory structure of Segment in an embodiment of the present invention. Figure 4 Structural schematic diagram of State in an embodiment of the present invention; Figure 5 Structural schematic diagram of Block in an embodiment of the present invention; Figure 6 Structural schematic diagram of Indicator in an embodiment of the present invention; Figure 7 Structural schematic diagram of ReadableInfo in an embodiment of the present invention. Detailed implementation manners

[0018] In order to enable those skilled in the art of the present technology to better understand the technical solutions of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0019] In one embodiment, as Figure 1 shown, a communication method based on shared memory for a communication middleware, the method includes the following steps: S100: Bind a channel identifier channel_id to each publisher. The subscriber registers the channel_id it is interested in with the Segmentmap. The publisher creates a shared memory area Segment between the publisher and the subscriber according to the channel_id, designs a dynamic allocation policy based on capacity rules, and dynamically adjusts the structure and capacity of the memory area according to the message size; S200: The publisher completes data writing by obtaining available Block and Buffer indexes in the Segment. The writing operation process controls read-write mutual exclusion through the lock_num_ field to ensure the security of concurrent access, and uses a circular rotation index mechanism (seq_ variable) to manage the writing position to avoid resource competition and deadlocks; S300: Create a global shared memory area Indicator as a notification queue in the same host to store the data writing notification information ReadableInfo of the publisher. ReadableInfo includes the host identifier host_id, channel_id, and Block index, which is convenient for the subscriber to quickly locate the data; S400: Design a distribution module for the receiving process, and monitor the ReadableInfo in the global shared memory area in real time through an independent thread. The subscriber locates the target Segment and Block in the SegmentMap according to the channel_id in the ReadableInfo to achieve efficient data reading and distribution.

[0020] Specifically, in a communication middleware, a data publisher sends data to a data subscriber. To achieve end-to-end transmission, a topic or a channel is usually set for identification and authentication between the two parties. Based on this communication model, the present invention proposes an end-to-end communication mechanism based on shared memory, and its communication process is as follows Figure 2 shown.

[0021] In one embodiment, S100 includes: S110: When a publisher sends data, it binds a specific channel_id. According to this channel_id, a shared memory area Segment is created. Each memory area includes a State, multiple Blocks, and corresponding Buffers. The State is used to control the shared state among current Segment processes. The Block is used to store message body data, and the Buffer is used to store metadata information; S120: The subscriber subscribes to at least one channel_id. When the subscriber starts, it registers the channel_id it subscribes to in the SegmentMap. The SegmentMap stores the index relationship between the channel_id and the corresponding shared memory Segment; S130: Design a dynamic allocation strategy based on capacity rules to dynamically adjust the structure and capacity of the Segment according to the message size.

[0022] Specifically, end-to-end communication is the communication between a publisher (Publisher) of one process and a subscriber (Subscriber) of another process. Since shared memory communication is used, the two processes are on the same host. When a publisher sends data, it binds a specific channel_id and creates a shared memory according to this channel_id. This shared memory is defined as a Segment. Each subscriber has its own subscribed channel_id. In the receiving process, a data structure of SegmentMap: <channel_id, Segment> is designed. The SegmentMap stores the index relationship between the channel_id and the corresponding shared memory Segment. When the subscriber obtains data on the corresponding channel_id, it first registers in the SegmentMap, and then obtains the data by finding the Segment according to the channel_id in the SegmentMap.

[0023] Furthermore, in the communication process described in the present invention, the shared memory created according to the channel_id is defined as a Segment, and the memory structure of the Segment is defined as followsFigure 3 As shown in: The content shared by each Segment includes: a State, n Blocks, and n buffers. The State controls the shared state among current Segment processes. The memory occupancy size of a Segment is closely related to the size of the message data to be transmitted. The specific rules are defined as follows.

[0024] In one embodiment, the rules in S130 include: For messages with a size between 0 - 10 kb, the structure and capacity of the Segment are set as follows: the number of Blocks is 512, and the size of each message is 16 kb; For messages with a size between 10 - 100 kb, the structure and capacity of the Segment are set as follows: the number of Blocks is 128, and the size of each message is 128 kb; For messages with a size between 100 kb - 1 MB, the structure and capacity of the Segment are set as follows: the number of Blocks is 64, and the size of each message is 1 MB; For messages with a size between 1 MB - 6 MB, the structure and capacity of the Segment are set as follows: the number of Blocks is 32, and the size of each message is 6 MB; For messages with a size between 6 MB - 10 MB, the structure and capacity of the Segment are set as follows: the number of Blocks is 16, and the size of each message is 16 MB; For messages with a size exceeding 10 MB, the structure and capacity of the Segment are set as follows: the number of Blocks is 8, and the size of each message is 32 MB.

[0025] Specifically, for example, assume that the size of a transmitted message is between 10 kb and 100 kb. Then, corresponding to the second rule, first, it is defined in the rule that the size of each Block is 1024 bytes, the number of Blocks in a Segment is 128, and the number of buffers also corresponds to 128. A buffer consists of two parts: a Message carrying the message and an additional MessageInfo. The size of MessageInfo is fixed at 1024 bytes. Therefore, the size of a buffer at this time is 128 kb + 1 kb = 129 kb.

[0026] It is also specified in the rule that the size of the State at the head of the Segment is 1024 bytes, and the size occupied by the unused part at the tail of the Segment is 1024 * 4 = 4 kb.

[0027] In one embodiment, asFigure 4 and 5 As shown in 5 , the State of Segment includes four atomic variables. need_remap_ is used to control whether the current Segment needs to be remapped. ceiling_msg_size_ represents the size of the message data, i.e., the number of bytes of a buffer. seq_ represents the index of the block that is currently being written. The sender writes a Block and a buffer every time it sends a message, and increments seq by 1 after each data write. reference_count_ represents the number of users using this piece of memory; A Block includes three atomic variables. lock_num_ is an atomic variable that is used to control the read-write mutex of the buffer corresponding to the Block, for achieving safe read and write between processes. msg_size_ and msg_info_size_ are the lengths of the message.

[0028] In one embodiment, S200 includes: S210: Before publishing data, the publisher creates a Segment according to the type and size combination rule of the data structure to be published; among them, in the Linux environment, shared memory can be created through either System V IPC or POSIX IPC; S220: Obtain the writable Block index based on the state variables of the State of Segment. Among them, the initial value of the state variable is 0. Each time a writable Block is obtained, the state variable is incremented by 1. If the value of the state variable exceeds the number of Blocks of the current Segment, a modulo operation is performed for rotation, so that the value of the state variable is limited between 0 and the number of Blocks; S230: Continuously obtain the writable Block until the index of the Block with lock_num_ equal to 0 is returned. Among them, lock_num_ is an atomic variable used to control the read-write mutex of the Block, and its initial value is 0, indicating that the Block is not occupied; S240: Write the data into the Buffer corresponding to the writable Block. After the writing is completed, release the Block by setting lock_num_ to 0 to ensure that other processes can continue to use this Block.

[0029] In one embodiment, S230 includes: When a process attempts to write to a Block, it atomically checks the value of lock_num_. If lock_num_ is 0, it sets it to a negative number, indicating that the Block is occupied. If lock_num_ is negative, it means the Block is already occupied by another process, and the current process cannot write. It continues to search for other writable Blocks until it finds a Block with lock_num_ equal to 0 and then returns the index of the Block.

[0030] Specifically, according to the communication process, one of the publishers of the sending process writes data to a channel, and the type of the data structure to be written is also determined in advance by the sender. Each channel corresponds to a Segment. Therefore, when the sender publishes data, it first creates a Segment according to the type and size of the data structure to be published according to the above rules. The Segment is a global shared memory. In the Linux environment, there are two ways to create shared memory, namely based on System V IPC and POSIX IPC, and either way can be used.

[0031] After creating the Segment, data can be written to this Segment. The goal of data writing is to fill data into one of the Blocks and the corresponding Buffer in the Segment. Therefore, an index of a Buffer needs to be obtained in advance, and seq_ in State records this index. The initial value of seq_ in State is 0. Each time an index of a writable Block is obtained, the value of seq_ is incremented by 1. If the value of seq_ exceeds the number of Blocks in the current Segment, it rotates from the beginning, that is, using the modulo operation, so that the value of seq_ is limited between 0 and the number of Blocks. At the same time, the operation of obtaining the index of a writable Block is a loop operation. Only when it is determined that the Block corresponding to the obtained index of the Block is writable will this index be returned to exit the loop operation.

[0032] The Block performs read-write mutual exclusion based on the lock_num_ in the header. lock_num_ is an atomic variable with an initial value of 0. Therefore, reading and writing to this variable is process-safe. Assuming the current Block is writable, it will atomically determine whether the value of lock_num_ is 0. If it is 0, it sets the value of lock_num_ to a negative number, indicating that a process has occupied this Block. If another process also wants to write to this Block at this time, it first checks the value of lock_num_ and finds that it is negative, so it cannot be written by this process.

[0033] Through the above operations, a writable Block and its corresponding Buffer can be obtained from the Segment. Next, data is written to this Block and Buffer. After writing the data, the exclusive ownership of this Block and Buffer needs to be released. Here, the value of lock_num_ of this Block is reset to 0, indicating that no process is writing data to this Block.

[0034] In one embodiment, S300 includes: S310: When the first publisher process starts, create a globally unique shared memory to store the Indicator. Here, the Indicator is a queue-shaped data structure used to store ReadableInfo. Each ReadableInfo corresponds to an index, and the index is stored in the seq array at the tail of the Indicator. The next_seq at the head of the Indicator is an atomic variable representing the index of the next writable ReadableInfo. S320: The publisher writes data to the Block and Buffer in the Segment corresponding to a certain channel_id, and requests a writable ReadableInfo index from the Indicator. S330: Before writing the ReadableInfo, the publisher atomically increments next_seq by 1. The publisher writes the ReadableInfo to the position corresponding to next_seq in the Indicator. The subscriber obtains this notification through the Indicator and reads the data. Here, the ReadableInfo contains the host identifier, the channel identifier, and the Block index where the data is located.

[0035] Specifically, after writing data to a Segment, the publisher sends a notification message indicating that it has finished writing the data, and the subscriber can obtain the data from this Segment.

[0036] To implement the message notification mechanism among publishers and subscribers in multiple processes, a special data structure Indicator is created. When the first publishing node starts, it will allocate a globally unique shared memory to store the Indicator. This Indicator is shared by all publishers and subscribers on the same host. The data format composition of the Indicator is as Figure 6 shown.

[0037] An Indicator can be understood as a queue-shaped data structure that can store 4,096 ReadableInfos. Each ReadableInfo corresponds to an index, and this index is written into the seq array at the tail of the Indicator. Therefore, the size of the seq array is also 4,096. The next_seq at the head is an atomic variable representing the index of the next writable ReadableInfo. Among them, the data format of the ReadableInfo is as Figure 7 shown. The host_id is the unique identifier of each host, the channel_id represents the channel_id to which the publisher writes data on the Segment, and the block_index represents the block index of the Segment where the data is written.

[0038] After the publisher finishes writing data to a certain Block and Buffer in the Segment on a channel_id, then the publisher needs to fill a ReadableInfo. Therefore, it first needs to request from the Indicator. At this time, the next_seq at the head of the global Indicator is the index of the writable ReadableInfo. Before the publisher writes the ReadableInfo, it will atomically increment next_seq by 1, and then write the notification information to the obtained index.

[0039] In one embodiment, S400 includes: S410: Design the Dispatcher, the distribution module of the receiving process, to read the ReadableInfos on the Indicator and perform data distribution. An independent thread is started inside the Dispatcher. The main body of the thread is a function with an infinite loop. A local seq index is separately saved in the thread content, and the initial value is 0; S420: The thread loop function loops to read the next_seq at the head of the Indicator and compares it with the local seq; if next_seq is not equal to the local seq, it means that other publishers have published data and written the ReadableInfo. According to the local seq index, the seq array of the Indicator is indexed to obtain the new seq value, update the local seq value, and read the corresponding ReadableInfo according to the seq; S430: Compare the channel_id to determine whether the current publisher process contains subscribers of this channel_id. If so, index the Segment according to the channel_id, and then read the data written by the publisher according to the block_index in the ReadableInfo.

[0040] Specifically, there is a Dispatcher module in the subscriber process. This module is used to monitor whether the data on the Segment has been updated and written, and reads the information of ReadableInfo on the Indicator for data distribution; a local seq index is separately saved inside the Dispatcher, and the initial value is 0. The thread loop function will loop to read the value of next_seq at the head of the Indicator and compare it with the local seq. If the value of next_seq is not equal to the local seq, it means that other publishers have published data and written it into ReadableInfo. At this time, the seq array in the Indicator can be indexed according to the locally saved seq, and then the local seq is updated to the value in the seq array. Then, the ReadableInfo corresponding to the local seq can be obtained. Finally, the channel_id is compared to determine whether the current publisher process contains subscribers of this channel_id. If it exists, the Segment is indexed according to the channel_id, and then the data written by the publisher is read out according to the block_index in the ReadableInfo.

[0041] A shared memory-based communication system for communication middleware, including a shared memory area creation module, a data writing module, a global shared memory area creation module, and a distribution module. Shared memory area creation module: Bind a channel identifier channel_id to each publisher. Subscribers register the channel_ids they are interested in with the Segmentmap. The publisher creates a shared memory area Segment between the publisher and the subscriber according to the channel_id, designs a dynamic allocation strategy based on capacity rules, and dynamically adjusts the structure and capacity of the memory area according to the message size. Data writing module: The publisher completes data writing by obtaining available Block and Buffer indexes in the Segment. The write operation process controls read-write mutual exclusion and uses a round-robin index mechanism to manage the write position. Global shared memory area creation module: Create a global shared memory area Indicator as a notification queue in the same host, store the data write notification information ReadableInfo of the publisher, and ReadableInfo includes the host identifier host_id, channel_id, and Block index, which is convenient for subscribers to quickly locate the data. Distribution module: a receiving process that monitors ReadableInfo in the global shared memory area in real time through an independent thread. Subscribers locate the target Segment and Block in the SegmentMap according to the channel_id in ReadableInfo to achieve efficient data reading and distribution.

[0042] For the specific limitations of a shared-memory-based communication system for communication middleware, reference can be made to the limitations of a shared-memory-based communication method for communication middleware described above, which will not be elaborated here. Each module in the above-mentioned shared-memory-based communication system for communication middleware can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in the processor of a computer device in hardware form or be independent of it, or can be stored in the memory of a computer device in software form to facilitate the processor to call and execute the operations corresponding to the above-mentioned modules.

[0043] For the above-mentioned shared-memory-based communication method and system for communication middleware, compared with the prior art, the present invention has the following beneficial effects: Through direct data transfer based on shared memory, multiple data copies and context switches are avoided, significantly improving data transmission efficiency and reducing latency. The dynamic allocation strategy and round-robin index mechanism based on capacity rules can efficiently adapt to different message sizes and communication requirements. Through the design of atomic variables and mutex locks, the security and stability of multi-process concurrent access are ensured. Through the unified notification mechanism and real-time distribution module, the requirements of high-real-time scenarios are met. The present invention is applicable to the design and implementation of high-performance communication middleware and can be widely used in fields such as real-time data processing, distributed system communication, and industrial Internet of Things.

[0044] The above has introduced in detail a shared-memory-based communication method and system for communication middleware provided by the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the core idea of the present invention. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present invention, several improvements and modifications can still be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A communication method based on shared memory for communication middleware, characterized in that: The method comprises the following steps: S100: Bind a channel identifier channel_id to each publisher, and the subscriber registers the channel_id it is interested in with Segmentmap. The publisher creates a shared memory area Segment between the publisher and the subscriber based on the channel_id, and designs a dynamic allocation strategy based on capacity rules to dynamically adjust the structure and capacity of the memory area according to the message size. S200: The publisher completes data writing by obtaining the available Block and Buffer indexes in the Segment. The writing operation process controls the read-write mutual exclusion and uses the circular rotation index mechanism to manage the writing position. S300: Create a global shared memory area Indicator in the same host as a notification queue to store the publisher's data write notification information ReadableInfo, which includes the host identifier host_id, channel_id and Block index, so that the subscriber can quickly locate the data; S400: Design the distribution module of the receiving process, monitor the ReadableInfo in the global shared memory area in real time through an independent thread, and the subscriber locates the target Segment and Block in the SegmentMap according to the channel_id in the ReadableInfo to achieve efficient data reading and distribution.

2. The method according to claim 1, characterized in that S100 includes: S110: When a publisher sends data, it will bind a certain channel_id and create a shared memory area Segment based on this channel_id. Each memory area includes a State, multiple Blocks and corresponding Buffers. The State is used to control the shared state between the current Segment processes, the Block is used to store the message body data, and the Buffer is used to store metadata information. S120: The subscriber follows at least one channel_id. When the subscriber starts, it registers the channel_id it follows to SegmentMap. SegmentMap stores the index relationship between the channel_id and the corresponding shared memory Segment. S130: Design a dynamic allocation strategy based on capacity rules to dynamically adjust the structure and capacity of the Segment according to the message size.

3. The method according to claim 2, characterized in that The rules in S130 include: For messages with a size between 0 and 10 KB, the Segment structure and capacity are set as follows: the number of blocks is 512, and the size of each message is 16 KB; For messages with a size between 10-100kb, the Segment structure and capacity are set as follows: the number of blocks is 128, and the size of each message is 128kb; For messages with a size between 100kb and 1MB, the Segment structure and capacity are set to: the number of blocks is 64, and the size of each message is 1MB; For messages with a size between 1MB and 6MB, the Segment structure and capacity are set as follows: the number of blocks is 32, and the size of each message is 6MB; For messages with a size between 6MB and 10MB, the Segment structure and capacity are set as follows: the number of blocks is 16, and the size of each message is 16MB; For messages larger than 10MB in size, the Segment structure and capacity are set as follows: the number of blocks is 8 and the size of each message is 32MB.

4. The method according to claim 3, characterized in that The Segment State includes four atomic variables: need_remap_ is used to control whether the current Segment needs to be remapped; ceiling_msg_size_ represents the size of the message data, that is, the number of bytes in a buffer; seq_ represents the index of the block currently being written. The sender writes a block and a buffer each time it sends a message. After each data is written, seq+1 is added; reference_count_ represents the number of users using this memory; Block includes three atomic variables. lock_num_ is an atomic variable. This atomic variable is used to control the read and write mutual exclusion of the buffer corresponding to the Block, which is used to achieve safe reading and writing between processes. msg_size_ and msg_info_size_ are the lengths of the message.

5. The method according to claim 4, characterized in that S200 includes: S210: Before publishing data, the publisher creates a segment according to the type and size of the data structure to be published and the rules. In the Linux environment, the shared memory can be created by System V IPC or POSIX IPC. S220: Obtain a writable Block index based on the state variable of the Segment State, where the initial value of the state variable is 0. Each time a writable Block is obtained, the state variable is incremented by 1. If the state variable value exceeds the number of Blocks in the current Segment, the state variable is rotated through the modulo operation so that the state variable value is limited between 0 and the number of Blocks. S230: Loop to obtain writable blocks until a block with lock_num_ being 0 is found and then return the index of the block, where lock_num_ is an atomic variable used to control the read-write mutual exclusion of the block, and its initial value is 0, indicating that the block is not occupied; S240: Write the data into the buffer corresponding to the writable block. After the writing is completed, release the block by setting lock_num_ to 0 to ensure that other processes can continue to use the block.

6. The method according to claim 5, characterized in that S230 includes: When a process tries to write to a Block, it atomically checks the value of lock_num_. If lock_num_ is 0, it sets it to a negative number, indicating that the Block is occupied. If lock_num_ is a negative number, it means that the Block is occupied by another process and the current process cannot write to it. It continues to look for other writable Blocks until it finds a Block with lock_num_ of 0 and returns the index of the Block.

7. The method according to claim 6, characterized in that S300 includes: S310: When the first publisher process starts, a globally unique shared memory is created to store Indicator, where Indicator is a queue-shaped data structure used to store ReadableInfo. Each ReadableInfo corresponds to an index, which is stored in the seq array at the end of Indicator. The next_seq at the head of Indicator is an atomic variable, indicating the index of the next writable ReadableInfo. S320: The publisher writes data to the Block and Buffer in the Segment corresponding to a channel_id, and requests a writable ReadableInfo index from the Indicator; S330: Before writing ReadableInfo, the publisher atomically increases next_seq by 1. The publisher writes ReadableInfo to the position corresponding to next_seq in Indicator. The subscriber obtains the notification through Indicator and reads the data. Among them, ReadableInfo includes the host identifier, channel identifier and the Block index where the data is located.

8. The method according to claim 7, characterized in that S400 includes: S410: Design the distribution module Dispatcher of the receiving process, which is used to read the ReadableInfo on the Indicator and distribute the data. An independent thread is started inside Dispatch. The thread body is an infinite loop function. The thread content saves a local seq index separately, and the initial value is 0. S420: The thread loop function loops to read the next_seq in the Indicator header and compares it with the local seq; if next_seq is not equal to the local seq, it means that other publishers have published data and written ReadableInfo, and the seq array of the Indicator is indexed according to the local seq, the new seq value is obtained, the local seq value is updated, and the corresponding ReadableInfo is read according to the seq; S430: Compare channel_id to determine whether the current publisher process contains a subscriber of this channel_id. If so, index Segment according to channel_id, and then read the data written by the publisher according to block_index in ReadableInfo.

9. A communication system based on shared memory for communication middleware, characterized in that: Including shared memory area creation module, data writing module, global shared memory area creation module and distribution module, Shared memory area creation module: bind a channel identifier channel_id to each publisher, and the subscriber registers the channel_id of its interest to Segmentmap. The publisher creates a shared memory area Segment between the publisher and the subscriber based on the channel_id, designs a dynamic allocation strategy based on capacity rules, and dynamically adjusts the structure and capacity of the memory area according to the message size; Data writing module: The publisher completes data writing by obtaining the available Block and Buffer indexes in the Segment. The writing operation process controls the read-write mutual exclusion and uses a circular rotation index mechanism to manage the writing position. Global shared memory area creation module: creates a global shared memory area Indicator as a notification queue in the same host, stores the publisher's data write notification information ReadableInfo, which contains the host identifier host_id, channel_id and Block index, so that subscribers can quickly locate data; Distribution module: The receiving process monitors the ReadableInfo in the global shared memory area in real time through an independent thread. The subscriber locates the target Segment and Block in the SegmentMap according to the channel_id in the ReadableInfo to achieve efficient data reading and distribution.

Images