A Linux pipeline communication performance optimization method

Abstract

The application provides a Linux pipeline communication performance optimization method, and belongs to the technical field of pipeline communication. The method is characterized in that a self-developed acceleration library is set and compiled into a dynamic library. Before application startup, the dynamic library is preloaded by using an LD_PRELOAD technology, so that functions in the dynamic library are preferentially called by the application. The self-developed acceleration library comprises the following interfaces: an mkfifo interface, a pipe interface, an open interface, a write interface and a read interface. The application is based on the LD_PRELOAD dynamic preloading technology, keeps the use interface of the pipeline communication unchanged, does not need to recompile the existing application code, performs transparent acceleration optimization on the traditional pipeline communication of the application by means of the shared memory technology, and improves the efficiency of inter-process data transmission.

Description

TECHNICAL FIELD

[0001] The present application relates to pipeline communication, in particular to a Linux pipeline communication performance optimization method, and belongs to the technical field of inter-process communication. BACKGROUND

[0002] In the inter-process communication (IPC) technology, the traditional pipeline is a commonly used communication method, which realizes streaming data transmission based on a kernel buffer, provides a standardized read / write operation interface, and is easy to use. A large number of application codes use the traditional pipeline standard interface to implement their respective businesses.

[0003] Shared memory is a high-efficiency inter-process communication method, which maps the same physical memory to the address space of different processes, avoids multiple copying of data between the kernel and the user space, and greatly improves the data transmission efficiency.

[0004] LD_PRELOAD is an environment variable of the Linux system, which allows a specified shared library to be loaded before a program runs, so as to use the functions defined in the library preferentially, and to realize the interception and replacement of system functions. This mechanism can transparently enhance the function without modifying the application code.

[0005] The traditional pipeline has certain performance overhead in the data transmission process. The data needs to be copied and context switched between the kernel and the user space multiple times, resulting in low transmission efficiency, especially in the large data transmission scenario, the performance bottleneck is obvious. The shared memory lacks standardized operation interfaces, and the developer needs to handle complex problems such as memory management and synchronization when using it, which increases the development difficulty and error risk, and is difficult to meet the needs of developers for ease of use. Therefore, how to use the shared memory technology to transparently accelerate the traditional pipeline communication has become a problem to be solved. SUMMARY

[0006] In view of the above problems, the present application provides a Linux pipeline communication performance optimization method to improve the efficiency of inter-process data transmission.

[0007] To achieve the above purpose, the technical scheme of the present application is as follows: a Linux pipeline communication performance optimization method, a self-developed acceleration library is set and compiled into a dynamic library, before the application starts, the dynamic library is loaded in advance by using the LD_PRELOAD technology, so that the functions in the dynamic library are called preferentially by the application; the self-developed acceleration library includes the following interfaces: mkfifo interface, pipe interface, open interface, write interface, and read interface, wherein:

[0008] The mkfifo interface is connected with a named pipe, a unique shared memory area name is dynamically generated by calling a function of the interface, the pipe path of the user is bound with the dynamically generated unique shared memory area name, and initialization of a global named pipe management structure fifo_mapping is completed, and the global named pipe management structure fifo_mapping records the correspondence between the shared memory area name and the pipe path;

[0009] The pipe interface is connected with an anonymous pipe, a shared memory area is created by calling a function of the interface, fd[2] is returned to the user for reading and writing, and the anonymous pipe global management structure is initialized based on the returned fd[2], and the anonymous pipe global management structure records the mapping relationship between the shared memory area corresponding fd[2] and the anonymous pipe;

[0010] The open interface is connected with a named pipe, an open function of the interface is called to find the bound unique corresponding shared memory area name of the pipe path transmitted through the interface, a shared memory area is created and the fd of the shared memory area mapping is returned, so as to read and write subsequently;

[0011] The write interface judges whether the input fd is a pipe operation, if yes, the shared memory area of the corresponding pipe is found, and the shared memory area is written; otherwise, a normal write operation is performed.

[0012] The read interface judges whether the input fd is a pipe operation, if yes, the shared memory area of the corresponding pipe is found, and the shared memory area is read; otherwise, a normal read operation is performed.

[0013] Further, the function prototype of the mkfifo interface is int mkfifo(const char pathname,mode_t mode).

[0014] Further, the workflow of the mkfifo interface includes the following:

[0015] User connects to the mkfifo interface through a named pipe, according to the given path, first find the existing global named pipe management structure fifo_mapping, if the global named pipe management structure fifo_mapping exists corresponding to the given path, call the system mkfifo function to return the existing mapping and end; if not, dynamically generate a unique shared memory area name, bind the user's pipe path to the shared memory area name, record to the global named pipe management structure fifo_mapping, further complete the initialization of the global named pipe management structure fifo_mapping, and then call the system mkfifo interface function to end the process.

[0016] Further, the function prototype of the pipe interface is int pipe(int fds[2]).

[0017] Further, the working process of the pipe interface includes the following:

[0018] User connects to the pipe interface through an anonymous pipe interface, calls the interface function to first dynamically generate a unique name as a shared memory area object name, open the shared memory area, obtain the read-write shared memory area handle fd, and then adjust the shared memory area size through ftruncate; After that, the shared memory area is mapped to the current process address space through mmap, and the header management data of the shared memory area is initialized, and then the opened shared memory area fd is stored in the pipe interface incoming parameter fd[2] array; Finally, record the opened shared memory area fd to the global related management data of the anonymous pipe, which is used for subsequent reading and writing through fd, and through traversing the global management structure to find out whether there is a corresponding fd, so as to judge whether it is a legal pipe operation.

[0019] Further, the function prototype of the open interface is int open(const char pathname, intflags,...).

[0020] Further, the working process of the open interface includes the following:

[0021] User connects to the open interface through a named pipe, finds the existing global named pipe management structure fifo_mapping through the incoming path, judges whether the mapping exists, if not, call the normal open function to return; if the mapping exists, find the shared memory area name corresponding to the incoming path and open it;

[0022] After that, the ftruncate function is used to adjust the size of the shared memory, the mmap function is used to map the shared memory area to the address space of the current process, and the shared memory area header management data is initialized, and finally the process ends.

[0023] Further, the function prototype of the write interface is ssize_t write (int fd, const void data, size_t len), and the parameters are fd, data buffer and size.

[0024] Further, the working process of the write interface includes the following:

[0025] According to the incoming fd, the existing global named pipe management structure fifo_mapping and the anonymous pipe global management structure are found, it is judged whether the mapping exists, if not, the normal write function is called to return; if it exists, the shared memory area associated with the incoming fd is found and opened, then the data is written in a loop, the write pointer is updated, finally the mapping is removed and the shared memory area is closed.

[0026] Further, the function prototype of the read interface is ssize_t read (int fd, void data, size_t len), and the parameters are fd, data buffer and size; the working process includes the following:

[0027] The existing global named pipe management structure fifo_mapping and the anonymous pipe global management structure are found through the incoming fd, it is judged whether the mapping exists, if not, the normal write function is called to return, if the mapping exists, the shared memory area associated with the incoming fd is found and opened, then the data is read in a loop and the read pointer is updated, finally the mapping is removed and the shared memory area is closed.

[0028] The Linux pipe communication performance optimization method has the following advantages:

[0029] The method for transparently accelerating and optimizing the existing Linux pipe communication application is based on the LD_PRELOAD dynamic preloading technology, and the application using the pipe for communication is transparently accelerated and optimized; the method maintains the use interface of the pipe communication unchanged, does not need to recompile the existing application source code, simultaneously based on the cross-process sharing of the shared memory, realizes the transparent acceleration of the existing traditional pipe communication application, and improves the efficiency of inter-process data transmission. BRIEF DESCRIPTION OF DRAWINGS

[0030] The application will be described in further detail below with reference to the drawings and specific embodiments.

[0031] Figure 1 is a schematic diagram of the mkfifo interface process in the embodiment of the application;

[0032] Figure 2 is a schematic diagram of the pipe interface process in the embodiment of the application;

[0033] Figure 3 is a schematic diagram of the open interface process in the embodiment of the application;

[0034] Figure 4 is a schematic diagram of the write interface process in the embodiment of the application;

[0035] Figure 5 is a schematic diagram of the read interface process in the embodiment of the application. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. In the following description, a large number of specific details are set forth in order to provide a thorough understanding of the application, but the application can also be implemented in other ways different from those described herein, and persons skilled in the art can make similar generalizations without departing from the connotation of the application, so the application is not limited by the specific embodiments disclosed below.

[0037] Embodiment 1

[0038] A Linux pipe communication performance optimization method mainly sets a self-developed acceleration library, the self-developed acceleration library maintains the same use interface (including name and interface prototype) as traditional pipe communication (named pipe and anonymous pipe); the self-developed acceleration library is compiled into a dynamic library, based on the LD_PRELOAD technology, before application startup, the function in the dynamic library is preloaded, so as to realize transparent replacement of the original traditional pipe interface at runtime. Specifically, the self-developed acceleration library mainly includes the following interfaces: mkfifo interface, pipe interface, open interface, write interface, and read interface. Among them,

[0039] mkfifo interface, named pipe scenario will use, usually used for cross-process communication; with the named pipe communication connection, by calling the function of the interface dynamically generates a unique shared memory area name, and then binds the user's pipe path with the dynamically generated unique shared memory area name, and completes the initialization of the global named pipe management structure fifo_mapping, which records the correspondence between the shared memory area name and the pipe path in the global named pipe management structure fifo_mapping; the interface will use a pipe path passed in by the user to create a named pipe, and different processes can use this path to open the named pipe afterwards, and then read and write can be performed.

[0040] pipe interface, anonymous pipe scenario will use, usually used for parent-child process communication; the interface is connected with the anonymous pipe communication, by calling the interface function, fd[2] (one fd for reading and one fd for writing) is returned to the user for reading and writing. Its internal implementation will dynamically create a shared memory area, record the fd[2] returned to the user, and initialize the anonymous pipe global management structure based on the returned fd[2], which records the mapping relationship between the shared memory area corresponding to the fd[2] and the anonymous pipe in the anonymous pipe global management structure. When the user uses the fd[2] for reading and writing later, the associated shared memory area can be found from the anonymous pipe global management structure, so as to complete efficient reading and writing based on the shared memory area.

[0041] open interface, named pipe scenario, used to open an already created named pipe; connected with the named pipe communication, by calling the open function of the interface, the pipe path passed in through the interface finds the uniquely corresponding shared memory area name bound, completes the creation of the shared memory area and returns the fd of the created shared memory area mapping, so as to read and write later.

[0042] write interface, for the passed-in fd, judge whether it is a pipe operation, if so, find the shared memory area of the corresponding pipe, and write to the shared memory area; otherwise, go through the normal write operation.

[0043] read interface, for the passed-in fd, judge whether it is a pipe operation, if so, find the shared memory area of the corresponding pipe, and read from the shared memory area; otherwise, go through the normal read operation.

[0044] Embodiment 2

[0045] Based on embodiment 1, combined with Figure 1 further refine the workflow of the mkfifo interface, the function prototype of the interface is int mkfifo(const char pathname, mode_t mode), the specific process includes the following:

[0046] Users connect to the mkfifo interface via named pipes. Following the given pipe path, the system first searches for an existing global named pipe management structure, fifo_mapping. If a mapping corresponding to the given path exists in fifo_mapping, the system mkfifo interface function is called to return the existing mapping and the process ends. If no mapping exists, a unique shared memory region name is dynamically generated, and the user's pipe path is bound to this shared memory region name, recorded in the fifo_mapping structure. This further initializes the fifo_mapping structure, and then the system mkfifo interface function is called to terminate the process.

[0047] Example 3

[0048] Based on Example 1, combined with Figure 2 The workflow of the pipe interface is further refined. The function prototype of the interface is int pipe(int fds[2]), and the specific process includes the following:

[0049] Users connect to the pipe interface through the anonymous pipe interface. The interface function first dynamically generates a unique name as the name of the shared memory area object, opens the shared memory area, and obtains the handle fd for reading and writing the shared memory area (fd is an identifier of the shared memory area, used when reading and writing the shared memory area); then adjusts the size of the shared memory area through ftruncate, then maps the shared memory area to the current process address space through mmap, and initializes the header management data of the shared memory area. Then, the opened shared memory area fd is stored in the fd[2] array of the input parameters of the pipe interface (users can use fd[2] to read and write later). Finally, the opened shared memory area fd is recorded to the global management data related to the anonymous pipe, which is used to check whether there is a corresponding fd by traversing the global management structure when reading and writing through fd later, so as to determine whether it is a legal pipe operation.

[0050] Example 4

[0051] Based on Example 2, combined with Figure 3 To further refine the workflow of the open interface, its function prototype is int open(const char). The process, which includes the following steps: pathname, int flags, ...), is as follows:

[0052] The user connects to the open interface through a named pipe, and through the incoming pathname, finds the existing global named pipe management structure fifo_mapping to determine whether a mapping already exists. If no mapping exists, it indicates a non-pipe operation, and the normal open function is directly called for return. If a mapping exists, the name of the shared memory area corresponding to the pathname is found, and the following operations are then performed:

[0053] 1) The shared memory area is created using the shm_open function, using the shared memory area name (for example, / my_shared_pipe), and is set to O_CREAT | O_RDWR opening mode, with file permissions of 0666. The ftruncate function is used to adjust the shared memory size to the specified value (for example, 1MB). The shared memory area is mapped to the address space of the current process through the mmap function, and the shared memory area structure pointer SharedPipe is obtained. pipe. The beginning of the SharedPipe shared memory area is a header structure, which stores basic management data, synchronization semaphores, and mutexes for accessing the shared memory area. The data format is as follows:

[0054] struct pipe_share_header {

[0055] pthread_mutex_t mutex;

[0056] int size; / / Total available size of current data

[0057] int current_size; / / Current size of data used

[0058] char read_pos; / / Current read position

[0059] char write_pos; / / Current write position

[0060] char data[0]; / / Logical ring buffer

[0061] };

[0062] 2) The header data in the shared memory area structure is initialized, setting the total size of the buffer to the size of the shared memory area (such as 1MB), the read pointer position read_pos and the write pointer position write_pos to 0, the read process count read_count and the write process count write_count to 0, the buffer full flag is_full to 0, and the buffer empty flag is_empty to 1. The mutex pthread_mutex_init (& pipe->header.mutex, NULL) and the semaphores sem_init (& pipe->header.read_sem, 1, 0) and sem_init (& pipe->header.write_sem, 1, 1024 are initialized. 1024)

[0063] 3) The handle fd of the shared memory area is returned, and the process ends.

[0064] Embodiment 5

[0065] On the basis of the above embodiments, in combination with Figure 4 the working process of the write interface is refined, and the write interface function prototype is ssize_t write (int fd, const void data, size_t len), the parameters being fd, data buffer, and size; the main process includes the following:

[0066] Through the passed-in fd (here, fd includes the fd of the named pipe (returned through the open interface) and the fd of the anonymous pipe (obtained through the pipe interface)), the existing global named pipe management structure fifo_mapping (the shared memory name is found through fd, and then searched based on the shared memory name) and the anonymous pipe global management structure (corresponding to Figure 4 the "anonymous and named global mapping structure" in the foregoing description) are found, it is judged whether the mapping exists, if not, it is a non-pipe operation, and the normal write function is directly called and returned. If it exists, the following operations are performed:

[0067] 1) The process to be written data obtains the shared memory area structure through the mmap (the passed-in parameter fd) operation. First, the shared memory area mutex is obtained by calling pthread_mutex_lock (& pipe->header.mutex), and then it is judged whether the size of the current used size plus the size of the to-be-written len is greater than the current total size, if yes, an overflow error is reported, and the process ends. Otherwise, the loop writing is started, and the core process is

[0068] while(len-- ){

[0069] write_pos= data++, write_pos = (write_pos + 1) % size); Meanwhile, update the current used size, increase it by len.

[0070] 2) Release the mutex pthread_mutex_unlock(&pipe->header.mutex), and unmap the shared memory area.

[0071] Embodiment 6

[0072] On the basis of the above embodiments, in combination with Figure 5 The read interface workflow is refined, and the interface function prototype is ssize_t read (int fd, void data, size_t len), the parameters are fd, data buffer and size, and the main process includes the following:

[0073] Through the passed-in fd, find the existing global named pipe management structure fifo_mapping and anonymous pipe global management structure (corresponding to Figure 5 "Anonymous and Named Global Mapping Structure" in the "Anonymous and Named Global Mapping Structure"), determine whether the mapping exists, if not, it is a non-pipe operation, directly call the normal read function to return. If it exists, perform the following operations:

[0074] 1) First, call pthread_mutex_lock(&pipe->header.mutex) to obtain the mutex of the shared memory area. Then start the loop to read (the core process is

[0075] while(len && current_size){

[0076] data++= read_pos, read_pos = (read_pos + 1) % size);

[0077] len--; / / read number minus 1

[0078] current_size--; / / current pipe usage number minus 1

[0079] }

[0080] 2) release mutex pthread_mutex_unlock(&pipe->header.mutex), unmap shared memory area.

[0081] It should be noted that the use flow of the anonymous pipe is: first, calling pipe to obtain fd, then using fd to perform read / write, and then close(fd). The use flow of the named pipe is: first, creating a named pipe by mkfifo, inputting path, then opening the named pipe (through its path) to obtain fd, and then using fd to perform read / write. That is, pipe and open will not be used in the same scenario, pipe is anonymous, and open is used for named pipe. The pipe interface prototype directly returns fd[2] to the user, and the user can directly initiate read / write subsequently, so it does not need to perform open; open is used for named pipe, and before use, it usually creates a named pipe by mkfifo to pass a path, and then calls open to open and use (actually, it also opens the shared memory area mapped therewith), two scenarios, so there are two different initialization processes.

[0082] In conclusion, the application proposes a method for accelerating and optimizing the existing Linux pipe communication application, which can directly and transparently accelerate and optimize the traditional pipe application without recompiling the existing application code, and meanwhile, the efficiency of inter-process data transmission is improved by using the shared memory technology.

[0083] Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

Claims

1. A Linux pipe communication performance optimization method, characterized in that, A self-developed acceleration library is set and compiled into a dynamic library, before application startup, the dynamic library is preloaded by using the LD_PRELOAD technology, so that the application preferentially calls the functions in the dynamic library; the self-developed acceleration library comprises the following interfaces: an mkfifo interface, a pipe interface, an open interface, a write interface and a read interface, wherein: The mkfifo interface is in communication connection with a named pipe, a function of the interface is called to dynamically generate a unique shared memory area name, then a pipe path of a user is bound with the dynamically generated unique shared memory area name, and initialization of a global named pipe management structure fifo_mapping is completed, the global named pipe management structure fifo_mapping records a corresponding relationship between the shared memory area name and the pipe path; The pipe interface is in communication connection with an anonymous pipe, a function of the interface is called to create a shared memory area, fd[2] for reading and writing is returned to a user, and based on the returned fd[2], an anonymous pipe global management structure is initialized, the anonymous pipe global management structure records a mapping relationship between the shared memory area corresponding to fd[2] and the anonymous pipe; The open interface is in communication connection with the named pipe, an open function of the interface is called to make a pipe path input through the interface find a uniquely corresponding shared memory area name, a shared memory area is created and an fd of the shared memory area mapping is returned, so as to facilitate subsequent reading and writing; The write interface judges whether the input fd is a pipe operation, if yes, a shared memory area corresponding to the pipe is found, and the shared memory area is written; otherwise, a normal write operation is performed; The read interface judges whether the input fd is a pipe operation, if yes, a shared memory area corresponding to the pipe is found, and the shared memory area is read; otherwise, a normal read operation is performed.

2. The Linux pipe communication performance optimization method of claim 1, wherein, The function prototype for the mkfifo interface is int mkfifo(const char pathname, mode_t mode).

3. The Linux pipe communication performance optimization method of claim 2, wherein, The working process of the mkfifo interface comprises the following steps: A user is connected to the mkfifo interface through a named pipe, according to a given path, firstly, a global named pipe management structure fifo_mapping is found, if the global named pipe management structure fifo_mapping exists in the given path, a system mkfifo function is called to return an existing mapping and end; if not, a unique shared memory area name is dynamically generated, a pipe path of the user is bound with the shared memory area name, recorded in the global named pipe management structure fifo_mapping, and initialization of the global named pipe management structure fifo_mapping is further completed, then a system mkfifo interface function is called to end the process.

4. The Linux pipe communication performance optimization method of claim 1, wherein, The function prototype of the pipe interface is int pipe(int fds[2]).

5. The Linux pipe communication performance optimization method of claim 4, wherein, The working process of the pipe interface comprises the following steps: The user connects to the pipe interface through an anonymous pipe interface, and the interface function is called to first dynamically generate a unique name as a shared memory area object name, open the shared memory area, obtain a read-write shared memory area handle fd, and then adjust the shared memory area size through ftruncate; thereafter, the shared memory area is mapped to the current process address space through mmap, and the header management data of the shared memory area is initialized, and then the opened shared memory area fd is stored in the incoming parameter fd[2] array of the pipe interface; finally, the opened shared memory area fd is recorded to the global management data of the anonymous pipe, which is used for subsequent reading and writing through the fd, and whether it is a legal pipe operation is determined by searching the global management structure.

6. The Linux pipe communication performance optimization method of claim 1, wherein, The function prototype for the open interface is int open(const char pathname, int flags,...).

7. The Linux pipe communication performance optimization method of claim 6, wherein, The workflow of the open interface includes the following: The user connects to the open interface through a named pipe, and finds the global named pipe management structure fifo_mapping of the existing named pipe through the incoming path to determine whether the mapping exists, and returns if the mapping does not exist; if the mapping exists, the shared memory area name corresponding to the incoming path is found and opened; Thereafter, the shared memory size is adjusted using the ftruncate function, the shared memory area is mapped to the address space of the current process through the mmap function, and the shared memory area header management data is initialized, and finally the process ends.

8. The Linux pipe communication performance optimization method of claim 1, wherein, The function prototype of the write interface is ssize_t write (int fd, const void data, size_t len), with parameters fd, data buffer, and size.

9. The Linux pipe communication performance optimization method of claim 8, wherein, The workflow of the write interface includes the following: According to the incoming fd, the existing global named pipe management structure fifo_mapping and the global anonymous pipe management structure are found to determine whether the mapping exists, and the normal write function is called to return if the mapping does not exist; if the mapping exists, the shared memory area associated with the incoming fd is found and opened, then the data is written in a loop and the write pointer is updated, and finally the mapping is released and the shared memory area is closed.

10. The Linux pipe communication performance optimization method of claim 1, wherein, The function prototype of the read interface is ssize_t read (int fd, void data, size_t len), with parameters fd, data buffer and size; the workflow includes the following: The existing global named pipe management structure fifo_mapping and the global anonymous pipe management structure are found through the incoming fd to determine whether the mapping exists, and the normal write function is called to return if the mapping does not exist; if the mapping exists, the shared memory area associated with the incoming fd is found and opened, then the data is read in a loop and the read pointer is updated, and finally the mapping is released and the shared memory area is closed.

Images