Linux pipeline communication performance optimization method

Abstract

The invention provides a Linux pipeline communication performance optimization method, and belongs to the technical field of pipeline communication.The method includes the steps that a self-developed acceleration library is set and compiled into a dynamic library, and before an application is started, the dynamic library is loaded in advance through the LDPRELOAD technology, so that the application preferentially calls functions in the dynamic library; and the self-developed acceleration library comprises the following interfaces: an mkfifo interface, a pipe interface, an open interface, a write interface and a read interface. Based on the LDPRELOAD dynamic preloading technology, a use interface of pipeline communication is kept unchanged, existing application codes do not need to be recompiled, transparent acceleration optimization is conducted on traditional pipeline communication of applications by means of the shared memory technology, and the efficiency of data transmission between processes is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to pipeline communication, in particular to a Linux pipeline communication performance optimization method, and belongs to the technical field of inter-process communication. BACKGROUND

[0002] In the inter-process communication (IPC) technology, the traditional pipeline is a commonly used communication method, which realizes streaming data transmission based on a kernel buffer, provides a standardized read / write operation interface, and is easy to use. A large number of application codes use the traditional pipeline standard interface to implement their respective businesses.

[0003] Shared memory is a high-efficiency inter-process communication method, which maps the same physical memory to the address space of different processes, avoids multiple copying of data between the kernel and the user space, and greatly improves the data transmission efficiency.

[0004] LD_PRELOAD is an environment variable of the Linux system, which allows a specified shared library to be loaded before a program runs, so as to use the functions defined in the library preferentially, and to realize the interception and replacement of system functions. This mechanism can transparently enhance the function without modifying the application code.

[0005] The traditional pipeline has certain performance overhead in the data transmission process. The data needs to be copied and context switched between the kernel and the user space multiple times, resulting in low transmission efficiency, especially in the large data transmission scenario, the performance bottleneck is obvious. The shared memory lacks standardized operation interfaces, and the developer needs to handle complex problems such as memory management and synchronization when using it, which increases the development difficulty and error risk, and is difficult to meet the needs of developers for ease of use. Therefore, how to use the shared memory technology to transparently accelerate the traditional pipeline communication has become a problem to be solved. SUMMARY

[0006] In view of the above problems, the present application provides a Linux pipeline communication performance optimization method to improve the efficiency of inter-process data transmission.

[0007] To achieve the above purpose, the technical scheme of the present application is as follows: a Linux pipeline communication performance optimization method, a self-developed acceleration library is set and compiled into a dynamic library, before the application starts, the dynamic library is loaded in advance by using the LD_PRELOAD technology, so that the application preferentially calls the functions in the dynamic library; the self-developed acceleration library includes the following interfaces: mkfifo interface, pipe interface, open interface, write interface, and read interface, wherein: The mkfifo interface is connected with a named pipe, a unique shared memory area name is dynamically generated by calling a function of the interface, the pipe path of the user is bound with the dynamically generated unique shared memory area name, and initialization of a global named pipe management structure fifo_mapping is completed, and the global named pipe management structure fifo_mapping records the correspondence between the shared memory area name and the pipe path; The pipe interface is connected with an anonymous pipe, a shared memory area is created by calling a function of the interface, fd[2] is returned to the user for reading and writing, and the global anonymous pipe management structure is initialized based on the returned fd[2], and the global anonymous pipe management structure records the mapping relationship between the fd[2] of the shared memory area and the anonymous pipe; The open interface is connected with a named pipe, and the open function of the interface is called to make the pipe path transmitted through the interface find the bound unique corresponding shared memory area name, create a shared memory area and return the fd of the shared memory area mapping, so as to facilitate subsequent reading and writing; The write interface judges whether the input fd is a pipe operation, if yes, the shared memory area of the corresponding pipe is found, and the shared memory area is written; otherwise, the normal writing operation is performed. The read interface judges whether the input fd is a pipe operation, if yes, the shared memory area of the corresponding pipe is found, and the shared memory area is read; otherwise, the normal reading operation is performed.

[0008] Further, the function prototype of the mkfifo interface is int mkfifo(const char pathname,mode_t mode).

[0009] Further, the working process of the mkfifo interface includes the following: The user connects to the mkfifo interface through a named pipe, according to the given path, first finds the existing global named pipe management structure fifo_mapping, if the global named pipe management structure fifo_mapping exists in the given path, the system mkfifo function is called to return the existing mapping and end; if not, a unique shared memory area name is dynamically generated, the pipe path of the user is bound with the shared memory area name, and the global named pipe management structure fifo_mapping is recorded, and the initialization of the global named pipe management structure fifo_mapping is further completed, and then the system mkfifo interface function is called to end the process.

[0010] Further, the function prototype of the pipe interface is int pipe(int fds[2]).

[0011] Further, the workflow of the pipe interface includes the following: The user connects to the pipe interface through the anonymous pipe interface, and calls the interface function to first dynamically generate a unique name as the shared memory object name, open the shared memory area, obtain the read-write shared memory area handle fd, and then adjust the shared memory area size through ftruncate. After that, the shared memory area is mapped to the current process address space through mmap, and the header management data of the shared memory area is initialized. Then, the opened shared memory area fd is stored in the pipe interface incoming parameter fd[2] array. Finally, the opened shared memory area fd is recorded to the global management data of the anonymous pipe, which is used for subsequent reading and writing through fd. The global management structure is traversed to check whether there is a corresponding fd, so as to determine whether it is a legal pipe operation.

[0012] Further, the function prototype of the open interface is int open(const char pathname, intflags,...).

[0013] Further, the workflow of the open interface includes the following: The user connects to the open interface through the named pipe, finds the global named pipe management structure fifo_mapping of the existing named pipe through the incoming path, and judges whether the mapping exists. If the mapping does not exist, the normal open function is called to return. If the mapping exists, the shared memory area name corresponding to the incoming path is found and opened. After that, the ftruncate function is used to adjust the shared memory size, the mmap function is used to map the shared memory area to the address space of the current process, and the shared memory area header management data is initialized based on this. Finally, the process is ended.

[0014] Further, the function prototype of the write interface is ssize_t write (int fd, const void data, size_t len), and the parameters are fd, data buffer and size.

[0015] Further, the workflow of the write interface includes the following: According to the incoming fd, find the existing global named pipe management structure fifo_mapping and anonymous pipe global management structure, judge whether the mapping exists, if not, call the normal write function to return, if exists, find the shared memory area associated with the incoming fd and open, then loop to write data and update the write pointer, finally unmapping and closing the shared memory area.

[0016] Further, the function prototype of the read interface is ssize_t read (int fd, void data,size_t len), the parameters are fd, data buffer and size; the workflow includes the following: According to the incoming fd, find the existing global named pipe management structure fifo_mapping and anonymous pipe global management structure, judge whether the mapping exists, if not, call the normal write function to return, if exists, find the shared memory area associated with the incoming fd and open, then loop to write data and update the write pointer, finally unmapping and closing the shared memory area.

[0017] The Linux pipe communication performance optimization method has the following beneficial effects: The method for transparently accelerating and optimizing the existing Linux pipe communication application is based on the LD_PRELOAD dynamic preloading technology, and the application using the pipe for communication is transparently accelerated and optimized; the method keeps the use interface of the pipe communication unchanged, does not need to recompile the existing application source code, simultaneously based on the shared memory cross-process sharing, realizes the transparent acceleration of the existing traditional pipe communication application, and improves the efficiency of inter-process data transmission. BRIEF DESCRIPTION OF DRAWINGS

[0018] The application will be further described in detail below in combination with the drawings and specific embodiments.

[0019] Figure 1 is the mkfifo interface process schematic diagram in the embodiment of the application; Figure 2 is the pipe interface process schematic diagram in the embodiment of the application; Figure 3 is the open interface process schematic diagram in the embodiment of the application; Figure 4 is the write interface process schematic diagram in the embodiment of the application; Figure 5 is the read interface process schematic diagram in the embodiment of the application. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application, but the present application can also be implemented in other manners different from those described herein, and those skilled in the art can make similar generalizations without departing from the spirit and scope of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0021] Embodiment 1

[0022] A Linux pipeline communication performance optimization method mainly sets a self-developed acceleration library, the self-developed acceleration library maintains the same use interface (including name and interface prototype) as traditional pipeline communication (named pipe and anonymous pipe); the self-developed acceleration library is compiled into a dynamic library, based on the LD_PRELOAD technology, before application startup, the function in the dynamic library is preloaded, so as to realize runtime transparent replacement of the original traditional pipeline interface. Specifically, the self-developed acceleration library mainly includes the following interfaces: mkfifo interface, pipe interface, open interface, write interface, and read interface. Among them, The mkfifo interface is used in the named pipe scenario, and is usually used for cross-process communication; the named pipe communication is connected, the function of the interface is called to dynamically generate a unique shared memory area name, the pipe path of the user is bound with the dynamically generated unique shared memory area name, and the initialization of the global named pipe management structure fifo_mapping is completed, the global named pipe management structure fifo_mapping records the correspondence between the shared memory area name and the pipe path; the interface uses a pipe path input by the user to create a named pipe, and different processes can use the path to open the named pipe thereafter, and then read and write.

[0023] The pipe interface is used in the anonymous pipe scenario, and is usually used for parent-child process communication; the interface is connected with the anonymous pipe communication, by calling the interface function, fd[2] (one fd for reading and one fd for writing) is returned to the user for reading and writing. The internal implementation will dynamically create a shared memory area, record the fd[2] returned to the user, and initialize the anonymous pipe global management structure based on the returned fd[2], the anonymous pipe global management structure records the mapping relationship between the shared memory area corresponding to the fd[2] and the anonymous pipe. When the user uses the fd[2] for reading and writing later, the associated shared memory area can be found from the anonymous pipe global management structure, so as to complete efficient reading and writing based on the shared memory area.

[0024] open interface, which is used in the well-known pipe scenario, is used to open an already created well-known pipe; the well-known pipe is connected in communication, and by calling the open function of the interface, the pipe path transmitted through the interface finds the bound unique corresponding shared memory area name, completes the creation of the shared memory area, and returns the fd of the created shared memory area mapping, so as to subsequently read and write.

[0025] write interface, for the incoming fd, judges whether it is a pipe operation, if yes, finds the shared memory area of the corresponding pipe, and writes to the shared memory area; otherwise, the normal write operation is performed.

[0026] read interface, for the incoming fd, judges whether it is a pipe operation, if yes, finds the shared memory area of the corresponding pipe, and reads the shared memory area; otherwise, the normal read operation is performed.

[0027] Embodiment 2

[0028] On the basis of embodiment 1, in combination with Figure 1 Further refine the working process of the mkfifo interface, the function prototype of the interface is int mkfifo(const char pathname, mode_t mode), and the specific process includes the following: The user connects to the mkfifo interface through the well-known pipe, according to the given pipe path, first finds the existing global well-known pipe management structure fifo_mapping, if the global well-known pipe management structure fifo_mapping exists in the mapping corresponding to the given path, calls the system mkfifo interface function to return the existing mapping and ends; if not, dynamically generates a unique shared memory area name, and binds the user's pipe path with the shared memory area name, records it to the global well-known pipe management structure fifo_mapping, further completes the initialization of the global well-known pipe management structure fifo_mapping, and then calls the system mkfifo interface function to end the process.

[0029] Embodiment 3

[0030] On the basis of embodiment 1, in combination with Figure 2 Further refine the working process of the pipe interface, the function prototype of the interface is int pipe(int fds[2]), and the specific process includes the following: The user connects to the pipe interface through the anonymous pipe interface, and calling the interface function first dynamically generates a unique name as the shared memory area object name, opens the shared memory area, and obtains the read-write shared memory area handle fd (fd is an identifier of the shared memory area, used when reading and writing the shared memory area); then adjusts the shared memory area size through ftruncate, and then maps the shared memory area to the current process address space through mmap, and initializes the header management data of the shared memory area, and then stores the opened shared memory area fd into the pipe interface incoming parameter fd[2] array (the user can use fd[2] for subsequent reading and writing), and finally records the opened shared memory area fd to the global related management data of the anonymous pipe, which is used for subsequent reading and writing through fd, and through traversing the global management structure to find whether there is a corresponding fd, so as to judge whether it is a legal pipe operation.

[0031] Embodiment 4

[0032] On the basis of embodiment 2, in combination with Figure 3 Further refine the open interface workflow, the function prototype of this interface is int open(const char pathname, int flags,...), the specific process includes the following: The user connects to the open interface through the named pipe, finds the existing global named pipe management structure fifo_mapping through the incoming path (pathname), judges whether the mapping exists, if not, it is a non-pipe operation, directly call the normal open function to return. If it exists, find the name of the shared memory area corresponding to the path, and then perform the following operations: 1) Use the shm_open function to create a shared memory area, use the shared memory area name (for example, / my_shared_pipe), set the opening mode as O_CREAT | O_RDWR, and the file permission as 0666. Use the ftruncate function to adjust the shared memory size to the specified value (for example, 1MB). Map the shared memory area to the address space of the current process through the mmap function, and obtain the shared memory area structure pointer SharedPipe pipe. The beginning of the SharedPipe shared memory area is the header structure, which stores the basic management data, synchronization semaphores and mutexes for accessing the shared memory area. The data format is as follows: struct pipe_share_header { pthread_mutex_t mutex; int size; / / total available size of current data int current_size; / / current used size of data char read_pos; / / current read position char write_pos; / / current write position char data[0]; / / logically circular buffer 2) Initialize the header data in the shared memory area structure, set the total size of the buffer to the size of the shared memory area (such as 1MB), the read pointer position read_pos and the write pointer position write_pos to 0, the read process count read_count and the write process count write_count to 0, the buffer full flag is_full to 0, and the buffer empty flag is_empty to 1. Initialize the mutex pthread_mutex_init(&pipe->header.mutex, NULL) and the semaphores sem_init(&pipe->header.read_sem, 1, 0) and sem_init(&pipe->header.write_sem, 1, 1024 1024) 3) Return the handle fd of the shared memory area and end the process.

[0033] Example 5

[0034] On the basis of the above examples, in combination Figure 4 with the write interface, the working process of the write interface is refined, and the write interface function prototype is ssize_t write (int fd, const void data, size_t len), the parameters are fd, data buffer and size; the main process includes the following: Through the passed-in fd (here fd includes the fd of the named pipe (returned by the open interface) and the fd of the anonymous pipe (obtained by the pipe interface)), find the existing global named pipe management structure fifo_mapping (specifically find the shared memory name based on the fd, and then find based on the shared memory name) and the anonymous pipe global management structure (corresponding to Figure 4 "Anonymous and Named Global Mapping Structure" in the above embodiment), judge whether the mapping exists, if not, it is a non-pipe operation, directly call the normal write function to return. If it exists, perform the following operations:​ 1) The process of data to be written gets the shared memory area structure through mmap (pass in parameter fd) operation. First, call pthread_mutex_lock (& pipe-> header. mutex) to get the mutex lock of the shared memory area, and then judge whether the size currently used plus the size of the data to be written is greater than the current total size. If so, report an overflow error and return. Otherwise, start the loop to write, and the core process is while(len-- ){ write_pos= data++, write_pos = (write_pos + 1) % size); At the same time, update the current used size, and increase len.

[0035] 2) Release the mutex pthread_mutex_unlock (& pipe-> header. mutex), and release the shared memory area mapping.

[0036] Embodiment 6

[0037] On the basis of the above embodiment, in combination with Figure 5 the read interface workflow is refined, and the interface function prototype is ssize_t read (int fd, void data, size_t len), the parameters are fd, data buffer and size, and the main process includes the following: Through the passed-in fd, find the existing global named pipe management structure fifo_mapping and anonymous pipe global management structure (corresponding to Figure 5 "Anonymous and named global mapping structure" in the above embodiment), judge whether the mapping exists. If not, it is a non-pipe operation, and the normal read function is directly called to return. If it exists, the following operations are performed: 1) First, call pthread_mutex_lock (& pipe-> header. mutex) to get the mutex lock of the shared memory area. Then start the loop to read (the core process is while(len && current_size){ data++= read_pos, read_pos = (read_pos + 1) % size); len--; / / read number minus 1 current_size--; / / current pipe usage number minus 1 } 2) release mutex pthread_mutex_unlock(&pipe->header.mutex), release shared memory area mapping.

[0038] It should be noted that the use flow of the anonymous pipe is: first, calling pipe to obtain fd, then using fd to perform read / write, and then close(fd). The use flow of the named pipe is: first, creating a named pipe by calling mkfifo and inputting path, then opening the named pipe (by the path) to obtain fd, and then using fd to perform read / write. That is, pipe and open will not be used in the same scenario, pipe is anonymous, and open is used for the named pipe. The pipe interface prototype directly returns fd[2] to the user, and the user can directly initiate read / write subsequently, so it is not necessary to perform open; open is used for the named pipe, and before use, a path is usually transmitted by calling mkfifo to create a named pipe, and then open is called to open and use (actually, the shared memory area mapped therewith is also opened) the named pipe, two scenarios, so there are two different initialization processes.

[0039] In conclusion, the application proposes a method for accelerating and optimizing the existing Linux pipe communication application, which can directly and transparently accelerate and optimize the traditional pipe application without recompiling the existing application code, and meanwhile, the efficiency of inter-process data transmission is improved by using the shared memory technology.

[0040] Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

Claims

1. A method for optimizing Linux pipe communication performance, characterized in that, A self-developed acceleration library is set up and compiled into a dynamic library. Before the application starts, the dynamic library is preloaded using LD_PRELOAD technology, so that the application calls functions in the dynamic library first. The self-developed acceleration library includes the following interfaces: mkfifo interface, pipe interface, open interface, write interface, and read interface, wherein: The mkfifo interface communicates with named pipes. It dynamically generates a unique shared memory region name by calling the function of this interface, then binds the user's pipe path to the dynamically generated unique shared memory region name, and completes the initialization of the global named pipe management structure fifo_mapping. The global named pipe management structure fifo_mapping records the correspondence between the shared memory region name and the pipe path. The pipe interface communicates with the anonymous pipe. By calling the function of the interface, a shared memory area is created and returned to the user for reading and writing fd[2]. The global management structure of the anonymous pipe is initialized based on the returned fd[2]. The global management structure of the anonymous pipe records the mapping relationship between the fd[2] corresponding to the shared memory area and the anonymous pipe. The open interface communicates with named pipes. By calling the open function of this interface, the unique corresponding shared memory region name is found through the pipe path passed through the interface, the shared memory region is created, and the file descriptor (fd) mapped to the shared memory region is returned for subsequent read and write operations. The write interface determines whether the passed file descriptor (fd) is a pipe operation. If it is, it finds the corresponding shared memory area of ​​the pipe and writes to the shared memory area; otherwise, it performs a normal write operation. The read interface determines whether the passed file descriptor (fd) is a pipe operation. If it is, it finds the corresponding shared memory area of ​​the pipe and reads from the shared memory area; otherwise, it performs a normal read operation.

2. The Linux pipe communication performance optimization method according to claim 1, characterized in that, The function prototype of the mkfifo interface is int mkfifo(const char pathname, mode_t mode).

3. The Linux pipe communication performance optimization method according to claim 2, characterized in that, The workflow of the mkfifo interface includes the following: Users connect to the mkfifo interface via named pipes. Following the given path, the system first searches for an existing global named pipe management structure, fifo_mapping. If a mapping corresponding to the given path exists in fifo_mapping, the system mkfifo function is called to return the existing mapping and the process ends. If no mapping exists, a unique shared memory area name is dynamically generated, and the user's pipe path is bound to this shared memory area name, recorded in the fifo_mapping structure. This further initializes the fifo_mapping structure, and then the system mkfifo interface function is called to end the process.

4. The Linux pipe communication performance optimization method according to claim 1, characterized in that, The function prototype of the pipe interface is int pipe(int fds[2]).

5. The Linux pipe communication performance optimization method according to claim 4, characterized in that, The workflow of the pipe interface includes the following: Users connect to the pipe interface through the anonymous pipe interface. The interface function first dynamically generates a unique name as the name of the shared memory area object, opens the shared memory area, obtains the handle fd for reading and writing the shared memory area, and then adjusts the size of the shared memory area through ftruncate. After that, the shared memory area is mapped to the current process address space through mmap, and the header management data of the shared memory area is initialized. Then, the opened shared memory area fd is stored in the fd[2] array of the input parameters of the pipe interface. Finally, the opened shared memory area fd is recorded to the global management data related to the anonymous pipe, which is used to check whether there is a corresponding fd by traversing the global management structure when reading and writing through fd in the future, so as to determine whether it is a legal pipe operation.

6. The Linux pipe communication performance optimization method according to claim 1, characterized in that, The function prototype of the open interface is int open(const char pathname, int flags, ...).

7. The Linux pipe communication performance optimization method according to claim 6, characterized in that, The workflow of the open interface includes the following: Users connect to the open interface through a named pipe. The interface searches for the existing global named pipe management structure fifo_mapping based on the passed path, and determines whether a mapping already exists. If no mapping exists, the normal open function is called and returned. If a mapping exists, the shared memory area name corresponding to the passed path is found and opened. Next, the ftruncate function is used to adjust the size of the shared memory, the mmap function is used to map the shared memory area to the address space of the current process, and the header management data of the shared memory area is initialized accordingly, and finally the process ends.

8. The Linux pipe communication performance optimization method according to claim 1, characterized in that, The function prototype of the write interface is ssize_t write(int fd, const void). The function is defined as follows: (data, size_t len), with parameters being fd, data buffer, and size.

9. A method for optimizing Linux pipe communication performance according to claim 8, characterized in that, The workflow of the write interface includes the following: Based on the passed-in file descriptor (fd), search for the existing global named pipe management structure fifo_mapping and the global anonymous pipe management structure, and determine whether a mapping already exists. If it does not exist, call the normal write function and return; if it exists, find the shared memory area associated with the passed-in file descriptor and open it, then write data in a loop and update the write pointer, and finally unmap and close the shared memory area.

10. A method for optimizing Linux pipe communication performance according to claim 1, characterized in that, The function prototype of the read interface is ssize_t read (int fd, void ... The function `data, size_t len` takes `fd` as its parameter, which represents the data buffer and its size. The workflow includes the following: The system searches for existing global named pipe management structures (fifo_mapping and anonymous pipe global management structures) using the passed-in file descriptor (fd). It checks if a mapping already exists. If not, it calls the normal write function and returns. If a mapping exists, it finds the shared memory area associated with the passed-in fd, opens it, then reads data in a loop and updates the read pointer. Finally, it unmaps the shared memory area and closes it.

Images