Cooperative communication method of storage host process and compression process and electronic equipment

Abstract

The invention discloses a cooperative communication method of a storage host process and a compression process and an electronic device, and relates to the technical field of process communication, by setting a preset communication channel between the storage host process and the compression process, a control instruction is separated from data processing, and the instruction is transmitted through the preset communication channel. According to the method, timely delivery and sequential execution of operation instructions are ensured, and data are exchanged through a shared memory area managed by a kernel, so that the problem of multiple data copying in pipeline transmission is avoided, ordered bidirectional cooperation between a storage host process and a compression process is realized through a standardized instruction interaction mechanism, and the operation efficiency is improved. The problem of memory management confusion caused by lack of a coordination mechanism in a traditional shared memory scheme is fundamentally solved, and dual improvement of inter-process communication efficiency and resource management efficiency is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of process communication, and in particular to a cooperative communication method of a storage main process and a compression process and an electronic device. BACKGROUND

[0002] There are mainly two implementation paths for the data compression architecture of the current storage system: one is to deeply integrate the compression function in the storage main process, and the other is to run the compression function as an independent process. However, deeply integrating the compression function in the storage main process causes the close coupling of the storage service and the compression algorithm, making it difficult to maintain and upgrade the system; running the compression function as an independent process causes the efficiency bottleneck of inter-process communication. In order to establish a communication channel between the independent compression process and the storage main process, the related technology often uses standard inter-process communication technologies such as pipe transmission or shared memory.

[0003] However, pipe transmission has significant data copy overhead when transmitting large-scale data, affecting the overall performance; while shared memory reduces data copying, but cannot support the two-way cooperation between the storage main process and the compression process, and the storage main process is difficult to pass the positioning identifier of the data to be compressed in the kernel shared memory to the compression process, and the compression process also cannot feedback the specific location information of the compression result in the kernel space to the storage main process through an effective way after processing, resulting in that the kernel shared memory space cannot be coordinated and utilized by the user process, which not only reduces the data processing efficiency, but also causes the disorder of memory resource management. SUMMARY

[0004] The present application provides a cooperative communication method of a storage main process and a compression process and an electronic device to at least solve the problem that the data compression process of the storage system in the related technology causes the disorder of memory resource management due to the lack of inter-process two-way coordination mechanism of shared memory.

[0005] The present application provides a cooperative communication method of a storage main process and a compression process, applied to the compression process, the method comprising: receiving a data operation instruction through a preset communication channel between the storage main process and the compression process; reading data from the shared memory based on the data operation instruction and processing the data to obtain a processing result; storing the processing result in the shared memory and generating a corresponding data operation completion instruction, wherein the storage main process reads the processing result from the shared memory based on the data operation completion instruction for processing.

[0006] The present application provides an electronic device comprising a memory, a processor, and a cooperative communication program of a storage main process and a compression process stored on the memory and executable on the processor. When the processor executes the cooperative communication program of the storage main process and the compression process, the cooperative communication method of the storage main process and the compression process described above is implemented.

[0007] The application separates the control instruction and data processing by setting a preset communication channel between the storage master process and the compression process, transmits the instruction through the preset communication channel, ensures the timely delivery and sequential execution of the operation instruction, and exchanges the data through the shared memory area managed by the kernel, thereby avoiding the multiple data copying problem in the pipeline transmission, realizing the ordered and bidirectional cooperation between the storage master process and the compression process through the standardized instruction interaction mechanism, and fundamentally solving the memory management confusion problem caused by the lack of coordination mechanism in the traditional shared memory scheme, and realizing the dual improvement of the inter-process communication efficiency and the resource management efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0008] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0009] Figure 1 The flow chart of the cooperative communication method of the storage master process and the compression process according to some embodiments of the present application; Figure 2 The structural schematic diagram of the cooperative communication system of the storage master process and the compression process according to some embodiments of the present application; Figure 3 The flow chart of the cooperative communication method of the storage master process and the compression process according to some embodiments of the present application; Figure 4 The interaction process schematic diagram between the storage master process and the compression process when writing data according to some embodiments of the present application; Figure 5 The interaction process schematic diagram between the storage master process and the compression process when reading data according to some embodiments of the present application; Figure 6 The block schematic diagram of the electronic device according to some embodiments of the present application. DETAILED DESCRIPTION

[0010] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0011] It should be noted that in the description of the present application, the terms "comprising", "including" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or further includes elements inherent to such a process, method, article or device. The terms "first", "second" and the like in the present application are applicable to distinguish similar objects, and are not used to describe a specific order or sequence.

[0012] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below in combination with the drawings and specific embodiments.

[0013] In combination with the specific application environment architecture or specific hardware architecture on which the cooperative communication method of the storage main process and the compression process and the electronic device are dependent, the specific application environment architecture or specific hardware architecture is described here.

[0014] The embodiments of the present application provide a cooperative communication method of a storage main process and a compression process, and the method is described in detail in combination with the execution flow of the cooperative communication method of the storage main process and the compression process.

[0015] With the wide application of cloud computing and big data technology, storage business is facing the challenge of order-level growth of data volume, and the daily data increment of enterprise-level storage system has reached TB level. Under this background, "data storage and compression optimization" has become a core requirement - to reduce storage occupation and transmission bandwidth consumption through compression technology, but at the same time, higher requirements are put forward for the cooperation efficiency of storage process and compression process, data security and system stability.

[0016] There are mainly two implementation paths for the data compression architecture of the current storage system: one is to deeply integrate the compression function into the storage main process, and the other is to run the compression function as an independent process.

[0017] The limitations of the integrated architecture are as follows: The main process needs to handle multiple tasks such as data read / write, metadata management and compression calculation, which causes the CPU (Central Processing Unit) load to rise significantly, and further causes the storage I / O (Input / Output) delay to increase; any update of the compression function, such as switching from LZ4 (Lempel-Ziv4, lossless data compression algorithm) to ZSTD (Zstandard, a data compression algorithm), needs to recompile the main process code, and cannot realize hot plug expansion; the exception of the compression module (such as memory leakage) will directly cause the storage main process to crash, and produce a chain failure risk.

[0018] The limitations of the independent process architecture are as follows: Data exchange through temporary files introduces two additional lower-level disk I / O operations, significantly increasing data processing delay; the lack of effective task instruction and state feedback mechanism makes it difficult for the storage master process to grasp the processing status of the compression process in real time, which may lead to data accumulation or task loss.

[0019] To solve the problem of inter-process data interaction, related technologies usually use general IPC (Inter-Process Communication) mechanisms such as FIFO (First In, First Out) and shared memory.

[0020] The limitations of FIFO are as follows: When used to transmit large-scale storage data, multiple user-kernel state data copying is required, resulting in a significant increase in transmission delay. In addition, FIFO cannot directly establish an association between instructions and data, and additional file path information is required to identify data location, increasing system complexity and error risk.

[0021] The limitations of shared memory are as follows: The shared area is exposed to the user space and is vulnerable to tampering by malicious processes or abnormal process exit; processes need to implement complex synchronization mechanisms (such as mutexes and semaphores) among themselves, which may lead to deadlocks and other problems if not designed properly.

[0022] Among them, the kernel shared memory scheme can provide isolation protection using the kernel space, but it is usually limited to single-process kernel data caching purposes and lacks organic integration with communication mechanisms such as FIFO, making it difficult to support complete collaboration between the storage master process and the compression process. The specific manifestations are as follows: the storage master process cannot pass data identifiers in the kernel shared memory in a lightweight manner; the compression process cannot effectively feedback the location information of the result data in the kernel space after processing, making it difficult for the kernel space data resources to be coordinated and utilized by the user process.

[0023] Based on this, the present application proposes a collaboration communication method for a storage master process and a compression process. Referring to Figure 1 The collaboration communication method for the storage master process and the compression process according to the embodiments of the present application can include the following steps: S110, receiving a data operation instruction through a preset communication channel between the storage master process and the compression process.

[0024] S120, reading data from the shared memory based on the data operation instruction and processing the data to obtain a processing result, wherein when the data operation instruction is a write data instruction, reading the to-be-compressed data from the first storage space based on the write data instruction and performing compression processing on the to-be-compressed data to obtain the processing result, wherein the first storage space is in the shared memory.

[0025] S130, storing the processing result into the shared memory and generating a corresponding data operation completion instruction, wherein the storage master process reads the processing result from the shared memory based on the data operation completion instruction for processing.

[0026] Specifically, referring to Figure 2 , the preset communication channel set between the storage master process and the compression process can be FIFO, and the compression process can receive the data operation instruction through the preset communication channel, such as receiving the write data instruction and the read data instruction.

[0027] After receiving the data operation instruction, the compression process reads data from the shared memory based on the data operation instruction and processes the data to obtain a processing result. For example, in the case that the compression process receives the write data instruction, it indicates that the storage master process needs to compress the data that the upper host needs to write to the lower disk, therefore, the compression process needs to read the data to be written from the shared memory and perform compression processing on the data to be written to obtain the compression result; in the case that the compression process receives the read data instruction, it indicates that the storage master process needs to decompress the data that needs to be read from the lower disk to the upper host, therefore, the compression process needs to read the data to be read from the shared memory and perform compression processing on the data to be read to obtain the decompression result.

[0028] Specifically, the data operation instruction includes a write data instruction, and in the case that the compression process receives the write data instruction, it indicates that the storage master process needs to compress the data that the upper host needs to write to the lower disk. Therefore, the storage master process writes the to-be-compressed data (the data that the upper host needs to write to the lower disk) into the shared memory, specifically into the first storage space of the shared memory, and the compression process receives the write data instruction through the preset communication channel, analyzes the write data instruction, determines the storage location of the to-be-compressed data, that is, determines the first storage space; then, reads the to-be-compressed data from the first storage space and uses a preset compression algorithm, such as LZ4, ZSTD, etc., to perform compression processing on the to-be-compressed data to obtain compressed data and metadata corresponding to the compressed data. The metadata corresponding to the compressed data refers to the attribute information of the compressed data, which specifically can include the storage address and length of the compressed data in the shared memory, the size of the original data before compression, the type of the compression algorithm used, the compression task sequence number, etc.

[0029] After the compression process reads data from the shared memory for processing to obtain a processing result, the processing result needs to be written into the shared memory, and a corresponding data operation completion instruction is generated, and then the data operation completion instruction is sent to the storage master process, so that the storage master process reads the processing result from the shared memory based on the data operation completion instruction for processing. For example, in the case of writing the compression result into the shared memory, a write data completion instruction is generated, and the write data completion instruction is sent to the storage master process, and after the storage master process receives the write data completion instruction, the compression result is read from the shared memory, and the read result is written into the lower layer disk; in the case of writing the decompression result into the shared memory, a decompression completion instruction is generated, and the decompression completion instruction is sent to the storage master process, and after the storage master process receives the decompression completion instruction, the decompression result is read from the shared memory, and the read result is returned to the upper host.

[0030] The present application separates the control instruction and the data processing by setting a preset communication channel between the storage master process and the compression process. The instruction is transmitted through the preset communication channel to ensure timely delivery and sequential execution of the operation instruction, while the data is exchanged through the shared memory area managed by the kernel. The orderly bidirectional cooperation between the storage master process and the compression process is realized through the standardized instruction interaction mechanism, which fundamentally solves the memory management confusion problem caused by the lack of coordination mechanism in the traditional shared memory scheme, and realizes the dual improvement of the inter-process communication efficiency and the resource management efficiency.

[0031] In some embodiments, the processing result includes compressed data and metadata corresponding to the compressed data, and storing the processing result into the shared memory includes: storing the compressed data and the metadata corresponding to the compressed data into a second storage space and a third storage space respectively, wherein the second storage space and the third storage space are in the shared memory.

[0032] Specifically, after the compression process compresses the to-be-compressed data to obtain a processing result, the processing result is stored into the shared memory. Specifically, the processing result includes compressed data and metadata corresponding to the compressed data, the compressed data is stored into a second storage space of the shared memory, and the metadata corresponding to the compressed data is stored into a third storage space of the shared memory.

[0033] In some embodiments, when the data operation completion instruction is a write data completion instruction, after the compressed data and the metadata corresponding to the compressed data are stored into the second storage space and the third storage space respectively, the method further includes: sending the write data completion instruction to the storage master process through the preset communication channel; wherein the write data completion instruction includes an address of the second storage space and an address of the third storage space.

[0034] Specifically, after the compression process stores the compressed data and the metadata corresponding to the compressed data into the second storage space and the third storage space respectively, the compression process needs to generate a data operation completion instruction, specifically a write data completion instruction, wherein the write data completion instruction specifically includes the address of the second storage space and the address of the third storage space; then, the compression process sends the write data completion instruction to the storage master process through the preset communication channel.

[0035] The present application stores the compressed data and the metadata physically separately in the shared memory, and transmits the write data completion instruction through the preset communication channel to transmit the address information of the storage positions of the compressed data and the metadata. In this way, the separate storage of the data and the metadata avoids the analysis complexity and potential data boundary errors in the mixed storage mode, improves the accuracy and security of data access, and greatly reduces the data volume of inter-process communication by transmitting only the storage addresses instead of the data itself, thereby fundamentally solving the copy overhead and delay problems caused by the transmission of large volume data in the traditional pipeline.

[0036] In some embodiments, when the data operation instruction is a read data instruction, the data is read from the shared memory based on the data operation instruction for processing to obtain a processing result, including: determining the logical address of the data to be read and the address of the fourth storage space based on the read data instruction, and searching the metadata based on the logical address of the data to be read to determine the physical block address of the data to be read, while generating a corresponding search completion instruction, wherein the fourth storage space is in the shared memory; sending the search completion instruction to the storage master process through the preset communication channel, wherein the search completion instruction includes the physical block address of the data to be read; receiving a read data completion instruction through the preset communication channel; reading the compressed data from the fourth storage space based on the read data completion instruction, and decompressing the compressed data to obtain the processing result, wherein the processing result includes the decompressed data, and the compressed data is read from the lower disk by the storage master process based on the search completion instruction and then written into the fourth storage space.

[0037] Specifically, the data operation instruction includes a read data instruction. After the compression process receives the read data instruction through the preset communication channel, the compression process analyzes the read data instruction, determines the logical address corresponding to the data to be read, and determines the fourth storage space address for temporarily storing the read result; then, the compression process queries and retrieves the metadata record associated with the determined logical address based on the determined logical address, analyzes the physical block address where the data to be read is actually stored from the metadata, and completes the conversion from the application layer logical address to the lower storage medium physical address; finally, the compression process generates a search completion instruction based on the physical block address, so that the storage master process reads the compressed data from the lower disk based on the search completion instruction and stores the compressed data into the fourth storage space, and then waits for the storage master process to generate a read data completion instruction.

[0038] The compression process reads the compressed data from the fourth storage space after receiving the read data completion instruction through the preset communication channel, and decompresses the compressed data based on a preset decompression algorithm to obtain a processing result, which specifically includes decompressed data. It should be noted that the preset decompression algorithm needs to correspond to the compression algorithm used when the data is compressed, and the compression algorithm of the data can be determined by querying the corresponding metadata. In this way, the accuracy and decompression speed of data decompression can be improved.

[0039] In some embodiments, storing the processing result in the shared memory includes: storing the decompressed data back to the fourth storage space.

[0040] Specifically, after the compression process decompresses the compressed data to obtain the processing result, the processing result is stored in the shared memory. Specifically, the processing result includes the decompressed data, and the decompressed data is stored back to the fourth storage space.

[0041] When the data operation completion instruction is a decompression completion instruction, after the decompressed data is stored back to the fourth storage space, the method further includes: sending the decompression completion instruction to the business process through the preset communication channel.

[0042] Specifically, after the compression process stores the decompressed data in the fourth storage space, a data operation completion instruction needs to be generated, specifically a decompression completion instruction; then, the decompression completion instruction is sent to the storage master process through the preset communication channel.

[0043] The present application optimizes the allocation of computing resources by assigning the physical address resolution task to the compression process; the find completion instruction and the read data completion instruction are transmitted through the preset communication channel, and the find completion instruction and the read data completion instruction cooperate to ensure operation reliability while avoiding unnecessary data copying; through the reuse of the fourth storage space, seamless connection of compressed and decompressed data is achieved, effectively improving memory usage efficiency.

[0044] Figure 3 The flowchart of the cooperative communication method of the storage master process and the compression process according to some other embodiments of the present application is shown in FIG. 10. Referring to FIG. 10, Figure 3 The cooperative communication method of the storage master process and the compression process according to the embodiments of the present application can further include the following steps: S210, generating a data operation instruction and sending the data operation instruction to the compression process through a preset communication channel between the storage master process and the compression process, wherein when the data operation instruction is a write data instruction, the write data instruction includes the address of the first storage space, wherein the first storage space is in the shared memory, and the first storage space is used to store the compressed data written by the storage master process.

[0045] S220, receiving the data operation completion instruction through the preset communication channel.

[0046] S230, processing the processing result read from the shared memory based on the data operation completion instruction.

[0047] Specifically, when the upper host needs to write data to the lower disk, the storage master process needs to generate a write data instruction and send the write data instruction to the compression process through the preset communication channel between the storage master process and the compression process. Specifically, when the storage master process needs to write the data of the upper host to the lower disk, the storage master process writes the data to be compressed to the shared memory, specifically to the first storage space, and generates a write data instruction, and transmits the write data instruction to the compression process through the preset communication channel. When the upper host needs to read the data of the lower disk back to the upper host, the storage master process needs to generate a read data instruction and send the read data instruction to the compression process through the preset communication channel.

[0048] Further, the compression process receives the data operation instruction, reads the data from the shared memory based on the data operation instruction, and processes the data to obtain a processing result, stores the processing result in the shared memory, and generates a corresponding data operation completion instruction, and transmits the data operation completion instruction to the storage master process through the preset communication channel. For example, in the case that the compression process receives the write data instruction, it indicates that the storage master process needs to compress the data to be written to the lower disk, so the compression process needs to read the data to be written from the shared memory and compress the data to be written to obtain a compression result; in the case that the compression process receives the read data instruction, it indicates that the storage master process needs to decompress the data to be read from the lower disk to the upper host, so the compression process needs to read the data to be read from the shared memory and compress the data to be read to obtain a decompression result.

[0049] After the compression process reads the data from the shared memory to obtain the processing result, it needs to write the processing result to the shared memory, generate a corresponding data operation completion instruction, and then send the data operation completion instruction to the storage master process, so that the storage master process processes the processing result read from the shared memory based on the data operation completion instruction. For example, in the case that the compression result is written to the shared memory, a write data completion instruction is generated and sent to the storage master process, and the storage master process receives the write data completion instruction, reads the compression result from the shared memory, and writes the read result to the lower disk; in the case that the decompression result is written to the shared memory, a decompression completion instruction is generated and sent to the storage master process, and the storage master process receives the decompression completion instruction, reads the decompression result from the shared memory, and returns the read result to the upper host.

[0050] The preset communication channel between the storage master process and the compression process is set, the control instruction and the data processing are separated, the instruction is transmitted through the preset communication channel, the timely delivery and sequential execution of the operation instruction are ensured, the data is exchanged through the shared memory area managed by the kernel, the problem of multiple data copying in the pipeline transmission is avoided, the ordered and bidirectional cooperation between the storage master process and the compression process is realized through the standardized instruction interaction mechanism, the memory management confusion problem caused by the lack of coordination mechanism in the traditional shared memory scheme is fundamentally solved, and the efficiency of inter-process communication and the effectiveness of resource management are improved.

[0051] In some embodiments, when the data operation completion instruction is a write data completion instruction, processing based on the processing result read from the shared memory based on the data operation completion instruction includes: writing the compressed data in the second storage space and the metadata in the third storage space to the lower layer disk based on the write data completion instruction, wherein the compressed data and the metadata of the compressed data are generated by the compression process and stored in the second storage space and the third storage space, respectively.

[0052] Specifically, after the compression process stores the compressed data and the metadata corresponding to the compressed data in the second storage space and the third storage space, respectively, the compression process needs to generate a data operation completion instruction, specifically a write data completion instruction, wherein the write data completion instruction specifically includes the address of the second storage space and the address of the third storage space; then, the write data completion instruction is sent to the storage master process through the preset communication channel.

[0053] After the storage master process receives the write data completion instruction, the storage master process parses the write data completion instruction to determine the address of the second storage space and the address of the second storage space, so as to write the compressed data in the second storage space and the metadata in the third storage space to the lower layer disk.

[0054] In this way, the storage master process only needs to pass the address of the to-be-compressed data in the shared memory through a lightweight write data instruction to trigger the compression process of the compression process, and after the compression is completed, the storage address of the compressed data and the metadata is returned through the write data completion instruction. In this way, the inter-process communication always remains at the lightweight interaction level of instructions and addresses, and the performance overhead of large-scale data copying and transmission between processes in the traditional scheme is completely avoided.

[0055] In some embodiments, when the data operation instruction is a read data instruction, the read data instruction includes a logical address of to-be-read data and an address of a fourth storage space, wherein the fourth storage space is in the shared memory.

[0056] Specifically, when the storage master process reads data in the lower layer disk back to the upper layer host, a read data instruction is generated, and the read data instruction includes a logical address of the data to be read and an address of the fourth storage space. The logical address of the data to be read identifies a logical position of the data to be read in the storage space, and the physical block address where the data to be read is actually stored can be parsed based on the logical address. The fourth storage space is used to temporarily store the read result.

[0057] After sending the read data instruction to the compression process through the preset communication channel, the method further includes: receiving a search completion instruction through the preset communication channel; determining the physical block address of the data to be read based on the search completion instruction, reading the compressed data from the lower layer disk based on the physical block address of the data to be read, and storing the compressed data into the fourth storage space, while generating a read data completion instruction; and sending the read data completion instruction to the compression process through the preset communication channel.

[0058] Specifically, after the storage master process sends the read data instruction to the compression process through the preset communication channel, the compression process parses the read data instruction, determines the logical address corresponding to the data to be read, and determines the fourth storage space address where the read result is temporarily stored. Then, the compression process queries and retrieves the metadata record associated with the logical address based on the determined logical address, parses the physical block address where the data to be read is actually stored from the metadata, and completes the conversion from the application layer logical address to the lower layer storage medium physical address. Finally, the compression process generates a search completion instruction based on the physical block address, and sends the search completion instruction to the storage master process through the preset communication channel.

[0059] The storage master process receives the search completion instruction through the preset communication channel, parses the search completion instruction, determines the physical block address of the data to be read, reads the compressed data from the lower layer disk based on the physical block address of the data to be read, and stores the compressed data into the fourth storage space, while generating a read data completion instruction, and sends the read data completion instruction to the compression process through the preset communication channel.

[0060] In some embodiments, when the data operation completion instruction is a decompression completion instruction, the processing result is processed based on the data operation completion instruction from the shared memory, including: returning the decompressed data in the fourth storage space to the upper layer host based on the decompression completion instruction, wherein the decompressed data is generated by the compression process and stored in the fourth storage space.

[0061] Specifically, after receiving a read data completion instruction through a preset communication channel, the compression process reads compressed data from the fourth storage space and decompresses the data based on a preset decompression algorithm to obtain the processing result, which includes the decompressed data. After storing the decompressed data in the fourth storage space, the compression process needs to generate a decompression completion instruction and send it to the storage master process through the preset communication channel.

[0062] After receiving the decompression completion instruction through the preset communication channel, the storage master process returns the decompressed data of the fourth storage space to the upper host based on the decompression completion instruction.

[0063] In this application, the storage master process provides the logical address and cache space address via a read data instruction, triggering the compression process to perform metadata lookup and address resolution. The compression process returns the physical block address via a lookup completion instruction, guiding the storage master process to accurately read the disk data. Subsequently, a read data completion instruction notifies the compression process to perform decompression, and finally, a decompression completion instruction completes the data callback. Thus, by using lightweight instructions to transmit metadata and address information, the performance loss caused by large-scale data copying in traditional inter-process communication is effectively avoided. Furthermore, the dynamic reuse mechanism of the fourth storage space eliminates the overhead of repeated memory allocation and release, achieving efficient utilization of storage resources.

[0064] Reference Figure 2 Both the storage master process and the compression process load this kernel module, manage memory resources by addressing them in 4K increments, and directly map a memory region (shared memory) managed by the kernel module into the virtual address spaces of the storage master process and the compression process, respectively. Furthermore, the storage master process and the compression process use two FIFO channels (pre-defined communication channels) for communication.

[0065] As a concrete example, refer to Figure 4 The interaction process between the storage master process and the compression process during data writing is as follows: Step 1: The main storage process generates a write data instruction. The write data instruction includes the address of the first storage space, which is located in shared memory and is used to store the data to be compressed written by the main storage process.

[0066] Step 2: The storage master process sends the write data command to the compression process.

[0067] Step 3: The compression process reads the data to be compressed from the first storage space based on the write data instruction, and performs compression processing on the data to be compressed to obtain the compressed data and the corresponding metadata.

[0068] Step 4: The compression process stores the compressed data and the corresponding metadata into the second storage space and the third storage space, respectively. The second storage space and the third storage space are in shared memory.

[0069] Step 5: The compression process generates a write data completion command.

[0070] Step 6: The compression process sends the write data completion command to the storage master process.

[0071] Step 7: The storage master process writes the compressed data in the second storage space and the metadata in the third storage space to the lower disk based on the write data completion instruction.

[0072] As a concrete example, refer to Figure 5 The interaction process between the storage master process and the compression process when reading data is as follows: Step 1: The main storage process generates a read data instruction. The read data instruction includes the logical address of the data to be read and the address of the fourth storage space, which is located in shared memory.

[0073] Step 2: The storage master process sends the read data command to the compression process.

[0074] Step 3: The compression process determines the logical address of the data to be read and the address of the fourth storage space based on the read data instruction, and searches for metadata based on the logical address of the data to be read to determine the physical block address of the data to be read, while generating the corresponding search completion instruction.

[0075] Step 4: The compression process sends a search completion command to the storage master process.

[0076] Step 5: The storage master process determines the physical block address of the data to be read based on the lookup completion instruction, reads the compressed data from the lower disk based on the physical block address of the data to be read, stores the compressed data in the fourth storage space, and generates a read data completion instruction at the same time.

[0077] Step 6: The storage master process sends the read data completion instruction to the compression process.

[0078] Step 7: The compression process reads compressed data from the fourth storage space based on the read data completion instruction, and decompresses the compressed data to obtain decompressed data. At the same time, the decompression is completed.

[0079] Step 8: The compression process sends a decompression completion command to the storage master process.

[0080] Step 9: The storage master process returns the decompressed data from the fourth storage space to the upper-layer host based on the decompression completion instruction.

[0081] Corresponding to the above embodiments, this application also proposes an electronic device.

[0082] See Figure 6 As shown, the electronic device 300 of this application includes a memory 310, a processor 320, and a collaborative communication program between the storage master process and the compression process stored in the memory 310 and executable on the processor 320. When the processor executes the collaborative communication program between the storage master process and the compression process, the aforementioned collaborative communication method between the storage master process and the compression process is implemented.

[0083] It should be noted that the above explanation of the embodiments and beneficial effects of the collaborative communication method between the storage master process and the compression process also applies to the electronic devices of the embodiments of this application. To avoid redundancy, they will not be elaborated in detail here.

[0084] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0085] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0086] The foregoing has provided a detailed description of a collaborative communication method and electronic device for a storage master process and a compression process, as provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for collaborative communication between a storage master process and a compression process, characterized in that, Applied to a compression process, the method includes: Data manipulation instructions are received through a pre-defined communication channel between the main storage process and the compression process; Data is read from shared memory based on the data operation instruction, and the data is processed to obtain a processing result. When the data operation instruction is a write data instruction, the data to be compressed is read from the first storage space based on the write data instruction, and the data to be compressed is compressed to obtain the processing result. The first storage space is located in the shared memory. The processing result is stored in the shared memory, and a corresponding data operation completion instruction is generated. The storage master process reads the processing result from the shared memory based on the data operation completion instruction and processes it.

2. The collaborative communication method between the storage master process and the compression process according to claim 1, characterized in that, The processing result includes compressed data and metadata corresponding to the compressed data. Storing the processing result into the shared memory includes: The compressed data and the corresponding metadata are stored in the second storage space and the third storage space, respectively, wherein the second storage space and the third storage space are located in the shared memory.

3. The collaborative communication method between the storage master process and the compression process according to claim 1, characterized in that, When the data operation completion instruction is a write data completion instruction, after storing the compressed data and the corresponding metadata into the second storage space and the third storage space respectively, the method further includes: A write data completion command is sent to the storage master process through the preset communication channel; The write data completion instruction includes the address of the second storage space and the address of the third storage space.

4. The collaborative communication method between the storage master process and the compression process according to claim 1, characterized in that, When the data operation instruction is a read data instruction, data is read from shared memory based on the data operation instruction for processing to obtain a processing result, including: Based on the read data instruction, the logical address of the data to be read and the address of the fourth storage space are determined, and the metadata is searched based on the logical address of the data to be read to determine the physical block address of the data to be read. At the same time, a corresponding search completion instruction is generated, wherein the fourth storage space is located in the shared memory. The search completion instruction is sent to the storage master process through the preset communication channel, wherein the search completion instruction includes the physical block address of the data to be read; The command to complete reading data is received through the preset communication channel; Based on the read data completion instruction, compressed data is read from the fourth storage space, and the compressed data is decompressed to obtain the processing result. The processing result includes decompressed data, which is written to the fourth storage space by the storage master process after being read from the lower disk based on the lookup completion instruction.

5. The collaborative communication method between the storage master process and the compression process according to claim 4, characterized in that, Storing the processing result into the shared memory includes: The decompressed data is stored back in the fourth storage space; When the data operation completion instruction is a decompression completion instruction, after storing the decompressed data back to the fourth storage space, the method further includes: The decompression completion command is sent to the business process through the preset communication channel.

6. A method for collaborative communication between a storage master process and a compression process, characterized in that, Applied to the storage master process, the method includes: A data operation instruction is generated and sent to the compression process through a preset communication channel between the storage master process and the compression process. When the data operation instruction is a write data instruction, the write data instruction includes the address of a first storage space, wherein the first storage space is located in shared memory and is used to store the data to be compressed written by the storage master process. The command to complete the operation is received by the preset communication channel; Based on the data operation completion instructions, the processing results are read from shared memory and processed.

7. The collaborative communication method between the storage master process and the compression process according to claim 6, characterized in that, When the data operation completion instruction is a write data completion instruction, the processing result is read from the shared memory based on the data operation completion instruction and processed, including: Based on the write data completion instruction, the compressed data in the second storage space and the metadata in the third storage space are written to the lower-level disk. The compressed data and the metadata of the compressed data are generated by the compression process and stored in the second storage space and the third storage space, respectively.

8. The collaborative communication method between the storage master process and the compression process according to claim 6, characterized in that, When the data operation instruction is a read data instruction, the read data instruction includes the logical address of the data to be read and the address of the fourth storage space, wherein the fourth storage space is located in shared memory; After sending the read data command to the compression process through the preset communication channel, the method further includes: Receive the search completion command through the preset communication channel; Based on the search completion instruction, the physical block address of the data to be read is determined. Based on the physical block address of the data to be read, compressed data is read from the lower disk and stored in the fourth storage space. At the same time, a read data completion instruction is generated. The read data completion instruction is sent to the compression process through the preset communication channel.

9. The collaborative communication method between the storage master process and the compression process according to claim 6, characterized in that, When the data operation completion instruction is a decompression completion instruction, the processing result is read from the shared memory based on the data operation completion instruction and processed, including: Based on the decompression completion instruction, the decompressed data in the fourth storage space is returned to the upper-layer host, wherein the decompressed data is generated by the compression process and stored in the fourth storage space.

10. An electronic device, characterized in that, The system includes a memory, a processor, and a collaborative communication program between a storage master process and a compression process, which is stored in the memory and can run on the processor. When the processor executes the collaborative communication program between the storage master process and the compression process, it implements the collaborative communication method between the storage master process and the compression process according to any one of claims 1-9.

Images