Method for cooperative communication between a storage master process and a compression process and electronic device

Abstract

The application discloses a kind of storage main process and the cooperative communication method of compressed process and electronic equipment, it is related to process communication technical field, by setting the preset communication channel between storage main process and compressed process, control instruction and data processing are separated, instruction is transmitted through preset communication channel, ensure that operation instruction is timely and sequentially executed, and data is exchanged through shared memory area managed by kernel, both avoid multiple data copy problems in pipeline transmission, realize the orderly bidirectional cooperation between storage main process and compressed process through standardized instruction interaction mechanism, fundamentally solve the memory management confusion problem caused by lack of coordination mechanism in traditional shared memory scheme, realize the dual promotion of interprocess communication efficiency and resource management efficiency.

Description

Technical Field

[0001] This application relates to the field of process communication technology, and in particular to a collaborative communication method and electronic device between a storage master process and a compression process. Background Technology

[0002] Current storage system data compression architectures primarily employ two implementation paths: one is to deeply integrate compression functionality into the main storage process, and the other is to run compression functionality as an independent process. However, deeply integrating compression functionality into the main storage process results in tight coupling between the storage service and the compression algorithm, making system maintenance and feature upgrades difficult; running compression functionality as an independent process leads to efficiency bottlenecks in inter-process communication. To establish a communication channel between the independent compression process and the main storage process, relevant technologies often employ standard inter-process communication techniques such as pipe transfer or shared memory.

[0003] However, pipelining incurs significant data copying overhead when transmitting large amounts of data, impacting overall performance. While shared memory reduces data copying, it cannot support bidirectional collaboration between the storage master process and the compression process. The storage master process struggles to pass the location identifier of the data to be compressed in the kernel shared memory to the compression process. Furthermore, the compression process cannot effectively return the specific location information of the compression result in the kernel space to the storage master process after processing. As a result, the kernel shared memory space cannot be coordinated and utilized by user-mode processes, which reduces data processing efficiency and causes chaotic memory resource management. Summary of the Invention

[0004] This application provides a collaborative communication method and electronic device between a storage master process and a compression process, to at least solve the problem of disordered memory resource management caused by the lack of a bidirectional inter-process coordination mechanism for shared memory in the data compression process of storage systems in related technologies.

[0005] This application provides a collaborative communication method between a storage master process and a compression process, applied to the compression process. The method includes: receiving data operation instructions through a preset communication channel between the storage master process and the compression process; reading data from shared memory based on the data operation instructions and processing the data to obtain a processing result; storing the processing result in the shared memory and generating a corresponding data operation completion instruction, wherein the storage master process reads the processing result from the shared memory based on the data operation completion instruction and processes it.

[0006] This application provides an electronic device, including a memory, a processor, and a collaborative communication program between a storage master process and a compression process stored in the memory and executable on the processor. When the processor executes the collaborative communication program between the storage master process and the compression process, it implements the aforementioned collaborative communication method between the storage master process and the compression process.

[0007] This application separates control commands from data processing by setting up a pre-defined communication channel between the storage master process and the compression process. Commands are transmitted through the pre-defined communication channel to ensure timely delivery and sequential execution of operation commands, while data is exchanged through a shared memory area managed by the kernel. This avoids the problem of multiple data copies in pipe transmission and achieves orderly bidirectional cooperation between the storage master process and the compression process through a standardized command interaction mechanism. It fundamentally solves the memory management chaos caused by the lack of coordination mechanism in traditional shared memory schemes, and achieves a dual improvement in inter-process communication efficiency and resource management performance. Attached Figure Description

[0008] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a flowchart illustrating a collaborative communication method between a storage master process and a compression process according to some embodiments of this application;

[0010] Figure 2 This is a schematic diagram of the structure of a collaborative communication system between a storage master process and a compression process according to some embodiments of this application;

[0011] Figure 3 This is a flowchart illustrating a collaborative communication method between a storage master process and a compression process according to other embodiments of this application;

[0012] Figure 4 This is a schematic diagram illustrating the interaction process between the storage master process and the compression process during data writing according to some embodiments of this application;

[0013] Figure 5 This is a schematic diagram illustrating the interaction process between the storage master process and the compression process when reading data according to some embodiments of this application;

[0014] Figure 6 This is a block diagram of an electronic device according to some embodiments of this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0016] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0017] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] This section describes the specific application environment architecture or specific hardware architecture, taking into account the collaborative communication methods between the storage master process and the compression process, as well as the specific application environment architecture or specific hardware architecture upon which the execution of the electronic device depends.

[0019] The embodiments of this application provide a collaborative communication method between a storage master process and a compression process. The method is described in detail below, taking into account the execution flow of the collaborative communication method between the storage master process and the compression process.

[0020] With the widespread application of cloud computing and big data technologies, storage services are facing the challenge of exponential growth in data volume and instructions, with daily data increments in enterprise-level storage systems reaching terabyte levels. Against this backdrop, "data storage and compression optimization" has become a core requirement—reducing storage footprint and transmission bandwidth consumption through compression technology, but simultaneously placing higher demands on the collaboration efficiency of storage and compression processes, data security, and system stability.

[0021] There are two main implementation paths for data compression architecture in current storage systems: one is to deeply integrate the compression function into the main storage process, and the other is to run the compression function as an independent process.

[0022] The limitations of integrated architecture are as follows:

[0023] The main process needs to handle multiple tasks simultaneously, including data reading and writing, metadata management, and compression calculations, which significantly increases the CPU (Central Processing Unit) load and consequently leads to increased storage I / O (Input / Output) latency. Any update to the compression function, such as switching from LZ4 (Lempel-Ziv4, a lossless data compression algorithm) to ZSTD (Zstandard, a data compression algorithm), requires recompiling the main process code, making hot-swappable expansion impossible. Anomalies in the compression module (such as memory leaks) can directly cause the storage main process to crash, creating a cascading failure risk.

[0024] The limitations of an independent process architecture are as follows:

[0025] Exchanging data via temporary files introduces two additional lower-level disk I / O operations, significantly increasing data processing latency; the lack of effective task instructions and status feedback mechanisms means that the storage master process cannot monitor the compression process's status in real time, which can easily lead to data backlog or task loss.

[0026] To solve the problem of data interaction between processes, related technologies usually adopt general IPC (Inter-Process Communication) mechanisms, such as FIFO (First In, First Out) and shared memory.

[0027] The limitations of FIFO are as follows:

[0028] When used to transfer large amounts of stored data, multiple data copies between user mode and kernel mode are required, leading to a significant increase in transmission latency. Furthermore, FIFOs cannot directly establish an association between instructions and data, requiring additional information such as file paths to identify data locations, which increases system complexity and the risk of errors.

[0029] The limitations of shared memory are as follows:

[0030] Shared areas are exposed in user space and are susceptible to tampering by malicious processes or abnormal process exits; processes need to implement complex synchronization mechanisms (such as mutexes and semaphores) themselves, and improper design can easily lead to deadlock and other problems.

[0031] While kernel shared memory schemes can provide isolation and protection using kernel space, they are typically limited to kernel data caching for a single process and lack organic integration with communication mechanisms such as FIFO, thus failing to support a complete collaborative process between the storage master process and the compression process. Specifically, the storage master process cannot pass data identifiers in kernel shared memory in a lightweight manner; and the compression process cannot effectively report the location information of the result data in kernel space after processing, resulting in kernel space data resources being unable to be coordinated and utilized by user-mode processes.

[0032] Based on this, this application proposes a collaborative communication method between the storage master process and the compression process. (Refer to...) Figure 1 The collaborative communication method between the storage master process and the compression process in this application embodiment may include the following steps:

[0033] S110 receives data operation instructions through a preset communication channel between the storage master process and the compression process.

[0034] S120: Read data from shared memory based on data operation instructions, and process the data to obtain a processing result. When the data operation instruction is a write data instruction, read the data to be compressed from the first storage space based on the write data instruction, and compress the data to be compressed to obtain a processing result. The first storage space is located in shared memory.

[0035] S130: The processing result is stored in shared memory, and a corresponding data operation completion instruction is generated. The storage master process reads the processing result from the shared memory based on the data operation completion instruction and processes it.

[0036] Specifically, refer to Figure 2 The preset communication channel between the storage master process and the compression process can be a FIFO. The compression process can receive data operation instructions, such as write data instructions and read data instructions, through the preset communication channel.

[0037] Upon receiving a data manipulation instruction, the compression process reads data from shared memory based on the instruction and processes the data to obtain the result. For example, if the compression process receives a write data instruction, it means the storage master process needs to compress data that the upper-level host needs to write to the lower-level disk. Therefore, the compression process needs to read the data to be written from shared memory and compress it to obtain the compression result. If the compression process receives a read data instruction, it means the storage master process needs to decompress data that needs to be read from the lower-level disk to the upper-level host. Therefore, the compression process needs to read the data to be read from shared memory and decompress it to obtain the decompression result.

[0038] Specifically, data operation instructions include write data instructions. When the compression process receives a write data instruction, it indicates that the storage master process needs to compress the data that the upper-layer host needs to write to the lower-layer disk. Therefore, the storage master process writes the data to be compressed (the data that the upper-layer host needs to write to the lower-layer disk) into shared memory, specifically into the first storage space of shared memory. After receiving the write data instruction through a preset communication channel, the compression process parses the write data instruction to determine the storage location of the data to be compressed, i.e., the first storage space. Subsequently, it reads the data to be compressed from the first storage space and uses a preset compression algorithm, such as LZ4 or ZSTD, to compress the data to obtain the compressed data and its corresponding metadata. The metadata corresponding to the compressed data is the attribute information of the data to be compressed, which may include the storage address and length of the compressed data in shared memory, the original data size before compression, the type of compression algorithm used, and the compression task sequence number.

[0039] After the compression process reads data from shared memory, processes it to obtain the result, writes the result back to shared memory, generates a corresponding data operation completion instruction, and then sends the instruction to the storage master process. The storage master process then reads the result from shared memory based on the instruction and processes it. For example, when writing the compression result to shared memory, a write data completion instruction is generated and sent to the storage master process. Upon receiving the instruction, the storage master process reads the compression result from shared memory and writes it to the lower-level disk. Similarly, when writing the decompression result to shared memory, a decompression completion instruction is generated and sent to the storage master process. Upon receiving the instruction, the storage master process reads the decompression result from shared memory and returns it to the upper-level host.

[0040] This application separates control commands from data processing by setting up a pre-defined communication channel between the storage master process and the compression process. Commands are transmitted through the pre-defined communication channel to ensure timely delivery and sequential execution of operation commands, while data is exchanged through a shared memory area managed by the kernel. This avoids the problem of multiple data copies in pipe transmission and achieves orderly bidirectional cooperation between the storage master process and the compression process through a standardized command interaction mechanism. It fundamentally solves the memory management chaos caused by the lack of coordination mechanism in traditional shared memory schemes, and achieves a dual improvement in inter-process communication efficiency and resource management performance.

[0041] In some embodiments, the processing result includes compressed data and metadata corresponding to the compressed data. Storing the processing result into shared memory includes: storing the compressed data and metadata corresponding to the compressed data into a second storage space and a third storage space, respectively, wherein the second storage space and the third storage space are in shared memory.

[0042] Specifically, after the compression process compresses the data to be compressed and obtains the processing result, it stores the result in shared memory. Specifically, the processing result includes compressed data and the corresponding metadata. The compressed data is stored in the second storage space of shared memory, and the corresponding metadata is stored in the third storage space of shared memory.

[0043] In some embodiments, when the data operation completion instruction is a write data completion instruction, after storing the compressed data and the corresponding metadata into the second storage space and the third storage space respectively, the method further includes: sending a write data completion instruction to the storage master process through a preset communication channel; wherein, the write data completion instruction includes the address of the second storage space and the address of the third storage space.

[0044] Specifically, after the compression process stores the compressed data and its corresponding metadata into the second and third storage spaces respectively, it needs to generate a data operation completion instruction, specifically a write data completion instruction. The write data completion instruction includes the addresses of the second and third storage spaces. Then, the write data completion instruction is sent to the storage master process through a preset communication channel.

[0045] This application physically separates compressed data and metadata in shared memory and uses a pre-defined communication channel to transmit write completion instructions, thereby conveying the address information of the compressed data and metadata storage locations. This separate storage of data and metadata avoids the parsing complexity and potential data boundary errors inherent in hybrid storage models, improving the accuracy and security of data access. By transmitting only the storage address, rather than the data itself, through the pre-defined communication channel, the amount of data exchanged between processes is significantly reduced, fundamentally solving the copy overhead and latency problems associated with traditional pipe transfers of large amounts of data.

[0046] In some embodiments, when the data operation instruction is a read data instruction, data is read from shared memory based on the data operation instruction for processing to obtain a processing result, including: determining the logical address of the data to be read and the address of the fourth storage space based on the read data instruction, and searching metadata based on the logical address of the data to be read to determine the physical block address of the data to be read, while generating a corresponding search completion instruction, wherein the fourth storage space is located in shared memory; sending the search completion instruction to the storage master process through a preset communication channel, wherein the search completion instruction includes the physical block address of the data to be read; receiving the read data completion instruction through the preset communication channel; reading compressed data from the fourth storage space based on the read data completion instruction, and decompressing the compressed data to obtain a processing result, wherein the processing result includes decompressed data, and the compressed data is read from the lower-level disk and written to the fourth storage space by the storage master process based on the search completion instruction.

[0047] Specifically, the data operation instructions include read data instructions. After receiving the read data instruction through a preset communication channel, the compression process parses the read data instruction to determine the logical address corresponding to the data to be read. The logical address identifies the logical location of the data to be read in the storage space and determines the address of the fourth storage space where the read result is temporarily stored. Subsequently, based on the determined logical address, the compression process queries and retrieves the associated metadata record, and parses the physical block address where the data to be read is actually stored from the metadata, completing the conversion from the application layer logical address to the underlying storage medium physical address. Finally, the compression process generates a lookup completion instruction based on the physical block address, so that the storage master process can read the compressed data from the lower disk based on the lookup completion instruction and store the compressed data in the fourth storage space, and then wait for the storage master process to generate a read data completion instruction.

[0048] After receiving a read data completion command through a preset communication channel, the compression process reads the compressed data from the fourth storage space and decompresses it based on a preset decompression algorithm to obtain the processing result, which specifically includes the decompressed data. It should be noted that the preset decompression algorithm must correspond to the compression algorithm used during the data compression process, which can be determined by querying the relevant metadata. This improves the accuracy and speed of data decompression.

[0049] In some embodiments, storing the processing result in shared memory includes: storing the decompressed data back to the fourth storage space.

[0050] Specifically, after the compression process decompresses the compressed data and obtains the processing result, it stores the result in shared memory. Specifically, the processing result includes the decompressed data, which is then stored back in the fourth storage space.

[0051] When the data operation completion instruction is a decompression completion instruction, after storing the decompressed data back to the fourth storage space, the method also includes: sending a decompression completion instruction to the business process through a preset communication channel.

[0052] Specifically, after the compression process stores the decompressed data into the fourth storage space, it needs to generate a data operation completion instruction, specifically a decompression completion instruction; then, the decompression completion instruction is sent to the storage master process through a preset communication channel.

[0053] This application optimizes the allocation of computing resources by delegating the physical address resolution task to the compression process; it transmits lookup completion instructions and read data completion instructions through a preset communication channel, and the lookup completion instructions and read data completion instructions work together to ensure operational reliability while avoiding unnecessary data copying; and it achieves seamless connection between compressed and decompressed data by reusing the fourth storage space, effectively improving memory utilization efficiency.

[0054] Figure 3 This is a flowchart illustrating a collaborative communication method between a storage master process and a compression process according to other embodiments of this application. (Refer to...) Figure 3 The collaborative communication method between the storage master process and the compression process in this application embodiment may further include the following steps:

[0055] S210, generate a data operation instruction and send the data operation instruction to the compression process through a preset communication channel between the storage master process and the compression process. When the data operation instruction is a write data instruction, the write data instruction includes the address of the first storage space, wherein the first storage space is located in shared memory and is used to store the data to be compressed written by the storage master process.

[0056] S220 receives data through a preset communication channel to complete the operation instructions.

[0057] S230 reads the processing results from shared memory based on the data operation completion instruction and processes them.

[0058] Specifically, when the upper-layer host needs to write data to the lower-layer disk, the storage master process needs to generate a write data instruction and send it to the compression process through a pre-defined communication channel between the storage master process and the compression process. Specifically, when the storage master process needs to write data from the upper-layer host to the lower-layer disk, it writes the data to be compressed to shared memory, specifically the first storage space, and simultaneously generates a write data instruction, which is then transmitted to the compression process through the pre-defined communication channel. When the upper-layer host needs to read data from the lower-layer disk back to the upper-layer host, the storage master process needs to generate a read data instruction and send it to the compression process through the pre-defined communication channel.

[0059] Furthermore, the compression process receives data operation instructions, reads data from shared memory based on these instructions, processes the data to obtain a processing result, stores the processing result back into shared memory, generates a corresponding data operation completion instruction, and transmits the data operation completion instruction to the storage master process through a preset communication channel. For example, when the compression process receives a write data instruction, it indicates that the storage master process needs to compress data that the upper-level host needs to write to the lower-level disk. Therefore, the compression process needs to read the data to be written from shared memory and compress it to obtain a compression result. When the compression process receives a read data instruction, it indicates that the storage master process needs to decompress data that needs to be read from the lower-level disk to the upper-level host. Therefore, the compression process needs to read the data to be read from shared memory and decompress it to obtain a decompression result.

[0060] After the compression process reads data from shared memory, processes it to obtain the result, writes the result back to shared memory, generates a corresponding data operation completion instruction, and then sends the instruction to the storage master process. The storage master process then reads the result from shared memory based on the instruction and processes it. For example, when writing the compression result to shared memory, a write data completion instruction is generated and sent to the storage master process. Upon receiving the instruction, the storage master process reads the compression result from shared memory and writes it to the lower-level disk. Similarly, when writing the decompression result to shared memory, a decompression completion instruction is generated and sent to the storage master process. Upon receiving the instruction, the storage master process reads the decompression result from shared memory and returns it to the upper-level host.

[0061] This application separates control commands from data processing by setting up a pre-defined communication channel between the storage master process and the compression process. Commands are transmitted through the pre-defined communication channel to ensure timely delivery and sequential execution of operation commands, while data is exchanged through a shared memory area managed by the kernel. This avoids the problem of multiple data copies in pipe transmission and achieves orderly bidirectional cooperation between the storage master process and the compression process through a standardized command interaction mechanism. It fundamentally solves the memory management chaos caused by the lack of coordination mechanism in traditional shared memory schemes, and achieves a dual improvement in inter-process communication efficiency and resource management performance.

[0062] In some embodiments, when the data operation completion instruction is a write data completion instruction, the processing result is read from the shared memory based on the data operation completion instruction and processed, including: writing the compressed data in the second storage space and the metadata in the third storage space to the lower disk based on the write data completion instruction, wherein the compressed data and the metadata of the compressed data are generated by the compression process and stored in the second storage space and the third storage space respectively.

[0063] Specifically, after the compression process stores the compressed data and its corresponding metadata into the second and third storage spaces respectively, it needs to generate a data operation completion instruction, specifically a write data completion instruction. The write data completion instruction includes the addresses of the second and third storage spaces. Then, the write data completion instruction is sent to the storage master process through a preset communication channel.

[0064] After receiving the write data completion instruction, the storage master process parses the write data completion instruction to determine the address of the second storage space and write the compressed data in the second storage space and the metadata in the third storage space to the lower disk.

[0065] In this way, the storage master process only needs to pass the address of the data to be compressed in shared memory through a lightweight write data instruction to trigger the compression process. After compression, the storage address of the compressed data and metadata is returned via the same write data completion instruction. This ensures that inter-process communication always remains at a lightweight interaction level of instructions and addresses, completely avoiding the performance overhead of large-scale data copying and transmission between processes in traditional solutions.

[0066] In some embodiments, when the data operation instruction is a read data instruction, the read data instruction includes the logical address of the data to be read and the address of the fourth storage space, wherein the fourth storage space is located in shared memory.

[0067] Specifically, when the storage master process needs to read data from the lower-level disk back to the upper-level host, it generates a read data instruction. The read data instruction includes the logical address of the data to be read and the address of the fourth storage space. The logical address of the data to be read identifies the logical location of the data to be read in the storage space. Based on the logical address, the physical block address where the data to be read is actually stored can be resolved. The fourth storage space is used to temporarily store the read results.

[0068] After sending a read data instruction to the compression process through a preset communication channel, the method further includes: receiving a search completion instruction through the preset communication channel; determining the physical block address of the data to be read based on the search completion instruction; reading compressed data from the lower-level disk based on the physical block address of the data to be read; storing the compressed data in the fourth storage space; and generating a read data completion instruction at the same time; and sending the read data completion instruction to the compression process through the preset communication channel.

[0069] Specifically, after the storage master process sends a read data instruction to the compression process through a preset communication channel, the compression process parses the read data instruction to determine the logical address corresponding to the data to be read. The logical address identifies the logical location of the data to be read in the storage space and determines the address of the fourth storage space where the read result is temporarily stored. Subsequently, based on the determined logical address, the compression process queries and retrieves the associated metadata record, and parses the physical block address where the data to be read is actually stored from the metadata, completing the conversion from the application layer logical address to the underlying storage medium physical address. Finally, the compression process generates a lookup completion instruction based on the physical block address and sends the lookup completion instruction to the storage master process through the preset communication channel.

[0070] The storage master process receives the search completion instruction through a preset communication channel, parses the search completion instruction, determines the physical block address of the data to be read, reads the compressed data from the lower disk based on the physical block address of the data to be read, and stores the compressed data in the fourth storage space. At the same time, it generates a read data completion instruction and sends the read data completion instruction to the compression process through the preset communication channel.

[0071] In some embodiments, when the data operation completion instruction is a decompression completion instruction, the processing result is read from the shared memory based on the data operation completion instruction and processed, including: returning the decompressed data of the fourth storage space to the upper-layer host based on the decompression completion instruction, wherein the decompressed data is generated by the compression process and stored in the fourth storage space.

[0072] Specifically, after receiving a read data completion instruction through a preset communication channel, the compression process reads compressed data from the fourth storage space and decompresses the data based on a preset decompression algorithm to obtain the processing result, which includes the decompressed data. After storing the decompressed data in the fourth storage space, the compression process needs to generate a decompression completion instruction and send it to the storage master process through the preset communication channel.

[0073] After receiving the decompression completion instruction through the preset communication channel, the storage master process returns the decompressed data of the fourth storage space to the upper host based on the decompression completion instruction.

[0074] In this application, the storage master process provides the logical address and cache space address via a read data instruction, triggering the compression process to perform metadata lookup and address resolution. The compression process returns the physical block address via a lookup completion instruction, guiding the storage master process to accurately read the disk data. Subsequently, a read data completion instruction notifies the compression process to perform decompression, and finally, a decompression completion instruction completes the data callback. Thus, by using lightweight instructions to transmit metadata and address information, the performance loss caused by large-scale data copying in traditional inter-process communication is effectively avoided. Furthermore, the dynamic reuse mechanism of the fourth storage space eliminates the overhead of repeated memory allocation and release, achieving efficient utilization of storage resources.

[0075] Reference Figure 2 Both the storage master process and the compression process load this kernel module, manage memory resources by addressing them in 4K increments, and directly map a memory region (shared memory) managed by the kernel module into the virtual address spaces of the storage master process and the compression process, respectively. Furthermore, the storage master process and the compression process use two FIFO channels (pre-defined communication channels) for communication.

[0076] As a concrete example, refer to Figure 4 The interaction process between the storage master process and the compression process during data writing is as follows:

[0077] Step 1: The main storage process generates a write data instruction. The write data instruction includes the address of the first storage space, which is located in shared memory and is used to store the data to be compressed written by the main storage process.

[0078] Step 2: The storage master process sends the write data command to the compression process.

[0079] Step 3: The compression process reads the data to be compressed from the first storage space based on the write data instruction, and performs compression processing on the data to be compressed to obtain the compressed data and the corresponding metadata.

[0080] Step 4: The compression process stores the compressed data and the corresponding metadata into the second storage space and the third storage space, respectively. The second storage space and the third storage space are in shared memory.

[0081] Step 5: The compression process generates a write data completion command.

[0082] Step 6: The compression process sends the write data completion command to the storage master process.

[0083] Step 7: The storage master process writes the compressed data in the second storage space and the metadata in the third storage space to the lower disk based on the write data completion instruction.

[0084] As a concrete example, refer to Figure 5 The interaction process between the storage master process and the compression process when reading data is as follows:

[0085] Step 1: The main storage process generates a read data instruction. The read data instruction includes the logical address of the data to be read and the address of the fourth storage space, which is located in shared memory.

[0086] Step 2: The storage master process sends the read data command to the compression process.

[0087] Step 3: The compression process determines the logical address of the data to be read and the address of the fourth storage space based on the read data instruction, and searches for metadata based on the logical address of the data to be read to determine the physical block address of the data to be read, while generating the corresponding search completion instruction.

[0088] Step 4: The compression process sends a search completion command to the storage master process.

[0089] Step 5: The storage master process determines the physical block address of the data to be read based on the lookup completion instruction, reads the compressed data from the lower disk based on the physical block address of the data to be read, stores the compressed data in the fourth storage space, and generates a read data completion instruction at the same time.

[0090] Step 6: The storage master process sends the read data completion instruction to the compression process.

[0091] Step 7: The compression process reads compressed data from the fourth storage space based on the read data completion instruction, and decompresses the compressed data to obtain decompressed data. At the same time, the decompression is completed.

[0092] Step 8: The compression process sends a decompression completion command to the storage master process.

[0093] Step 9: The storage master process returns the decompressed data from the fourth storage space to the upper-layer host based on the decompression completion instruction.

[0094] Corresponding to the above embodiments, this application also proposes an electronic device.

[0095] See Figure 6 As shown, the electronic device 300 of this application includes a memory 310, a processor 320, and a collaborative communication program between the storage master process and the compression process stored in the memory 310 and executable on the processor 320. When the processor executes the collaborative communication program between the storage master process and the compression process, the aforementioned collaborative communication method between the storage master process and the compression process is implemented.

[0096] It should be noted that the above explanation of the embodiments and beneficial effects of the collaborative communication method between the storage master process and the compression process also applies to the electronic devices of the embodiments of this application. To avoid redundancy, they will not be elaborated in detail here.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0098] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0099] The foregoing has provided a detailed description of a collaborative communication method and electronic device for a storage master process and a compression process, as provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method of storing cooperative communication of a main process and a compression process, characterized by, Applied to a compression process, the method includes: Data manipulation instructions are received through a pre-defined communication channel between the main storage process and the compression process; Data is read from shared memory based on the data operation instruction, and the data is processed to obtain a processing result. When the data operation instruction is a write data instruction, the data to be compressed is read from the first storage space based on the write data instruction, and the data to be compressed is compressed to obtain the processing result. The first storage space is located in the shared memory. The processing result is stored in the shared memory. When the data operation instruction is a write data instruction, the processing result includes compressed data and metadata corresponding to the compressed data. The compressed data and metadata corresponding to the compressed data are stored in the second storage space and the third storage space, respectively. The second storage space and the third storage space are located in the shared memory. Generate a corresponding data operation completion instruction. When the data operation completion instruction is a write data completion instruction, send the write data completion instruction to the storage master process through the preset communication channel. The write data completion instruction includes the address of the second storage space and the address of the third storage space. The storage master process reads the processing result from the shared memory and processes it based on the data operation completion instruction.

2. The method of claim 1, wherein the main process and the compression process are stored in a memory. When the data operation instruction is a read data instruction, data is read from shared memory based on the data operation instruction for processing to obtain a processing result, including: Based on the read data instruction, the logical address of the data to be read and the address of the fourth storage space are determined, and the metadata is searched based on the logical address of the data to be read to determine the physical block address of the data to be read. At the same time, a corresponding search completion instruction is generated, wherein the fourth storage space is located in the shared memory. The search completion instruction is sent to the storage master process through the preset communication channel, wherein the search completion instruction includes the physical block address of the data to be read; The command to complete reading data is received through the preset communication channel; Based on the read data completion instruction, compressed data is read from the fourth storage space, and the compressed data is decompressed to obtain the processing result. The processing result includes decompressed data, which is written to the fourth storage space by the storage master process after being read from the lower disk based on the lookup completion instruction.

3. The method of claim 2, wherein the compressed process is a process that is compressed by a compression program. Storing the processing result into the shared memory includes: The decompressed data is stored back in the fourth storage space; When the data operation completion instruction is a decompression completion instruction, after storing the decompressed data back to the fourth storage space, the method further includes: The decompression completion command is sent to the business process through the preset communication channel.

4. A method of storing cooperative communication of a main process and a compression process, characterized by, Applied to the storage master process, the method includes: A data operation instruction is generated and sent to the compression process through a preset communication channel between the storage master process and the compression process. When the data operation instruction is a write data instruction, the write data instruction includes the address of a first storage space, wherein the first storage space is located in shared memory and is used to store the data to be compressed written by the storage master process. The command to complete the operation is received by the preset communication channel; Based on the data operation completion instruction, the processing result is read from the shared memory and processed. When the data operation completion instruction is a write data completion instruction, the write data completion instruction is parsed to determine the address of the second storage space and the address of the third storage space, so as to write the compressed data in the second storage space and the metadata in the third storage space to the lower disk. The compressed data and the metadata of the compressed data are generated by the compression process and stored in the second storage space and the third storage space respectively.

5. The method of claim 4, wherein the compressed process is a process that is compressed by a compression program. When the data operation instruction is a read data instruction, the read data instruction includes the logical address of the data to be read and the address of the fourth storage space, wherein the fourth storage space is located in shared memory; After sending the read data command to the compression process through the preset communication channel, the method further includes: Receive the search completion command through the preset communication channel; Based on the search completion instruction, the physical block address of the data to be read is determined. Based on the physical block address of the data to be read, compressed data is read from the lower disk and stored in the fourth storage space. At the same time, a read data completion instruction is generated. The read data completion instruction is sent to the compression process through the preset communication channel.

6. The method of claim 4, wherein the main process and the compression process are stored in a memory. When the data operation completion instruction is a decompression completion instruction, the processing result is read from the shared memory based on the data operation completion instruction and processed, including: Based on the decompression completion instruction, the decompressed data in the fourth storage space is returned to the upper-layer host, wherein the decompressed data is generated by the compression process and stored in the fourth storage space.

7. An electronic device, comprising: The system includes a memory, a processor, and a collaborative communication program between a storage master process and a compression process, which is stored in the memory and can run on the processor. When the processor executes the collaborative communication program between the storage master process and the compression process, it implements the collaborative communication method between the storage master process and the compression process according to any one of claims 1-6.

Images