Management method and device of shared memory, electronic equipment and storage medium
By dynamically adjusting the number of shared memory blocks managed by the proxy service process, the problems of resource waste and insufficiency in the proxy service process are solved, thereby improving the utilization of host machine memory resources and the input/output performance of virtual machine processes.
Patent Information
- Application Number
- CN202511476929.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-16
- Publication Date
- 2026-01-23
AI Technical Summary
In existing technologies, the shared memory in the proxy service process is fixed and cannot be dynamically adjusted, resulting in resource waste and insufficiency, which affects the utilization rate of the host machine's memory resources.
By dynamically adjusting the number of shared memory blocks managed by the agent service process, creating or reclaiming shared memory blocks, and mapping and releasing them according to the needs of the virtual machine process, resource waste can be avoided.
It improves the utilization of host machine memory resources, avoids resource shortages and waste, and improves the input/output performance of virtual machine processes.
Smart Images
Figure CN121387589A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of memory management technology, and in particular to a method, apparatus, electronic device and storage medium for managing shared memory. Background Technology
[0002] In the field of data storage, the block storage service client consists of two components: Proxy and librbd. The Qemu virtual machine process sends input and output data to the Proxy service process through librbd, and the Proxy service process writes the input and output data into the block storage service.
[0003] Normally, when a Qemu virtual machine process connects to the Proxy process, it maps all shared memory and allocates a fixed portion of the managed area for its own maintenance and use. However, this shared memory is of fixed size within the Proxy service process. Even if no virtual machines are running on the host machine, this shared memory remains resident in the Proxy service process and cannot be returned to the operating system, thus wasting the host machine's memory resources. Summary of the Invention
[0004] This application provides a shared memory management method, apparatus, electronic device, and storage medium, which can dynamically adjust the number of shared memory blocks managed by the agent service process, thereby improving the utilization rate of the host machine's memory resources.
[0005] In a first aspect, one embodiment of this application provides a shared memory management method applied to a proxy service process, the method comprising: Determine a first number of free shared memory blocks managed by the agent service process; wherein the agent service process is used to instruct the allocation of shared memory; If the first quantity is less than the first threshold, a new shared memory block is created through the agent service process, and a first notification message is sent to each virtual machine process. Each virtual machine process is used to start at least one virtual machine. The first notification message is used to instruct each virtual machine process to establish a mapping relationship with the new shared memory block. The first threshold is used to indicate the minimum number of shared memory blocks required to maintain the normal operation of the agent service process. Among the virtual machine processes and shared memory blocks with mapping relationships, at least one virtual machine started by the virtual machine process has the permission to use the shared memory block.
[0006] In this embodiment, when the first number of free shared memory blocks managed by the agent service process is less than the minimum number of shared memory blocks required to maintain the normal operation of the agent service, it indicates that the shared memory blocks of the agent service process are about to run out. New shared memory blocks can be created in advance, and a first notification message is sent to each virtual process. In this way, each virtual machine process establishes a mapping relationship with the new shared memory block, so that virtual machines started by each virtual machine process can use the new shared memory block. This avoids temporary allocation and improves the efficiency of page frame allocation. Therefore, by dynamically adjusting the number of shared memory blocks managed by the agent service process, the utilization rate of host machine memory resources is improved.
[0007] In one alternative implementation, creating a new shared memory block via a proxy service process includes: New shared memory blocks are created by changing the holder of a portion of the shared memory blocks managed by the operating system to a proxy service process.
[0008] The above embodiment describes creating a new shared memory block by modifying the holder of the shared memory block, a process that is simple to implement.
[0009] In one alternative implementation, the method further includes: If the first quantity is greater than the second threshold, then some of the free shared memory blocks are reclaimed to the memory pool managed by the agent service process, and the mapping relationship between the reclaimed shared memory blocks and the associated virtual machine process is released; where the second threshold is the maximum number of free shared memory blocks allowed by the agent service process; the second threshold is greater than the first threshold. Release the shared memory blocks that have been reclaimed from the memory pool back to the operating system.
[0010] In the above embodiments, if the first quantity exceeds the maximum number of free shared memory blocks allowed by the agent service process, in order to avoid wasting memory resources, the free shared memory blocks are reclaimed to the memory pool managed by the agent service process, and the mapping relationship between the reclaimed shared memory blocks and the associated virtual machine processes is released. This design further improves resource utilization.
[0011] In one alternative implementation, releasing shared memory blocks reclaimed from the memory pool back to the operating system includes: Change the holder of the shared memory blocks reclaimed to the memory pool to the operating system.
[0012] The above embodiment describes a method of releasing shared memory blocks reclaimed from the memory pool to the operating system by modifying the holder of the shared memory blocks. This method is simple to implement.
[0013] In one optional implementation, before determining whether the first quantity is greater than the second threshold, the method further includes: The amount of input / output data that the virtual machine process needs to process has decreased, or some virtual machine processes have performed the operation of exiting and starting the virtual machine.
[0014] In the above embodiments, the above-mentioned behavior of the virtual machine process can lead to an increase in the number of free shared memory blocks managed by the agent service process. Therefore, when the above-mentioned behavior of the virtual machine is determined, the relationship between the first quantity and the second threshold is then determined to improve processing efficiency.
[0015] In one alternative implementation, for each shared memory block in the partially free shared memory blocks, the following reclamation operation is performed: If a page swapping operation is detected, determine that some page frames in the page frames holding the shared memory block belong to the target shared memory block. If the holder of the target shared memory block is the proxy service process, then after the data transfer is completed, some page frames are cleared, and the page frames of the shared memory block are controlled to point to the pages of the shared memory block, and the cleared page frames are controlled to hold the pages of the target shared memory block. If the holder of the target memory block is a virtual machine process, a second notification message is sent to the virtual machine process. The second notification message is used to instruct the virtual machine process to clear some page frames after the data transfer is completed, and to control the page frames of the shared memory block to point to the pages of the shared memory block, and to control the cleared page frames to hold the pages of the target shared memory block.
[0016] The above embodiment describes the process of reclaiming shared memory blocks. In this case, considering the possibility of page swapping, after the data transfer is completed, the page frames are returned, and the swapped page frames and pages are swapped back.
[0017] In one alternative implementation, the method further includes: Unmap each virtual machine process and its associated shared memory block, and release the unmapped shared memory block to the operating system.
[0018] The above embodiments release the shared memory of the contact mapping to the operating system, thereby improving resource utilization.
[0019] In one alternative implementation, the method further includes: If a proxy service process restart operation is detected, for any shared memory block, read the bitmap record information of the shared memory block and determine the usage status of the shared memory block; If a shared memory block is in an occupied state, then establish a mapping between the shared memory block and its owner; If the holder is a virtual machine process, then the shared memory block is loaded according to its original mapping relationship; If the holder is a proxy service process, then the page frame is initialized and loaded according to the original mapping of the shared memory block.
[0020] In the above embodiments, after the proxy service process restarts, leveraging the characteristic that shared memory blocks, due to their mapping, are not reclaimed by the operating system, appropriate operations can be performed based on the shared memory block's occupancy status and whether its holder is a virtual machine process or the proxy service process. This enables the reconstruction of shared memory blocks after the proxy service process restarts, quickly reclaiming business data and avoiding retries for input / output operations.
[0021] In one alternative implementation, the method includes: For any virtual machine process, if the number of free memory pages in the shared memory block allocated by the virtual machine process is less than the third threshold, then the first request from the virtual machine process is received; where the third threshold is the minimum number of memory pages that the virtual machine can handle input and output business data. Based on the first request, allocate a target shared memory block for the virtual machine in the memory pool; wherein, the memory pool is a resource pool composed of idle resources managed by the agent service.
[0022] In the above embodiments, when the number of idle memory pages in the shared memory block allocated to the virtual machine process is less than the minimum number of memory pages for the virtual machine to handle input and output business data, in order to avoid the virtual machine process running out of shared memory blocks, it can request the proxy service process to allocate other shared memory blocks, which can be called target shared memory blocks, to avoid the virtual machine process having to temporarily request them during subsequent use, thereby improving efficiency.
[0023] In one alternative implementation, the method further includes: If the number of free memory pages in a shared memory block is greater than the fourth threshold, a second request from the virtual machine process is received; wherein the fourth threshold is greater than the number of memory pages included in each shared memory block. At least one free co-occurring memory block is reclaimed into the memory pool according to the second request.
[0024] In the above embodiments, when the number of idle memory pages in the shared memory blocks allocated by the virtual machine process is greater than the number of memory pages included in each shared memory block, at least one idle shared memory block can be reclaimed to the memory pool of the agent service process to avoid wasting memory resources.
[0025] In one optional implementation, before determining whether the first quantity is less than a first threshold, the method further includes: The number of virtual machine processes has increased, or the amount of input / output data that at least one virtual machine process needs to process has increased.
[0026] In the above embodiments, the above-mentioned behavior of the virtual machine process can lead to a reduction in the number of free shared memory blocks managed by the agent service process. Therefore, when the above-mentioned behavior of the virtual machine is determined, the relationship between the first quantity and the first threshold is then determined to improve processing efficiency.
[0027] Secondly, one embodiment of this application provides a shared memory management device that integrates a proxy service process. The device includes: The determination module is used to: determine a first number of free shared memory blocks managed by the agent service process; wherein the agent service process is used to instruct the allocation of shared memory; The processing module is configured to: determine whether the first quantity is less than a first threshold; if so, create a new shared memory block through the agent service process and send a first notification message to each virtual machine process; wherein each virtual machine process is used to start at least one virtual machine; the first notification message is used to instruct each virtual machine process to establish a mapping relationship with the new shared memory block; the first threshold is used to indicate the minimum number of shared memory blocks required to maintain the normal operation of the agent service process; among the virtual machine processes and shared memory blocks with mapping relationships, at least one virtual machine started by the virtual machine process has the permission to use the shared memory block.
[0028] Thirdly, one embodiment of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above methods.
[0029] Fourthly, one embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of any of the above methods.
[0030] Fifthly, one embodiment of this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods. Attached Figure Description
[0031] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0032] Figure 1This is a schematic diagram of a shared memory structure provided in an embodiment of this application; Figure 2 A flowchart illustrating a shared memory management method provided in one embodiment of this application; Figure 3 A schematic diagram illustrating a memory resource reclamation process according to an embodiment of this application; Figure 4 A schematic diagram illustrating another memory resource reclamation process provided in an embodiment of this application; Figure 5 A schematic flowchart illustrating a reconstruction process provided in an embodiment of this application; Figure 6 A schematic flowchart illustrating another reconstruction process provided in an embodiment of this application; Figure 7 A schematic diagram of a shared memory management device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0033] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.
[0034] For ease of understanding, the terms used in the embodiments of this application are explained below: (1) In the field of computer science, the core of IO (Input / Output) refers to the process of data transmission between external devices and core computer components (such as memory and CPU), and is more focused on the operation and mechanism of data transmission than the data itself.
[0035] The core computer components are responsible for data processing and temporary storage, but they cannot directly interact with the external world. For example, reading keyboard input, displaying screen content, storing files to the hard drive, and connecting to a network to send and receive data all require I / O operations. For instance, an input operation refers to an external device transmitting data to the core computer components; an output operation is the core computer components sending processed data back to an external device.
[0036] In the embodiments of this application, IO operation refers to the data transmission process, and IO data refers to input and output data.
[0037] (2) Qemu, an open-source machine emulator and virtualization tool, is an upper-layer application that uses librbd and Proxy to transmit input and output data. In this embodiment, Qemu refers to the Qemu virtual machine process, or simply the virtual machine process, which can be used to start the virtual machine.
[0038] (3) Proxy, the core component of the block storage service client, writes Qemu's input and output data to the backend storage service, implementing the logic for the use and management of shared memory. In this embodiment, Proxy refers to the Proxy proxy service process, or the proxy service process.
[0039] (4) Shared memory: In the proxy scenario, shared memory is an efficient inter-process communication mechanism used to solve the data exchange problem between the proxy process and other components (such as backend services, caching modules, monitoring processes, etc.). It avoids the overhead of traditional IO or network communication by allowing multiple processes to directly access the same physical memory area, and is especially suitable for high-frequency data interaction scenarios.
[0040] (5)Librbd, a .so library, is a component of the block client. Qemu reads and writes are completed through the Librbd interface to implement the logic of shared memory usage.
[0041] (6) Pgg, the smallest granularity of shared memory blocks, which the Proxy expands and reclaims.
[0042] (7) Page: The shared memory block is further divided into memory pages.
[0043] In this embodiment, the shared memory of the proxy is often one or more large memory blocks. This shared memory is then divided into a management area and a data area, with the data area further subdivided into numerous pages at a 4k granularity. Specifically, in the block storage service client, the proxy manages the shared memory blocks at a 64MB granularity (Pgg), with each Pgg divided into 16k pages at a 4k granularity. When a Qemu virtual machine process connects to the proxy service process, the proxy service process allocates a shared memory block to the virtual machine process.
[0044] The number of any elements in the accompanying drawings is for illustrative purposes only and not as a limitation, and any naming is for distinction only and has no limiting meaning.
[0045] Typically, the host machine hosts multiple Qemu virtual machine processes and one proxy service process, which rapidly exchange input and output data via shared memory. The librbd instance in the current Qemu virtual machine process maps the shared memory in all proxy service processes and negotiates the use of a portion of the shared memory with the proxy service processes. When a Qemu virtual machine process exits, the proxy service process reclaims this shared memory but does not release it back to the operating system. Normally, when a Qemu virtual machine process connects to the proxy process, it maps all the shared memory and allocates a fixed management area for its own maintenance and use. However, the following two problems exist: First, Qemu uses a fixed amount of shared memory, which cannot be dynamically expanded or reduced according to the pressure of the business, resulting in insufficient resources and wasted memory resources.
[0046] Secondly, the shared memory in the proxy is of a fixed size. Even if no virtual machines are running on the host machine, this shared memory remains resident in the proxy service process and cannot be returned to the operating system, thus wasting the host machine's memory resources.
[0047] To address this, this application provides a shared memory management method. In this method, the shared memory is reorganized into smaller units (Pgg), and a certain number of Pggs are allocated to Qemu based on the business pressure. When Qemu exits or the business pressure decreases, the Pggs are cleaned up, returned, and released to the operating system.
[0048] After introducing the design concept of the embodiments of this application, the following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application can be applied. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.
[0049] refer to Figure 1 This is a schematic diagram of a shared memory structure provided in an embodiment of this application.
[0050] See Figure 1This demonstrates the shared memory structure managed by the extended page group manager `xpg_manager` in the agent service process. `xpg_manager_t` contains `xpg_param_t`, a Bitmap, and a `list_head`. Multiple `Page_group_t`s are linked through `list_head`, and each `Page_group_t` has an array of `ofsc_pg_desc_t` pointers and an array of `Page` pointers. The Qemu virtual machine process interacts with shared memory for page allocation via Librbd, and multiple Qemu instances have corresponding allocation processes.
[0051] Specifically, a Pgg is a contiguous memory space containing three parts: Page_group_t, ofsc_pg_desc_t, and Page. ofsc_pg_desc_t is a page frame, which points to a Page, a 4KB memory space (also called a memory page). When Qemu reads or writes input / output data, it allocates a certain number of page frames and places the data into the Page through these page frames. To avoid copying, the Proxy performs page swapping when processing this input / output data, that is, it swaps the Page with a new page frame from Qemu's page frames.
[0052] Of course, the methods provided in the embodiments of this application are not limited to... Figure 1 The application scenarios shown can also be used in other possible scenarios, and this application does not impose any limitations. Figure 1 The functions that each device in the application scenario shown can achieve will be described in subsequent method embodiments, and will not be elaborated on here.
[0053] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, the method may include more or fewer operation steps based on conventional or non-inventive methods. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application.
[0054] The following is combined Figure 1 The application scenarios shown illustrate the technical solutions provided in the embodiments of this application.
[0055] refer to Figure 2 This application provides a shared memory management method applied to a proxy service process, which instructs the allocation of shared memory. The method includes the following steps: S201: Determine the first number of free shared memory blocks managed by the agent service process.
[0056] S202: Determine if the first quantity is less than the first threshold. If so, execute S203.
[0057] S203: Create a new shared memory block through the agent service process and send the first notification message to each virtual machine process.
[0058] In this embodiment, when the first number of free shared memory blocks managed by the agent service process is less than the minimum number of shared memory blocks required to maintain the normal operation of the agent service, it indicates that the shared memory blocks of the agent service process are about to run out. New shared memory blocks can be created in advance, and a first notification message is sent to each virtual process. In this way, each virtual machine process establishes a mapping relationship with the new shared memory block, so that virtual machines started by each virtual machine process can use the new shared memory block. This avoids temporary allocation and improves the efficiency of page frame allocation. Therefore, by dynamically adjusting the number of shared memory blocks managed by the agent service process, the utilization rate of host machine memory resources is improved.
[0059] See Figure 2 This mainly explains the process of dynamically expanding and shrinking Pgg in the Proxy.
[0060] Regarding S201, determine the number of shared memory blocks managed by the agent service process, and determine the first number of free shared memory blocks among them.
[0061] Regarding S202 and S203, when the number of virtual machine processes increases, or when the amount of input / output data that at least one virtual machine process needs to process increases (increased business pressure), too many shared memory blocks are allocated to the proxy service process, resulting in a decrease in the number of free shared memory blocks (the first quantity) in the proxy service process. At this time, it can be determined whether the first quantity is less than a first threshold, which is the minimum number of shared memory blocks required to maintain the normal operation of the proxy service process. If the first quantity is less than the first threshold, it means that the shared memory blocks of the proxy service process are about to be insufficient. In this way, a new shared memory block can be created in advance, and a first notification message can be sent to each virtual process. In this way, each virtual machine process establishes a mapping relationship with the new shared memory block, so that the virtual machines started by each virtual machine process can use the new shared memory block. This avoids temporary allocation and improves the efficiency of page frame allocation.
[0062] In the example above, each virtual machine process is used to start at least one virtual machine. In a group of virtual machine processes and shared memory blocks with a mapping relationship, at least one virtual machine started by a virtual machine process has the permission to use the shared memory block.
[0063] Building upon the aforementioned technical solutions, new shared memory blocks can be created by changing the holder of a portion of the shared memory blocks managed by the operating system to a proxy service process. In practical applications, this can be achieved by modifying the status bits or flag bits of certain memory blocks.
[0064] Additionally, when the amount of input / output data that a virtual machine process needs to process decreases, or when some virtual machine processes perform the operation of exiting and starting the virtual machine, a further judgment can be made to determine the subsequent operation. This process can be implemented through steps A1-A2: A1: Determine if the first quantity is greater than the second threshold. The second threshold is the maximum number of free shared memory blocks allowed by the proxy service process, and the second threshold is greater than the first threshold. In this case, to avoid wasting memory resources, some free shared memory blocks can be reclaimed to the memory pool managed by the proxy service process, and the mapping relationship between the reclaimed shared memory blocks and the associated virtual machine processes can be released.
[0065] Optionally, for each shared memory block in the partially free shared memory blocks, perform the following operations A11-A13 to implement the reclamation process: A11: If a page swapping operation is detected, determine that some page frames in the page frames holding the shared memory block belong to the target shared memory block.
[0066] Among the pages pointed to by the page frames of each page in the shared memory block, some pages do not belong to the shared memory block, confirming that page swapping operations have occurred.
[0067] Optionally, iterate through each page frame corresponding to each page in the shared memory block, and determine whether all page frames point to memory pages of this shared memory block. If page swapping has occurred, some page frames will point to pages of other shared memory blocks. In other words, page frames of other shared memory blocks may also hold memory pages of this shared memory block. In this case, they cannot be reclaimed, and it is necessary to find out which shared memory blocks hold memory pages of this shared memory block.
[0068] Taking shared memory block N1 as an example, it includes n1 pages, corresponding to n1 page frames. Here, "corresponding" means that n1 logical addresses correspond to n1 physical addresses. Through this correspondence, data can be read and written by accessing the corresponding page frame through a single page. However, in actual applications, page swapping operations may occur. This can change the original correspondence between pages and page frames; that is, a page frame may point to a new page, and these new pages may not belong to the current shared memory block.
[0069] A12: If the holder of the target shared memory block is the proxy service process, then after the data transfer is completed, some page frames are cleared, and the page frames of the shared memory block are controlled to point to the pages of the shared memory block, and the cleared page frames are controlled to hold the pages of the target shared memory block.
[0070] In this scenario, the page frame can be returned after the data transmission is complete, and the swapped page frame and page can be swapped back.
[0071] A13: If the holder of the target memory block is a virtual machine process, then send a second notification message to a virtual machine process.
[0072] The second notification message is used to instruct a virtual machine process to clear some page frames after the data transfer is completed, and to control the page frames of the shared memory block to point to the pages of the shared memory block, and to control the cleared page frames to hold the pages of the target shared memory block.
[0073] Optionally, in this case, a second notification message can be used to notify the virtual machine process to return the page frames after the data transfer is completed, and to swap the exchanged page frames and pages back.
[0074] In addition, to avoid wasting resources, once all pages have been returned, the agent service process will notify all virtual machine processes to unmap the shared memory block.
[0075] A2: Release the shared memory blocks that have been reclaimed to the memory pool back to the operating system.
[0076] After all mappings are removed, the proxy service process will release the shared memory block and return it to the operating system.
[0077] For example, the specific implementation of the recycling process could be to change the holder of the shared memory blocks recycled to the memory pool to the operating system.
[0078] By reclaiming shared memory blocks until the number of free shared memory blocks falls below a second threshold, the host machine's memory usage can be reduced, thus saving memory.
[0079] In a specific example Figure 3 This is a schematic diagram illustrating a memory resource reclamation process provided in an embodiment of this application.
[0080] See Figure 3 The extended page group resource recycling manager xpg_rclm_Pgg process first checks whether all pages are in place. If they are in place, it synchronously notifies all Qemu to unmap them, then releases shared memory and cleans up the Bitmap. If they are not in place, it finds the Pgg to be swapped, triggers the PggA / PggB page swapping state machine xpg_rclm_Pgg_xch, and finally the process ends.
[0081] Figure 4 For another flowchart illustrating memory resource reclamation provided in this application embodiment, please refer to... Figure 4 The xpg_rclm_Pgg_xch process first determines whether Pgg is used by Qemu. If it is used, a message is sent to Pgg to initiate an asynchronous callback for page swapping. If it is not used, it iterates through Pgg to find pages in the free state that can be swapped and swaps them. Then it enters the state machine xpg_rclm_Pgg, and the process ends.
[0082] In practical applications, the agent service process may experience abnormal exits and restarts. In such cases, upon restarting, the agent service process needs to restore the previous shared memory block allocation and page swapping state to prevent the shared memory blocks, pages, and page frames used by the current virtual machine process from becoming invalid, resulting in input / output data loss. When the agent service process restarts, because the shared memory blocks are still mapped by the virtual machine process, this shared memory will not be reclaimed by the operating system. During agent service process reinitialization, the mapping is re-executed. Optionally, the Pgg reconstruction process is explained in steps B1-B4.
[0083] B1: If a proxy service process restart operation is detected, for any shared memory block, read the bitmap record information of the shared memory block and determine the usage status of the shared memory block.
[0084] The bitmap records information about the current shared memory block's usage status, such as 1 indicating that it is occupied and 0 indicating that it is not occupied.
[0085] B2: If the shared memory block is in an occupied state, a mapping will be established between the shared memory block and its owner to map the shared memory block.
[0086] In this scenario, the shared memory block needs to be initialized. This requires mapping the shared memory of the block first. Additionally, the Page_group_t section in the shared memory block's description information can be used to determine whether the current owner of the shared memory block is a proxy service process or a virtual machine process.
[0087] B3: If the holder is a virtual machine process, then the shared memory block is loaded according to its original mapping.
[0088] In this case, there is no need to reinitialize the page frame.
[0089] B4: If the holder is a proxy service process, initialize the page frame and load it according to the original mapping of the shared memory block.
[0090] In this case, the page frame needs to be reinitialized.
[0091] In both cases B3 and B4, it is not necessary to re-initialize the mapping between page frames and pages; the mapping relationship before the restart is maintained.
[0092] In a specific example Figure 5 A flowchart illustrating a reconstruction process provided in this application embodiment is shown below. Figure 5 The xpg_init_Pages process initializes the extended page group by traversing the Bitmap. If a Bit is occupied, xpg_init_Page is executed. Then, it determines whether to continue traversing until the traversal is complete.
[0093] Figure 6 A flowchart illustrating another reconstruction process provided in this application embodiment is shown below. Figure 6 The extended page group page initialization process (xpg_init_Page) first opens shared memory using the shared memory open function (shm_open) and adds it to the management list. Then, it checks if reinitialization is needed. If so, it reinitializes the Pgg. Next, it initializes each extended page group description structure (ofsc_pg_desc) and adds it to the Pgg. If reinitialization is not needed, it checks if the Pgg is in use. If it is, it synchronously notifies all Qemu mappings to map this Pgg. If it is not in use, it adds it to the free list (freelist). Finally, the process ends.
[0094] The above embodiment describes the processing procedure of the proxy service process. Steps C1-C2 are the processing procedure for the virtual machine process: C1: For any virtual machine process, if the number of free memory pages in the shared memory block allocated by the virtual machine process is less than the third threshold, then the first request from the virtual machine process is received.
[0095] The third threshold is the minimum number of memory pages required for the virtual machine to handle input / output data. In this case, the virtual machine process can send the first request to the agent service process via a Unix socket.
[0096] C2: Allocate the target shared memory block for the virtual machine in the memory pool based on the first request.
[0097] The memory pool is a resource pool consisting of idle resources managed by the agent service. This process involves virtual machine processes requesting additional shared memory blocks.
[0098] In a specific example, the virtual machine process uses memory pages to pass input and output data. When there are not enough idle memory pages, for example, when the number drops to a certain level (the third threshold), the virtual machine process can send a first request to the agent service process via a Unix socket to request an additional shared memory block (the target shared memory block).
[0099] If, after a period of time, the number of free memory pages exceeds the reclamation waterline (fourth threshold), resource reclamation can be achieved through steps D1-D2: D1: If the number of free memory pages in the shared memory block is greater than the fourth threshold, then accept the second request from the virtual machine process.
[0100] The fourth threshold is greater than the number of memory pages included in each shared memory block.
[0101] D2: Reclaim at least one free co-occurring memory block into the memory pool according to the second request.
[0102] In the above technical solutions, an idle memory page usually refers to a memory page that does not carry incomplete input / output data.
[0103] Typically, depending on the workload (the amount of input and output data being transmitted), virtual machine processes with different workloads will request and hold different numbers of shared memory blocks.
[0104] In summary, the embodiments of this application have the following beneficial effects: Shared memory resources in virtual machine processes can be dynamically expanded and reduced to avoid resource shortages that could degrade input / output data transmission performance and waste memory resources. Similarly, shared memory resources in proxy service processes can be dynamically expanded and reduced. Across the entire host machine, the proxy service process adjusts memory usage based on the number of virtual machine processes and workload, preventing memory waste. Shared memory can be reclaimed when page swapping occurs. Shared memory blocks can also be rebuilt after the proxy service process restarts, quickly reclaiming business logic and avoiding retries of input / output operations. The method of rebuilding shared memory blocks leverages the characteristic that mapped shared memory is not reclaimed by the operating system. Resource allocation and return are controlled through the high and low watermarks of page levels and shared memory blocks in virtual machine processes, dialing service processes, and proxy service processes.
[0105] like Figure 7 As shown, based on the same inventive concept as the above-described shared memory management method, this application embodiment also provides a shared memory management device, which integrates a proxy service process and includes a determination module 71 and a processing module 72.
[0106] The determining module 71 is used to: determine a first number of free shared memory blocks managed by the agent service process; wherein the agent service process is used to instruct the allocation of shared memory; The processing module 72 is configured to: determine whether the first quantity is less than a first threshold; if so, create a new shared memory block through the agent service process and send a first notification message to each virtual machine process; wherein each virtual machine process is used to start at least one virtual machine; the first notification message is used to instruct each virtual machine process to establish a mapping relationship with the new shared memory block; the first threshold is used to indicate the minimum number of shared memory blocks required to maintain the normal operation of the agent service process; among the virtual machine processes and shared memory blocks with mapping relationships, at least one virtual machine started by the virtual machine process has the permission to use the shared memory block.
[0107] In one alternative implementation, the processing module 72 is specifically used for: New shared memory blocks are created by changing the holder of a portion of the shared memory blocks managed by the operating system to a proxy service process.
[0108] In an optional implementation, the processing module 72 is further configured to: Determine if the first quantity is greater than the second threshold. If so, reclaim some of the free shared memory blocks to the memory pool managed by the agent service process, and unmap the reclaimed shared memory blocks from the associated virtual machine process. The second threshold is the maximum number of free shared memory blocks allowed by the agent service process. Release the shared memory blocks that have been reclaimed from the memory pool back to the operating system.
[0109] In one alternative implementation, the processing module 72 is specifically used for: Change the holder of the shared memory blocks reclaimed to the memory pool to the operating system.
[0110] In one optional implementation, before determining whether the first quantity is greater than the second threshold, the processing module 72 is further configured to: The amount of input / output data that the virtual machine process needs to process has decreased, or some virtual machine processes have performed the operation of exiting and starting the virtual machine.
[0111] In an alternative implementation, the processing module 72 is further configured to perform the following reclamation operation for each of the partially free shared memory blocks: If a page swapping operation is detected, determine that some page frames in the page frames holding the shared memory block belong to the target shared memory block. If the holder of the target shared memory block is the proxy service process, then after the data transfer is completed, some page frames are cleared, and the page frames of the shared memory block are controlled to point to the pages of the shared memory block, and the cleared page frames are controlled to hold the pages of the target shared memory block. If the holder of the target memory block is a virtual machine process, a second notification message is sent to the virtual machine process. The second notification message is used to instruct the virtual machine process to clear some page frames after the data transfer is completed, and to control the page frames of the shared memory block to point to the pages of the shared memory block, and to control the cleared page frames to hold the pages of the target shared memory block.
[0112] In an optional implementation, the processing module 72 is further configured to: Unmap each virtual machine process and its associated shared memory block, and release the unmapped shared memory block to the operating system.
[0113] In an optional implementation, the processing module 72 is further configured to: If a proxy service process restart operation is detected, for any shared memory block, read the bitmap record information of the shared memory block and determine the usage status of the shared memory block; If a shared memory block is in an occupied state, then establish a mapping between the shared memory block and its owner; If the holder is a virtual machine process, then the shared memory block is loaded according to its original mapping relationship; If the holder is a proxy service process, then the page frame is initialized and loaded according to the original mapping of the shared memory block.
[0114] In an optional implementation, the processing module 72 is further configured to: For any virtual machine process, if the number of free memory pages in the shared memory block allocated by the virtual machine process is less than the third threshold, then the first request from the virtual machine process is received; where the third threshold is the minimum number of memory pages that the virtual machine can handle input and output business data. Based on the first request, allocate a target shared memory block for the virtual machine in the memory pool; wherein, the memory pool is a resource pool composed of idle resources managed by the agent service.
[0115] In an optional implementation, the processing module 72 is further configured to: If the number of free memory pages in a shared memory block is greater than the fourth threshold, a second request from the virtual machine process is received; wherein the fourth threshold is greater than the number of memory pages included in each shared memory block. At least one free co-occurring memory block is reclaimed into the memory pool according to the second request.
[0116] In one optional implementation, before determining whether the first quantity is less than the first threshold, the processing module 72 is further configured to: The number of virtual machine processes has increased, or the amount of input / output data that at least one virtual machine process needs to process has increased.
[0117] The shared memory management device proposed in this application embodiment adopts the same inventive concept as the shared memory management method described above, and can achieve the same beneficial effects, so it will not be described again here.
[0118] Based on the same inventive concept as the shared memory management method described above, this application also provides an electronic device, which may specifically be a desktop computer, portable computer, smartphone, tablet computer, personal digital assistant (PDA), server, etc. Figure 8 As shown, the electronic device may include a processor 801 and a memory 802.
[0119] The processor 801 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.
[0120] Memory 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 802 may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.
[0121] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned computer storage medium can be any available medium or data storage device that a computer can access, including but not limited to: mobile storage devices, random access memory (RAM), magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)) and other media capable of storing program code.
[0122] Alternatively, if the integrated units described above in this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes: mobile storage devices, random access memory (RAM), magnetic memory (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical memory (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor memory (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs), etc.) and other media capable of storing program code.
[0123] Based on the same inventive concept, this application also provides a computer program product, which includes computer program code that, when executed on a computer, causes the computer to perform any of the shared memory management methods discussed above. Since the principle by which the above computer program product solves the problem is similar to that of the shared memory management method, the implementation of the above computer program product can be referred to the implementation of the method, and repeated details will not be elaborated further.
[0124] The above embodiments are only used to provide a detailed description of the technical solutions of this application. However, the description of the above embodiments is only for the purpose of helping to understand the methods of the embodiments of this application and should not be construed as a limitation on the embodiments of this application. Any changes or substitutions that can be easily conceived by those skilled in the art should be covered within the protection scope of the embodiments of this application.
Claims
1. A management method of shared memory, characterized by, The method applied to a proxy service process comprises: determining a first number of idle shared memory blocks managed by the proxy service process, wherein the proxy service process is used to instruct allocation of shared memory; determining whether the first number is less than a first threshold value, if yes, creating a new shared memory block through the proxy service process, and sending a first notification message to each virtual machine process, wherein each virtual machine process is used to start at least one virtual machine, the first notification message is used to instruct each virtual machine process to establish a mapping relationship with the new shared memory block, and the first threshold value is used to indicate a minimum number of shared memory blocks required to maintain normal operation of the proxy service process; the at least one virtual machine started by the virtual machine process in a group of virtual machine processes and shared memory blocks with the mapping relationship has a permission to use the shared memory block.
2. The method of claim 1, wherein, The new shared memory block is created through the proxy service process, comprising: creating the new shared memory block by modifying a holder of a part of the shared memory blocks managed by an operating system to the proxy service process.
3. The method of claim 1, wherein, The method further comprises: determining whether the first number is greater than a second threshold value, if yes, recycling part of the idle shared memory blocks to a memory pool managed by the proxy service process, and removing a mapping relationship between the recycled shared memory blocks and associated virtual machine processes; wherein the second threshold value is a maximum number of idle shared memory blocks allowed by the proxy service process, and the second threshold value is greater than the first threshold value; releasing the shared memory blocks recycled to the memory pool to an operating system.
4. The method of claim 3, wherein, The shared memory blocks recycled to the memory pool are released to the operating system, comprising: modifying a holder of the shared memory blocks recycled to the memory pool to the operating system.
5. The method of claim 3, wherein, Before determining whether the first number is greater than the second threshold value, the method further comprises: determining that a data amount of input and output data to be processed by the virtual machine process is reduced, or part of the virtual machine processes perform an operation of exiting a started virtual machine.
6. The method of claim 3, wherein, The method further comprises, for each shared memory block in the part of the idle shared memory blocks, performing the following recycling operation: in a case where it is detected that a page swap operation occurs, determining that part of page frames in page frames holding pages of the shared memory block belong to a target shared memory block; if a holder of the target shared memory block is the proxy service process, after data transmission ends, emptying the part of the page frames, and controlling page frames of the shared memory block to point to pages of the shared memory block, and controlling the emptied part of the page frames to hold pages of the target shared memory block; if the holder of the target memory block is one virtual machine process, sending a second notification message to the one virtual machine process, wherein the second notification message is used to instruct the one virtual machine process to, after data transmission ends, empty the part of the page frames, and control page frames of the shared memory block to point to pages of the shared memory block, and control the emptied part of the page frames to hold pages of the target shared memory block.
7. The method of claim 6, wherein, The method further comprises: The mapping of each virtual machine process and the associated shared memory block is released, and the unmapped shared memory block is released to the operating system.
8. The method according to any one of claims 1 to 7, characterized in that, The method further comprises: If the operation of restarting the agent service process is detected, the bitmap record information of any shared memory block is read, and the use state of the shared memory block is determined; If the shared memory block is in the occupied state, the mapping of the shared memory block and its holder is established; If the holder is a virtual machine process, the shared memory block is loaded according to the original mapping relationship of the shared memory block; If the holder is an agent service process, a page frame is initialized, and the shared memory block is loaded according to the original mapping relationship of the shared memory block.
9. The method according to any one of claims 1 to 7, characterized in that, The method comprises: For any virtual machine process, if the number of memory pages in the idle state in the shared memory block allocated by the virtual machine process is less than a third threshold value, a first request from the virtual machine process is received; wherein the third threshold value is the minimum number of memory pages for the virtual machine to handle input and output service data. According to the first request, a target shared memory block is allocated for the virtual machine in a memory pool; wherein the memory pool is a resource pool composed of idle resources managed by the agent service.
10. The method of claim 9, wherein, The method further comprises: If the number of memory pages in the idle state in the shared memory block is greater than a fourth threshold value, a second request from the virtual machine process is received; wherein the fourth threshold value is greater than the number of memory pages included in each shared memory block; According to the second request, at least one idle shared memory block is recycled to the memory pool.
11. The method according to any one of claims 1 to 7, characterized in that, Before the judgment of whether the first number is less than the first threshold value, the method further comprises: It is determined that the number of virtual machine processes increases, or the amount of input and output data that at least one virtual machine process needs to process increases.
12. A management apparatus of shared memory, characterized by, The device integrates an agent service process, and the device comprises: A determination module is configured to determine a first number of idle shared memory blocks managed by the agent service process; wherein the agent service process is used to indicate the allocation of shared memory. A processing module is configured to judge whether the first number is less than a first threshold value, and if so, create a new shared memory block through the agent service process, and send a first notification message to each virtual machine process; wherein each virtual machine process is used to start at least one virtual machine; the first notification message is used to indicate that each virtual machine process establishes a mapping relationship with the new shared memory block; the first threshold value is used to indicate the minimum number of shared memory blocks required to maintain the normal operation of the agent service process; among a group of virtual machine processes and shared memory blocks with a mapping relationship, at least one virtual machine started by the virtual machine process has the permission to use the shared memory block.
13. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 11.
14. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions are executed by the processor to implement the steps of the method of any one of claims 1 to 11.