Memory use method and device, equipment and medium

Abstract

The invention provides a memory use method and device, equipment and a medium, relates to the technical field of storage systems, and is applied to a distributed storage system.The method comprises the steps that after business processing services are started, a corresponding shared memory object is created for each business processing service; when data interaction needs to be carried out, establishing a link between the business processing service and the business request service; based on first network card information of a first physical node where the business processing service is located and second network card information of a second physical node where the business request service is located, whether the first physical node and the second physical node are the same physical node or not is judged; and when the nodes are the same physical node, performing data interaction by using the shared memory object. The copying of data interaction between the business request service and the business processing service deployed in the same physical node is reduced, and the copying by using a memory of a network card is not needed, so that the performance of the distributed storage system is improved.

Description

Technical Field

[0001] The present application relates to the technical field of storage systems, and in particular to a memory use method, device, equipment and medium. Background Art

[0002] With the development of current AI deep learning and artificial intelligence and high-performance computing tasks, HPC (High Performance Computing) scenarios are increasingly being used. In the future, with the continuous development and popularization of HPC technology, storage systems will also face the adaptation and iteration of HPC scenarios. For example, HPC storage systems are required to support data storage and management of high-performance computing tasks, massive data storage for large-scale data analysis and mining, and storage of large-scale data sets and model parameters to support various scenarios such as machine learning and deep learning tasks and reasoning.

[0003] In order to adapt to the above scenarios, HPC storage systems also require the use of high-performance hardware devices, such as high-speed storage devices (SSD, NVMe, etc.), high-speed network connections (InfiniBand, Ethernet), high-performance CPUs, etc. Among the requirements for high-speed network connections, in order to meet the above high performance, the network connection needs to meet high bandwidth and low latency. The industry currently commonly uses RMDA (Remote Direct Memory Access) support. RDMA technology can reduce memory copies, reduce data transmission delays and CPU load. For the selection of high-performance CPUs, multi-NUMA architecture is preferred to reduce memory access conflicts, improve parallel processing capabilities, and adapt to high-performance computing.

[0004] In the above-mentioned CPU usage, some memory copies between NUMAs are inevitable under the multi-NUMA architecture. Especially during network transmission, the network card can only be bound to one NUMA (non-uniform memory access). When the data of other NUMAs is sent through the network card, the data needs to be copied from the current NUMA to the network card NUMA. After the cluster performance of the distributed storage system reaches a certain specification, the impact of this memory copy on the cluster performance of the distributed storage system will be more obvious. Because data needs to be sent through the network card, and the network card is usually bound to a specific NUMA, when multiple NUMAs under the same physical node need to use the network card to realize data interaction, it will affect the performance of the distributed storage system. Summary of the invention

[0005] In view of this, embodiments of the present application provide a memory use method, apparatus, device and medium to overcome the above problems or at least partially solve the above problems.

[0006] A first aspect of an embodiment of the present application provides a memory use method, which is applied to a distributed storage system, wherein the distributed storage system includes a plurality of physical nodes, and a plurality of business processing services and a plurality of business request services are deployed in the plurality of physical nodes. The method includes:

[0007] After the business processing service is started, for each business processing service, a shared memory object corresponding to the business processing service is created;

[0008] When the business request service needs to interact with the business processing service, a link is established between the business processing service and the business request service;

[0009] Based on the first network card information of the first physical node where the business processing service is located and the second network card information of the second physical node where the business request service is located, determining whether the first physical node and the second physical node are the same physical node;

[0010] When the first physical node and the second physical node are the same physical node, using the shared memory object to perform the business request service requires data interaction with the business processing service.

[0011] Optionally, a cluster control service is also deployed on a physical node of the distributed storage system; after creating a shared memory object corresponding to the business processing service, the method further includes:

[0012] Based on the first identifier of the business processing service, create a second identifier of the shared memory object;

[0013] reporting the second identifier to the cluster control service through the business processing service, so that the cluster control service records the second identifier in a pre-created first data structure;

[0014] After the business request service subscribes to the first data structure, pushing the first data structure to the business request service through the cluster control service;

[0015] The first data structure is used to manage multiple shared memory objects corresponding to the multiple business processing services.

[0016] Optionally, the method further comprises:

[0017] After the business processing service is started, selecting the first network card information of the business processing service;

[0018] Recording first information of the business processing service in the first data structure, the first information including a first address, a first port number, and a first process number of the business processing service;

[0019] Based on the first information, monitoring the business processing service;

[0020] After the business request service is started, selecting the second network card information of the business request service;

[0021] Recording the second information of the business request service and the second network card information in a second data structure, wherein the second information includes a second address, a second port number, and a second process number of the business request service;

[0022] The business request service is monitored based on the second information.

[0023] Optionally, the determining whether the first physical node and the second physical node are the same physical node based on the first network card information of the first physical node where the business processing service is located and the second network card information of the second physical node where the business request service is located includes:

[0024] Acquire, through the service request service, from the first data structure, the first network card information of the first physical node where the service processing service is located, and acquire, from the second data structure, the second network card information;

[0025] Matching the first network card information with the second network card information;

[0026] Determine whether at least one first network card address and at least one second network card address are the same in the first network card information and the second network card information;

[0027] When the at least one first network card address and the at least one second network card address are identical, determining that the first physical node and the second physical node are the same physical node;

[0028] When the first network card address and the second network card address do not exist in the same manner, determining that the first physical node and the second physical node are not the same physical node;

[0029] The first network card information includes at least a first network card address, and the second network card information includes at least a second network card address.

[0030] Optionally, the method further comprises:

[0031] After determining that the first physical node and the second physical node are the same physical node, marking the link between the business processing service and the business request service as a first set value, wherein the first set value is used to indicate that data is exchanged between the business processing service and the business request service through the shared memory object;

[0032] Obtaining a first identifier of the business processing service through the business request service, and determining a second identifier of the shared memory object based on the first identifier;

[0033] Querying the shared memory object according to the second identifier through the business request service;

[0034] Through the business request service, use the memory application interface of the shared memory object to apply for a memory size of the target length, and record a memory pointer corresponding to the memory size of the target length applied for;

[0035] The method also includes: after determining that the first physical node and the second physical node are not the same physical node, marking the link between the business processing service and the business request service as a second set value, wherein the second set value is used to indicate that data interaction is performed between the business processing service and the business request service through the memory of the network card.

[0036] Optionally, the method further comprises:

[0037] When the data interaction is a write request, the data corresponding to the write request is written to the location pointed to by the memory pointer in the shared memory object through the business request service;

[0038] Sending a first message to the business processing service through the business request service, where the first message carries a first memory pointer corresponding to the write request;

[0039] Writing the data corresponding to the write request in the shared memory object pointed to by the first memory pointer to the disk through the business processing service, and sending a second message to the business request service, wherein the second message is used to inform the business request service that the data writing is completed;

[0040] Through the business request service, the memory release interface of the shared memory object is called to release the memory size of the target length corresponding to the first memory pointer.

[0041] Optionally, the method further comprises:

[0042] When the data interaction is a read request, sending a third message to the business processing service through the business request service, where the third message carries a second memory pointer corresponding to the read request;

[0043] Reading the data corresponding to the read request from the disk to the location pointed to by the second memory pointer in the shared memory object through the business processing service;

[0044] Sending a fourth message to the business request service through the business processing service, where the fourth message is used to notify the business request service to read data;

[0045] Through the business request service, the data corresponding to the read request in the shared memory object pointed to by the second memory pointer is read, and after reading, the memory size of the target length corresponding to the second memory pointer is released.

[0046] A second aspect of an embodiment of the present application provides a memory use device, which is applied to a distributed storage system, wherein the distributed storage system includes a plurality of physical nodes, wherein a plurality of business processing services and a plurality of business request services are deployed in the plurality of physical nodes, and the device includes:

[0047] A creation module, used for creating a shared memory object corresponding to each business processing service after the business processing service is started;

[0048] An establishing module, used for establishing a link between the business processing service and the business request service when the business request service needs to interact with the business processing service;

[0049] A judgment module, configured to judge whether the first physical node and the second physical node are the same physical node based on the first network card information of the first physical node where the business processing service is located and the second network card information of the second physical node where the business request service is located;

[0050] An execution module is used for, when the first physical node and the second physical node are the same physical node, using the shared memory object to perform data interaction between the business request service and the business processing service.

[0051] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the memory usage method as described in the first aspect.

[0052] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the memory usage method as described in the first aspect.

[0053] Beneficial effects of this application:

[0054] An embodiment of the present application provides a memory usage method, which is applied to a distributed storage system, wherein the distributed storage system includes multiple physical nodes, and multiple business processing services and multiple business request services are deployed in the multiple physical nodes. The method includes: after the business processing service is started, for each business processing service, creating a shared memory object corresponding to the business processing service; when the business request service needs to interact with the business processing service for data, establishing a link between the business processing service and the business request service; based on the first network card information of the first physical node where the business processing service is located and the second network card information of the second physical node where the business request service is located, determining whether the first physical node and the second physical node are the same physical node; when the first physical node and the second physical node are the same physical node, using the shared memory object to interact with the business request service for data.

[0055] Through the technical solution of the present application, a shared memory object is pre-created for each business processing service. When other business request services interact with the business processing service, it is possible to determine whether the business request service and the business processing service that need to interact are deployed on the same physical node based on whether the network card information corresponding to the business request service and the business processing service can match. When the business request service and the business processing service are deployed on the same physical node, the shared memory object pre-created by the business processing service in the physical node can be directly used for data interaction without using a network card, thereby reducing the copying of data interaction between the business request service and the business processing service deployed on the same physical node, and eliminating the need to use the memory of the network card for copying, thereby improving the performance of the distributed storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application.

[0057] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0058] Figure 1 is a flow chart of a memory usage method provided by an embodiment of the present application;

[0059] Figure 2 This is a schematic diagram of the interaction of various services in a memory usage method provided by an embodiment of the present application;

[0060] Figure 3 This is a flowchart of determining whether a business processing service and a business request service are deployed on the same physical node in a memory usage method provided in an embodiment of the present application;

[0061] Figure 4 It is a schematic diagram of a framework of a memory use device provided by an embodiment of the present application;

[0062] Figure 5 It is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0063] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0064] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] In the NUMA architecture, multiple NUMA nodes can be on the same physical node, but the network card can only be bound to the same NUMA node. Therefore, when data from other NUMA nodes needs to be transmitted to another NUMA node, the data must first be copied from the NUMA where it is located to the network card NUMA, and then the message is transmitted through the rmda memory requested by the network card NUMA. When the storage cluster is large, this data transmission method will affect the storage performance.

[0066] Based on this, the present application proposes a memory usage method, device, equipment and medium. Figure 1 is a flowchart of a memory usage method provided by an embodiment of the present application, such as Figure 1 shown.

[0067] An embodiment of the present application provides a memory usage method, which is applied to a distributed storage system, wherein the distributed storage system includes multiple physical nodes, and multiple business processing services and multiple business request services are deployed in the multiple physical nodes. The method includes:

[0068] Step S101, after the business processing service is started, for each business processing service, a shared memory object corresponding to the business processing service is created;

[0069] Step S102, when the business request service needs to interact with the business processing service, a link is established between the business processing service and the business request service;

[0070] Step S103, based on the first network card information of the first physical node where the business processing service is located and the second network card information of the second physical node where the business request service is located, determining whether the first physical node and the second physical node are the same physical node;

[0071] Step S104: when the first physical node and the second physical node are the same physical node, using the shared memory object to perform data interaction between the business request service and the business processing service.

[0072] First, the memory usage method of the present application is applied to a distributed storage system, which is a system that stores data in a dispersed manner on multiple physical nodes (computers or servers). Each physical node may contain part of the data, and the system provides complete data storage services by coordinating these nodes. The distributed storage system includes multiple physical nodes, such as independent computers or servers, and each physical node can be responsible for storing data, processing requests or other computing tasks. Multiple physical nodes are interconnected through a network to jointly provide a distributed storage function. Among them, multiple business processing services and multiple business request services are deployed in multiple physical nodes. One or more business processing services and / or one or more business request services may be deployed on each physical node, that is, the same physical node may include both business processing services and business request services, or may include business processing services or only business request services.

[0073] Among them, the business processing service is a service running on the physical node, responsible for executing various business logic and data processing operations. For example, the business processing service can be responsible for executing data calculations, processing user requests, running data analysis tasks, etc. The business request service is used to receive and process requests from users or other systems. After receiving the request, the business request service forwards it to the corresponding business processing service for processing. For example, the business request service can be responsible for processing API calls, user queries, data upload and download requests, etc. Both the business processing service and the business request service are processes.

[0074] First, in step S101, for each business processing service, when the business processing service is started and in operation, the system will create a shared memory object for the business processing service. This shared memory object is a memory area in which the business processing service can store and share data. At the same time, other services or processes can access this shared memory object to perform interactive tasks such as data transfer.

[0075] Further, in step S102, when a business processing service of a physical node A needs to interact with a business processing service of another physical node B, the physical node A is the client, the physical node B is the server, and the business request service will establish a link between the physical business processing service and the business request service. For example, the business processing service of physical node B is an OSS service (Object Storage Service), and the business request processing on physical node A is a client service. When the client service needs to request data interaction with the OSS service, the client service will establish a link between the OSS service and the client service to achieve communication between the client service and the OSS service.

[0076] Further, in step S103, the business processing service is deployed on the first physical node, and the business request service is deployed on the second physical node. According to the first network card information of the first physical node where the business processing service is located and the second network card information of the second physical node where the business request service is located, the first network card information is the network card information of the network card configured on the first physical node, and the second network card information is the network card information of the network card configured on the second physical node. According to the first network card information and the second network card information, it is determined whether the first physical node and the second physical node are the same physical node.

[0077] Finally, in step S104, when it is determined that the first physical node and the second physical node are the same physical node according to the first network card information and the second network card information, since the business processing service and the business request service are both deployed on the same physical node, the shared memory object created after the business processing service is started can be used to perform data interaction between the business processing service and the business request service. In other words, the business request service and the business processing service can directly exchange data through the shared memory object without the need for transmission over the network.

[0078] For NUMA architecture, when creating a link, the network card information of the services at both ends is used to determine whether the services at both ends belong to the same physical node. If they belong to the same physical node, shared memory objects can be used for data transmission. If they do not belong to the same physical node, the network card rmda is used for message exchange. This reduces unnecessary memory copies between NUMAs in the same node and improves storage performance.

[0079] Through the above embodiment, a shared memory object is pre-created for each business processing service. When other business request services interact with the business processing service, it is possible to determine whether the business request service and the business processing service that need to interact are deployed on the same physical node based on whether the network card information corresponding to the business request service and the business processing service can match. When the business request service and the business processing service are deployed on the same physical node, the shared memory object pre-created by the business processing service in the physical node can be directly used for data interaction without using the network card, thereby reducing the copying of data interaction between the business request service and the business processing service deployed on the same physical node, and eliminating the need to use the memory of the network card for copying, thereby improving the performance of the distributed storage system.

[0080] Optionally, a cluster control service is also deployed on a physical node of the distributed storage system; after creating a shared memory object corresponding to the business processing service, the method further includes:

[0081] Based on the first identifier of the business processing service, create a second identifier of the shared memory object;

[0082] reporting the second identifier to the cluster control service through the business processing service, so that the cluster control service records the second identifier in a pre-created first data structure;

[0083] After the business request service subscribes to the first data structure, pushing the first data structure to the business request service through the cluster control service;

[0084] The first data structure is used to manage multiple shared memory objects corresponding to the multiple business processing services.

[0085] Figure 2 is a schematic diagram of the interaction of various services in a memory usage method provided by an embodiment of the present application, such as Figure 2 shown.

[0086] Specifically, in one embodiment, a cluster control service (CCS) is also deployed on a physical node of the distributed storage system. After a business processing request creates a shared memory object, in order to facilitate the identification and management of the shared memory objects corresponding to each business processing request, a second identifier of the shared memory object can be created according to the first identifier of the business processing service to mark the shared memory object. The first identifier is used to identify and manage different business processing services, and the second identifier is used to identify and manage shared memory objects corresponding to different business processing services.

[0087] For example, when the business processing service is an OSS service, the naming format of the second identifier of each shared memory object is "shm_oss.$id". For example, if the first identifier of the OSS service is 1, the name of the shared memory object corresponding to the OSS service is created as shm_oss.1, and shm_oss.1 is used to uniquely identify this shared memory object.

[0088] Further, the second identifier is reported to the cluster control service through the business processing service, so that the cluster control service records the second identifier in a pre-created first data structure, wherein the first data structure may be a mapping table including the second identifier of the shared memory object identifier and the first identifier of the corresponding business processing service. The first data structure is used to centrally manage the shared memory objects corresponding to all business processing services, so that the system can effectively track and manage all shared memory objects.

[0089] The first data structure records the first identifier of each business processing service, the second identifier of the shared memory object corresponding to each business processing service, and the like.

[0090] Furthermore, the business request service will subscribe to the first data structure. After the business request service subscribes to the first data structure, when the first data structure changes, the cluster control service will push the first data structure to the business request service so that the business request service receives the relevant information of the business processing service and the relevant information of the shared memory object in the new first data structure.

[0091] Through the above embodiment, in the distributed storage system, the cluster control service manages the identification of the business processing service and its shared memory object, and pushes this information to the business request service. In this way, the business request service can effectively use the shared memory object for data interaction, thereby improving system performance and efficiency.

[0092] Optionally, the method further comprises:

[0093] After the business processing service is started, selecting the first network card information of the business processing service;

[0094] Recording first information of the business processing service in the first data structure, the first information including a first address, a first port number, and a first process number of the business processing service;

[0095] The cluster control service monitors the business processing service through the first information;

[0096] After the business request service is started, selecting the second network card information of the business request service;

[0097] Recording the second information of the business request service and the second network card information in a second data structure, wherein the second information includes a second address, a second port number, and a second process number of the business request service;

[0098] The business request service is monitored based on the second information.

[0099] Specifically, in one embodiment, the first data structure is also used to record the first network card information of each business processing service, that is, after the business processing service is started, the first network card information of the business processing service is selected and recorded in the first data structure, wherein the first information includes the first address, the first port number and the first process number of the business processing service, and the business request service can establish a link with the corresponding business processing service by accessing the first network card information of the business processing service that needs to interact with data in the first data structure. The cluster control service monitors the business processing service through the first information, including the network activity and process status of the business processing service.

[0100] The first address is the IP address of the first physical node where the business processing service is located, the second address is the IP address of the second physical node where the business request service is located, and the third address is the IP address of the third physical node where the cluster control service is located.

[0101] For the business request service, after the business request service is started, it is also necessary to select the second network card information of the business request service, and record the second information of the business request service and the second network card information in the second data structure, wherein the second information includes the second address, the second port number and the second process number of the business request service. The second data structure is used to record the relevant information of each business request service, and monitor the business request service through the second information, including the network activity and process status of the business request service.

[0102] The network card information at least includes the network card address.

[0103] Through the above embodiments, in the distributed storage system, in order to ensure that the business processing service and the business request service can communicate correctly, it is necessary to record and monitor the network and process information of the business processing service and the business request service. By recording the address, port number, process number and network card information of the service and performing corresponding monitoring, it is possible to effectively coordinate the communication between services, optimize data interaction and solve potential faults.

[0104] Optionally, the determining whether the first physical node and the second physical node are the same physical node based on the first network card information of the first physical node where the business processing service is located and the second network card information of the second physical node where the business request service is located includes:

[0105] Acquire, through the service request service, from the first data structure, the first network card information of the first physical node where the service processing service is located, and acquire, from the second data structure, the second network card information;

[0106] Matching the first network card information with the second network card information;

[0107] Determine whether at least one first network card address and at least one second network card address are the same in the first network card information and the second network card information;

[0108] When the at least one first network card address and the at least one second network card address are identical, determining that the first physical node and the second physical node are the same physical node;

[0109] When the first network card address and the second network card address do not exist in the same manner, determining that the first physical node and the second physical node are not the same physical node;

[0110] The first network card information includes at least a first network card address, and the second network card information includes at least a second network card address.

[0111] Specifically, in one embodiment, when determining in the above steps whether the first physical node and the second physical node are the same physical node, first, through the business request service, from the subscribed first data structure, obtain the first network card information of the first physical node where the business processing service that the business request service needs to connect is located, and obtain the second network card information of the second physical node where the business processing service is located from the second data structure.

[0112] Further, the first network card information of the first physical node and the second network card information of the second physical node are matched, wherein the first network card information includes at least a first network card address, and the second network card information includes at least a second network card address.

[0113] Determine whether there is at least one first network card address and at least one second network card address that are the same in the first network card information and the second network card information; when there is at least one first network card address and at least one second network card address that are the same, determine that the first physical node and the second physical node are the same physical node, wherein the first network card address and the second network card address both refer to network card IP addresses.

[0114] When there is no first network card address that is identical to any second network card address, it is determined that the first physical node and the second physical node are not the same physical node.

[0115] Through the above embodiments, in a distributed system, by comparing whether the same network card address exists in the network card information of the business processing service and the business request service, it can be determined whether the business processing service and the business request service are deployed on the same physical node. If the network card addresses are the same, it is confirmed that the two services are on the same physical node; if they are different, it is confirmed that they are not on the same physical node. This helps to optimize communication and data exchange between services.

[0116] Optionally, in addition to comparing the network card IP addresses of the physical nodes where the business request service and the business processing wall are respectively located to determine whether the two physical nodes are the same physical node, the following method can also be used to determine whether the two physical nodes are the same physical node.

[0117] MAC address: Each network card has a unique MAC address, which can be obtained through network management tools. If two devices have the same MAC address, they are likely the same device. Note that this method may not be reliable in some cases because MAC addresses can be forged.

[0118] Hardware serial number: Most computer hardware (such as motherboard, CPU, hard disk, etc.) has a unique serial number. You can get these serial numbers for comparison. For example, in Linux system, you can get the serial number of the motherboard through the dmidecode command, or get the serial number of the CPU through the lscpu command.

[0119] UUID and system ID: The operating system can read the device's UUID (Universally Unique Identifier) ​​or other unique identifier. These identifiers can usually uniquely identify the device. For Linux systems, you can get the machine ID through cat / etc / machine-id or hostnamectl.

[0120] Hardware configuration: By comparing the hardware configuration (such as CPU model, memory size, hard disk capacity, etc.), you can also identify whether they are the same physical node. You can use tools such as lshw or inxi to view detailed hardware information and compare them.

[0121] Operating system and software environment: Check the operating system version, installed software packages, application versions, etc. If the operating system and major software environment of two nodes are almost identical, they are likely different configuration snapshots of the same device.

[0122] Storage hardware information: Check the storage device information, such as the serial number, model, and other attributes of the SSD / HDD. This information can usually also be obtained through system commands (such as lsblk, smartctl).

[0123] BIOS information: Check the BIOS version and serial number. You can use the dmidecode -t bios command to get detailed BIOS information, which can be used to confirm the uniqueness of the hardware.

[0124] Physical location: In some cases, it is also a good idea to consider the physical location of the nodes. For example, if two devices are located in the same cabinet and have the same network topology and power supply number, it may indicate that they are different physical connections of the same device.

[0125] Network topology: Analyze network traffic and connection status to see if there is any abnormal traffic or exposure. If two nodes in the same network environment show similar behavior, they may be two interfaces of the same device.

[0126] Cloud platform identification: If it is a node existing in a cloud environment, check the resource identification information of the cloud service provider. Usually, each cloud host has a unique ID.

[0127] Through the above method, it is possible to more accurately determine whether two physical nodes are the same physical node. The method to be selected depends on the available information, environmental characteristics and actual needs. When one method cannot be used to determine, multiple methods can be combined to determine.

[0128] Optionally, the method further comprises:

[0129] After determining that the first physical node and the second physical node are the same physical node, marking the link between the business processing service and the business request service as a first set value, wherein the first set value is used to indicate that data is exchanged between the business processing service and the business request service through the shared memory object;

[0130] Obtaining a first identifier of the business processing service through the business request service, and determining a second identifier of the shared memory object based on the first identifier;

[0131] Querying the shared memory object according to the second identifier through the business request service;

[0132] Through the business request service, use the memory application interface of the shared memory object to apply for a memory size of the target length, and record a memory pointer corresponding to the memory size of the target length applied for;

[0133] The method also includes: after determining that the first physical node and the second physical node are not the same physical node, marking the link between the business processing service and the business request service as a second set value, wherein the second set value is used to indicate that data interaction is performed between the business processing service and the business request service through the memory of the network card.

[0134] Figure 3 This is a flowchart of determining whether a business processing service and a business request service are deployed on the same physical node in a memory usage method provided in an embodiment of the present application. Figure 3 shown.

[0135] Specifically, in one embodiment, when it is determined that the business processing service and the business request service are deployed on the same physical node, it is also necessary to mark the link between the business processing service and the business request service as data interaction through shared memory, so as to further process the details of the shared memory object.

[0136] First, the link between the business processing service and the business request service is marked as a first set value, which indicates that data can be exchanged between the business processing service and the business request service through a shared memory object. And through the business request service, in the first data structure, the first identifier of the business processing service is read, and through the first identifier, the second identifier of the shared memory object is determined. In this process, for example, after the Client service receives the information in the first data structure and obtains the first identifier "1" of the specific OSS service, it uses the naming format "shm_oss.$id" of the second identifier of the shared memory object and combines the first identifier to open the shared memory object with the second identifier "shm_oss.1" used by the OSS service.

[0137] Further, after finding the shared memory object corresponding to the second identifier through the business request service, use the memory application interface provided by the shared memory object. Request to allocate the target length of memory size, the target length is the memory size required by the business request service, for example, 100MB, 1GB, etc. After the memory application is successful, the system will return a memory pointer, and the memory pointer points to the memory area actually corresponding to the target length of memory size requested for allocation in the shared memory object. The business request service records this memory pointer for use in subsequent data interactions to perform corresponding operations (such as data writing and reading).

[0138] In addition, when it is determined based on the first network card information and the second network card information that the first physical node and the second physical node are not the same physical node, at this time, the link between the business processing service and the business request service is marked as a second set value. The second set value is used to indicate that the business processing service and the business request service are not on the same physical node, and data interaction is required through the network card's memory.

[0139] Through the above embodiment, after confirming that two services are deployed on the same physical node, the system will mark the link between them as data exchange through shared memory. Then, the business request service will obtain the identifier of the business processing service, determine the identifier of the shared memory object, find the corresponding shared memory object, and exchange data by applying for a memory interface. This method optimizes the efficiency and performance of data exchange.

[0140] Optionally, the method further comprises:

[0141] When the data interaction is a write request, the data corresponding to the write request is written to the location pointed to by the memory pointer in the shared memory object through the business request service;

[0142] Sending a first message to the business processing service through the business request service, where the first message carries a first memory pointer corresponding to the write request;

[0143] Writing the data corresponding to the write request in the shared memory object pointed to by the first memory pointer to the disk through the business processing service, and sending a second message to the business request service, wherein the second message is used to inform the business request service that the data writing is completed;

[0144] Through the business request service, the memory release interface of the shared memory object is called to release the memory size of the target length corresponding to the first memory pointer.

[0145] Specifically, in one embodiment, the memory pointer includes a first memory pointer and a second memory pointer, the first memory pointer is a memory pointer corresponding to a write request, and the second memory pointer is a memory pointer corresponding to a read request.

[0146] When the data interaction required by the business request service is a write request, it means that the data corresponding to the write request needs to be written into the shared memory object, and then the business processing service further processes the data. The business request service writes the data corresponding to the write request to the location pointed to by the memory pointer in the shared memory object, through the specific memory location pointed to by the memory pointer. This step ensures that the data is correctly placed in the shared memory and is ready for use by the business processing service. The business request service sends the memory pointer (pointing to the memory location where the data corresponding to the write request is written) and related information to the business processing service through a first message. The first message is used to notify the business processing service that the data corresponding to the write request has been written into the shared memory object, and provides the first memory pointer for the business processing service to read the data corresponding to the write request in the shared memory object. The business processing service reads the data corresponding to the write request from the shared memory object according to the first memory pointer, and writes the data corresponding to the write request to the disk, transferring the data from the memory to the persistent storage. After writing to the disk, the business processing service sends a second message to the business request service, and the second message is used to inform the business request service that the write operation of the data corresponding to the write request has been completed. This allows the business request service to know that its write request has been processed. Finally, the business request service calls the memory release interface provided by the shared memory object to release the memory size of the target length previously requested, ensuring that the memory that is no longer in use can be recycled to avoid memory leaks.

[0147] Through the above embodiment, when processing a write request, the business request service writes the data to the specified location through the shared memory object, and then notifies the business processing service of the memory location of the data. The business processing service reads the data in the shared memory, writes it to the disk, and notifies the business request service that the write operation is complete. Finally, the business request service releases the previously requested memory. This process ensures the complete flow of data from the business request service to the disk and effectively manages memory resources.

[0148] Optionally, the method further comprises:

[0149] When the data interaction is a read request, sending a third message to the business processing service through the business request service, where the third message carries a second memory pointer corresponding to the read request;

[0150] Reading the data corresponding to the read request from the disk to the location pointed to by the second memory pointer in the shared memory object through the business processing service;

[0151] Sending a fourth message to the business request service through the business processing service, where the fourth message is used to notify the business request service to read data;

[0152] Through the business request service, the data corresponding to the read request in the shared memory object pointed to by the second memory pointer is read, and after reading, the memory size of the target length corresponding to the second memory pointer is released.

[0153] Specifically, in one embodiment, when the data interaction required by the business request service is a read request, it indicates that the data corresponding to the read request needs to be read from the hard disk into the shared memory object, and the data is finally returned to the business request service. After receiving the read request, the business request service will send a third message to the business processing service, and the third message carries a second memory pointer corresponding to the read request, and the second memory pointer points to a memory location in a shared memory object, which is used to cache the data to be read from the hard disk. After the business processing service receives the third message, the business processing service will use this second memory pointer to read the data corresponding to the read request from the disk to the memory location pointed to by the second memory pointer in the shared memory object, so that the data corresponding to the read request is stored in the location pointed to by the second memory pointer after being read. After the data is read into the shared memory object, the business processing service will send a fourth message to the business request service, and the purpose of the fourth message is to notify the business request service that the operation of reading the data corresponding to the read request from the hard disk has been completed. The business request service then uses the second memory pointer to read the data corresponding to the read request in the shared memory object. After the reading is completed, the memory size of the target length of the memory area pointed to by the second memory pointer will also be released to avoid memory leaks.

[0154] Through the above embodiments, efficient data access and memory management can be achieved, ensuring that data can be read and returned quickly and correctly when processing a read request.

[0155] Figure 4 is a schematic diagram of a memory usage device provided by an embodiment of the present application. Figure 4 shown.

[0156] Based on the same inventive concept, another embodiment of the present application further provides a memory usage device, which is applied to a distributed storage system, wherein the distributed storage system includes a plurality of physical nodes, wherein a plurality of business processing services and a plurality of business request services are deployed in the plurality of physical nodes, and the device includes:

[0157] A creation module 11, configured to create a shared memory object corresponding to each business processing service after the business processing service is started;

[0158] An establishing module 12, configured to establish a link between the business processing service and the business request service when the business request service needs to interact with the business processing service;

[0159] A judgment module 13, configured to judge whether the first physical node and the second physical node are the same physical node based on the first network card information of the first physical node where the business processing service is located and the second network card information of the second physical node where the business request service is located;

[0160] The execution module 14 is used for using the shared memory object to perform data interaction between the business request service and the business processing service when the first physical node and the second physical node are the same physical node.

[0161] Optionally, a cluster control service is also deployed on a physical node of the distributed storage system; and the device further includes:

[0162] A second identification creation module, configured to create a second identification of the shared memory object based on the first identification of the business processing service after creating the shared memory object corresponding to the business processing service;

[0163] A second identification recording module, configured to report the second identification to the cluster control service through the business processing service, so that the cluster control service records the second identification in a pre-created first data structure;

[0164] a push module, configured to push the first data structure to the business request service through the cluster control service after the business request service subscribes to the first data structure;

[0165] The first data structure is used to manage multiple shared memory objects corresponding to the multiple business processing services.

[0166] Optionally, the device further comprises:

[0167] A first network card information selection module, used to select the first network card information of the business processing service after the business processing service is started;

[0168] A first information recording module, configured to record first information of the business processing service in the first data structure, wherein the first information includes a first address, a first port number, and a first process number of the business processing service;

[0169] A first monitoring module, used for monitoring the business processing service based on the first information;

[0170] A second network card information selection module, used for selecting the second network card information of the service request service after the service request service is started;

[0171] A second information recording module, used to record the second information of the business request service and the second network card information in a second data structure, wherein the second information includes a second address, a second port number and a second process number of the business request service;

[0172] The second monitoring module is used to monitor the business request service based on the second information.

[0173] Optionally, the judging module 13 includes:

[0174] an acquiring unit, configured to acquire, through the business request service, from the first data structure, first network card information of the first physical node where the business processing service is located, and to acquire the second network card information from the second data structure;

[0175] a matching unit, configured to match the first network card information with the second network card information;

[0176] a judging unit, configured to judge whether at least one first network card address and at least one second network card address are the same in the first network card information and the second network card information;

[0177] A first determining unit, configured to determine that the first physical node and the second physical node are the same physical node when the at least one first network card address and the at least one second network card address are identical;

[0178] a second determining unit, configured to determine that the first physical node and the second physical node are not the same physical node when the first network card address and the second network card address do not exist;

[0179] The first network card information includes at least a first network card address, and the second network card information includes at least a second network card address.

[0180] Optionally, the device further comprises:

[0181] A first set value setting device, used for marking the link between the business processing service and the business request service as a first set value after determining that the first physical node and the second physical node are the same physical node, wherein the first set value is used to indicate that data is exchanged between the business processing service and the business request service through the shared memory object;

[0182] A second identification determining device, configured to obtain the first identification of the business processing service through the business request service, and determine the second identification of the shared memory object based on the first identification;

[0183] A shared memory object query device, configured to query the shared memory object according to the second identifier through the business request service;

[0184] An application device, used to apply for a target length of memory size by using the application memory interface of the shared memory object through the business request service, and record a memory pointer corresponding to the applied target length of memory size;

[0185] The device also includes: a second set value setting device, which is used to mark the link between the business processing service and the business request service as a second set value after determining that the first physical node and the second physical node are not the same physical node, and the second set value is used to indicate that data interaction is performed between the business processing service and the business request service through the memory of the network card.

[0186] Optionally, the device further comprises:

[0187] A first writing module, configured to, when the data interaction is a write request, write the data corresponding to the write request to the location pointed to by the memory pointer in the shared memory object through the business request service;

[0188] A first message sending module, configured to send a first message to the business processing service through the business request service, wherein the first message carries a first memory pointer corresponding to the write request;

[0189] A second writing module is used to write the data corresponding to the write request in the shared memory object pointed to by the first memory pointer to the disk through the business processing service, and send a second message to the business request service, wherein the second message is used to inform the business request service that the data writing is completed;

[0190] The first release module is used to call the memory release interface of the shared memory object through the business request service to release the memory size of the target length corresponding to the first memory pointer.

[0191] Optionally, the device further comprises:

[0192] A third message sending module is used to send a third message to the business processing service through the business request service when the data interaction is a read request, wherein the third message carries a second memory pointer corresponding to the read request;

[0193] A first reading module, configured to read the data corresponding to the read request from the disk to a position pointed to by the second memory pointer in the shared memory object through the business processing service;

[0194] A second reading module, used for sending a fourth message to the business request service through the business processing service, wherein the fourth message is used for notifying the business request service to read data;

[0195] The second release module is used to read the data corresponding to the read request in the shared memory object pointed to by the second memory pointer through the business request service, and release the memory size of the target length corresponding to the second memory pointer after reading.

[0196] Based on the same inventive concept, another embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the memory usage method as described in any of the above embodiments.

[0197] Based on the same inventive concept, another embodiment of the present application further provides a computer program product, including a computer program, which is executed by a processor to perform the memory usage method described in any of the above embodiments.

[0198] Among them, electronic equipment refers to Figure 5 , Figure 5 is a schematic diagram of an electronic device provided by an embodiment of the present application. Figure 5 As shown, the electronic device 500 includes: a memory 510 and a processor 520. The memory 510 and the processor 520 are connected via a bus communication. A computer program is stored in the memory 510. The computer program can be run on the processor 520 to implement the steps in the digital memory usage method disclosed in the above embodiment of the present application.

[0199] Based on the same inventive concept, another embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, wherein when the program is executed by a processor, the memory usage method as described in any of the above embodiments is implemented.

[0200] As for the device, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0201] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0202] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0203] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0204] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0205] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0206] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0207] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0208] The above is a detailed introduction to a memory usage method, device, equipment and medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A memory usage method, characterized in that: Applied to a distributed storage system, the distributed storage system includes a plurality of physical nodes, a plurality of business processing services and a plurality of business request services are deployed in the plurality of physical nodes, the method includes: After the business processing service is started, for each business processing service, a shared memory object corresponding to the business processing service is created; When the business request service needs to interact with the business processing service, a link is established between the business processing service and the business request service; Based on the first network card information of the first physical node where the business processing service is located and the second network card information of the second physical node where the business request service is located, determining whether the first physical node and the second physical node are the same physical node; When the first physical node and the second physical node are the same physical node, using the shared memory object to perform the business request service requires data interaction with the business processing service.

2. The memory use method according to claim 1, characterized in that: A cluster control service is also deployed on a physical node of the distributed storage system; after creating a shared memory object corresponding to the business processing service, it also includes: Based on the first identifier of the business processing service, create a second identifier of the shared memory object; reporting the second identifier to the cluster control service through the business processing service, so that the cluster control service records the second identifier in a pre-created first data structure; After the business request service subscribes to the first data structure, pushing the first data structure to the business request service through the cluster control service; The first data structure is used to manage multiple shared memory objects corresponding to the multiple business processing services.

3. The memory use method according to claim 2, characterized in that: The method further comprises: After the business processing service is started, selecting the first network card information of the business processing service; Recording first information of the business processing service in the first data structure, the first information including a first address, a first port number, and a first process number of the business processing service; Based on the first information, monitoring the business processing service; After the business request service is started, selecting the second network card information of the business request service; Recording the second information of the business request service and the second network card information in a second data structure, wherein the second information includes a second address, a second port number, and a second process number of the business request service; The business request service is monitored based on the second information.

4. The memory use method according to claim 3, characterized in that: The determining, based on the first network card information of the first physical node where the business processing service is located and the second network card information of the second physical node where the business request service is located, whether the first physical node and the second physical node are the same physical node includes: Acquire, through the service request service, from the first data structure, the first network card information of the first physical node where the service processing service is located, and acquire, from the second data structure, the second network card information; Matching the first network card information with the second network card information; Determine whether at least one first network card address and at least one second network card address are the same in the first network card information and the second network card information; When the at least one first network card address and the at least one second network card address are identical, determining that the first physical node and the second physical node are the same physical node; When the first network card address and the second network card address do not exist in the same manner, determining that the first physical node and the second physical node are not the same physical node; The first network card information includes at least a first network card address, and the second network card information includes at least a second network card address.

5. The memory use method according to claim 2, characterized in that: The method further comprises: After determining that the first physical node and the second physical node are the same physical node, marking the link between the business processing service and the business request service as a first set value, wherein the first set value is used to indicate that data is exchanged between the business processing service and the business request service through the shared memory object; Obtaining a first identifier of the business processing service through the business request service, and determining a second identifier of the shared memory object based on the first identifier; Querying the shared memory object according to the second identifier through the business request service; Through the business request service, use the memory application interface of the shared memory object to apply for a memory size of the target length, and record a memory pointer corresponding to the memory size of the target length applied for; The method also includes: after determining that the first physical node and the second physical node are not the same physical node, marking the link between the business processing service and the business request service as a second set value, wherein the second set value is used to indicate that data interaction is performed between the business processing service and the business request service through the memory of the network card.

6. The memory use method according to claim 5, characterized in that: The method further comprises: When the data interaction is a write request, the data corresponding to the write request is written to the location pointed to by the memory pointer in the shared memory object through the business request service; Sending a first message to the business processing service through the business request service, where the first message carries a first memory pointer corresponding to the write request; Writing the data corresponding to the write request in the shared memory object pointed to by the first memory pointer to the disk through the business processing service, and sending a second message to the business request service, wherein the second message is used to inform the business request service that the data writing is completed; Through the business request service, the memory release interface of the shared memory object is called to release the memory size of the target length corresponding to the first memory pointer.

7. The memory use method according to claim 6, characterized in that: The method further comprises: When the data interaction is a read request, sending a third message to the business processing service through the business request service, where the third message carries a second memory pointer corresponding to the read request; Reading the data corresponding to the read request from the disk to the location pointed to by the second memory pointer in the shared memory object through the business processing service; Sending a fourth message to the business request service through the business processing service, where the fourth message is used to notify the business request service to read data; Through the business request service, the data corresponding to the read request in the shared memory object pointed to by the second memory pointer is read, and after reading, the memory size of the target length corresponding to the second memory pointer is released.

8. A memory usage device, characterized in that: Applied to a distributed storage system, the distributed storage system includes a plurality of physical nodes, a plurality of business processing services and a plurality of business request services are deployed in the plurality of physical nodes, the device includes: A creation module, used for creating a shared memory object corresponding to each business processing service after the business processing service is started; An establishing module, used for establishing a link between the business processing service and the business request service when the business request service needs to interact with the business processing service; A judgment module, configured to judge whether the first physical node and the second physical node are the same physical node based on the first network card information of the first physical node where the business processing service is located and the second network card information of the second physical node where the business request service is located; An execution module is used for, when the first physical node and the second physical node are the same physical node, using the shared memory object to perform data interaction between the business request service and the business processing service.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the memory use method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by a processor, the memory using method as described in any one of claims 1-7 is implemented.

Images