A method and device for improving high performance computing

By adopting shared memory technology and memory hot and cold data hierarchical algorithm in high-performance computing clusters, dynamic and efficient addition and deletion of computing nodes is achieved, solving the efficiency of computing node management under dynamic load, and improving computing performance and memory utilization Rate.

CN118550735BActive Publication Date: 2024-11-26CHINA TELECOM CLOUD TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411025615.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-26
Estimated Expiration
2044-07-30

AI Technical Summary

Technical Problem

In dynamic load scenarios, how to quickly and efficiently add or hibernate computing nodes to improve the performance of high-performance computing, especially when load conditions change frequently, the existing technology is difficult to effectively solve this problem.

Method used

By adopting shared memory technology in high-performance computing clusters, multiple computing nodes time-sharing multiplex the same shared memory, and using the memory hot and cold data hierarchy algorithm to dynamically adjust the node load under elastic load conditions, achieving efficient addition of computing nodes and delete.

Benefits of technology

It improves the performance and memory utilization of high-performance computing, solves the problem of uneven memory usage after a single computing node is expanded, and improves the dynamic addition/deletion capabilities of nodes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118550735B_ABST
    Figure CN118550735B_ABST
Patent Text Reader

Abstract

The present invention relates to a method and device for improving high-performance computing, belonging to the field of high-performance computing technology, the method comprising: configuring shared memory as a dax device in multiple computing nodes; mapping the dax device to the current process when the program starts; the process obtains the virtual address of the first address of the dax device, initializes the first area in the shared memory, calls the malloc function in the so file shared library, the process seizes bytes in the first area, and calculates the first offset; according to the virtual address of the first address mapped by the dax device in the current process and the first offset, the virtual address of the first address of the shared memory area seized this time in the current process is obtained; the computing node writes the calculation result into the shared memory area to obtain a second offset; when obtaining data, the virtual address of the usable data block is obtained, and data interaction is performed in a read-write memory manner. The present invention enables multiple computing nodes to time-share multiplex the same shared memory.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of high performance computing, and in particular, relates to a method and device for improving high performance computing. Background Art

[0002] With the advent of the era of information explosion, high-performance computing technology has been increasingly used in various scientific research or various corporate production, such as climate simulation and weather forecasting, oil exploration, analysis of genomic data, etc. Under increasingly complex application scenarios, the amount of data has also increased exponentially. Therefore, for high-performance computing technology, the ultra-large capacity demand for memory is a key bottleneck restricting its rapid development.

[0003] In dynamic load scenarios such as dynamic model training and real-time reasoning, nodes will be added / sleeped according to the load conditions. This way, the capabilities of each node can be fully utilized under high load conditions. Under low load conditions, sleeping some nodes can save expenses. When the load conditions change frequently, how to quickly and efficiently add nodes / sleeping nodes to effectively improve the performance of high-performance computing technology is a technical problem that needs to be solved. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the invention is to provide a method, device, electronic device and storage medium for improving high-performance computing. In the present invention, for a high-performance computing cluster with inelastic load, multiple computing nodes can time-share the same shared memory, and for a high-performance computing cluster with elastic load, the effect of dynamically and efficiently adding / deleting computing nodes can be achieved, thereby effectively improving the performance of high-performance computing technology.

[0005] In a first aspect of the present invention, a method for improving high performance computing is provided, which is applied to a system including: a plurality of computing nodes connected to a shared memory via an interconnection bus, the method comprising:

[0006] Determine that the computing nodes are nodes in the non-elastic load cluster, and configure the shared memory as a dax device in multiple computing nodes;

[0007] Write a so file shared library, which maps the dax device to this process when the program starts;

[0008] The process obtains the virtual address of the first address of the dax device, initializes the first area in the shared memory, calls the malloc function in the so file shared library, the process grabs a byte in the first area, and calculates the first offset based on the ID of the grabbed byte;

[0009] According to the virtual address of the first address mapped by the dax device in the current process and the first offset, the virtual address of the first address of the shared memory area seized this time in the current process is obtained;

[0010] The computing node writes the computing result into the shared memory area, and obtains the second offset according to the virtual address difference of the data and the virtual address of the first address of the dax device;

[0011] When the computing node obtains data, it obtains the virtual address of the usable data block according to the second offset and the virtual address of the first address of the dax device, and performs data interaction in the manner of reading and writing memory.

[0012] Furthermore, the above method for improving high performance computing further includes:

[0013] Determine that the computing node is a node in the elastic load balancing cluster, and use the hot and cold memory tiering algorithm to place the common data of the computing node into the shared memory applied for by the computing node;

[0014] When the load increases to be greater than a first preset threshold, some computing nodes are scheduled to participate in the calculation, and the newly awakened nodes directly read the data in the shared memory of other nodes except the newly awakened nodes.

[0015] Furthermore, the above method for improving high performance computing further includes:

[0016] Determine that the computing node is a node in the elastic load balancing cluster, and use the hot and cold memory tiering algorithm to place the common data of the computing node into the shared memory applied for by the computing node;

[0017] When the load is reduced to less than a second preset threshold, some computing nodes are put into sleep mode, and data that may be used later by the sleeping computing nodes but is currently stored in the shared memory is compressed;

[0018] The second area in the shared memory seizes the memory area, and the compressed data is placed in the seized memory area as backup data, and the released shared memory is recorded as a temporary use area;

[0019] When other computing nodes that are still running cannot apply for memory, they can use the temporary area. Then wake up the dormant computing nodes again and check whether the temporary area is occupied.

[0020] If it is occupied, the dedicated memory is re-occupied, and the backup data is copied back from the memory area for decompression and continued use. If the temporary use area is not occupied, it is used directly to release the backup data in the memory area.

[0021] Further, in the above method for improving high performance computing, configuring the shared memory as a dax device in the plurality of computing nodes is configured through the ndctl command;

[0022] The so file shared library maps the dax device to this process through mmap when the program starts;

[0023] Initialize the first area in the shared memory by writing 1, 2, 3...n per byte, where n = (maximum capacity of shared memory / 1GiB) - 1;

[0024] The process grabs a byte in the first area by CAS.

[0025] Furthermore, in the above method for improving high performance computing, the computing result at least includes: intermediate data, output data, and status information that needs to be interacted with.

[0026] Furthermore, in the above method for improving high performance computing, the second offset is obtained according to the virtual address difference of the data and the virtual address of the first address of the dax device, which is obtained by subtracting the virtual address difference of the data from the virtual address of the first address of the dax device.

[0027] Furthermore, in the above method for improving high performance computing, the virtual address of the usable data block is obtained according to the second offset and the virtual address of the first address of the dax device by summing the second offset and the virtual address of the first address of the dax device.

[0028] In a second aspect of the present invention, a device for improving high performance computing is provided, which is applied to a system including: a plurality of computing nodes connected to a shared memory via an interconnection bus, the device including:

[0029] Configuration module: used to determine whether a computing node is a node in a non-elastic load cluster and configure shared memory as a dax device in multiple computing nodes;

[0030] Mapping module: used to write the so file shared library, which maps the dax device to the current process when the program starts;

[0031] Calculation module: used by the process to obtain the virtual address of the first address of the dax device, initialize the first area in the shared memory, call the malloc function in the so file shared library, the process grabs a byte in the first area, and calculates the first offset according to the ID of the grabbed byte;

[0032] The first obtaining module is used to obtain the virtual address of the first address of the shared memory area seized this time in the current process according to the virtual address of the first address mapped by the dax device in the current process and the first offset;

[0033] The second obtaining module is used for the computing node to write the calculation result into the shared memory area, and obtain the second offset according to the virtual address difference of the data and the virtual address of the first address of the dax device;

[0034] The third obtaining module: when the computing node obtains data, the virtual address of the usable data block is obtained according to the second offset and the virtual address of the first address of the dax device, and data interaction is performed in the form of reading and writing memory.

[0035] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor and a memory;

[0036] The processor is used to execute a method for improving high-performance computing as described in any of the above items by calling the program or instruction stored in the memory.

[0037] In a fourth aspect of the present invention, a computer-readable storage medium is further proposed, wherein the computer-readable storage medium stores a program or instruction, wherein the program or instruction enables a computer to execute a method for improving high-performance computing as described in any of the above items.

[0038] The beneficial effects of the present invention are as follows: the present invention connects each computing node in a high-performance computing cluster with non-dynamic load to the same ultra-large shared memory through an interconnection bus, and the computing nodes can realize time-sharing multiplexing of the entire shared memory, which not only solves the capacity expansion problem of a single computing node, but also solves the problem of waste caused by uneven memory usage of each node after simply expanding the capacity of each computing node. Moreover, since each computing node is connected to the same shared memory, the computing data between the nodes can also be quickly exchanged through the shared memory. For computing nodes in a high-performance computing cluster with dynamic loads, multi-node common data is written only to shared memory through similar memory hot-cold tiering technology. Other nodes can give priority to reading directly from the shared memory when starting up. When a computing node is put into hibernation to save energy, the memory in the node can be deeply reduced in power consumption without retaining it. The local memory of the node only needs to be compressed and sent to a specific area of ​​the shared memory. The memory previously used in the shared memory is marked as temporary memory. When the shared memory is sufficient, it does not need to be released. In this way, when the dormant computing node is restarted later, the data retained in the shared memory can be directly used. This not only improves memory utilization, but also improves the ability to dynamically add / delete nodes, thereby improving the performance of high-performance computing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are only used to illustrate specific embodiments and are not considered to limit the present invention. In the entire drawings, the same reference symbols represent the same components. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0040] Figure 1 A method for improving high performance computing provided by an embodiment of the present invention Figure 1 ;

[0041] Figure 2 A method for improving high performance computing provided by an embodiment of the present invention Figure 2 ;

[0042] Figure 3 A method for improving high performance computing provided by an embodiment of the present invention Figure 3 ;

[0043] Figure 4 A diagram of a device for improving high performance computing provided by an embodiment of the present invention;

[0044] Figure 5 A schematic block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present invention.

[0046] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0047] In the description of the present invention, the terms "first", "second", and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be internal communication between two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0048] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0049] Before introducing the embodiments of the present invention, professional terms involved in the present invention are first introduced.

[0050] HPC (High-Performance Computing) is a technology that uses high-performance computers and computing clusters to solve complex problems that require a large amount of computing resources. A computing cluster is a system consisting of multiple computers connected together through a high-speed network, and each computer is called a node. Through distributed computing technology, a computing cluster can process large-scale computing tasks as a whole.

[0051] Compute Express Link (CXL) is an industry-supported cache-coherent interconnect technology standard protocol that is an open industrial standard between processors and devices such as accelerators, memory buffers, intelligent network interfaces, persistent memory, and solid-state drives.

[0052] CAS (compare and swap) compares and exchanges, which is generally used to assign the set value to the memory only when the value in the specified memory is equal to the expected value, and is used as a lock-free resource competition method.

[0053] A shared library usually ends with .so (Shared Object) in UNIX and UNIX-like systems (such as Linux). It is a library that can be shared among multiple programs. Shared libraries allow programs to be dynamically loaded and linked at runtime, thereby reducing memory usage and disk space usage, and facilitating software updates and maintenance.

[0054] LD_PRELOAD is an environment variable that allows the user to specify a shared library when running a program. This shared library will be loaded before the standard library. This is very useful for injecting custom code or replacing existing library functions while the program is running.

[0055] The present invention proposes a method, device, electronic device and storage medium for improving high-performance computing. For a high-performance computing cluster with inelastic load, multiple computing nodes can time-share the same shared memory. For a high-performance computing cluster with elastic load, dynamic and efficient addition / deletion of computing nodes can be achieved, thereby effectively improving the performance of high-performance computing technology.

[0056] Method Embodiment

[0057] Figure 1 A method for improving high performance computing provided by an embodiment of the present invention Figure 1 .

[0058] In a first aspect of the present invention, a method for improving high performance computing is provided, which is applied to a system comprising: a plurality of computing nodes connected to a shared memory via an interconnect bus, Figure 1 The method includes six steps from S1 to S6:

[0059] S1: Determine that the computing node is a node in a non-elastic load cluster, and configure the shared memory as a dax device in multiple computing nodes.

[0060] Exemplarily, a high-performance computing system includes N computing nodes, N is greater than 1, and the N computing nodes are nodes in a non-elastic load cluster. All the N computing nodes are connected to a shared memory through an interconnection bus, and the shared memory is configured as a dax device in the N computing nodes through the ndctl command.

[0061] S2: Write a so file shared library. The so file shared library maps the dax device to this process when the program starts.

[0062] Specifically, in an embodiment of the present invention, a so file shared library is written, and when the program starts, the so file shared library is loaded through the LD_PRELOAD command. When the program starts, the so file shared library maps the shared memory dax device to the current process through mmap.

[0063] S3: The process obtains the virtual address of the first address of the dax device, initializes the first area in the shared memory, calls the malloc function in the so file shared library, the process grabs a byte in the first area, and calculates the first offset based on the ID of the grabbed byte.

[0064] Specifically, in an embodiment of the present invention, the dax device of the shared memory is mapped to the present process by mmap, and the process can obtain the virtual address of the first address of the dax device and initialize the first area in the shared memory, where the first area is a specific area in the shared memory. The first area in the shared memory is initialized by writing 1, 2, 3...n per byte, where n=(maximum capacity of the shared memory / 1GiB)-1. Next, when the process uses malloc to apply for memory in the shared memory, the so can intercept the malloc function and call the malloc function of the same name in the so file shared library instead. The process grabs 1 byte in the specific area in the shared memory by CAS, and calculates the first offset according to the id of the grabbed byte, where the first offset is the offset of the grabbed memory.

[0065] S4: Obtain the virtual address of the first address of the shared memory area seized this time in the current process according to the virtual address of the first address mapped by the dax device in the current process and the first offset.

[0066] Specifically, in an embodiment of the present invention, the virtual address of the first address of the shared memory area seized this time in this process is obtained according to the sum of the virtual address of the first address mapped by the dax device in this process and the first offset, and then this memory area can be used separately in this process.

[0067] S5: The computing node writes the computing result into the shared memory area, and obtains the second offset according to the virtual address difference of the data and the virtual address of the first address of the dax device.

[0068] Specifically, in an embodiment of the present invention, since multiple computing nodes are connected through an interconnection bus, some calculation results calculated by the computing nodes, such as intermediate data, output data, status information that needs to be interacted with, etc., can be directly written into the shared memory area applied for by this thread, and then the second offset is obtained based on the difference between the virtual address of the data and the virtual address of the first address of the dax device.

[0069] S6: When the computing node obtains data, it obtains the virtual address of the usable data block according to the second offset and the virtual address of the first address of the dax device, and performs data interaction in the manner of reading and writing memory.

[0070] Specifically, in an embodiment of the present invention, when other computing nodes need to obtain these data, the virtual address of the usable data block can be obtained by adding the second offset to the virtual address of the first address of the dax device, and data interaction can be performed by reading and writing memory.

[0071] Figure 2 A method for improving high performance computing provided by an embodiment of the present invention Figure 2.

[0072] Furthermore, the above method for improving high performance computing further includes two steps S21 to S22:

[0073] S21: Determine that the computing node is a node in the elastic load balancing cluster, and use a memory hot and cold data tiering algorithm to place the common data of the computing node into the shared memory applied for by the computing node.

[0074] S22: When the load increases to be greater than a first preset threshold, some computing nodes are scheduled to participate in the calculation, and the newly awakened node directly reads data in the shared memory of other nodes except the newly awakened node.

[0075] Specifically, for the computing nodes in the elastic load cluster, when the load increases to be greater than the first preset threshold and the amount of computing increases and more computing nodes need to be scheduled and started to participate in the calculation, the newly awakened nodes can directly read the data in the shared memory of other nodes, and there is no need to re-perform the initial calculation.

[0076] Figure 3 A method for improving high performance computing provided by an embodiment of the present invention Figure 3 .

[0077] Furthermore, the above method for improving high performance computing is combined with Figure 3 , and also includes five steps from S31 to S35:

[0078] S31: Determine that the computing node is a node in the elastic load balancing cluster, and use a memory hot and cold data tiering algorithm to place the common data of the computing node into the shared memory applied for by the computing node.

[0079] S32: When the load is reduced to less than a second preset threshold, some computing nodes are put into sleep mode, and data that may be used later by the sleeping computing nodes but is currently stored in the shared memory is compressed.

[0080] S33: occupying a memory area in the second area of ​​the shared memory, placing the compressed data in the occupied memory area as backup data, and recording the released shared memory as a temporary use area.

[0081] S34: When other computing nodes that are still running cannot apply for memory, they can use the temporary usage area. Later, the dormant computing nodes are woken up again to check whether the temporary usage area is occupied.

[0082] S35: If it is occupied, re-occupy the dedicated memory, copy the backup data back from the memory area, decompress it and continue to use it. If the temporary use area is not occupied, use it directly and release the backup data in the memory area.

[0083] Specifically, in an embodiment of the present invention, for computing nodes in an elastic load cluster, when the load is reduced to a second preset threshold, some computing nodes are put into sleep mode, and some computing nodes are put into sleep mode. The data that may be used later by the sleeping computing nodes but is now placed in the shared memory is compressed through a suitable algorithm, and a memory area is seized in the second area in the shared memory, with 200M as a unit. Then, the data is placed in the memory area as backup data, and the released shared memory is recorded as a temporary use area. This area can be used when other computing nodes that are still running cannot apply for memory. Subsequently, this node is re-awakened to check whether the temporary use area used before is occupied. If it is occupied, the exclusive memory is seized again, and the backed-up compressed data is copied back from the memory area for decompression and continued use. If the previous temporary use area is not used, it is used directly to release the backup data in the memory area.

[0084] Further, in the above method for improving high performance computing, configuring the shared memory as a dax device in the plurality of computing nodes is configured through the ndctl command;

[0085] The so file shared library maps the dax device to this process through mmap when the program starts;

[0086] Initialize the first area in the shared memory by writing 1, 2, 3...n per byte, where n = (maximum capacity of shared memory / 1GiB) - 1;

[0087] The process grabs a byte in the first area by CAS.

[0088] Furthermore, in the above method for improving high performance computing, the computing result at least includes: intermediate data, output data, and status information that needs to be interacted with.

[0089] Furthermore, in the above method for improving high performance computing, the second offset is obtained according to the virtual address difference of the data and the virtual address of the first address of the dax device, which is obtained by subtracting the virtual address difference of the data from the virtual address of the first address of the dax device.

[0090] Specifically, in the embodiment of the present invention, a difference value obtained by subtracting the virtual address difference value of the data from the virtual address of the first address of the dax device is the second offset.

[0091] Furthermore, in the above method for improving high performance computing, the virtual address of the usable data block is obtained according to the second offset and the virtual address of the first address of the dax device by summing the second offset and the virtual address of the first address of the dax device.

[0092] Specifically, in the embodiment of the present invention, the virtual address of the usable data block can be obtained by adding the second offset to the virtual address of the first address of the dax device on the computing node.

[0093] On the local server, / pmem+qemu is used to simulate the CXL3.0 shared memory environment. Three virtual machines are started on the server as three computing nodes of the high-performance computing cluster. The three virtual machines have only 100M local memory. In each virtual machine, the / pmem on the host machine is converted to a dax device size of 8G through the ndctl command. We prepare some subset data related to weather simulation for the high-performance computing cluster to parse. According to the logic described above, a libnav.so shared library is written. When the relevant computing process is started, the shared library we specially wrote is loaded into the process by adding LD_PRELOAD="libnav.so". In this way, we can perform some memory-related initialization operations, and the cluster can run normally. We check the memory usage of the three computing nodes and find that the actual usage of the processes on the three virtual machines exceeds 2G, which is requested from the shared memory. We reduce one computing node, and the other two nodes can normally seize the memory released by this node. When a node is added again, that node can also read data from the memory of other nodes normally and run normally.

[0094] Device Embodiment

[0095] Figure 4 A diagram of a device for improving high performance computing provided by an embodiment of the present invention.

[0096] In a second aspect of the present invention, a device for improving high performance computing is provided, which is applied to a system comprising: a plurality of computing nodes connected to a shared memory via an interconnection bus, Figure 4 , the device comprises:

[0097] Configuration module 41: used for determining that the computing node is a node in a non-elastic load cluster, and configuring the shared memory as a dax device in the multiple computing nodes.

[0098] Exemplarily, a high-performance computing system includes N computing nodes, N is greater than 1, and the N computing nodes are nodes in a non-elastic load cluster. All N computing nodes are connected to a shared memory through an interconnection bus. The configuration module 41 configures this shared memory as a dax device in the N computing nodes through the ndctl command.

[0099] Mapping module 42: used to write a so file shared library, which maps the dax device to the current process when the program starts.

[0100] Specifically, in the embodiment of the present invention, a so file shared library is written, and when the program starts, the so file shared library is loaded through the LD_PRELOAD command. When the program starts, the so file shared library mapping module 42 maps the shared memory dax device to the current process through mmap.

[0101] Calculation module 43: used for the process to obtain the virtual address of the first address of the dax device, initialize the first area in the shared memory, call the malloc function in the so file shared library, the process grabs a byte in the first area, and calculates the first offset according to the ID of the grabbed byte.

[0102] Specifically, in an embodiment of the present invention, the dax device of the shared memory is mapped to the present process by mmap, and the process can obtain the virtual address of the first address of the dax device and initialize the first area in the shared memory, where the first area is a specific area in the shared memory. The first area in the shared memory is initialized by writing 1, 2, 3...n per byte, where n=(maximum capacity of the shared memory / 1GiB)-1. Next, when the process uses malloc to apply for memory in the shared memory, the so can intercept the malloc function and call the malloc function of the same name in the so file shared library instead. The process grabs 1 byte in the specific area in the shared memory by CAS, and the calculation module 43 calculates the first offset according to the id of the grabbed byte, where the first offset is the offset of the grabbed memory.

[0103] The first obtaining module 44 is used to obtain the virtual address of the first address of the shared memory area seized this time in the current process according to the virtual address of the first address mapped by the dax device in the current process and the first offset.

[0104] Specifically, in the embodiment of the present invention, the first obtaining module 44 obtains the virtual address of the first address of the shared memory area preempted this time in the current process according to the sum of the virtual address of the first address mapped by the dax device in the current process and the first offset, and then the memory area can be used separately in the current process.

[0105] The second obtaining module 45 is used for the computing node to write the calculation result into the shared memory area, and obtain the second offset according to the virtual address difference of the data and the virtual address of the first address of the dax device.

[0106] Specifically, in the embodiment of the present invention, since multiple computing nodes are connected through an interconnection bus, some calculation results calculated by the computing nodes, such as intermediate data, output data, status information that needs to be interacted with, etc., can be directly written into the shared memory area applied for by this thread, and then the second obtaining module 45 obtains the second offset based on the difference between the virtual address of the data and the virtual address of the first address of the dax device.

[0107] The third obtaining module 46 is used to obtain the virtual address of the usable data block according to the second offset and the virtual address of the first address of the dax device when the computing node obtains data, and performs data interaction in the manner of reading and writing memory.

[0108] Specifically, in the embodiment of the present invention, when other computing nodes need to obtain these data, the third obtaining module 46 can obtain the virtual address of the usable data block by adding the virtual address of the first address of the dax device to the second offset, and can perform data interaction by reading and writing memory.

[0109] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor and a memory;

[0110] The processor is used to execute any one of the methods for improving high performance computing as described above by calling the program or instruction stored in the memory.

[0111] In a fourth aspect of the present invention, a computer-readable storage medium is further proposed, wherein the computer-readable storage medium stores a program or instruction, wherein the program or instruction enables a computer to execute a method for improving high-performance computing as described in any one of the above items.

[0112] Figure 5 It is a schematic block diagram of an electronic device provided by an embodiment of the present invention.

[0113] like Figure 5 As shown, the electronic device includes: at least one processor 501, at least one memory 502 and at least one communication interface 503. The various components in the electronic device are coupled together through a bus system 504. The communication interface 503 is used to transmit information between external devices. It can be understood that the bus system 504 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 504 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 5 Various buses are labeled as bus system 504 .

[0114] It can be understood that the memory 502 in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.

[0115] In some implementations, the memory 502 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system and application programs.

[0116] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. A program that implements any method of a method for improving high-performance computing provided in an embodiment of the present invention can be included in the application.

[0117] In an embodiment of the present invention, the processor 501 calls a program or instruction stored in the memory 502, specifically, a program or instruction stored in an application program, and the processor 501 is used to execute the steps of each embodiment of a method for improving high-performance computing provided in an embodiment of the present invention.

[0118] Determine that the computing nodes are nodes in the non-elastic load cluster, and configure the shared memory as a dax device in multiple computing nodes;

[0119] Write a so file shared library, which maps the dax device to this process when the program starts;

[0120] The process obtains the virtual address of the first address of the dax device, initializes the first area in the shared memory, calls the malloc function in the so file shared library, the process grabs a byte in the first area, and calculates the first offset based on the ID of the grabbed byte;

[0121] According to the virtual address of the first address mapped by the dax device in the current process and the first offset, the virtual address of the first address of the shared memory area seized this time in the current process is obtained;

[0122] The computing node writes the computing result into the shared memory area, and obtains the second offset according to the virtual address difference of the data and the virtual address of the first address of the dax device;

[0123] When the computing node obtains data, it obtains the virtual address of the usable data block according to the second offset and the virtual address of the first address of the dax device, and performs data interaction in the manner of reading and writing memory.

[0124] Any method in a method for improving high performance computing provided in an embodiment of the present invention can be applied to the processor 501, or implemented by the processor 501. The processor 501 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 501 or an instruction in the form of software. The above processor 501 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0125] The steps of any method in a method for improving high performance computing provided by an embodiment of the present invention can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software units in the decoding processor. The software unit can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 502, and the processor 501 reads the information in the memory 502 and completes the steps of the method in combination with its hardware.

[0126] Those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present invention and to form different embodiments.

[0127] Those skilled in the art will appreciate that the description of each embodiment has its own emphasis, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0128] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope defined by the appended claims. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present invention, and these modifications or substitutions should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

[0129] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for improving high performance computing, characterized in that: Applied to a system including: a plurality of computing nodes connected to a shared memory via an interconnection bus, the method includes: Determine that the computing nodes are nodes in the non-elastic load cluster, and configure the shared memory as a dax device in multiple computing nodes; Write a so file shared library, which maps the dax device to this process when the program starts; The process obtains the virtual address of the first address of the dax device, initializes the first area in the shared memory, calls the malloc function in the so file shared library, the process grabs a byte in the first area, and calculates the first offset based on the ID of the grabbed byte; Obtaining the virtual address of the first address of the shared memory area seized this time in the current process according to the virtual address of the first address of the dax device and the first offset; The computing node writes the computing result into the shared memory area, and obtains the second offset according to the virtual address difference of the data and the virtual address of the first address of the dax device; When the computing node acquires data, the virtual address of the usable data block is obtained according to the second offset and the virtual address of the first address of the dax device, and data interaction is performed in a manner of reading and writing memory; Determine that the computing node is a node in the elastic load balancing cluster, and use the hot and cold memory tiering algorithm to place the common data of the computing node into the shared memory applied for by the computing node; When the load increases to be greater than a first preset threshold, some computing nodes are scheduled to participate in the calculation, and the newly awakened nodes directly read the data in the shared memory of other nodes except the newly awakened nodes; Determine that the computing node is a node in the elastic load balancing cluster, and use the hot and cold memory tiering algorithm to place the common data of the computing node into the shared memory applied for by the computing node; When the load is reduced to less than a second preset threshold, some computing nodes are put into sleep mode, and data that may be used later by the sleeping computing nodes but is currently stored in the shared memory is compressed; The second area in the shared memory seizes the memory area, and the compressed data is placed in the seized memory area as backup data, and the released shared memory is recorded as a temporary use area; When other computing nodes that are still running cannot apply for memory, they can use the temporary area. Then wake up the dormant computing nodes again and check whether the temporary area is occupied. If it is occupied, the dedicated memory is re-occupied, and the backup data is copied back from the memory area for decompression and continued use. If the temporary use area is not occupied, it is used directly to release the backup data in the memory area.

2. A method for improving high performance computing according to claim 1, characterized in that: The configuration of the shared memory as a dax device in the multiple computing nodes is configured by the ndctl command; The so file shared library maps the dax device to the current process by mmap when the program starts; Initialize the first area in the shared memory by writing 1, 2, 3...n per byte, where n = (maximum capacity of shared memory / 1GiB) - 1; The process grabs a byte in the first area by CAS.

3. A method for improving high performance computing according to claim 1, characterized in that: The calculation result includes at least: intermediate data, output data and status information that needs to be interacted with.

4. A method for improving high performance computing according to claim 1, characterized in that: The second offset is obtained according to the virtual address difference of the data and the virtual address of the first address of the dax device by subtracting the virtual address difference of the data from the virtual address of the first address of the dax device.

5. The method for improving high performance computing according to claim 1, characterized in that: The virtual address of the usable data block is obtained according to the second offset and the virtual address of the first address of the dax device by summing the second offset and the virtual address of the first address of the dax device.

6. A device for improving high performance computing, characterized in that: Applicable to a system including: a plurality of computing nodes connected to a shared memory via an interconnection bus, the device includes: Configuration module: used to determine whether a computing node is a node in a non-elastic load cluster and configure shared memory as a dax device in multiple computing nodes; Mapping module: used to write the so file shared library, which maps the dax device to the current process when the program starts; Calculation module: used by the process to obtain the virtual address of the first address of the dax device, initialize the first area in the shared memory, call the malloc function in the so file shared library, the process grabs a byte in the first area, and calculates the first offset according to the ID of the grabbed byte; A first obtaining module is used to obtain the virtual address of the first address of the shared memory area seized this time in the current process according to the virtual address of the first address of the dax device and the first offset; The second obtaining module is used for the computing node to write the calculation result into the shared memory area, and obtain the second offset according to the virtual address difference of the data and the virtual address of the first address of the dax device; The third obtaining module is used for obtaining the virtual address of the usable data block according to the second offset and the virtual address of the first address of the dax device when the computing node obtains data, and performs data interaction in a manner of reading and writing memory; The device is also used to: determine that the computing node is a node in the elastic load cluster, and use a memory hot and cold data tiering algorithm to place the common data of the computing node into the shared memory applied for by the computing node; When the load increases to be greater than a first preset threshold, some computing nodes are scheduled to participate in the calculation, and the newly awakened nodes directly read the data in the shared memory of other nodes except the newly awakened nodes; Determine that the computing node is a node in the elastic load balancing cluster, and use the hot and cold memory tiering algorithm to place the common data of the computing node into the shared memory applied for by the computing node; When the load is reduced to less than a second preset threshold, some computing nodes are put into sleep mode, and data that may be used later by the sleeping computing nodes but is currently stored in the shared memory is compressed; The second area in the shared memory seizes the memory area, and the compressed data is placed in the seized memory area as backup data, and the released shared memory is recorded as a temporary use area; When other computing nodes that are still running cannot apply for memory, they can use the temporary area. Then wake up the dormant computing nodes again and check whether the temporary area is occupied. If it is occupied, the dedicated memory is re-occupied, and the backup data is copied back from the memory area for decompression and continued use. If the temporary use area is not occupied, it is used directly to release the backup data in the memory area.

7. An electronic device, characterized in that: include: Processor and memory; The processor is used to execute a method for improving high-performance computing as described in any one of claims 1 to 5 by calling the program or instruction stored in the memory.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute a method for improving high-performance computing as described in any one of claims 1 to 5.