Memory system, data processing method and device, storage medium and program product

Abstract

The invention discloses a memory system, a data processing method and device, a storage medium and a program product, and relates to the technical field of storage. The method comprises the steps of determining a plurality of data writing modes for writing to-be-stored data into a far-end memory according to a verification data determination mode and / or a writing condition corresponding to the to-be-stored data. And determining the storage cost of each data writing mode according to the storage parameter information of the to-be-stored data corresponding to the target storage space of the far-end memory system, the read-write operation frequency and the read-write cost of each data writing mode. And storing the to-be-stored data to the far-end memory system by using the target data writing mode with the lowest storage cost, so that the problems that the related technology is limited by the distance between the memory node and the computing node and the optimal memory performance under different conditions cannot be ensured can be solved, and high-reliability storage and read-write processing of the far-end memory data can be realized.

Description

Technical Field

[0001] The present invention relates to the field of storage technologies, and in particular, to a data processing method, an electronic device, a computer-readable storage medium, a computer program product, and a remote memory system. Background Art

[0002] Remote memory expansion technology allows a computer system to utilize remotely connected memory resources. To achieve highly reliable remote memory data storage and read / write, related technologies selectively protect a subset of the memory space, and implement memory protection at the cache page level through batch write-back and set associative mapping. However, this method has requirements for the distance between the memory node and the computing node, and the set associative mapping optimization method cannot guarantee optimal memory performance in different scenarios. Summary of the Invention

[0003] The present invention provides a data processing method, an electronic device, a computer-readable storage medium, a computer program product, and a remote memory system, which can achieve highly reliable remote memory data storage and read / write, and ensure optimal memory performance in different scenarios.

[0004] To solve the above technical problems, the present invention provides the following technical solutions: The present invention provides a data processing method, including: Determining multiple data writing methods for storing the data to be stored in the remote memory system according to the check data determination method corresponding to the data to be stored and / or the writing condition for writing the data to be stored into the remote memory system; respectively counting the number of read / write operations of each data writing method, and determining the storage cost of each data writing method according to the storage parameter information of the target storage space corresponding to the data to be stored in the remote memory system, the number of read / write operations of each data writing method, and the corresponding read / write cost; using the target data writing method with the lowest storage cost to store the data to be stored in the remote memory system; wherein, at least one of the check data determination method and the writing condition of different data writing methods is different, and the storage parameter information at least includes the storage capacity of the target storage space and the amount of data associated with the data to be stored.

[0005] The present invention also provides an electronic device, including a memory and a processor, and the processor is configured to implement the steps of any one of the above data processing methods when executing a computer program stored in the memory.

[0006] The present invention also provides a computer-readable storage medium, on which a computer program is stored, and the computer program is configured to implement the steps of any one of the above data processing methods when executed by a processor.

[0007] The present invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of any of the above data processing methods.

[0008] Finally, the present invention also provides a remote memory system, which at least includes a remote memory, a buffer, a read-write processor, and a program memory; the memory is connected to the buffer, and the read-write processor is connected to the buffer and the program memory; the remote memory at least includes one stripe, and the stripe is used to store corresponding data to be stored; the buffer serves as the memory space of the read-write processor, including a read operation buffer space and / or a write operation buffer space, and the occupied capacities of the read operation buffer space and the write operation buffer space are determined according to the current application requirements and load characteristics; the read-write processor, when executing the computer program stored in the program memory, implements the steps of any of the above data processing methods.

[0009] The advantages of the technical solution provided by the present invention are as follows: for the data to be stored in the remote memory system, the remote memory system replaces the computing node to complete the verification calculation, write cost evaluation, and write method selection of the data to be written. In this way, the read-write performance and memory performance of the remote memory system will not be limited by the distance between the computing node and the remote memory system, so as to realize remote memory expansion in any distance scenario. From the two aspects of verification data calculation and write timing, the influence of the storage situation and service operation situation of the remote memory system in different scenarios on the write method is reflected. By evaluating the costs of different write methods, the data write method that best matches the current storage performance can be determined, so as to ensure that the memory performance of the remote memory system reaches the optimal level in different scenarios, and the high-reliability remote memory can realize efficient data storage and read-write functions. In addition, the present invention also provides corresponding implementation electronic devices, computer-readable storage media, computer program products, and remote memory systems for the data processing method, further making the method more practical, and the electronic devices, computer-readable storage media, computer program products, and remote memory systems have corresponding advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the present invention or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0011] Figure 1 It is a schematic diagram of the hardware composition framework applicable to the data processing method provided by the present invention; Figure 2Flow schematic diagram of a data processing method provided by the present invention; Figure 3 Data interaction schematic diagram corresponding to various data writing methods provided by the present invention; Figure 4 Structural framework diagram under an exemplary embodiment of a data processing device provided by the present invention; Figure 5 Structural framework diagram of an exemplary embodiment of a remote memory system provided by the present invention; Figure 6 Data interaction schematic diagram between a remote memory system and a computing node under an exemplary scenario provided by the present invention. Detailed implementation manners

[0012] In order to enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation manners. Among them, the terms "first", "second", "third", "fourth", etc. in the specification and the above drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. The term "exemplary" means "serving as an example, embodiment, or illustration". Any embodiment described herein as "exemplary" should not be construed as being superior to or better than other embodiments.

[0013] With the rapid development of big data analysis, HPC (High Performance Computing), and AI (Artificial Intelligence) technologies, memory-intensive applications have increasingly high requirements for the memory capacity, memory access performance, and flexibility of computer systems. In order to support the needs of large-scale data processing and computing tasks and improve the overall performance of computer systems, a memory system based on remote memory expansion technology has replaced the traditional memory system and is widely used in memory-intensive applications.

[0014] Remote memory expansion technology can dynamically adjust memory resources according to application requirements through memory allocation technology, avoiding a large one-time investment in local memory. It not only achieves flexible resource management but also better controls costs. In addition, remote memory expansion can share data among multiple computing units and maintain data consistency, which reasonably improves the data availability of the computing system. Although remote memory expansion technology can increase memory capacity and improve the performance of computer systems, since this technology uses the memory resources of multiple storage nodes or computing nodes based on network connections, once there is a network failure or a node failure, the corresponding memory resources cannot be accessed, affecting the stability and reliability of the entire computer system. Secondly, remote memory access has higher latency and lower bandwidth, and is not suitable for application scenarios with high requirements for latency and bandwidth, such as memory-sensitive applications. In addition, multiple computing nodes simultaneously reading and writing remote memory is very likely to cause faults to spread more widely within the computer system, which means that a small number of memory faults may lead to errors or crashes of a large amount of computing resources.

[0015] To address the above risks and achieve highly reliable remote memory data storage and high-performance remote memory reading and writing, related technologies usually adopt remote memory fault tolerance strategies, which selectively protect a subset of the memory space and achieve memory protection at the cache page level through batch write-back and set-associative mapping. Among them, set-associative mapping includes memory interleaved mapping and cache line interleaved mapping. However, this method involves data exchange between the memory and the last-level cache of the CPU (Central Processing Unit), has requirements for the distance between the memory node and the computing node, and is not suitable for remote memory expansion scenarios where the two are far apart. In addition, various set-associative mapping optimization methods cannot guarantee that the memory performance is optimal in different situations.

[0016] In view of this, the present invention builds a processing unit with a check calculation function, a write cost evaluation function, and a write method selection function in the remote memory system. For the data to be stored that needs to be stored thereon, multiple data write methods for writing it to the remote memory are determined according to the check data determination method and / or the write timing corresponding to the data to be stored. According to the storage parameter information of the target storage space of the data to be stored in the remote memory system, the number of read and write operations and the read and write costs of each data write method, the storage costs of each data write method are determined. The data to be stored is stored in the remote memory system using the target data write method with the lowest storage cost, thereby ensuring that the memory performance of the remote memory system reaches the optimal level in different scenarios and enabling highly reliable and efficient data storage and read and write functions for remote memory. Combining with the specific application environment architecture or specific hardware architecture on which the execution of the data processing method depends, the specific application environment architecture or specific hardware architecture is described herein. Next, in combination with Figure 1Some possible application scenarios related to the technical solution of the present invention are introduced by way of example, which may include the following: The server 11 is connected to the remote memory system 12 through a network. The remote memory system 12 at least includes a processing unit 121 and a memory 122. The memory 122 serves as the remote memory space of the remote memory system 12 and provides remote memory resources to the server 11, so that the server 11 can use the remote memory resources of the remote memory system 12 through the network. When the server 11 needs to store data in the remote memory system 12, it can send a data write request to the remote memory system 12. The data write request carries the data to be stored or data storage location information. The remote memory system 12 receives the data write request and reads the data to be stored. The processing unit 121 determines various data writing methods for storing the data to be stored in the remote memory system according to the check data determination method corresponding to the data to be stored and / or the writing timing of the data to be stored written into the remote memory system 12; respectively counts the read and write operation times of each data writing method, and determines the storage cost of each data writing method according to the storage parameter information of the target storage space of the data to be stored in the remote memory system, the read and write operation times of each data writing method, and the corresponding read and write costs; finally, uses the target data writing method with the lowest storage cost to store the data to be stored in the remote memory system, which can achieve efficient data writing while ensuring the high reliability of the remote memory system 12, and further realize the efficient data storage and read and write functions of the high-reliability remote memory.

[0017] It should be noted that the above application scenarios are only shown for the convenience of understanding the ideas and principles of the present invention, and the embodiments of the present invention are not limited in this regard. On the contrary, the embodiments of the present invention can be applied to any applicable scenario. After introducing the technical solution of the present invention, the various non-limiting embodiments of the present invention will be described in detail below with reference to the drawings and specific embodiments. First, please refer to Figure 2 , the data processing method provided in this embodiment may include the following content: S201: Determine various data writing methods for storing the data to be stored in the remote memory system according to the check data determination method corresponding to the data to be stored and / or the writing condition for writing the data to be stored into the remote memory system.

[0018] Among them, the data to be stored is the data that needs to be stored in the remote memory by a computing node that performs remote memory expansion through a network. The computing node can be, for example, a general-purpose server, an artificial intelligence server, various types of host ends and user ends, etc. When writing the data to be stored into the remote memory system, since the data to be stored is associated with corresponding check data, in order to ensure the consistency of the data stored in the remote memory space of the remote memory system and its check data, each write not only needs to update the data to be written, that is, the data to be stored, but also needs to update the corresponding check data. There are many calculation methods for check data, and different calculation methods for check data have different impacts on write performance and data reliability. The check data determination method is the method for calculating check data, and the write condition refers to the write timing for writing the data to be stored back to the remote memory system. For example, whenever the data to be stored is received, it is immediately written to the corresponding storage location, or a certain specific condition is reached and it is written to the remote memory space together with other data. A data write method includes at least one of a check data determination method or a write condition. By arbitrarily combining multiple check data determination methods and multiple write conditions, multiple data write methods can be obtained, that is, the check data determination methods and write conditions of different data write methods are at least one different.

[0019] S202: Respectively count the number of read and write operations of each data write method, and determine the storage cost of each data write method according to the storage parameter information of the target storage space corresponding to the data to be stored in the remote memory system, the number of read and write operations of each data write method, and the corresponding read and write costs.

[0020] It can be understood that different data writing methods have their own advantages and different impacts on the memory performance of the remote memory system. While ensuring the reliability of the data to be stored, reducing the impact on the writing performance, and ensuring that the memory performance of the remote memory system is the best under various conditions, this step will dynamically select the data writing method that best matches the current conditions from the multiple data writing methods determined in S201 in combination with the storage parameter information of the remote memory system. The matching measurement rule is based on the storage cost or the writing cost. The target storage space refers to the location in the remote memory system used to store the data to be stored. Considering that the remote memory system usually consists of multiple memory devices to provide rich remote memory resources, such as multiple memory modules or multiple memory cards, the target storage space refers to the storage space corresponding to the memory device where the data to be stored is located. Of course, if the remote memory system has only one memory device, the target storage space is the remote memory space of the remote memory system. The storage parameter information includes at least the storage capacity of the target storage space and the amount of data associated with the data to be stored. Associated data such as dirty data or clean data in the same storage page as the data to be stored, and data in other storage pages related to this storage page in the target storage space. The number of read and write operations includes the number of read operations and the number of write operations, that is, from the time the remote memory system receives the data to be stored until the data to be stored is written into the target storage space, the number of data read operations and the number of data write operations involved in different data writing methods. The read and write cost includes the read cost and the write cost, which refers to the resource consumption or cost required when reading or writing data from the remote memory system, measured from multiple dimensions such as time delay, energy consumption, hardware wear, and economic cost (such as the lifespan of the storage medium).

[0021] S203: Use the target data writing method with the lowest storage cost to store the data to be stored in the remote memory system.

[0022] When the storage costs corresponding to different data writing methods are calculated in the previous step, the data writing method with the lowest storage cost is the data writing method that best matches or is the optimal one for the current conditions. This step is defined as the target data writing method. Using the target data writing method to write the data to be stored into the remote memory system can not only ensure the reliability of the data, but also ensure that the data is efficiently written while having the least impact on the storage performance of the remote storage system, that is, it can ensure that the remote storage system can maintain good memory performance throughout the process.

[0023] In the technical solution provided in this embodiment, for the data to be stored that needs to be stored on the remote memory system, the remote memory system replaces the computing node to complete the verification calculation, write cost evaluation, and write method selection of the data to be written. In this way, the read and write performance and memory performance of the remote memory system will not be limited by the distance between the computing node and the remote memory system, thereby realizing remote memory expansion in any distance scenario. From the two aspects of verification data calculation and write timing, the influence of the storage situation and business operation situation of the remote memory system in different scenarios on the write method is reflected. By evaluating the costs of different write methods, the data write method that best matches the current storage performance can be determined, so as to ensure that the memory performance of the remote memory system reaches the optimal level in different scenarios, and the efficient data storage and read and write functions of the highly reliable remote memory can be realized.

[0024] It should be noted that there is no strict order of execution between the steps in the present invention. As long as it conforms to the logical order, these steps can be executed simultaneously or in a certain preset order. Figure 2 This is just a schematic way and does not mean that it can only be in such an execution order.

[0025] In the above embodiment, the calculation method of the verification data is not limited. In order to improve the write efficiency of the data to be written, this embodiment also provides two verification value calculation methods to achieve the efficient writing of the data to be stored, which may include the following content: As an exemplary verification value calculation method, in this method, the remote storage space of the remote storage system is composed of multiple stripes, that is, the data to be stored is stored in the stripes, the target storage space is the stripe, the data to be stored is the dirty data to be written, and the target associated data belonging to the same stripe as the dirty data to be written is obtained; the exclusive OR sum between the dirty data to be written and each target associated data is calculated as the verification value of the dirty data to be written. Among them, the target associated data is the dirty data or clean data located in the same stripe as the dirty data to be written, and the exclusive OR sum refers to the exclusive OR operation between the dirty data to be written and each target associated data. For the convenience of description, the verification data determination method recorded in this embodiment can be defined as the stripe calculation verification value method, that is, the stripe calculation verification value method is to use the dirty data to be written and other data belonging to the same stripe to calculate the exclusive OR sum together as the new verification value and write it into the remote memory system together.

[0026] Another way to calculate the check value parallel to the above embodiments: According to the target memory address to which the dirty data to be written is correspondingly written into the remote memory system, read the old data stored in the target memory address at the current moment and before, and the old check data of the old data from the remote memory system; calculate the difference between the dirty data to be written and the old data, and calculate the exclusive OR value of the difference and the old check data as the check value of the dirty data to be written. Wherein, the target memory address refers to the address corresponding to the storage location of the data to be stored in the remote memory system, that is, the address to be written. The old data refers to the memory data stored before the address to be written. For the convenience of description, the check data determination method described in this embodiment can be defined as the differential calculation check value method, that is, the differential calculation check value method is to use the dirty data to be written, the old memory data at the write position and the corresponding old check value data, calculate the difference at the write position and add it to the old check value to obtain a new check value, and write them into the memory together.

[0027] As can be seen from the above, this embodiment provides two ways to determine the check data, which can efficiently implement the data writing process, ensure the reliability of the data to be stored through the check value, and thus reduce the impact on the writing performance while ensuring the data reliability.

[0028] For the consistency of the written data and the check data, the present invention stores the corresponding check data while storing the data, which makes the memory data writing operation not only need to write the original data, that is, the data to be stored, but also need to update the relevant check data. Therefore, additional check value calculations are required. For the remote memory scenario, the distance between the computing node and the remote memory system is relatively far. The check value calculation, long-distance transmission of the check value, memory write latency and time, etc. at the computing node will all affect the I / O (Input / Output) performance of the remote memory system. Based on this, on the basis of deploying and executing the above check value calculation, cost evaluation and dynamic selection of the writing method in the remote memory system in the above embodiments, in order to further improve the read and write performance of the remote memory, a cache can also be built in the remote memory system near the remote memory node. The cache device provides a cache space, and check value calculation, data prefetching, etc. are performed in the cache space, which may include the following contents: Calculating the check value in the cache space may include: in the stripe calculation check value method, all other relevant data in the stripe where the dirty data to be written in the remote memory system is located may be read into the cache space, and the check value may be calculated together with the dirty data to be written, and then the dirty data and the obtained check value may be written back to the remote storage space of the remote memory system. In the differential calculation check value method, the corresponding old check value of the old data at the address to be written in the remote memory system may be read into the cache space, the difference between the old data at the address to be written and the dirty data to be written may be calculated, and added to the old check value to obtain a new check value, and then the dirty data and the new check value may be written back to the remote storage space of the remote memory system.

[0029] To further improve the flexibility of data writing, based on the embodiments, according to the different timings of the remote storage space of the remote memory system, the writing conditions may include the direct writing method and the group write-back method based on the cache space. For the stripe calculation check value method, the process of the direct writing method may include: for the scenario of storing the dirty data to be written into a pre-constructed cache space; based on the target memory address in the remote memory system where the dirty data to be written is written, in the remote memory system, determine each target data belonging to the same stripe as the dirty data to be written, and read each target data into the cache space. Of course, if the target data is already in the cache space, there is no need to write it again. Calculate the first exclusive OR sum between the dirty data to be written and each target data; write the dirty data to be written to the target memory address, and write the first exclusive OR sum to the first memory check address in the remote memory system. For the scenario where the dirty data to be written is not stored in a pre-constructed cache space, based on the target memory address in the remote memory system where the dirty data to be written is written, determine each target data belonging to the same stripe as the dirty data to be written in the remote memory system, calculate the first exclusive OR sum between the dirty data to be written and each target data; write the dirty data to be written to the target memory address, and write the first exclusive OR sum to the first memory check address in the remote memory system. Among them, the first memory check address refers to the storage location in the remote memory system for storing the first exclusive OR sum, that is, the check value of the data to be stored in the data writing method determined by the stripe calculation check value method and the direct writing method. For example, when a remote memory write request occurs, the dirty data T to be written and all other relevant data in the same stripe, that is, each target data , where n represents the number of target data. If in the form of cache pages, n represents the number of mutually associated data pages in the stripe. Calculate the first exclusive OR sum of these data through the relational expression as the check value P1 of the dirty data to be written. Write the dirty data T to be written to the target memory address, and write the new check value P1 to the first memory check address, completing all steps of the stripe calculation check value method for writing a single dirty data back to the remote memory system.

[0030] For the method of calculating the check value for stripes, the existence of the cache allows the write-back operations of associated addresses in the same stripe to share a single check value update. The process of the group write-back method based on the cache space may include: storing the dirty data to be written into a pre-constructed cache space; when the group write-back condition is met, writing the dirty data to be written to the target memory address in the remote memory system, determining the target data in the remote memory space of the remote memory system that belongs to the same stripe as the dirty data to be written, and reading the target data into the cache space; determining the target dirty data to be written and the target clean data in the cache space that belong to the same stripe as the dirty data to be written according to the target memory address; calculating the second exclusive OR sum among the dirty data to be written, the target dirty data to be written, and the target clean data as the check value for the dirty data to be written and the target dirty data to be written; writing the dirty data to be written and the target dirty data to be written to the corresponding memory addresses in the remote memory system respectively, and writing the second target exclusive OR sum to the second memory check address in the remote memory system. Among them, the second memory check address refers to the storage location in the remote memory system for storing the second exclusive OR sum, that is, the check value of the data to be stored and each target dirty data to be written under the data writing method determined by the method of calculating the check value for stripes and the group write-back method. The group write-back condition may include, for example, passing through a preset t clock cycles or the occupancy of the cache space allocated for the write operation reaching or exceeding a preset ratio. For example, the dirty data to be written is stored in the cache close to the remote memory, and wait until the following preset conditions are met: i. Do not wait and directly proceed to the next step (this method is the direct write-back method); ii. Pass through a preset t clock cycles; iii. The occupancy of the cache space allocated for the write operation reaches or exceeds a preset ratio. Then, according to the target memory address of the dirty data to be written, find the dirty data to be written and the clean data pre-read into the cache at other addresses that belong to the same stripe in the cache space. Then find the other addresses that belong to the same stripe and are not in the cache space, and read the data at these addresses from the remote memory system and store them in the cache space. Finally, according to the dirty data to be written, each target dirty data to be written and each target clean data , where d is the number of other dirty data in the cache space that belong to the same stripe as the dirty data to be written, that is, the total number of target dirty data to be written, and n represents the number of mutually associated data pages in the stripe. Based on the relational expression calculate the second exclusive OR sum of these data as the check value P2 for the dirty data to be written and each target dirty data to be written. Write the dirty data to be written and each target dirty data to be written Write to the corresponding addresses respectively, and write P2 to the second memory check address. In addition, the above write operation does not necessarily always wait until the group write-back condition is satisfied to trigger subsequent steps. It may also be that the corresponding group write-back condition is initiated due to the dirty data write operations of other related addresses, resulting in the write operation of the dirty data to be written being completed in advance.

[0031] As can be seen from the above, in this embodiment, by combining the cache space and two writing opportunities, while achieving efficient data writing through the method of calculating check values by stripes, the flexibility and practicality are improved through various writing methods, and the efficiency of the write operation of the remote memory system is effectively improved through the group write-back method, which is beneficial to realizing the efficient data storage and read-write functions of a highly reliable remote memory, and can be directly applied to various computing system scenarios with memory expansion requirements and high reliability requirements.

[0032] Similarly, for the differential calculation check value method, the write conditions may include the direct write method and the group write-back method based on the cache space. That is, the differential calculation check value method + the direct write method constitutes a data write method, and the differential calculation check value method + the group write-back method based on the cache space constitutes a data write method. Among them, for the differential calculation check value method + the direct write method constituting a data write method: for the scenario of storing the dirty data to be written into the pre-constructed cache space, based on the target memory address where the dirty data to be written is written into the remote memory system, in the remote memory system, read the old data and the corresponding check data of the target memory address, and read the old data and the corresponding check data into the cache space. Of course, if the old data and the corresponding check data are already in the cache space, there is no need to write again. Calculate the difference between the dirty data T to be written and the old data and use the exclusive OR value of the difference and the old check value as the new check value P3. For example, the new check value can be calculated through Finally, write the dirty data to be written to the target memory address, and write P3 to the third memory check address of the remote memory system to complete the steps of the differential calculation check value method for writing back a single dirty data to the remote memory system.

[0033] For the data writing method composed of the differential calculation check value method + the group write-back method based on the cache space: store the dirty data to be written into the pre-constructed cache space; when the group write-back condition is met, write the dirty data to be written into the first target memory address of the remote memory system, and determine in the cache space the target dirty data to be written that belongs to the same stripe as the dirty data to be written, and the second target memory address corresponding to the target dirty data to be written into the remote memory system; read the old data stored in the first target memory address and the second target memory address at the current moment and before in the remote memory space of the remote memory system, and the corresponding old check data; calculate the difference between the dirty data to be written and the corresponding old data, the difference between the target dirty data to be written and the corresponding old data, and the third exclusive OR sum of the old check data as the check value of the dirty data to be written and the target dirty data to be written; write the dirty data to be written into the first target memory address, write the target dirty data to be written into the second target memory address, and write the third exclusive OR sum into the corresponding fourth memory check address of the remote memory system.

[0034] For example, when a remote memory write request occurs, the write-back operations of the associated addresses in the same stripe share one check value update. To improve the write operation efficiency and reduce the write cost, the steps of the differential calculation check value method for the write-back operations of the associated addresses in the same stripe to the remote memory system include: A1. Store the dirty data to be written into the cache space of the cache device close to the remote memory. A2. Wait until one of the following pre-set conditions is met: after a pre-set t clock cycles; or the occupancy of the cache space allocated for the write operation reaches or exceeds the pre-set ratio. A3. According to the first target memory address of the dirty data to be written, find each target dirty data to be written of the related other addresses belonging to the same stripe in the cache space. . A4. Read from the remote memory system the dirty data T to be written and each target dirty data to be written , the old data in the corresponding memory addresses , and read the old check value data of the corresponding check address , and store them in the high cache space. A5. According to the dirty data to be written, each target dirty data to be written and the corresponding old data , use the relational expression to calculate the new check value P4 corresponding to the dirty data to be written and each target dirty data to be written, where d represents the total number of target dirty data to be written. Finally, write the target dirty data to be written and each target dirty data to be written Write them to the corresponding addresses in the remote memory system respectively, and write the new check value P4 to the corresponding fourth memory check address. Among them, the first memory check address, the second memory check address, the third memory check address, and the fourth memory check address are used to distinguish the storage locations of the check values calculated under different scenarios. For the actual situation, when the data to be stored is written to the remote memory system, only the check value corresponding to the optimal data writing method is stored. Similarly, in A2, the write operation does not necessarily wait until the condition is met to trigger subsequent steps. It may also be that the subsequent A4 is initiated due to the write operation of the dirty data to be written at other related addresses, resulting in the premature completion of the current write operation of the dirty data to be written.

[0035] Under different data writing methods, the data interaction between the cache space and the remote memory space can be as Figure 3 shown. In Figure 3 , for the convenience of description, the data writing method composed of the strip calculation check value method + the direct writing method is defined as Policy A, the data writing method composed of the strip calculation check value method + the group write-back method is defined as Policy B, the data writing method composed of the differential calculation check value method + the direct writing method is defined as Policy C, and the data writing method composed of the differential calculation check value method + the group write-back method is defined as Policy D. The first row of data represents all the data in the same strip. T represents the dirty data to be written, D represents other dirty data in the cache space, that is, the target dirty data to be written, C represents the clean data in the cache space, the dotted line represents the data not in the cache space, P represents the strip check data. The upward arrow in each policy represents the data read from the remote memory system to the cache space, and the downward arrow represents the data written from the cache space to the remote memory space of the remote memory system.

[0036] As can be seen from the above, in this embodiment, by combining the cache space and two writing timings, while achieving efficient data writing through the strip calculation check value method, the flexibility and practicality are improved through multiple writing methods, and the efficiency of the write operation of the remote memory system is effectively improved through the group write-back method, which is beneficial to realizing the efficient data storage and read / write functions of the highly reliable remote memory, and can be directly applied to various computing system scenarios with memory expansion requirements and high reliability requirements.

[0037] Furthermore, in order to improve the effective utilization rate of the cache space and further enhance the data read / write efficiency, based on the above embodiments, the cache space provided by the cache can be divided into a read operation cache space and a write operation cache space. The read operation cache space is used to support the prefetching of the remote memory system. For example, other cache lines of the cache page to which the target cache line to be read belongs can be prefetched into the read operation cache space, and other cache pages of the stripe where the cache page to which the target cache line to be read belongs can be prefetched into the read operation cache space. The write operation cache space is used to support the remote memory write-back as described in the above embodiments. For example, the data to be stored can be temporarily stored in the write operation cache space and wait for a fixed clock cycle to complete the write operation of writing the data back to the remote memory space; the data to be stored can be temporarily stored in the write operation cache space, and when the occupancy of the write operation cache space reaches or exceeds a fixed ratio, the write operation of writing the data back to the remote memory space is performed. This embodiment may include the following content:

[0038] For the data reading scenario: Due to spatial locality, subsequent data read requests are more likely to directly hit in the cache space. In order to reduce memory addressing operations and shorten the response time of data read requests, when a data read request is received, determine the target data cache page where the data to be read corresponding to the data read request is located, and determine the target stripe to which the target data cache page belongs; prefetch and store the target data cache page and other data cache pages in the target stripe except the target data cache page into the pre-constructed read operation cache space. In this embodiment, when there is a data read request to the remote memory system, data in other memory spaces with high spatial locality with the target cache line space can be prefetched into the cache. For example, cache lines belonging to the same page and other pages belonging to the same stripe can not only reduce memory addressing operations, effectively shorten the response time of data read requests, and thus improve the data reading speed of the remote memory system, but also, even considering eviction in the cache, the probability of data belonging to the same stripe existing in the cache is still relatively high, which is beneficial to quickly calculating the check value during the write operation.

[0039] For the data writing scenario: When a data write request is received, the data to be written corresponding to the data write request is stored in a pre-constructed write operation cache space; when the data writing condition is met, the data in the write operation cache space is written into the remote memory space; wherein, the write operation cache space and the remote memory space belong to the remote memory system. In this embodiment, the remote memory space is the actual storage location of the data to be stored, that is, the memory space of the remote memory system. When there is a data write request to the remote memory system, the data to be written is temporarily stored in the cache. When a specific condition for data writing is met, such as after a certain number of clock cycles or when the amount of data to be written reaches a certain level, the data in the write operation cache space is written into the remote memory space. In this way, for scenarios where the possibility and quantity of dirty data to be written belonging to the same stripe are relatively large, since the operation of updating the check value is shared when writing dirty data of the same stripe, the writing of multiple dirty data can share the overhead cost of check value calculation, thereby reducing the write cost. While improving the data write speed, it can also ensure the high performance of the remote storage system.

[0040] As can be seen from the above, in this embodiment, a cache hardware structure is built near the remote memory node. On the one hand, it can be used as a cache space for completing computing tasks. On the other hand, it can be used as a cache space for writing data into the remote memory system, enhancing the performance of remote memory data writing. On the other hand, it can also be used as a cache space for data prefetching, improving the data reading performance of the remote memory system, which is conducive to realizing the efficient data storage and read / write functions of a highly reliable remote memory system.

[0041] Based on the above embodiments, the remote memory system can provide better memory expansion capabilities and more flexible memory allocation and scheduling capabilities for computing nodes. However, once the remote memory system fails or data is lost, the impact on the computing node will be very serious. In view of this, in this embodiment, according to the principle of erasure code, an independent redundant memory array data storage method is established in the remote memory system, enabling the remote memory system to have the ability to recover or reconstruct data when a hardware failure occurs, improving the reliability of data storage, which may include the following:

[0042] In this embodiment, the remote memory space of the remote memory system is composed of stripes. The remote memory space includes multiple stripes. The target storage space is a stripe. The process of storing the data to be stored into the corresponding stripe includes: dividing the data to be stored into multiple data blocks to be stored, and storing each data block to be stored into the corresponding first memory space; calculating the exclusive-or checksum between the data blocks to be stored, and storing the exclusive-or checksum into the second memory space; combining each first memory space and the second memory space into a stripe for storing the data to be stored. The first memory space can be the storage space provided by any kind of memory device, such as a memory stick or a memory card. Each first memory space is independent of each other and independent of the second memory space. In this way, when there is a failure or data loss in a memory space, such as the memory device where one of the first memory spaces is located or the memory device where the second memory space is located, the faulty data can be recovered through the remaining normal data and the redundant checksum. Taking the first memory space and the second memory space as the memory spaces provided by DIMM (Dual-Inline-Memory-Modules), the data to be stored is divided into several parts, and each part is stored on a DIMM memory stick of the remote memory system. The exclusive-or checksum between the several parts of the stored data is calculated and stored on another independent memory stick, forming an "n + 1" type redundancy. The n + 1 memory sticks are a stripe. Further, the check data of different stripes are evenly staggered in different memory spaces, that is, the check pages of different stripes are evenly staggered in different DIMMs, which can avoid the problem of uneven hardware wear.

[0043] As can be seen from the above, the data storage method of the stripe-based memory redundancy mechanism constructed according to the erasure code principle in this embodiment can be effectively recovered after data loss or errors, improving the data reliability of the remote memory system. In addition, by evenly staggering the check data of different stripes in different memory spaces, the problem of uneven hardware wear caused by different access frequencies of the memory data read and write operations to the data cache space and the check space can be avoided.

[0044] Further, in order to achieve address continuity, the present invention also provides a stripe-based memory redundancy mechanism based on page granularity, enabling the operating system to map continuous virtual addresses to physical data addresses and skip the check page addresses, thereby maintaining the continuity of virtual addresses. It may include the following content:

[0045] Divide the data to be stored into multiple data blocks to be stored, and store each data block to be stored into the corresponding data cache page; calculate the exclusive-or checksum between the data blocks to be stored, and store the exclusive-or checksum into the check page; combine each data cache page and the check page into a stripe for storing the data to be stored; where each data cache page is independent of each other and independent of the check page.

[0046] Since the mapping between physical memory addresses and virtual addresses is in page-sized units, the striping mechanism at the page granularity enables the operating system to map consecutive virtual addresses to physical data addresses and skip the verification page addresses, thus maintaining the continuity of virtual addresses. If striping is implemented at the cache line granularity, then for every n cache lines of virtual addresses, one cache line is required to store verification line information. This cache line cannot be used for writing data, and the continuity of the virtual address space cannot be achieved. In this embodiment, a memory device can provide one data cache page or one verification page. The memory device can be a DIMM, that is, one verification page is constructed for the cache pages of every n different DIMMs, forming an information redundancy of n + 1. The data of the n data cache pages in the same stripe are respectively , and the verification page is obtained through exclusive OR (XOR) operation based on the data of each data cache page, that is, the data P of the verification page can be constructed through . When a DIMM fails, resulting in data errors or loss, the striping memory redundancy mechanism enables the data cache page in this DIMM to be recovered through calculation using other data pages and the verification page in the same stripe, and the verification page can be recalculated through all the data pages in the same stripe. In addition, the striping memory redundancy mechanism and the memory ECC (Error Checking and Correcting) mechanism are complementary memory fault tolerance mechanisms and can coexist. ECC is oriented towards faults such as random bit flips in a single memory and can detect and recover a small number of data bit errors; while the striping memory redundancy mechanism is oriented towards data faults and losses at the cache page level and can recover one or more data pages.

[0047] As can be seen from the above, the striping memory redundancy mechanism of this embodiment is implemented at the cache page level, which can ensure the continuity of virtual addresses. Through "n + 1" memory redundancy, faulty data can be recovered, guaranteeing the high reliability of the remote memory system. The verification pages of different stripes are evenly staggered in different DIMMs, which can avoid the problem of uneven hardware wear caused by different access frequencies of data cache pages and verification pages during memory data read and write operations.

[0048] Based on the above embodiments, where the different data writing methods are evaluated for the writing cost based on the average number of reads and writes required for a single cache line write operation, this embodiment also provides a more convenient dynamic selection method, which may include the following content: Calculate the write cost corresponding to each data write method in advance according to the read operation times and read cost, write operation times and write cost of each data write method, and determine the write cost of the same check data determination method under different write conditions, and determine the target write condition with a low write cost; determine the target check data determination method with a low write cost according to the storage parameter information of the target storage space of the data to be stored in the remote memory system; use the data write method determined by the target check data determination method and the target write condition as the target data write method with the lowest storage cost.

[0049] In this embodiment, the number of data pages associated with the same stripe is n. When the cache space is triggered to write to the remote memory system, the other dirty data in the cache space that is associated with the dirty data to be written and belongs to the same stripe but does not include the dirty data itself that triggers the write mechanism, that is, the number of target dirty data to be written is d, and the number of clean data belonging to the same stripe is c. Then the number of other data in the same stripe that is not in the cache space is Taking the data write method composed of the stripe check value calculation method + direct write method as Policy A, defining the data write method composed of the stripe check value calculation method + group write-back method as Policy B, defining the differential check value calculation method + direct write method as Policy C, and defining the differential check value calculation method + group write-back method as Policy D as examples of these four data write methods, the determination process of the read and write operation times of different data write methods includes: For Policy A, after initiating a write of dirty data to the remote memory system, except for the clean data already in the cache space, read all other data under the same stripe from the remote memory system into the cache space, and then write the dirty data to be written and the updated check data to the remote memory system after completing the check value calculation. Therefore, a total of read operations and 2 write operations are required to achieve the write of 1 dirty data. For Policy B, in the case where neither the existing dirty data nor the clean data in the cache space needs to be read, after initiating a write of dirty data to the remote memory system, it is necessary to read all other data under the same stripe that is not in the cache space from the remote memory system into the cache space, and then write all the dirty data under the same stripe in the cache space and the updated check data to the remote memory system after completing the check value calculation. Therefore, a total of read operations and write operations are required to achieve Writing of one dirty data. For policy C, after initiating a write operation of dirty data to the remote memory system, it is necessary to read the old data on the dirty data bit to be written and the old data of the parity bit corresponding to the same stripe from the remote memory system. After completing the calculation of the check value, the dirty data to be written and the updated check data are written to the remote memory system. Therefore, a total of 2 read operations and 2 write operations are required, realizing the writing of 1 dirty data. For policy D, after initiating a write operation of dirty data to the remote memory system, it is necessary to read the old data on the corresponding bits of all dirty data in the cache space that belong to the same stripe as the dirty data to be written (including the dirty data itself to be written) and the old data of the parity bit corresponding to the same stripe from the remote memory system. After completing the calculation of the check value, all the dirty data in the cache space of this stripe and the updated check data are written to the remote memory system. Therefore, a total of read operations and write operations are required, realizing the writing of dirty data.

[0050] After determining the number of read and write operations for different data writing methods, the read cost corresponding to one read operation from the remote memory system to the cache space is , and the write cost corresponding to one write operation from the cache space to the remote memory system is . The average write costs of policies A to D are respectively expressed as , representing the read and write costs of writing 1 dirty data to the remote memory system on average. The write costs of various policies can be respectively expressed as: ; ; ; .

[0051] Among them, for any , there is , so under the condition that other factors are the same, whether it is the strip calculation check value method or the differential calculation check value method, the write condition of the group write-back method is better than that of the direct write-back method. In this embodiment, the target write condition is the group write-back method. When the target write condition is determined, and for the scenario where the remote memory system includes multiple strips and the target storage space is a strip, the target check data determination method can be determined according to the quantitative relationship among the number of associated data pages of the strip, the number of other dirty data associated with the dirty cache line to be written in the cache space, and the number of clean data associated with the dirty cache line to be written. Exemplarily, the number of data cache pages included in the strip to which the dirty data to be written belongs, the first number of target dirty data to be written belonging to the same strip as the dirty data to be written, and the second number of clean data belonging to the same strip as the dirty data to be written can be obtained; by comparing the first number, the second number with the number of data cache pages, the target check data determination method with low write cost is determined.

[0052] In this embodiment, the first number is d, the second number is c, and the number of data cache pages is n. By comparing with n to determine the target check data determination method: when the strip calculation check value policy A of group write-back is better than the differential calculation check value policy D of group write-back; when the differential calculation check value policy of group write-back is better than the strip calculation check value policy of group write-back; when the efficiency of the strip calculation check value policy of group write-back and the differential calculation check value policy of group write-back is the same.

[0053] As can be seen from the above, in this embodiment, an evaluation model for the cost of the data write method based on the write timing is first established, and then for each data write request, the check value calculation method is selected according to the storage parameter information, so as to be able to select a more efficient data write method in real time, thereby realizing the dynamic selection of the data write method of the highly reliable remote memory system and improving the efficiency of data writing.

[0054] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. The present invention also provides a corresponding device for the data processing method, further making the method more practical. Among them, the device can be described from the perspective of functional modules and the perspective of hardware respectively. The data processing device provided by the present invention will be introduced below. The device is used to implement the data processing method provided by the present invention. For the description of the features in the corresponding embodiments of the data processing device, reference can be made to the relevant descriptions in the corresponding embodiments of the data processing method, and the device embodiments will not be elaborated one by one.

[0055] First, from the perspective of functional modules, please refer to Figure 4 , Figure 4 which is a structural diagram of the data processing device provided in this embodiment in a specific implementation manner. The device may include: A write mode determination module 401, configured to determine multiple data write modes for storing the data to be stored in the remote memory system according to the check data determination method corresponding to the data to be stored, and / or the write condition for writing the data to be stored into the remote memory system.

[0056] A cost evaluation module 402, configured to respectively count the read and write operation times of each data write mode, and determine the storage cost of each data write mode according to the storage parameter information of the target storage space corresponding to the data to be stored in the remote memory system, the read and write operation times of each data write mode, and the corresponding read and write costs; wherein, at least one of the check data determination methods and write conditions of different data write modes is different.

[0057] A data write module 403, configured to store the data to be stored in the remote memory system by using the target data write mode with the lowest storage cost; the storage parameter information includes at least the storage capacity of the target storage space and the data volume associated with the data to be stored.

[0058] Exemplarily, in some embodiments of this embodiment, the above write mode determination module 401 may also be configured to: obtain each target associated data belonging to the same stripe as the dirty data to be written; calculate the exclusive OR sum between the dirty data to be written and each target associated data as the check value of the dirty data to be written.

[0059] In an exemplary implementation of the above embodiment, the write mode determination module 401 may further be configured to: store the dirty data to be written into a pre-constructed cache space; based on the target memory address in the remote memory system where the dirty data to be written is written, determine, in the remote memory system, each target data belonging to the same stripe as the dirty data to be written, and read each target data into the cache space; calculate the first exclusive OR sum between the dirty data to be written and each target data; write the dirty data to be written to the target memory address, and write the first exclusive OR sum to the first memory check address in the remote memory system.

[0060] In another exemplary implementation of the above embodiment, the write mode determination module 401 may further be configured to: store the dirty data to be written into a pre-constructed cache space; when the group write-back condition is satisfied, based on the target memory address in the remote memory system where the dirty data to be written is written, determine, in the remote memory space of the remote memory system, the target data belonging to the same stripe as the dirty data to be written, and read the target data into the cache space; based on the target memory address, determine, in the cache space, the target dirty data to be written and the target clean data belonging to the same stripe as the dirty data to be written; calculate the second exclusive OR sum between the dirty data to be written, the target dirty data to be written, and the target clean data, as the check value of the dirty data to be written and the target dirty data to be written; write the dirty data to be written and the target dirty data to be written to the corresponding memory addresses in the remote memory system respectively, and write the second target exclusive OR sum to the second memory check address in the remote memory system.

[0061] Exemplarily, in some other implementations of this embodiment, the write mode determination module 401 may further be configured to: based on the target memory address corresponding to the dirty data to be written and written into the remote memory system, read, from the remote memory system, the old data stored at the target memory address at the current time and before, and the old check data of the old data; calculate the difference between the dirty data to be written and the old data, and calculate the exclusive OR value between the difference and the old check data, as the check value of the dirty data to be written.

[0062] Exemplarily, in some other implementations of this embodiment, the apparatus may further include a storage module, and this module may be configured to: the remote memory system includes multiple stripes, and the process of storing the data to be stored in the target storage space being a stripe includes: dividing the data to be stored into multiple data blocks to be stored, and storing each data block to be stored into the corresponding first memory space respectively; calculating the exclusive OR checksum between each data block to be stored, and storing the exclusive OR checksum into the second memory space; combining each first memory space and the second memory space into a stripe for storing the data to be stored; wherein, each first memory space is independent of each other and independent of the second memory space.

[0063] Exemplarily, in some other embodiments of this embodiment, the above device may further include a storage module, which can be used for: The remote memory system includes multiple stripes. The process of storing the data to be stored into the corresponding stripes includes: dividing the data to be stored into multiple data blocks to be stored, and storing each data block to be stored into the corresponding data cache page respectively; calculating the exclusive OR checksum between each data block to be stored, and storing the exclusive OR checksum into the check page; combining each data cache page and the check page into a stripe for storing the data to be stored; wherein, each data cache page is independent of each other and independent of the check page.

[0064] Exemplarily, in some other embodiments of this embodiment, the above device may further include a read processing module, which can be used for: when receiving a data read request, determining the target data cache page where the data to be read corresponding to the data read request is located, and determining the target stripe to which the target data cache page belongs; pre-reading and storing the target data cache page and other data cache pages in the target stripe except the target data cache page into a pre-constructed read operation cache space.

[0065] Exemplarily, in some other embodiments of this embodiment, the above device may further include a write processing module, which can be used for: when receiving a data write request, storing the data to be written corresponding to the data write request into a pre-constructed write operation cache space; when the data write condition is satisfied, writing the data in the write operation cache space into the remote memory space; wherein, the write operation cache space and the remote memory space belong to the remote memory system.

[0066] Exemplarily, in some other embodiments of this embodiment, the above cost evaluation module 402 can also be used for: calculating the write cost corresponding to each data write method in advance according to the number of read operations and read cost, and the number of write operations and write cost of each data write method, and determining the target write condition with a low write cost according to the write cost of the same check data determination method under different write conditions; determining the target check data determination method with a low write cost according to the storage parameter information of the target storage space of the data to be stored in the remote memory system; taking the data write method determined by the target check data determination method and the target write condition as the target data write method with the lowest storage cost.

[0067] As an exemplary embodiment of the above embodiment, the above cost evaluation module 402 can further be used for: The remote memory system includes multiple stripes, the target storage space is a stripe, obtaining the number of data cache pages included in the stripe to which the dirty data to be written belongs, the first number of target dirty data to be written that belongs to the same stripe as the dirty data to be written, and the second number of clean data that belongs to the same stripe as the dirty data to be written; determining the target check data determination method with a low write cost by comparing the first number, the second number with the number of data cache pages.

[0068] The data processing device mentioned above is described from the perspective of functional modules. Further, the present invention also provides an electronic device, which is described from the perspective of hardware. The electronic device includes a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned data processing method embodiments.

[0069] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. Wherein, the computer program is configured to execute the steps in any of the above-mentioned data processing method embodiments when running.

[0070] In an exemplary embodiment, the above-mentioned computer-readable storage medium may include, but is not limited to: various media such as USB flash drives, read-only memories (ROM for short), random access memories (RAM for short), mobile hard disks, magnetic disks, or optical discs that can store computer programs.

[0071] An embodiment of the present application also provides a computer program product. The above-mentioned computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above-mentioned data processing method embodiments.

[0072] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above-mentioned data processing method embodiments.

[0073] Finally, the present invention also provides a remote memory system. Please refer to Figure 5 , the remote memory system 12 may at least include a remote memory 123, a buffer 124, a read-write processor 125, and a program memory 126; the remote memory 123 is connected to the buffer 124, and the read-write processor 125 is connected to the buffer 124 and the program memory 126.

[0074] Among them, the remote memory storage 123 includes at least one stripe, and the stripe is used to store the corresponding data to be stored. When storing data, the stripe can refer to the striped memory redundancy method recorded in the above embodiment. The cache 124 is the memory space of the read-write processor 125. It can use any high-speed cache device to realize the memory function when calculating the check value recorded in the above method embodiment, and the function of improving the read and write performance of the remote memory by supporting specific read and write strategies. Further, the cache 124 can be divided into a read operation cache space and / or a write operation cache space, and the occupied capacity of the read operation cache space and the write operation cache space is determined according to the current application requirements and load characteristics. In other words, the cache space provided by the cache 124 can be set to a ratio of read and write space, and a part of the cache space is allocated to the read operation, which is defined as the read operation cache space, and the other part of the cache space is allocated to the write operation, which is defined as the write operation cache space, which is used to support specific read strategies or write strategies respectively. Generally speaking, for applications with frequent read operations, more cache can be allocated to read operations, and for applications with frequent write operations, more cache can be allocated to write operations. The setting of this ratio needs to be determined according to specific application requirements and load characteristics. Among them, the read strategy includes: when reading data, pre-reading data in other memory spaces with high locality with the target cache line space currently being read into the cache space, such as cache lines belonging to the same page and other pages belonging to the same stripe. The write strategy includes: when writing data, temporarily storing the data to be written in the cache space, and when a specific condition is met, such as after a certain clock cycle, or when the amount of data to be written reaches a certain level, then writing the data in the cache space to the remote memory storage 123, multiple dirty data of the same stripe are written to share the update check value operation, and the overhead cost of the check value calculation is amortized. The program memory 126 is used to store a computer program that implements the steps of the data processing method recorded in the above embodiment, which can be a read-only memory. The read-write processor 125, for example, can select a CPU with relatively general performance, which is used to implement the steps of the data processing method recorded in the above embodiment when executing the computer program stored in the program memory 126, such as completing the check value calculation, logical judgment, addressing and other operations of the above-mentioned striped memory redundancy mechanism.

[0075] In order to make those skilled in the art more clearly understand the technical solution of the present invention, the present invention also provides a remote memory system in an exemplary embodiment, such as Figure 6As shown, the remote memory system serves as the remote memory resource of the computing node, and there is data interaction with the computing node through the network. The computing node can be a general server or an artificial intelligence server, and is used to complete corresponding computing tasks based on the data read from the remote memory system or writing data to the remote storage system. The remote memory system 12 may include a remote memory storage 123, a buffer 124, a read / write processor 125, and a program memory 126. The remote memory storage 123 adopts a data storage method of striping memory redundancy mechanism and serves as the memory space for the remote memory system 12 to store data. Among them, the data block marked with "p" represents the redundant data block. The buffer 124 adopts a cache, and the program memory 126 adopts a ROM (Read-Only Memory), and its corresponding read-only space stores computer program codes for implementing redundant data calculation, verification, and data writing. The read / write processor 125 completes the redundant data calculation task, verification task, cost evaluation task for implementing the data writing method, and dynamic selection task of the remote memory system. Based on Figure 6 Taking the strip of the remote memory storage 123 shown, which includes 7 data cache pages (which can be represented as page0 - page6) and 1 parity page (which can be represented as page7) as an example, and the size of each page is 4K, and they are located on different memory strips respectively, then the strip can store 28K of valid data and 4K of redundant data. Denotes the dirty cache line data in the cache space, which is originated from the data to be written sent by the server. Denotes the clean cache line data in the cache space, which is originated from the data read by the server from the remote memory system 12. Denotes the calculated parity data. Denotes the parity data stored in the remote memory system 12, and the group write-back condition is that the occupancy of the cache space for write operation exceeds 50%. The data storage and read / write implementation process of the remote memory system may include: The server writes data to the 3rd cache line of the first data cache page page0 of the remote memory system, that is, the dirty data to be written can be represented as the dirty cache line. The remote memory system stores in the cache space of the buffer 124 and sends an acknowledgment message to the server. When the occupancy ratio of the write operation cache space is less than 50%, the buffer 124 does not write to the strip of the remote memory system. During the waiting period, the server writes the dirty cache line and to the remote memory system. The corresponding target memory address is the 3rd cache line of the third data cache page page3. The corresponding target memory address is the 4th cache line of the fourth data cache page, page4. Meanwhile, the server also reads the cache line data from the remote memory system. The remote memory system reads the content of the entire cache page into the cache space. When the occupancy ratio of the write operation cache space is greater than or equal to 50%, the remote memory system continues to process the and uncompleted write operations. According to the request order, it first processes the write operation. In the stripe where is located, the data associated with can be expressed as . For , the first number of the subscript of each letter corresponds to the identifier of the data cache page, and the second number represents the line number in the data cache page where it is located. Among them, exists in the cache space, the corresponding dirty data to be written also exists in the cache space. Therefore, the number of other dirty data associated with in the cache space is , the number of clean data belonging to the same stripe is , and the number of stripe data pages is . Correspondingly, , so the check value is calculated by the conditional calculation check value method and written using the group write-back method. That is, the currently dynamically selected data write method is the data write method composed of the stripe calculation check value method + the group write-back method. Read data from the remote memory system into the cache space, calculate the check cache line , and then write the dirty cache line as well as the check cache line have been written into the remote memory system, continue to process the write operation: In the stripe where is located, the data associated with can be expressed as . Among them, exists in the cache space. Therefore, at this time, the number of other dirty data associated with in the cache space is , the number of clean data belonging to the same stripe is , and the number of stripe data pages is , so the method of calculating the check value by difference is adopted. Considering that , the cost of the write-back group mode and the direct write-back mode is the same. Therefore, the currently dynamically selected data write mode is the data write mode composed of the check value calculation by difference + the group write-back mode or the data write mode composed of the check value calculation by difference + the direct mode. Read data from the remote memory system and to the cache space, calculate the check cache line , and then write the dirty cache line and the check cache line to the remote memory system.

[0076] As can be seen from the above, the remote memory system provides multiple data write modes based on the register, and through the dynamic selection of different data write modes, it can achieve the efficient data storage and read / write functions of a highly reliable remote memory system, and can be applied to the computing system scenarios with memory expansion requirements and high reliability requirements.

[0077] The above has introduced in detail a data processing method, an electronic device, a computer-readable storage medium, a computer program product, and a remote memory system provided by the present invention. Each embodiment in this specification is described in a progressive manner, and the key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts between each embodiment can be referred to each other. Whether the units and algorithm steps of each example described in the disclosed embodiments are executed in the form of electronic hardware or computer software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, and such implementation should not be considered to exceed the scope of the present invention. Without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: Applications in remote memory systems, including: Determine multiple data writing methods for storing the data to be stored in the remote memory system according to a method for determining verification data corresponding to the data to be stored and / or a writing condition for writing the data to be stored into the remote memory system; The number of read and write operations of each data writing method is counted respectively, and the storage cost of each data writing method is determined according to the storage parameter information of the target storage space of the data to be stored in the remote memory system, the number of read and write operations of each data writing method and the corresponding read and write cost; Storing the data to be stored in the remote memory system using a target data writing method with the lowest storage cost; Among them, at least one of the verification data determination method and the writing condition of different data writing methods is different, and the storage parameter information at least includes the storage capacity of the target storage space and the data volume associated with the data to be stored.

2. The data processing method according to claim 1, characterized in that: The method of determining the verification data corresponding to the data to be stored includes: Obtaining target-related data that belongs to the same stripe as the dirty data to be written; An exclusive OR sum between the dirty data to be written and each target associated data is calculated to serve as a check value of the dirty data to be written.

3. The data processing method according to claim 2, characterized in that: Calculating the XOR sum between the dirty data to be written and each target associated data includes: Storing the dirty data to be written into a pre-built cache space; Based on the target memory address where the dirty data to be written is written into the remote memory system, in the remote memory system, each target data belonging to the same stripe as the dirty data to be written is determined, and each target data is read into the cache space; Calculate a first XOR sum between the dirty data to be written and each target data; The dirty data to be written is written to the target memory address, and the first XOR sum is written to a first memory check address of the remote memory system.

4. The data processing method according to claim 2, characterized in that: Calculating the XOR sum between the dirty data to be written and each target associated data includes: Storing the dirty data to be written into a pre-built cache space; When a group write-back condition is met, according to the target memory address of the dirty data to be written being written to the remote memory system, target data belonging to the same stripe as the dirty data to be written is determined in the remote memory space of the remote memory system, and the target data is read into the cache space; According to the target memory address, determine in the cache space target dirty data to be written and target clean data that belong to the same stripe as the dirty data to be written; Calculating a second XOR sum of the dirty data to be written, the target dirty data to be written, and the target clean data as a check value of the dirty data to be written and the target dirty data to be written; The dirty data to be written and the target dirty data to be written are respectively written into corresponding memory addresses of the remote memory system, and the second target XOR sum is written into a second memory check address of the remote memory system.

5. The data processing method according to claim 1, characterized in that: The method of determining the verification data corresponding to the data to be stored includes: According to the target memory address of the remote memory system corresponding to the dirty data to be written, old data stored at the target memory address at and before the current time and old verification data of the old data are read from the remote memory system; The difference between the dirty data to be written and the old data is calculated, and an exclusive OR value of the difference and the old verification data is calculated as a verification value of the dirty data to be written.

6. The data processing method according to claim 1, characterized in that: The remote memory system includes a plurality of stripes, the target storage space is a stripe, and the process of storing the data to be stored in the corresponding stripe includes: Dividing the data to be stored into a plurality of data blocks to be stored, and storing each data block to be stored in a corresponding first memory space; Calculating the XOR checksum between each data block to be stored, and storing the XOR checksum in the second memory space; combining each of the first memory space and the second memory space into a stripe for storing the data to be stored; The first memory spaces are independent of each other and independent of the second memory space.

7. The data processing method according to claim 1, characterized in that: The remote memory system includes a plurality of stripes, and the process of storing the to-be-stored data in the corresponding stripes includes: Dividing the data to be stored into a plurality of data blocks to be stored, and storing each data block to be stored in a corresponding data cache page; Calculating the XOR checksum between each data block to be stored, and storing the XOR checksum in a check page; Combining each data cache page and the check page into a stripe storing the data to be stored; The data cache pages are independent of each other and the check page.

8. The data processing method according to claim 1, characterized in that: Also includes: When a data read request is received, determining a target data cache page where the to-be-read data corresponding to the data read request is located, and determining a target stripe to which the target data cache page belongs; The target data cache page and other data cache pages in the target stripe except the target data cache page are pre-read and stored in a pre-built read operation cache space.

9. The data processing method according to claim 1, characterized in that: Also includes: When a data write request is received, the to-be-written data corresponding to the data write request is stored in a pre-built write operation cache space; When the data writing condition is met, the data in the write operation cache space is written to the remote memory space; wherein the write operation cache space and the remote memory space belong to a remote memory system.

10. The data processing method according to any one of claims 1 to 9, characterized in that: Determining the storage cost of each data writing method according to the storage parameter information of the target storage space of the remote memory system corresponding to the data to be stored, the number of read and write operations of each data writing method and the corresponding read and write cost, including: Calculate the write cost corresponding to each data writing method in advance based on the number of read operations and read cost, the number of write operations and write cost of each data writing method, and determine the target writing condition with low write cost based on the write cost of the same verification data determination method under different writing conditions; Determine a method for determining target verification data with low writing cost according to storage parameter information corresponding to the target storage space of the remote memory system for the data to be stored; The data writing method determined by the target verification data determination method and the target writing condition is used as the target data writing method with the lowest storage cost.

11. The data processing method according to claim 10, characterized in that: The remote memory system includes a plurality of stripes, the target storage space is a stripe, and according to the storage parameter information of the target storage space of the remote memory system corresponding to the data to be stored, a method for determining the target verification data with low writing cost is determined, including: Obtain the number of data cache pages included in the stripe to which the dirty data to be written belongs, the first number of target dirty data to be written that belongs to the same stripe as the dirty data to be written, and the second number of clean data that belongs to the same stripe as the dirty data to be written; By comparing the first number, the second number and the number of data cache pages, a target verification data determination method with low writing cost is determined.

12. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data processing method according to any one of claims 1 to 11 when executing the computer program.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 11 are implemented.

14. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the data processing method according to any one of claims 1 to 11 are implemented.

15. A remote memory system, characterized in that: At least includes remote memory storage, cache, read-write processor and program memory; The remote memory storage is connected to the buffer, and the read-write processor is connected to the buffer and the program storage; The remote memory storage device includes at least one stripe, which is used to store corresponding data to be stored; the cache serves as the memory space of the read-write processor, including a read operation cache space and / or a write operation cache space, and the occupied capacity of the read operation cache space and the write operation cache space is determined according to current application requirements and load characteristics; the read-write processor is used to implement the steps of the data processing method as described in any one of claims 1 to 11 when executing the computer program stored in the program memory.

Images