Data processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure provide a data processing method and device, electronic equipment and storage medium. The method comprises: receiving first access data sent by at least one client, the first access data representing instruction records of remote direct data read instructions sent by the client to target data cached in a non-coherent memory access structure; obtaining data heat of the target data according to the first access data, the data heat representing a frequency of access of the target data by the remote direct data read instructions; and based on the data heat of the target data, caching the target data to a target position in a data storage unit implemented based on the non-coherent memory access structure, or migrating the target data out of the data storage unit, wherein the target position has a data read-write speed corresponding to the data heat. The method improves the caching efficiency of the target data by the cache node, avoids forming a cache hotspot, and further improves the overall performance of the database.

Description

Technical Field

[0001] This disclosure relates to the field of distributed storage technology, and in particular to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] The compute-deposit separation architecture in databases solves a series of problems such as resource contention and high expansion costs by separating data processing and storage, and is a common distributed architecture today. To further improve the performance of databases based on compute-deposit separation architectures, existing technologies use distributed caching nodes to reduce direct access to the database, thereby reducing network load and improving database performance.

[0003] However, in practical applications, existing data caching solutions still suffer from the problem of unreasonable data caching locations, leading to cached data hotspots and affecting database access efficiency and performance. Summary of the Invention

[0004] This disclosure provides a data processing method, apparatus, electronic device, and storage medium to overcome the problem of unreasonable data cache location affecting database performance.

[0005] In a first aspect, embodiments of this disclosure provide a data processing method, the method comprising:

[0006] The system receives first access data sent by at least one client, the first access data representing a record of remote direct data read instructions sent by the client for target data cached within a non-consistent storage access structure; based on the first access data, it obtains the data popularity of the target data, the data popularity representing the frequency with which the target data is accessed by remote direct data read instructions; based on the data popularity of the target data, it caches the target data to a target location in a data storage unit implemented based on a non-consistent storage access structure, or migrates the target data out of the data storage unit, wherein the target location has a data read / write speed corresponding to the data popularity.

[0007] In a second aspect, embodiments of this disclosure provide a data processing apparatus, the apparatus comprising:

[0008] A receiving module is configured to receive first access data sent by at least one client, wherein the first access data represents an instruction record of a remote direct data read instruction sent by the client for target data cached within a non-consistent storage access structure;

[0009] The processing module is used to obtain the data popularity of the target data based on the first access data, wherein the data popularity characterizes the frequency of the target data being accessed by remote direct data read commands;

[0010] A caching module is used to cache the target data to a target location in a data storage unit implemented based on a non-consistent storage access structure, or to migrate the target data out of the data storage unit, based on the data popularity of the target data, wherein the target location has a data read / write speed corresponding to the data popularity.

[0011] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0012] A processor, and a memory communicatively connected to the processor;

[0013] The memory stores computer-executed instructions;

[0014] The processor executes computer execution instructions stored in the memory to implement the data processing method described in the first aspect and various possible designs of the first aspect.

[0015] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the data processing method described in the first aspect and various possible designs of the first aspect.

[0016] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the data processing method described in the first aspect and various possible designs of the first aspect.

[0017] The data processing method, apparatus, electronic device, and storage medium provided in this embodiment receive first access data sent by at least one client. This first access data represents a record of remote direct data read instructions sent by the client for target data cached within a non-consistent storage access structure. Based on the first access data, the data popularity of the target data is obtained, whereby the data popularity represents the frequency with which the target data is accessed by remote direct data read instructions. Based on the data popularity of the target data, the target data is either cached at a target location within a data storage unit implemented based on the non-consistent storage access structure, or the target data is moved out of the data storage unit. The target location has a data read / write speed corresponding to the data popularity. By receiving the first access data sent by the client to determine the data popularity of the target data, and then processing it accordingly—caching it at a target location in a data storage unit with a matching read / write speed or moving it out of the data storage unit—the caching efficiency of the cache nodes for the target data is improved, cache hotspots are avoided, and the overall performance of the database is enhanced. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is an application scenario diagram of the data processing method provided in the embodiments of this disclosure;

[0020] Figure 2 Flowchart of the data processing method provided in the embodiments of this disclosure Figure 1 ;

[0021] Figure 3 for Figure 2 A flowchart illustrating the specific implementation of step S102 in the illustrated embodiment;

[0022] Figure 4 for Figure 3 A flowchart illustrating the specific implementation of step S1021 in the illustrated embodiment;

[0023] Figure 5 A schematic diagram illustrating a process for obtaining the average number of visits provided in an embodiment of this disclosure;

[0024] Figure 6 Flowchart of the data processing method provided in the embodiments of this disclosure Figure 2 ;

[0025] Figure 7 A schematic diagram illustrating the mapping relationship between data popularity and access medium, provided for an embodiment of this disclosure;

[0026] Figure 8 for Figure 6 A flowchart illustrating the specific implementation of step S206 in the illustrated embodiment;

[0027] Figure 9 This is a schematic diagram illustrating data processing in a non-consistent storage access structure within a cache node, provided by an embodiment of the present disclosure.

[0028] Figure 10 This is a structural block diagram of a data processing apparatus provided in an embodiment of the present disclosure;

[0029] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;

[0030] Figure 12 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0033] The application scenarios of the embodiments of this disclosure are explained below:

[0034] Figure 1This diagram illustrates an application scenario of the data processing method provided in this disclosure. The data processing method can be applied to application scenarios involving data caching in distributed storage systems, and more specifically, to application scenarios involving distributed data caching based on Remote Direct Memory Access (RDMA) technology. Specifically, the method provided in this disclosure can be applied to cache nodes, which can be implemented through physical devices, such as server devices, or through virtual computing units created within physical devices, such as instances or containers. Figure 1 As shown, in a distributed storage system, data is stored in a distributed file system. Multiple distributed cache nodes communicate with the distributed file system to cache the original data in the data storage unit inside the cache node. On the other hand, they receive data read requests sent by clients and send the cached data to the clients, thereby reducing the number of clients directly accessing the distributed storage system, reducing network load, and improving the overall performance of the distributed database.

[0035] In applications involving data caching in distributed storage systems, existing technologies utilize remote direct data read commands (commands used to read data in remote direct data access technology) to effectively reduce the CPU usage of computing nodes and better utilize the read / write bandwidth of distributed memory, thereby reducing data read / write latency and improving database performance. However, due to the characteristic that one-sided read operations are not visible to the server, the server (e.g., cache nodes) cannot obtain the specific data content requested by the client. Consequently, it is impossible to manage hot and cold data based on the frequency of client-requested data, meaning that hot data cannot be stored in appropriate locations within the data storage units of the cache node to achieve load balancing and matching of access medium read / write performance. This leads to cache hotspot issues and impacts database access efficiency and performance. This disclosure provides a data processing method to address the above problems.

[0036] refer to Figure 2 , Figure 2 Flowchart of the data processing method provided in the embodiments of this disclosure Figure 1 The method of this embodiment can be applied to cache nodes, such as servers or instances running within servers. This data processing method includes:

[0037] Step S101: Receive first access data sent by at least one client, the first access data representing the instruction record of a remote direct data read instruction sent by the client for target data cached within the non-consistent storage access structure.

[0038] For example, refer to Figure 1 The illustrated application scenario diagram shows that when a client needs to access data, it sends a data read request to a cache node. The cache node includes data storage units implemented using a Non-Uniform Memory Access (NUMA) architecture. In this embodiment, the cache node contains multiple interconnected NUMA structures for accessing memory data. The specific implementation principle of the NUMA structure is existing technology and will not be elaborated here. More specifically, the client sends a one-sided read request based on the Remote Direct Data Access (RDMA) protocol (i.e., a remote direct data read instruction), or a read request based on Remote Procedure Call (RPC), to the cache node to quickly obtain the corresponding target data. It is understood that the client sending a data write request to the cache node is similar to sending a data read request, and will not be elaborated here. During this process, the client generates corresponding first access data based on the content contained in the sent data read request, i.e., the instruction record for the target data (i.e., the access record). The instruction record for the target data includes information such as the name of the target data, access time, number of accesses, and access frequency. For example, the first access data may also include information such as the cache node where the target data is located and the specific data storage unit within the cache node where the target data is stored. Subsequently, the client sends the first access data to the cache node. Upon receiving the first access data, the cache node processes it and executes subsequent steps. Optionally, the client sends the first access data to the cache node asynchronously, and the cache node can asynchronously obtain the first access data through a queue. Specific implementation methods can be configured as needed and will not be elaborated here.

[0039] Step S102: Based on the first access data, obtain the data heat of the target data. The data heat represents the frequency of the target data being accessed by remote direct data read commands.

[0040] For example, after receiving the first access data, the cache node parses the first access data according to the corresponding protocol. Based on the content of the first access data, such as the name of the target data, access time, number of accesses, access frequency, or one or more other information as shown in the previous steps, it determines the data heat, which represents the frequency of remote direct data read commands accessing the target data. Specifically, for example, based on multiple access times of the target data, a time interval is determined, and then the sum of the number of all access requests for the target data within that time interval is counted to obtain the data heat. The higher the data heat of the target data, the more frequently the target data is accessed; conversely, the lower the data heat of the target data, the less frequently the target data is accessed. Based on the data heat and a preset heat threshold, the data can be divided into "hot data" and "cold data." The data heat can be implemented in various ways, such as based on the number of accesses within a preset time period; or, for example, the frequency of periodic accesses to the data. The numerical value corresponding to the data heat can be an actual quantity, such as the number of accesses, or a normalized value representing the frequency; no specific restrictions are placed here.

[0041] In one possible implementation, the first access data includes the first access count of the target data within a first time interval, such as... Figure 3 As shown, the specific implementation of step S102 includes:

[0042] Step S1021: Calculate the average number of visits based on the first number of visits within the first time interval and the second number of visits within the second time interval, wherein the second time interval is located before the first time interval.

[0043] Step S1022: Obtain the data popularity of the target data based on the average number of visits.

[0044] For example, the first time interval is a real-time statistical interval for the first access count. In one possible implementation, the first time interval is the time interval consisting of the first moment when the client generates the first access data and the preset duration before that first moment. Therefore, the first access count within the first time interval can be understood as the instruction record for the target data generated by the client in the current time period. The second time interval is the time interval before the first time interval, i.e., the historical time interval. Therefore, the second access count within the second time interval can be understood as the historical instruction record of the target data. After the cache node obtains the first access count within the first time interval through the first access data, it combines it with the second access count within the second time interval, i.e., the historical instruction record of the target data, and performs a weighted calculation to obtain data that can average the access count of the target data in the current time period and the historical time period, i.e., the average access count. Then, based on the average access count, it is mapped to the corresponding data popularity. The higher the average access count, the higher the data popularity, and vice versa.

[0045] The second time interval and the second access count corresponding to the second time interval can be preset in the cache node or in the external access medium for communication between the cache node and the cache node. The cache node can directly read the second access count corresponding to the second time interval and maintain and update it. The specific implementation method is not limited here.

[0046] In this embodiment, the number of first accesses within the first time interval is corrected by combining historical instruction records to obtain the average number of accesses. Then, the data popularity is obtained based on the average number of accesses, reducing data popularity fluctuations. Since the target data needs to be migrated based on data popularity in subsequent exemplary steps, reducing data popularity fluctuations can reduce the frequency of migration of target data, reduce system computing resources and bandwidth consumption, and improve system stability.

[0047] Further, exemplarily, the second time interval includes at least one historical sub-interval, and the second access count includes the interval access count corresponding to each historical sub-interval, such as... Figure 4 As shown, the specific implementation of step S1021 includes:

[0048] Step S1021A: Based on the historical time corresponding to each historical sub-interval, obtain the corresponding weighting coefficient, wherein the weighting coefficient is inversely proportional to the time distance from the historical time to the current time.

[0049] Step S1021B: Based on the weighting coefficients of each historical sub-interval, calculate the weighted sum of the interval access counts corresponding to each historical sub-interval to obtain the weighted access count.

[0050] Step S1021C: Calculate the average number of visits based on the weighted number of visits and the first number of visits.

[0051] For example, the second time interval, which serves as the historical time interval, is composed of multiple historical sub-intervals. Each historical sub-interval corresponds to a specific number of accesses. These historical sub-intervals can be obtained by the cache node through statistics of first access data sent by one or more clients within previous historical time periods. In calculating the average number of accesses, a weighting coefficient is determined based on the time distance between the historical moment corresponding to each historical sub-interval and the current moment. The longer the time distance, the smaller the weighting coefficient of the corresponding access count and the smaller its proportion in the generated average number of accesses. In short, access data from a more distant period is less important. Then, based on the weighting coefficients of each historical sub-interval, the access counts of the corresponding historical sub-intervals are weighted and summed to obtain the weighted access count for the second time interval. Finally, the weighted sum of the weighted access count and the first access count yields the average access count after weighting based on historical time.

[0052] Figure 5 This is a schematic diagram illustrating a process for obtaining the average number of visits, as provided in an embodiment of this disclosure. Figure 5As shown, after the cache node obtains the first time interval T1 and the corresponding first access count C1 (denoted as TI[C1] in the figure), it obtains multiple historical sub-intervals (i.e., the second time interval) before the first time interval T1 and the interval access count corresponding to each historical sub-interval, as shown in the figure. This includes the historical sub-interval T0_1 and the corresponding interval access count C0_1 (denoted as T0_1[C0_1] in the figure), the historical sub-interval T0_2 and the corresponding interval access count C0_2 (denoted as T0_2[C0_2] in the figure), and the historical sub-interval T0_3 and the corresponding interval access count C0_3 (denoted as T0_3[C0_2] in the figure). The first time interval T1, and the historical sub-intervals T0_1, T0_2, T0_3, and T0_4 have the same interval length, L. The historical times corresponding to the historical sub-intervals T0_1, T0_2, T0_3, and T0_4 are the end times of their respective historical sub-intervals: t0_1, t0_2, t0_3, and t0_4. The current time is the end time t1 of the first time interval T1. Based on the visual distance between the end time of each historical sub-interval and the current time, the normalized weighting coefficients of each historical sub-interval are coef_01 = 1 / 10, coef_02 = 2 / 10, coef_03 = 3 / 10, and coef_04 = 4 / 10 (correspondence is shown in the figure). Then, based on the number of visits corresponding to each historical sub-interval, a weighted sum is performed to obtain the corresponding weighted number of visits; then, the sum of the weighted number of visits and the first number of visits (i.e., the weighted number of visits and the first number of visits have the same weight) is calculated to obtain the average number of visits.

[0053] Of course, it is understandable that the weighted method described above is only one of many possible ways to calculate the average number of visits. Other weighted methods can be used as needed to obtain the average number of visits. For example, a corresponding weighting coefficient can be set for the first time interval T1. The historical sub-intervals T0_1, T0_2, T0_3, T0_4, and the first time interval T1 can be normalized together to obtain the weighting coefficients for each historical sub-interval T0_1, T0_2, T0_3, T0_4, and the first time interval T1, respectively: coef_01 = 1 / 15, coef_02 = 2 / 15, coef_03 = 3 / 15, coef_04 = 4 / 15, coef_1 = 4 / 15. Then, the first visit count is weighted based on coef_1 and summed with the weighted visit count (i.e., the weights of the weighted visit count and the first visit count are different) to obtain the average number of visits.

[0054] In this embodiment, multiple weighting is applied based on the time distance between the historical time corresponding to the historical sub-interval and the current time. The instruction records generated at different historical times are smoothed by sliding, reducing the jumps in the average number of accesses, thereby making the calculation of the data heat of the target data more stable and accurate.

[0055] Step S103: Based on the data popularity of the target data, cache the target data in the target location of the data storage unit implemented based on the non-consistent storage access structure, or migrate the target data out of the data storage unit, wherein the target location has a data read and write speed corresponding to the data popularity.

[0056] For example, after obtaining the data popularity of the target data, the target data is processed accordingly based on this popularity, specifically including replacing and migrating the target data. For example, replacement refers to caching the target data to a data storage unit and / or migrating the target data out of the data storage unit. In one possible implementation, when the data popularity of the target data is greater than a first preset value, the target data is determined to be hot data. If the target data is not cached in a data storage unit within a cache node, it is cached from the database to a data storage unit within the cache node. When the data popularity of the target data is less than the first preset value, the target data is determined to be cold data. If the target data is already cached in a data storage unit within a cache node, it is migrated out of the data storage unit. Further, when the available space of the data storage unit is greater than a space threshold, the steps of caching the target data to the data storage unit and migrating the target data out of the data storage unit can be executed independently; however, when the available space of the data storage unit is less than the space threshold, the steps of migrating the target data out of the data storage unit and caching the target data to the data storage unit must be executed sequentially.

[0057] For example, migration refers to moving target data from a non-target location within a data storage unit to a target location. In one possible implementation, when the data popularity of the target data exceeds a first preset value, the target data is determined to be hot data. In this case, the target data is migrated from the first storage location of the data storage unit to the second storage location (target location), thereby improving the access efficiency of the target data. Conversely, when the data popularity of the target data is less than the second preset value, the target data is determined to be cold data. In this case, the target data is migrated from the second storage location (target location) of the data storage unit to the first storage location, thereby releasing storage resources and improving the overall access efficiency of the data storage unit.

[0058] Optionally, when the target data is not cached in the data storage unit, before step S103, the following steps are also included:

[0059] Step S104: Obtain the migration duration of the target data. The migration duration represents the duration during which the target data is migrated out of the data storage unit.

[0060] Accordingly, step S103 is implemented as follows: when the migration duration is longer than the second duration, the target data is cached in the target location of the data storage unit.

[0061] For example, after the data storage unit performs a data migration operation on the target data, since data migration is more time-consuming and has higher processing costs, in order to avoid repeated migration in and out, in this embodiment, after the target data is migrated out, a second judgment is made based on the migration time of the target data. If the migration time is longer than the second time, that is, the target data is not migrated out in a short period of time, then the step of caching the target data to the target location of the data storage unit is executed; if the migration time is shorter than the second time, that is, the target data has just been migrated out of the data storage unit, in this case, the step of caching the target data to the target location of the data storage unit is not executed, thereby avoiding the situation where the target data is repeatedly migrated in and out of the data storage unit, reducing ineffective resource consumption, and improving the storage efficiency of the data storage unit.

[0062] In this embodiment, by receiving first access data sent by at least one client, the first access data represents the client's instruction record for the target data; based on the first access data, the data popularity of the target data is obtained, which represents the frequency of access to the target data; based on the data popularity of the target data, the target data is either cached in the target location of the data storage unit or moved out of the data storage unit. By determining the data popularity of the target data by receiving the first access data sent by the client, and then processing it accordingly, either caching it in the target location of the data storage unit or moving it out of the data storage unit, the caching efficiency of the cache node for the target data is improved, the formation of cache hotspots is avoided, and thus the overall performance of the database is improved.

[0063] refer to Figure 6 , Figure 6 Flowchart of the data processing method provided in the embodiments of this disclosure Figure 2 This embodiment is in Figure 2 Based on the illustrated embodiment, the data processing method is further refined to address the scenario where target data is migrated within the data storage unit. In this embodiment, the data processing method includes:

[0064] Step S201: Receive first access data sent by at least one client, the first access data representing the instruction record of a remote direct data read instruction sent by the client for target data cached within the non-consistent storage access structure.

[0065] Step S202: Based on the first access data, obtain the data popularity of the target data. The data popularity represents the frequency of the target data being accessed by remote direct data read commands.

[0066] Step S203: Based on the data popularity of the target data, determine the target access medium on which the data storage unit is mounted. The target access medium has a data read / write speed corresponding to the data popularity.

[0067] For example, in this embodiment, the data storage unit in the cache node is a Non-Uniform Memory Access (NUMA) structure. In this embodiment, the cache node contains multiple interconnected NUMA structures for accessing memory data. Further, the NUMA structure implements data caching by mounting access media. The NUMA structure mounts access media of different types located at different locations (local or remote), and these different types of access media have different data read / write speeds. For example, the access media mounted on the NUMA structure include: Dynamic Random Access Memory (DRAM); Persistent Memory (PMEM) located locally on the NUMA structure; Persistent Memory located remotely on the NUMA structure; Solid State Disk (SSD) located locally on the NUMA structure; and Solid State Disk located remotely on the NUMA structure. The data read / write speeds of the aforementioned access media decrease sequentially. Given the limited space of the access media, allocating access media with high data read / write speeds to store frequently accessed data, and allocating access media with low data read / write speeds to store less frequently accessed data, can improve the overall data caching efficiency of the non-consistent storage access structure.

[0068] In one possible implementation, data popularity is represented by a level identifier that indicates the level of popularity. Figure 7 This is a schematic diagram illustrating the first type of mapping relationship between data popularity and access medium provided in this embodiment of the disclosure, such as... Figure 7As shown, data popularity is divided into L1 to L5. More specifically, for example, based on the first access data, when the average number of accesses to the target data within the third time period is 1 <= N < 100, the corresponding data popularity is generated as L0. The access medium corresponding to L0 is the solid-state drive (SSD1) at the remote end of the non-consistent storage access structure. Similarly, when the average number of accesses to the target data within the third time period is 100 <= N < 1000, the corresponding data popularity is generated as L1. The access medium corresponding to L1 is the solid-state drive (SSD2) at the local end of the non-consistent storage access structure. When the average number of accesses to the target data within the third time period is 100 <= N < 1000, the corresponding data popularity is generated as L1. When the number of accesses to the target data is less than 10,000, the corresponding data heat level is L2. The access medium for L2 is persistent memory located at the far end of the non-uniform memory access structure (shown as PEME1 in the diagram). When the average number of accesses to the target data within the third time period is 10,000 <= N < 100,000, the corresponding data heat level is L3. The access medium for L3 is persistent memory located locally within the non-uniform memory access structure (shown as PEME2 in the diagram). When the average number of accesses to the target data within the third time period is N >= 100,000, the corresponding data heat level is L4. The access medium for L4 is dynamic random access memory (shown as DRAM in the diagram). Based on... Figure 7 The mapping relationship shown can be mapped to the corresponding target access medium after obtaining the data popularity of the target data.

[0069] In one possible implementation, the first accessed data includes granularity information, which represents the classification granularity of the target data. A larger classification granularity results in a larger cached target data volume; conversely, a smaller classification granularity results in a smaller cached target data volume. When the classification granularity is large, caching large volumes of target data can increase the cache hit rate during subsequent client accesses. However, this comes at the cost of significant space consumption on the access medium. In another possible implementation, the granularity information can be represented by the data identifier of the target data, such as by the data name of the target data, to indicate the classification granularity of the target data.

[0070] In this embodiment, before determining the target access medium mounted on the data storage unit based on the data popularity of the target data, the method further includes: obtaining the data storage unit for caching the target data based on granularity information.

[0071] For example, based on the granularity information contained in the first access data, the cache node can determine a data storage unit that matches it in terms of performance and remaining space to store the target data, thereby improving data caching efficiency, balancing the load among data storage units, and thus improving the overall performance of the database.

[0072] Optionally, after step S203, the method further includes:

[0073] Step S204: Obtain the average heat of the data cached in the first access medium, where the first access medium is the access medium for the current cached target data.

[0074] For example, when the target data has been cached in the data storage unit, the data popularity of other data stored in the first access medium where the target data resides is first obtained, thereby obtaining the average popularity of the data cached in the first access medium. The data popularity of other data is information pre-generated and maintained within the cache node, and its generation method is the same as that used to obtain the data popularity of the target data in the above steps, and will not be repeated here. Furthermore, the average popularity can be information pre-calculated and stored locally on the cache node, or it can be information calculated in real-time by the cache node after receiving the first access data; there is no limitation here. Subsequently, this average popularity is used as a criterion for determining whether to trigger data migration between access media for the target data.

[0075] Step S205: Based on the amount of target data, obtain the migration time. The migration time represents the time required to migrate the target data to the target access medium.

[0076] For example, further, the first access data may contain the amount of target data, or the cache node may query the local or distributed storage system of the target data using the data identifier of the target data to obtain the amount of target data. Then, the cache node determines the migration time based on the amount of target data, and the data read / write speeds (read speed and write speed) of the first access medium and the target access medium. Specifically, this can be calculated based on the lower of the data read / write speeds of the first access medium and the target access medium; the specific process will not be elaborated further.

[0077] Step S206: If the triggering conditions are met, the target data is cached in the target access medium. The triggering conditions include: the target data's popularity exceeding a first threshold of average popularity, and / or the migration time being less than a first duration.

[0078] For example, after one or all of the above steps S204 and S205 are completed, a judgment is made based on the corresponding execution result. If the triggering condition is met, that is, if the data popularity of the target data is higher than the average popularity first threshold, and / or the migration time is less than the first duration, then the target data is migrated from the current first access medium to the target access medium, thereby making the access medium where the target data is located more popular.

[0079] Specifically, when the data popularity of the target data exceeds the average popularity threshold, indicating that the data popularity of the target data is significantly higher than the average popularity of the current access medium, the migration of the target data is triggered. On the one hand, the preset threshold limit can prevent the problem of triggering the migration of the target data when the data popularity fluctuates only slightly, thus avoiding the waste of computing and network resources caused by frequent and inefficient data migration. On the other hand, due to the fluctuation of client access volume during busy / idle hours, the access volume will fluctuate. The scheme of determining the timing of data migration by comparing the real-time data popularity of the target data with the fixed popularity threshold can be affected by the fluctuation phenomenon and erroneously trigger data migration, causing data to gather in the high-speed access medium, thereby affecting the data caching effect. In this embodiment, the scheme of determining the timing of data migration by using the average heat of the first access medium where the target data is located can avoid the influence of the above-mentioned tidal phenomenon. That is, when the overall access volume of the client is high (busy time), the heat of the target data will increase. At the same time, the heat of other data in the first access medium will also increase, which will lead to a synchronous increase in the average heat of the first access medium. Using the average heat as the criterion for judging the data migration of the target data can avoid the false triggering of data migration and improve the data caching effect.

[0080] On the other hand, by comparing the migration time with the first duration, it can be determined whether the migration process of the target data in the data storage unit can be completed within the first duration. This avoids the problem of data popularity losing its effectiveness due to excessive migration time and inability to complete within a detection cycle, thereby improving the effectiveness of caching target data based on data popularity and the access hit rate of cached target data.

[0081] Optionally, such as Figure 8 As shown, the specific implementation of step S206 includes:

[0082] Step S2061: Obtain the data cache speed based on the operating load of the data storage unit.

[0083] Step S2062: Based on the data caching speed, cache the target data in the target access medium.

[0084] For example, during the migration of target data, the cache node can further adjust the data migration speed (i.e., data caching speed) based on the operating load of the data storage unit where the target data resides. This avoids bandwidth and computing resource exhaustion, which could affect the stability of data access. Specifically, the operating load can be determined by the current available thread resources, CPU utilization, and available bandwidth resources of the data storage unit, and then mapped to the corresponding data caching speed. In one possible implementation, the operating load can be a normalized value or a percentage value. During the migration of target data, the cache node monitors the operating load in real time. When the operating load exceeds a load threshold, the current data caching speed is reduced to avoid resource exhaustion.

[0085] Figure 9 This is a schematic diagram illustrating data processing in a non-consistent storage access structure within a cache node, as provided in an embodiment of this disclosure. Figure 9 As illustrated, for example, multiple clients (e.g., client #1, client #2, client #3) send corresponding first access data to the distributed cache node P1. This first access data includes data identifiers for different target data (e.g., data D1 to data D5), as well as their corresponding frequency, storage medium, and data storage unit. Then, based on the first access data sent by each client, the data is sorted by popularity, resulting in a data popularity ranking table represented by data identifiers and a corresponding data index table. The data index table includes the identifiers of the data storage units corresponding to each target data (NUMA_0 and NUMA_1 in the diagram) and the corresponding storage media (SSD, DRAM, and PMEM in the diagram). Next, based on the data popularity ranking table and the corresponding data index table, a migration data index is obtained. This migration data index is used to characterize the data identifiers of the target data that needs to be migrated, as well as the target location for migration. Subsequently, data migration and replacement coordination between data storage units and within data storage units is implemented based on this migration data index.

[0086] In this embodiment, the specific implementation of steps S201 and S202 is the same as described above. Figure 2 In the illustrated embodiment, steps S101 and S102 are implemented in the same way; please refer to the detailed discussion for further information. Figure 2 The relevant descriptions in the illustrated embodiments will not be repeated here.

[0087] Corresponding to the data processing method in the above embodiments, Figure 10 This is a structural block diagram of a data processing apparatus provided according to embodiments of the present disclosure. For ease of explanation, only the parts relevant to embodiments of the present disclosure are shown. (Refer to...) Figure 10The data processing device 3 is applied to the cache node, which includes at least one data storage unit, comprising:

[0088] The receiving module 31 is configured to receive first access data sent by at least one client, wherein the first access data represents an instruction record of a remote direct data read instruction sent by the client for target data cached within a non-consistent storage access structure;

[0089] Processing module 32 is used to obtain the data popularity of the target data based on the first access data, wherein the data popularity characterizes the frequency of the target data being accessed by remote direct data read instructions;

[0090] The caching module 33 is used to cache the target data to a target location in a data storage unit based on a non-consistent storage access structure, or to migrate the target data out of the data storage unit, based on the data popularity of the target data, wherein the target location has a data read / write speed corresponding to the data popularity.

[0091] In one embodiment of this disclosure, the first access data includes a first access count of the target data within a first time interval: the processing module 32 is specifically used to: obtain an average access count based on the first access count within the first time interval and a second access count within a second time interval, wherein the second time interval is located before the first time interval; and obtain the data popularity of the target data based on the average access count.

[0092] In one embodiment of this disclosure, the second time interval includes at least one historical sub-interval, and the second access count includes the interval access count corresponding to each historical sub-interval; when the processing module 32 obtains the average access count based on the first access count in the first time interval and the second access count in the second time interval, it is specifically used to: obtain the corresponding weighting coefficient based on the historical time corresponding to each historical sub-interval, wherein the weighting coefficient is inversely proportional to the time distance from the historical time to the current time; calculate the weighted sum of the interval access counts corresponding to each historical sub-interval based on the weighting coefficient of each historical sub-interval to obtain the weighted access count; and obtain the average access count based on the weighted access count and the first access count.

[0093] In one embodiment of this disclosure, the caching module 33 is specifically used to: determine the target access medium mounted on the data storage unit based on the data popularity of the target data, wherein the target access medium has a data read / write speed corresponding to the data popularity; and cache the target data in the target access medium.

[0094] In one embodiment of this disclosure, the target access medium mounted on the data storage unit includes at least one of the following: dynamic random access memory; persistent memory located locally on the non-consistent storage access structure; persistent memory located at a remote end of the non-consistent storage access structure; solid-state drive located locally on the non-consistent storage access structure; and solid-state drive located at a remote end of the non-consistent storage access structure.

[0095] In one embodiment of this disclosure, when the target data is cached in the data storage unit, the caching module 33 is further configured to: obtain the average heat of the data cached in the first access medium, wherein the first access medium is the access medium for caching the target data; when caching the target data into the target access medium, the caching module 33 is specifically configured to: when the heat of the target data is higher than the average heat of a first threshold, migrate the target data to the target access medium, wherein the first threshold is determined at least based on the amount of the target data.

[0096] In one embodiment of this disclosure, before caching the target data into the target access medium, the caching module 33 is further configured to: obtain the migration time based on the amount of the target data, wherein the migration time represents the time taken to migrate the target data into the target access medium; when caching the target data into the target access medium, the caching module 33 is specifically configured to: if the migration time is less than a first duration, then cache the target data into the target access medium.

[0097] In one embodiment of this disclosure, the caching module 33 is specifically used to: obtain the data caching speed based on the operating load of the data storage unit; and cache the target data into the target access medium based on the data caching speed.

[0098] In one embodiment of this disclosure, when the target data is not cached in the data storage unit, the caching module 33 is further configured to: obtain the migration duration of the target data, wherein the migration duration represents the duration during which the target data is migrated out of the data storage unit; the caching module 33 is specifically configured to: cache the target data to the target location of the data storage unit when the migration duration is greater than the second duration.

[0099] In one embodiment of this disclosure, the first access data includes granularity information, which characterizes the classification granularity of the target data; the processing module 32 is further configured to: obtain a data storage unit for caching the target data based on the granularity information.

[0100] In one embodiment of this disclosure, at least one of the following is further included: the first access data is generated based on a remote direct data access request or a remote procedure call request sent by the client to the cache node; the first access data includes at least one of the following: the identifier of the cache node, the identifier of the data storage unit, and the identifier of the access medium on which the data storage unit is mounted.

[0101] The receiving module 31, processing module 32, and buffer module 33 are connected in sequence. The data processing device 3 provided in this embodiment can execute the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0102] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, such as... Figure 11 As shown, the electronic device 4 includes:

[0103] Processor 41, and memory 42 communicatively connected to processor 41;

[0104] Memory 42 stores instructions executed by the computer;

[0105] The processor 41 executes computer execution instructions stored in the memory 42 to achieve, for example, Figures 2-9 The data processing method in the illustrated embodiment.

[0106] Optionally, the processor 41 and the memory 42 are connected via a bus 43.

[0107] For relevant instructions, please refer to the corresponding text. Figures 2-9 The relevant descriptions and effects of the steps in the corresponding embodiments are understood, and will not be elaborated on here.

[0108] This disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement this disclosure. Figures 2-9 The data processing method provided in any of the corresponding embodiments.

[0109] This disclosure provides a computer program product, including a computer program that, when executed by a processor, implements as follows: Figures 2-9 The data processing method in the illustrated embodiment,

[0110] refer to Figure 12 The diagram illustrates a structural schematic of an electronic device 900 suitable for implementing embodiments of the present disclosure. The electronic device 900 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 12 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0111] like Figure 12 As shown, the electronic device 900 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the electronic device 900. The processing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0112] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 12 An electronic device 900 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0113] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the methods of embodiments of this disclosure.

[0114] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0115] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0116] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0117] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0119] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0120] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0121] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0122] In a first aspect, according to one or more embodiments of the present disclosure, a data processing method is provided, applied to a cache node, the cache node including at least one data storage unit, comprising:

[0123] The system receives first access data sent by at least one client, the first access data representing a record of remote direct data read instructions sent by the client for target data cached within a non-consistent storage access structure; based on the first access data, it obtains the data popularity of the target data, the data popularity representing the frequency with which the target data is accessed by remote direct data read instructions; based on the data popularity of the target data, it caches the target data to a target location in a data storage unit implemented based on a non-consistent storage access structure, or migrates the target data out of the data storage unit, wherein the target location has a data read / write speed corresponding to the data popularity.

[0124] According to one or more embodiments of this disclosure, the first access data includes a first access count of the target data within a first time interval; obtaining the data popularity of the target data based on the first access data includes: obtaining an average access count based on the first access count within the first time interval and a second access count within a second time interval, wherein the second time interval is located before the first time interval; and obtaining the data popularity of the target data based on the average access count.

[0125] According to one or more embodiments of this disclosure, the second time interval includes at least one historical sub-interval, and the second access count includes the interval access count corresponding to each of the historical sub-intervals; the step of obtaining the average access count based on the first access count within the first time interval and the second access count within the second time interval includes: obtaining a corresponding weighting coefficient based on the historical time corresponding to each of the historical sub-intervals, wherein the weighting coefficient is inversely proportional to the time distance from the historical time to the current time; calculating a weighted sum of the interval access counts corresponding to each of the historical sub-intervals based on the weighting coefficient of each of the historical sub-intervals to obtain a weighted access count; and obtaining the average access count based on the weighted access count and the first access count.

[0126] According to one or more embodiments of this disclosure, caching the target data to a target location in a data storage unit implemented based on a non-consistent storage access structure based on the data popularity of the target data includes: determining the target access medium mounted on the data storage unit based on the data popularity of the target data, wherein the target access medium has a data read / write speed corresponding to the data popularity; and caching the target data in the target access medium.

[0127] According to one or more embodiments of this disclosure, the target access medium mounted on the data storage unit includes at least one of the following: dynamic random access memory; persistent memory located locally on the non-consistent storage access structure; persistent memory located at a remote end of the non-consistent storage access structure; solid-state drive located locally on the non-consistent storage access structure; and solid-state drive located at a remote end of the non-consistent storage access structure.

[0128] According to one or more embodiments of this disclosure, when the target data is cached in the data storage unit, the method further includes: obtaining the average popularity of the cached data in a first access medium, wherein the first access medium is the current access medium for caching the target data; the step of caching the target data in the target access medium includes: when the data popularity of the target data is higher than the average popularity first threshold, migrating the target data to the target access medium, wherein the first threshold is determined at least based on the data volume of the target data.

[0129] According to one or more embodiments of this disclosure, before caching the target data into the target access medium, the method further includes: obtaining a migration time based on the amount of the target data, wherein the migration time characterizes the time taken to migrate the target data into the target access medium; caching the target data into the target access medium includes: if the migration time is less than a first duration, then caching the target data into the target access medium.

[0130] According to one or more embodiments of this disclosure, caching the target data into the target access medium includes: obtaining a data caching speed based on the operating load of the data storage unit; and caching the target data into the target access medium based on the data caching speed.

[0131] According to one or more embodiments of this disclosure, when the target data is not cached in the data storage unit, the method further includes: obtaining the migration duration of the target data, the migration duration representing the duration during which the target data is migrated out of the data storage unit; and caching the target data to a target location in the data storage unit based on the data popularity of the target data, including: caching the target data to the target location in the data storage unit when the migration duration is greater than a second duration.

[0132] According to one or more embodiments of this disclosure, the first access data includes granularity information, the granularity information representing the classification granularity of the target data; the method further includes: obtaining a data storage unit for caching the target data based on the granularity information.

[0133] According to one or more embodiments of this disclosure, it further includes at least one of the following: the first access data is generated based on a remote direct data access request or a remote procedure call request sent by the client to the cache node; the first access data includes at least one of the following: the identifier of the cache node, the identifier of the data storage unit, and the identifier of the access medium on which the data storage unit is mounted.

[0134] Secondly, according to one or more embodiments of the present disclosure, a data processing apparatus is provided, comprising:

[0135] A receiving module is configured to receive first access data sent by at least one client, wherein the first access data represents an instruction record of a remote direct data read instruction sent by the client for target data cached within a non-consistent storage access structure;

[0136] The processing module is used to obtain the data popularity of the target data based on the first access data, wherein the data popularity characterizes the frequency of the target data being accessed by remote direct data read commands;

[0137] A caching module is used to cache the target data to a target location in a data storage unit implemented based on a non-consistent storage access structure, or to migrate the target data out of the data storage unit, based on the data popularity of the target data, wherein the target location has a data read / write speed corresponding to the data popularity.

[0138] According to one or more embodiments of this disclosure, the first access data includes a first access count of the target data within a first time interval: the processing module is specifically configured to: obtain an average access count based on the first access count within the first time interval and a second access count within a second time interval, wherein the second time interval is located before the first time interval; and obtain the data popularity of the target data based on the average access count.

[0139] According to one or more embodiments of this disclosure, the second time interval includes at least one historical sub-interval, and the second access count includes the interval access count corresponding to each of the historical sub-intervals; when the processing module obtains the average access count based on the first access count in the first time interval and the second access count in the second time interval, it is specifically configured to: obtain a corresponding weighting coefficient based on the historical time corresponding to each of the historical sub-intervals, wherein the weighting coefficient is inversely proportional to the time distance from the historical time to the current time; calculate the weighted sum of the interval access counts corresponding to each of the historical sub-intervals based on the weighting coefficient of each of the historical sub-intervals to obtain the weighted access count; and obtain the average access count based on the weighted access count and the first access count.

[0140] According to one or more embodiments of this disclosure, the caching module is specifically configured to: determine the target access medium mounted on the data storage unit based on the data popularity of the target data, wherein the target access medium has a data read / write speed corresponding to the data popularity; and cache the target data in the target access medium.

[0141] According to one or more embodiments of this disclosure, the target access medium mounted on the data storage unit includes at least one of the following: dynamic random access memory; persistent memory located locally on the non-consistent storage access structure; persistent memory located at a remote end of the non-consistent storage access structure; solid-state drive located locally on the non-consistent storage access structure; and solid-state drive located at a remote end of the non-consistent storage access structure.

[0142] According to one or more embodiments of this disclosure, when the target data is cached in the data storage unit, the caching module is further configured to: obtain the average popularity of the data cached in the first access medium, wherein the first access medium is the access medium currently caching the target data; when caching the target data in the target access medium, the caching module is specifically configured to: when the data popularity of the target data is higher than the average popularity first threshold, migrate the target data to the target access medium, wherein the first threshold is determined at least based on the data volume of the target data.

[0143] According to one or more embodiments of this disclosure, before caching the target data into the target access medium, the caching module is further configured to: obtain the migration time based on the data volume of the target data, wherein the migration time characterizes the time taken to migrate the target data into the target access medium; and when caching the target data into the target access medium, the caching module is specifically configured to: if the migration time is less than a first duration, then cache the target data into the target access medium.

[0144] According to one or more embodiments of this disclosure, the caching module is specifically configured to: obtain a data caching speed based on the operating load of the data storage unit; and cache the target data in the target access medium based on the data caching speed.

[0145] According to one or more embodiments of this disclosure, when the target data is not cached in the data storage unit, the caching module is further configured to: obtain the migration duration of the target data, the migration duration representing the duration during which the target data is migrated out of the data storage unit; the caching module is specifically configured to: cache the target data to the target location of the data storage unit when the migration duration is greater than a second duration.

[0146] According to one or more embodiments of this disclosure, the first access data includes granularity information, the granularity information representing the classification granularity of the target data; the processing module is further configured to: obtain a data storage unit for caching the target data based on the granularity information.

[0147] According to one or more embodiments of this disclosure, it further includes at least one of the following: the first access data is generated based on a remote direct data access request or a remote procedure call request sent by the client to the cache node; the first access data includes at least one of the following: the identifier of the cache node, the identifier of the data storage unit, and the identifier of the access medium on which the data storage unit is mounted.

[0148] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, including: a processor, and a memory communicatively connected to the processor;

[0149] The memory stores computer-executed instructions;

[0150] The processor executes computer execution instructions stored in the memory to implement the data processing method described in the first aspect and various possible designs of the first aspect.

[0151] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the data processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0152] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the data processing method described in the first aspect and various possible designs of the first aspect.

[0153] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0154] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0155] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A data processing method, characterized in that, A cache node is applied to a distributed storage system, wherein the distributed storage system includes a distributed file system and the cache node is structured in a distributed manner, and the data storage unit inside the cache node is used to cache the original data of the distributed file system. The method includes: The system receives first access data sent by at least one client. The first access data is generated by the client based on the access record of target data in a cache node using a non-consistent storage access structure, according to the remote direct data read instruction. The first access data includes information about the cache node where the target data is located and information about the data storage unit in the cache node that stores the target data. The first access data is sent asynchronously by the at least one client to the cache node and received asynchronously by the cache node through a queue. Based on the first access data, the data popularity of the target data is obtained, and the data popularity represents the frequency with which the target data is accessed by remote direct data read commands; Based on the data popularity of the target data, the target data is cached in a target location within a data storage unit implemented based on a non-consistent storage access structure, or the target data is migrated out of the data storage unit, wherein the target location has a data read / write speed corresponding to the data popularity.

2. The method according to claim 1, characterized in that, The first access data includes the first access count of the target data within a first time interval: The step of obtaining the data popularity of the target data based on the first access data includes: The average number of visits is obtained based on the first number of visits within the first time interval and the second number of visits within the second time interval, wherein the second time interval is located before the first time interval; The data popularity of the target data is obtained based on the average number of visits.

3. The method according to claim 2, characterized in that, The second time interval includes at least one historical sub-interval, and the second access count includes the interval access count corresponding to each of the historical sub-intervals; the step of obtaining the average access count based on the first access count within the first time interval and the second access count within the second time interval includes: Based on the historical time corresponding to each of the aforementioned historical sub-intervals, a corresponding weighting coefficient is obtained, wherein the weighting coefficient is inversely proportional to the time distance from the historical time to the current time; Based on the weighting coefficients of each historical sub-interval, calculate the weighted sum of the interval access counts corresponding to each historical sub-interval to obtain the weighted access count; The average number of visits is obtained based on the weighted number of visits and the first number of visits.

4. The method according to claim 1, characterized in that, The step of caching the target data to a target location in a data storage unit implemented based on a non-consistent storage access structure, based on the data popularity of the target data, includes: Based on the data popularity of the target data, the target access medium on which the data storage unit is mounted is determined, and the target access medium has a data read / write speed corresponding to the data popularity; The target data is cached in the target access medium.

5. The method according to claim 4, characterized in that, The target access medium mounted on the data storage unit includes at least one of the following: Dynamic random access memory; Persistent memory located locally within a non-uniform storage access structure; Persistent memory located at the far end of a non-uniform memory access structure; Solid-state drives located locally on non-uniform storage access structures; Solid-state drives located at the far end of a non-uniform storage access architecture.

6. The method according to claim 4, characterized in that, When the target data is cached in the data storage unit, the method further includes: Obtain the average heat of the data cached in the first access medium, where the first access medium is the current access medium for caching the target data; The step of caching the target data in the target access medium includes: When the data popularity of the target data is higher than the average popularity first threshold, the target data is migrated to the target access medium, wherein the first threshold is determined at least based on the data volume of the target data.

7. The method according to claim 4, characterized in that, Before caching the target data into the target access medium, the method further includes: Based on the amount of the target data, the migration time is obtained, which represents the time required to migrate the target data to the target access medium. The step of caching the target data in the target access medium includes: If the migration time is less than the first duration, the target data is cached in the target access medium.

8. The method according to claim 4, characterized in that, The step of caching the target data in the target access medium includes: The data caching speed is obtained based on the operating load of the data storage unit; Based on the data caching speed, the target data is cached in the target access medium.

9. The method according to claim 1, characterized in that, When the target data is not cached in the data storage unit, the method further includes: The migration duration of the target data is obtained, wherein the migration duration represents the duration during which the target data is migrated out of the data storage unit; The step of caching the target data to the target location in the data storage unit based on the data popularity of the target data includes: If the migration duration is longer than the second duration, the target data is cached in the target location of the data storage unit.

10. The method according to claim 1, characterized in that, The first access data includes granularity information, which characterizes the classification granularity of the target data; the method further includes: Based on the granularity information, a data storage unit for caching the target data is obtained.

11. The method according to claim 1, characterized in that, It also includes at least one of the following: The first access data is generated based on the remote direct data access request or remote procedure call request sent by the client to the cache node; The first access data includes at least one of the following: The identifiers of the cache nodes, data storage units, and access media mounted on the data storage units.

12. A data processing apparatus, characterized in that, A cache node deployed in a distributed storage system, the distributed storage system including a distributed file system and the cache node with a distributed structure, wherein the data storage unit inside the cache node is used to cache the original data of the distributed file system, the device comprising: A receiving module is configured to receive first access data sent by at least one client. The first access data is generated by the client based on the access record of target data in a cache node with a non-consistent storage access structure according to the remote direct data read instruction. The first access data includes information about the cache node where the target data is located and information about the data storage unit in the cache node that stores the target data. The first access data is sent asynchronously by the at least one client to the cache node and received asynchronously by the cache node through a queue. The processing module is used to obtain the data popularity of the target data based on the first access data, wherein the data popularity characterizes the frequency of the target data being accessed by remote direct data read commands; A caching module is used to cache the target data to a target location in a data storage unit implemented based on a non-consistent storage access structure, or to migrate the target data out of the data storage unit, based on the data popularity of the target data, wherein the target location has a data read / write speed corresponding to the data popularity.

13. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the data processing method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the data processing method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the data processing method according to any one of claims 1 to 11.

Images