Data manager, data management method, electronic device and storage medium

Abstract

The present application discloses a data manager, a data management method, an electronic device and a storage medium, which relate to the field of data processing technology. A processor determines the data types of multiple data according to a preset algorithm, and a controller stores data of different data types in different memories or different partitions of the same memory according to the data types. Therefore, the problem of low efficiency of data management can be solved. While meeting the storage capacity, the access delay of data is reduced, and the efficiency of data management is improved.

Description

Technical Field

[0001] The technical field of data processing of the present application particularly relates to a data manager, a data management method, an electronic device and a storage medium. Background Art

[0002] With the rapid development of data analytics, the requirements for system storage capacity and access latency are increasing. Different types of memory have different storage characteristics. For example, solid-state drives (SSDs) offer high-capacity storage, but suffer from high access latency and a low write lifespan. Dynamic random-access memory (DRAM) offers low latency, but its physical density and cost limit its application in large-scale data processing.

[0003] In related technologies, the requirements of storage capacity and access latency cannot be met simultaneously, resulting in low efficiency of data management. Summary of the Invention

[0004] The present application provides a data manager, a data management method, an electronic device, and a storage medium to at least solve the problem of low efficiency of data management in related technologies.

[0005] The present application provides a data manager, the data manager includes a processor and a controller, the controller is connected to a first memory and a second memory respectively;

[0006] a processor configured to determine, from among a plurality of data, a plurality of first data, a plurality of second data, and a plurality of third data using a preset algorithm based on access logs within a preset time period, wherein a predicted access probability corresponding to the second data is less than a predicted access probability corresponding to the first data and greater than a predicted access probability corresponding to the third data, and the plurality of data are data stored in the first memory and the second memory;

[0007] The controller is used to store multiple first data in the first memory, store multiple second data in the cache area of ​​the second memory, perform data compression processing on the third data, store the compressed multiple third data in the compression storage area of ​​the second memory, and update the address conversion relationship.

[0008] This application also provides a data management method, including:

[0009] Get access logs within a preset period;

[0010] Perform feature extraction on access logs to obtain multi-dimensional features;

[0011] Determining a plurality of first data from a plurality of data according to a plurality of dimensional features and an access prediction model;

[0012] Based on multiple dimensional features and multiple first data, multiple second data and multiple third data are determined, wherein the predicted access probability corresponding to the second data is smaller than the predicted access probability corresponding to the first data, and greater than the predicted access probability corresponding to the third data.

[0013] The present application also provides a data management device, comprising:

[0014] The acquisition module is used to obtain access logs within a preset period;

[0015] The extraction module is used to extract features from access logs to obtain multiple dimensional features;

[0016] A first determining module, configured to determine a plurality of first data from a plurality of data according to a plurality of dimensional features and an access prediction model;

[0017] The second determination module is used to determine multiple second data and multiple third data based on multiple dimensional features and multiple first data, wherein the predicted access probability corresponding to the second data is smaller than the predicted access probability corresponding to the first data, and greater than the predicted access probability corresponding to the third data.

[0018] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any one of the above-mentioned data management methods when executing the computer program.

[0019] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data management methods are implemented.

[0020] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data management methods when executed by a processor.

[0021] Through this application, the processor determines the data types of multiple data according to a preset algorithm, and the controller stores data of different data types in different memories or different partitions of the same memory according to the data type. Therefore, the problem of low efficiency of data management can be solved. While meeting the storage capacity, the access delay of data is reduced, and the efficiency of data management is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 A schematic diagram of the structure of a data manager provided in an embodiment of the present application;

[0024] Figure 2 A flowchart of a data management method provided in an embodiment of the present application;

[0025] Figure 3 A schematic diagram of a second threshold adjustment curve provided in an embodiment of the present application;

[0026] Figure 4 A flowchart of a model incremental training method provided in an embodiment of the present application;

[0027] Figure 5 A schematic diagram of the structure of a data management device provided in an embodiment of the present application;

[0028] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0029] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0030] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0031] With the rapid development of data analytics, the requirements for system storage capacity and access latency are increasing. Different types of memory have different storage characteristics. For example, solid-state drives (SSDs) offer high-capacity storage, but suffer from high access latency and a low write lifespan. Dynamic random-access memory (DRAM) offers low latency, but its physical density and cost limit its application in large-scale data processing.

[0032] In related technologies, the requirements of storage capacity and access latency cannot be met simultaneously, resulting in low efficiency of data management.

[0033] To address the above technical issues, an embodiment of the present application provides a data manager, comprising a processor and a controller, wherein the controller is connected to a first memory and a second memory, respectively. The processor determines the data types of multiple data items according to a preset algorithm, and the controller stores data of different data types in different memories or in different partitions of the same memory, based on the data types. In this way, the controller intelligently classifies, stores, and manages data, thereby reducing data access latency while meeting storage capacity requirements and improving data management efficiency.

[0034] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0035] Example 1

[0036] Figure 1 This is a structural diagram of a data manager provided in an embodiment of the present application. Figure 1 , Figure 1 At least the data manager is included.

[0037] The data manager includes a processor and a controller, and the controller is connected to the first memory and the second memory respectively;

[0038] a processor configured to determine, from among a plurality of data, a plurality of first data, a plurality of second data, and a plurality of third data using a preset algorithm based on access logs within a preset time period, wherein a predicted access probability corresponding to the second data is less than a predicted access probability corresponding to the first data and greater than a predicted access probability corresponding to the third data, and the plurality of data are data stored in the first memory and the second memory;

[0039] The controller is used to store multiple first data in the first memory, store multiple second data in the cache area of ​​the second memory, perform data compression processing on the third data, store the compressed multiple third data in the compression storage area of ​​the second memory, and update the address conversion relationship.

[0040] Among them, the data manager can be used to manage and optimize data storage and access.

[0041] The processor may be configured to classify and analyze the plurality of data stored in the first memory and the second memory to determine a storage strategy for the plurality of data.

[0042] The processor may be a central processing unit or other types of processors, which are not limited here.

[0043] The controller may be used to manage specific operations of the first memory and the second memory, and the specific operations may include reading, writing, and storage location allocation of data.

[0044] The controller may be a field programmable gate array or other types of controllers, which are not limited here.

[0045] The controller may include an address translation engine and a compression engine.

[0046] The address translation engine can be used to translate between virtual addresses and physical addresses.

[0047] In the operating system, virtual addresses can be used to manage data, and in hardware devices, the physical addresses corresponding to the virtual addresses can be used to access the actual physical locations.

[0048] The address translation engine may include a translation lookaside buffer (TLB).

[0049] The TLB can be used to store address translation relationships to quickly find the mapping relationship between virtual addresses and physical addresses.

[0050] TLB can simultaneously store 1024 virtual address to physical address mapping relationships, greatly improving the hit rate of address translation and reducing the number of page table accesses.

[0051] TLB can set up a multi-stage pipeline architecture, which can process different tasks in parallel to increase the speed of address conversion and reduce latency.

[0052] TLB can be used to support direct conversion of virtual addresses to physical addresses, avoiding the overhead of page table traversal and improving the processing speed and efficiency of address translation.

[0053] The TLB and pipeline architecture significantly improves the speed of address translation and reduces latency, with mapping latency reaching 95 nanoseconds.

[0054] The compression engine can be used to compress and decompress data.

[0055] Data compression processing can significantly reduce the storage space occupied by data to improve storage efficiency and transmission speed.

[0056] Data compression processing can be achieved through a preset compression algorithm.

[0057] The preset compression algorithm can be implemented through a dedicated hardware circuit to realize the hardware of the preset compression algorithm, and the parallel processing capability of the hardware is utilized to improve the compression efficiency.

[0058] The compression engine can set up a streaming processing architecture to support real-time data compression and decompression.

[0059] The first memory may be a high-performance, low-latency storage device, which is not limited here.

[0060] The first memory may be used to store data with a high access probability, which requires a fast access speed to meet the high performance requirements of the system.

[0061] The second memory may be a storage device with a larger capacity and a slower reading and writing speed than the first memory, which is not limited here.

[0062] The second memory may include a buffer area and a compression storage area.

[0063] The second memory may be used to store data with a medium access probability, which still requires a relatively fast access speed.

[0064] The second memory may also be used to store compressed data, which has a low access probability and can save storage space through compression.

[0065] The second memory can reduce the actual write volume through data compression processing, thereby increasing the number of full disk writes per day.

[0066] The second memory can monitor the wear status of the storage unit in real time and dynamically adjust the data writing location to evenly distribute the write operation to all available units to avoid local excessive wear.

[0067] Access logs can be used to record access information for each data within a preset period of time.

[0068] The preset period may be a preset time range, for example, the preset period is the past 60 seconds.

[0069] Pre-set algorithms can be used to analyze access logs and classify data according to specific logic.

[0070] The first data may be data with a high access probability and may be stored in a high-performance first memory to ensure fast access.

[0071] The second data may be data with a medium access probability, between the first data and the third data, and may be stored in a storage device with slightly lower performance but lower cost to ensure faster access.

[0072] The third data may be data with a low access probability and may be compressed and stored to save space.

[0073] Since access logs change dynamically, the data type is also adjusted dynamically, allowing data management to adapt to different application scenarios and access modes, thereby improving data management efficiency.

[0074] For example, Table 1 can be used to illustrate multiple data:

[0075] Table 1

[0076]

[0077] In one possible implementation, the preset algorithm includes accessing a prediction model, and the processor is specifically configured to:

[0078] Perform feature extraction on access logs to obtain multi-dimensional features, including the number of visits and survival time.

[0079] determining a plurality of first data according to the number of visits and the visit prediction model;

[0080] Determining multiple third data based on the survival time;

[0081] A plurality of second data is determined based on the plurality of first data and the plurality of third data.

[0082] The access prediction model may be a pre-trained learning model, which is not limited here.

[0083] The access log may include the following information: virtual address and physical address, access type (read / write), timestamp, thread ID, process ID, processor ID, etc.

[0084] Multiple dimensional features may include temporal locality, spatial correlation, thread affinity, access intensity, number of accesses, survival time, access time interval, access duration, data size, data type, data access pattern, data access heat change rate, processor utilization, load rate, etc., which are not limited here.

[0085] The plurality of first data may be determined based on the number of accesses to the data, or may be determined based on the output result of an access prediction model.

[0086] The plurality of third data may be determined according to the survival time of the data.

[0087] The plurality of second data may be data other than the plurality of first data and the plurality of third data.

[0088] In a possible implementation, the processor is further configured to send an operation instruction to the controller, where the operation instruction includes a virtual address;

[0089] The controller is further configured to determine the physical address corresponding to the virtual address according to the address conversion relationship, and transmit the data corresponding to the physical address to the processor through a preset transmission mode.

[0090] The operation instruction may be a write operation, a read operation, etc., which is not limited here.

[0091] The preset transmission mode may be a transmission mode that allows the hardware subsystem to directly access the system.

[0092] The controller can receive the virtual address sent by the processor and query whether the virtual address exists in the address translation relationship. If so, the physical address is directly determined. If not, the page table is accessed and the address translation relationship is updated. The data corresponding to the physical address is transmitted to the processor through a preset transmission mode.

[0093] The address translation relationship can significantly reduce the number of page table accesses, reduce latency, reduce the CPU burden through the preset transmission mode, and improve the overall performance of the system.

[0094] In one possible implementation, the controller is specifically configured to:

[0095] Segmenting the plurality of third data to obtain a plurality of data blocks;

[0096] Performing dictionary encoding and compression processing on the plurality of data blocks by a target compression algorithm to obtain a plurality of compressed data blocks;

[0097] A plurality of compressed data blocks are determined as a plurality of compressed third data.

[0098] The plurality of third data may be segmented and processed according to the order of their timestamps.

[0099] Dictionary encoding can reduce the amount of data by finding repeated strings or patterns and replacing them with shorter symbols or indices.

[0100] Compression processing can be a process of further reducing data redundancy based on dictionary encoding.

[0101] This not only saves storage space, but also supports efficient decompression and data recovery, and is suitable for data storage scenarios with low access frequency.

[0102] Example 2

[0103] Figure 2 This is a flow chart of a data management method provided in an embodiment of the present application. The execution subject of the embodiment of the present application may be a processor. The processor may be implemented by software or by a combination of software and hardware. Figure 2 , the method comprising:

[0104] S201: Obtain access logs within a preset time period.

[0105] You can determine a preset time period and target object, and use the log collection tool to obtain the access logs of the target object within the preset time period.

[0106] The target object can be any one or more of a server, database, operating system, network device, and application program.

[0107] Optionally, the processor may be provided with a kernel module to implement monitoring of access logs and control of the controller so that the controller can perform direct memory access.

[0108] S202: Perform feature extraction on the access log to obtain multiple dimensional features.

[0109] Multiple files in the access log can be parsed and converted into log data in a preset data format. The log data can be processed by feature extraction algorithm to obtain multiple dimensional features.

[0110] The feature extraction algorithm may be a preset algorithm and is not limited here.

[0111] S203. Determine a plurality of first data from a plurality of data according to a plurality of dimensional features and an access prediction model.

[0112] Optionally, multiple dimensional features may be input into the access prediction model to determine multiple first data from the multiple data.

[0113] Optionally, multiple dimensional features include access frequency and processor utilization, and the multiple dimensional features can be input into the access prediction model to obtain multiple prediction data; the multiple prediction data are determined as multiple first data, and the multiple data other than the multiple prediction data are determined as multiple first processed data; for any first processed data, it is determined whether the access frequency corresponding to the first processed data is greater than the second threshold, and if so, the first processed data is determined as the first data.

[0114] The predicted data is data with an access probability greater than a first threshold within a preset time period.

[0115] The second threshold may be determined based on processor utilization.

[0116] Optionally, the second threshold may be determined by the following formula:

[0117] HOT_THRESH = 1000 × (1 + CPU_util / 100)

[0118] Here, HOT_THRESH may represent the second threshold, and CPU_util may represent processor utilization.

[0119] The second threshold can be adjusted dynamically through this formula. Whenever the processor utilization increases by 1%, the threshold is increased by 1%.

[0120] Next, combine Figure 3 , the second threshold is illustrated with an example.

[0121] Figure 3 This is a schematic diagram of a second threshold adjustment curve provided in an embodiment of the present application. Figure 3 , Figure 3 A graph of the relationship between processor utilization and a second threshold is included.

[0122] The X-axis of the relationship graph may represent the processor utilization, and the Y-axis may represent the second threshold.

[0123] In this way, more hot data can be retained when the load is low, and high-frequency access data can be prioritized when the load is high.

[0124] Optionally, when determining multiple first data, the survival time corresponding to each first data is set to 0.

[0125] It should be noted that multiple first data can be determined according to any feasible implementation method, and the embodiments of the present application are not limited to this.

[0126] S204: Determine multiple second data and multiple third data based on multiple dimensional features and multiple first data.

[0127] The predicted access probability corresponding to the second data is smaller than the predicted access probability corresponding to the first data, and is larger than the predicted access probability corresponding to the third data.

[0128] Optionally, data other than the multiple first data can be determined as multiple second processed data, and the multiple dimensional features corresponding to the multiple second processed data can be input into the cold data prediction model to determine multiple third data, and multiple second data can be determined based on the multiple third data and the multiple first data.

[0129] The cold data prediction model may be a pre-trained model, which is not limited here.

[0130] Optionally, the multiple dimensional features also include survival time. Among the multiple data, the data other than the multiple first data are determined as multiple second processed data; for any second processed data, it is determined whether the survival time corresponding to the second processed data is greater than a third threshold. If so, the second processed data is determined as the third data; the data other than the multiple first data and the multiple third data are determined as multiple second data.

[0131] The third threshold is determined according to the load rate of the second memory.

[0132] Optionally, the third threshold may be determined by the following formula:

[0133] AGE_MAX = 5000 / (1 + IOPS / 10000)

[0134] Here, AGE_MAX may represent the third threshold, and IOPS may represent the load rate.

[0135] In this way, when the memory load rate is higher, the third data retention time is shorter, thereby improving data management efficiency.

[0136] The implementation content of each step in the embodiment of the present application can refer to the description of the above embodiment 1, and the repeated content will not be repeated.

[0137] This embodiment provides a data management method that obtains access logs within a preset time period; performs feature extraction on the access logs to obtain multiple dimensional features; determines multiple first data items from multiple data items based on the multiple dimensional features and an access prediction model; and determines multiple second data items and multiple third data items based on the multiple dimensional features and the multiple first data items, wherein the predicted access probability corresponding to the second data item is less than the predicted access probability corresponding to the first data item and greater than the predicted access probability corresponding to the third data item. In this way, a controller intelligently classifies, stores, and manages data, reducing data access latency while meeting storage capacity requirements and improving data management efficiency.

[0138] In one possible implementation, at least one incremental training step is performed on the access prediction model to determine whether the access prediction model converges; if so, the converged access prediction model is determined as the optimized access prediction model; if not, the incremental training step is repeated until the access prediction model converges.

[0139] In which, at least one incremental training step may be performed on the access prediction model at intervals of the first duration.

[0140] The convergence condition may be a pre-set condition.

[0141] For example, the convergence condition is that the accuracy of the access prediction model reaches a threshold.

[0142] Next, combine Figure 4 , explaining the process of incremental training steps.

[0143] Figure 4 A flow chart of a model incremental training method provided in the embodiment of the present application. Based on the above embodiment, please refer to Figure 4 , the method comprising:

[0144] S401. Determine a plurality of training data according to the access log.

[0145] The training data includes multiple dimensional features and data labels within a preset time length.

[0146] Data labels can represent the actual access situation of each training data.

[0147] Among them, multiple training data can be divided into training set, validation set, and test set according to 8:1:1.

[0148] Feature extraction can be performed on the access log to determine multiple training data.

[0149] S402: Input multiple training data into the access prediction model, perform forward propagation, and obtain multiple model output values.

[0150] The multiple model output values ​​are multiple prediction data output by the access prediction model.

[0151] Accessing the network structure of the prediction model can include:

[0152] ① Input layer: input multiple dimensional features and preset duration.

[0153] ②The first long short-term memory layer: consists of 128 units, the activation function is the hyperbolic tangent function, and returns the complete sequence.

[0154] ③ Regularization layer: dropout rate 0.2 to prevent overfitting.

[0155] ④ The second long short-term memory layer: consists of 128 units, the activation function is the hyperbolic tangent function, and returns the final output.

[0156] ⑤Fully connected layer: The output dimension is N, where N is the N data that may be accessed within the preset time period in the future.

[0157] S403. Determine a loss value of a loss function based on multiple model output values ​​and multiple data labels, and determine a gradient value of an access prediction model based on the loss value.

[0158] Among them, the loss function can be a classification task function or a regression task function.

[0159] For example, the classification task function can be the cross entropy loss function, and the regression task function can be the mean squared error.

[0160] The training data can be input into the model to obtain the output value of the model. According to the selected loss function, the model output value and the data label are substituted into the loss function to calculate the loss value. Based on the loss value, the gradient value of the loss function with respect to each parameter is calculated through the back propagation algorithm.

[0161] S404: Determine update parameters of the access prediction model through the optimizer according to the gradient value, and update the access prediction model according to the update parameters.

[0162] The preset learning rate of the optimizer can be 0.001, and the batch size can be 256, which are not limited here.

[0163] The optimizer can calculate the updated value of each parameter based on the gradient value and the preset learning rate, and update the access prediction model based on the updated parameters.

[0164] Elastic Weight Consolidation (EWC) can be used to prevent catastrophic forgetting when accessing prediction models.

[0165] The implementation content of each step in the embodiment of the present application can refer to the description of the corresponding steps or operations in the above method embodiment, and repeated content will not be repeated.

[0166] This embodiment provides a model incremental training method, which determines whether the access prediction model has converged by performing at least one incremental training step on the access prediction model; if so, the converged access prediction model is determined as the optimized access prediction model; if not, the incremental training step is repeated until the access prediction model converges; wherein the incremental training step includes: determining multiple training data based on the access log, the training data including multiple dimensional features and data labels within a preset time length; inputting the multiple training data into the access prediction model, performing forward propagation, and obtaining multiple model output values; determining the loss value of the loss function based on the multiple model output values ​​and the multiple data labels, and determining the gradient value of the access prediction model based on the loss value; determining the update parameters of the access prediction model through the optimizer based on the gradient value, and updating the access prediction model based on the updated parameters. In this way, the access prediction model can continuously optimize its own parameters based on the latest access log data, thereby improving prediction accuracy.

[0167] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0168] Figure 5 This is a structural diagram of a data management device provided in an embodiment of the present application. Figure 5 The data management device 500 includes an acquisition module 501, an extraction module 502, a first determination module 503 and a second determination module 504, wherein:

[0169] Acquisition module 501, used to obtain access logs within a preset period;

[0170] Extraction module 502, used to perform feature extraction processing on the access log to obtain multiple dimensional features;

[0171] A first determining module 503 is configured to determine a plurality of first data from a plurality of data according to a plurality of dimensional features and an access prediction model;

[0172] The second determination module 504 is used to determine multiple second data and multiple third data based on multiple dimensional features and multiple first data, wherein the predicted access probability corresponding to the second data is smaller than the predicted access probability corresponding to the first data and greater than the predicted access probability corresponding to the third data.

[0173] In a possible implementation, the multiple dimensional features include access frequency and processor utilization, and the first determining module 503 is specifically configured to:

[0174] Inputting multiple dimensional features into an access prediction model to obtain multiple prediction data, where the prediction data is data with an access probability greater than a first threshold within a preset time period;

[0175] determining the plurality of predicted data as a plurality of first data, and determining a plurality of data other than the plurality of predicted data as a plurality of first processed data;

[0176] For any first processed data, it is determined whether the access frequency corresponding to the first processed data is greater than a second threshold. If so, the first processed data is determined as the first data. The second threshold is determined according to the processor utilization.

[0177] In a possible implementation, the apparatus further includes a training module 505, which is configured to:

[0178] Perform at least one incremental training step on the access prediction model to determine whether the access prediction model has converged;

[0179] If so, the converged access prediction model is determined as the optimized access prediction model;

[0180] If not, the incremental training steps are repeated until the access prediction model converges;

[0181] The incremental training steps include:

[0182] Determine multiple training data based on the access logs. The training data includes multiple dimensional features and data labels within a preset time period.

[0183] Input multiple training data into the access prediction model, perform forward propagation, and obtain multiple model output values;

[0184] Determine the loss value of the loss function based on multiple model output values ​​and multiple data labels, and determine the gradient value of the access prediction model based on the loss value;

[0185] According to the gradient value, the update parameters of the access prediction model are determined by the optimizer, and the access prediction model is updated according to the updated parameters.

[0186] In a possible implementation, the multiple dimensional features further include survival time, and the second determining module 504 is specifically configured to:

[0187] Determining data other than the plurality of first data among the plurality of data as a plurality of second processed data;

[0188] For any second processed data, determining whether the survival time corresponding to the second processed data is greater than a third threshold; if so, determining the second processed data as the third data, where the third threshold is determined according to the load rate of the second memory;

[0189] Data other than the plurality of first data and the plurality of third data are determined as a plurality of second data.

[0190] For the description of the features in the embodiment corresponding to the data management device, reference can be made to the relevant description of the embodiment corresponding to the data management method, which will not be repeated here.

[0191] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device 60 provided in this embodiment includes: at least one processor 601 and a memory 602. Optionally, the electronic device 60 further includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus.

[0192] During the specific implementation process, at least one processor 601 executes the computer-executable instructions stored in the memory 602, so that the at least one processor 601 executes the above-mentioned data management method embodiment.

[0193] The specific implementation process of the processor 601 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0194] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the application may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0195] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0196] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0197] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned data management method embodiments when running.

[0198] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0199] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above data management method embodiments are implemented.

[0200] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned data management method embodiments are implemented.

[0201] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0202] The above is a detailed introduction to a data management method provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core ideas of the present application. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, several improvements and modifications may be made to the present application, and such improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data manager, characterized in that: The data manager includes a processor and a controller, wherein the controller is connected to the first memory and the second memory respectively; The processor is configured to determine, from a plurality of data, a plurality of first data, a plurality of second data, and a plurality of third data using a preset algorithm based on access logs within a preset time period, wherein a predicted access probability corresponding to the second data is smaller than the predicted access probability corresponding to the first data and greater than the predicted access probability corresponding to the third data, and the plurality of data are data stored in the first memory and the second memory; The controller is configured to store the plurality of first data in the first memory, store the plurality of second data in a cache area of ​​the second memory, perform data compression processing on the third data, store the compressed plurality of third data in a compression storage area of ​​the second memory, and update the address translation relationship; The preset algorithm includes an access prediction model, and the processor is specifically configured to: Performing feature extraction on the access log to obtain multiple dimensional features, wherein the multiple dimensional features include the number of accesses and the survival time; Determining a plurality of first data according to the number of accesses and the access prediction model, wherein the access prediction model is used to predict access probabilities of respective data within a preset time period in the future; determining the plurality of third data according to the survival time; The plurality of second data are determined based on the plurality of first data and the plurality of third data.

2. The data manager according to claim 1, wherein: The processor is further configured to send an operation instruction to the controller, wherein the operation instruction includes a virtual address; The controller is further configured to determine a physical address corresponding to the virtual address according to the address conversion relationship, and transmit the data corresponding to the physical address to the processor through a preset transmission mode.

3. The data manager according to claim 1, wherein: The controller is specifically used for: Segmenting the plurality of third data to obtain a plurality of data blocks; Performing dictionary encoding and compression processing on the plurality of data blocks using a target compression algorithm to obtain a plurality of compressed data blocks; The plurality of compressed data blocks are determined as a plurality of compressed third data.

4. A data management method, characterized in that: include: Get access logs within a preset period; Performing feature extraction processing on the access log to obtain multiple dimensional features; Determining a plurality of first data from a plurality of data based on the plurality of dimensional features and an access prediction model; the access prediction model is used to predict access probabilities of each data within a preset time period in the future; Determining, based on the multiple dimensional features and the multiple first data, multiple second data and multiple third data, wherein the predicted access probability corresponding to the second data is smaller than the predicted access probability corresponding to the first data and larger than the predicted access probability corresponding to the third data; The multiple dimensional features further include survival time. Determining multiple second data and multiple third data based on the multiple dimensional features and the multiple first data includes: Determining data other than the plurality of first data among the plurality of data as a plurality of second processed data; For any second processed data, determining whether the survival time corresponding to the second processed data is greater than a third threshold; if so, determining the second processed data as third data, where the third threshold is determined according to a load rate of the second memory; Data other than the plurality of first data and the plurality of third data are determined as a plurality of second data.

5. The method according to claim 4, characterized in that The multiple dimensional features include access frequency and processor utilization, and determining multiple first data from multiple data based on the multiple dimensional features and the access prediction model includes: Inputting the multiple dimensional features into the access prediction model to obtain multiple prediction data, wherein the prediction data is data with an access probability greater than a first threshold within a preset time period; Determining the plurality of predicted data as a plurality of first data, and determining a plurality of data other than the plurality of predicted data as a plurality of first processed data; For any first processed data, it is determined whether the access frequency corresponding to the first processed data is greater than a second threshold. If so, the first processed data is determined as the first data. The second threshold is determined according to the processor utilization.

6. The method according to claim 5, characterized in that The method further comprises: Performing at least one incremental training step on the access prediction model to determine whether the access prediction model has converged; If so, the converged access prediction model is determined as the optimized access prediction model; If not, repeating the incremental training step until the access prediction model converges; Wherein, the incremental training step includes: Determining a plurality of training data according to the access log, wherein the training data includes a plurality of dimensional features and data labels within a preset time period; Inputting the plurality of training data into the access prediction model, performing forward propagation, and obtaining a plurality of model output values; Determining a loss value of a loss function according to the multiple model output values ​​and the multiple data labels, and determining a gradient value of the access prediction model according to the loss value; According to the gradient value, an optimizer determines an update parameter of the access prediction model, and updates the access prediction model according to the update parameter.

7. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the data management method according to any one of claims 4 to 6 when executing the computer program.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data management method according to any one of claims 4 to 6.

Images