Data processing method, device, electronic device and storage medium

Abstract

The present application provides a data processing method, which includes: obtaining an activity parameter value of a current file; the activity parameter value is related to at least two different activity indicators of the current file; comparing the activity parameter value with a threshold value; and determining a processing method for the current file based on the comparison result, wherein the processing method represents loading the current file into a cache system or deleting the current file from the cache system. The present application also provides a data processing device, an electronic device, and a storage medium. It can accurately determine whether the current file is cold data or hot data, thereby improving the efficiency of file preloading.

Description

Technical Field

[0001] The present invention relates to data processing technology, and in particular to a data processing method, device, electronic equipment and storage medium. Background Art

[0002] In distributed caching systems like Alluxio, high-performance caches can increase file read and write rates, thereby improving overall system performance. However, high-performance caches are expensive and are not typically deployed in large quantities in typical systems. Limited cache capacity means that the most useful data must be stored in high-performance caches. To address this, the industry has developed the concept of hot and cold data, defining frequently accessed data as hot data and rarely accessed data as cold data. However, existing solutions cannot preload current files into the Alluxio cache in a timely manner. Summary of the Invention

[0003] In order to solve the existing technical problems, the technical solution of this application is implemented as follows:

[0004] According to one aspect of the present application, a data processing method is provided, the method comprising:

[0005] Obtaining an activity parameter value of a current file; the activity parameter value is related to at least two different activity indicators of the current file;

[0006] Comparing the activity parameter value with a threshold;

[0007] A processing method for the current file is determined according to the comparison result, where the processing method represents loading the current file into a cache system or deleting the current file from the cache system.

[0008] In the above solution, the threshold includes a first threshold and a second threshold, and the second threshold is greater than the first threshold;

[0009] Determining a processing method for the current file based on the comparison result includes at least one of the following:

[0010] Determining, based on the comparison result, that the activity parameter value is less than or equal to the first threshold, deleting the current file from the cache system;

[0011] Determining, based on the comparison result, that the activity parameter value is between the first threshold value and the second threshold value, and maintaining the storage state of the current file unchanged;

[0012] According to the comparison result, it is determined that the activity parameter value is greater than or equal to the second threshold, and the current file is added to the cache system.

[0013] In the above solution, deleting the current file from the cache system includes:

[0014] The current file is moved from a current cache layer in the cache system to a next-level cache layer in the cache system.

[0015] In the above solution, adding the current file to the cache system includes:

[0016] The current file is moved from the current cache layer of the cache system to the upper cache layer of the cache system.

[0017] In the above solution, there are multiple current files;

[0018] Adding the current file to the cache system includes:

[0019] Determining the current remaining storage capacity of the cache system;

[0020] If the remaining storage capacity is less than the capacity threshold, sorting multiple activity parameter values ​​of multiple current files;

[0021] The target file with the highest activity parameter value is loaded into the cache system according to the sorting result.

[0022] In the above solution, obtaining the activity parameter value of the current file includes:

[0023] Scanning a current file based on a file activity instruction to determine an activity factor of the current file and a time-dependent difference parameter of the activity of the current file;

[0024] An activity parameter value of the current file is determined based on the activity factor and the differential parameter.

[0025] In the above solution, determining the activity factor of the current file includes:

[0026] Obtaining multiple activity indicators related to the current file in the cache system, the multiple activity indicators including at least two of the following: a number of sub-files stored in the cache system in the same directory as the current file, a number of other files loaded into the cache system simultaneously with the current file, a number of historical accesses to the current file, a generation time of the current file, and a temperature attribute of the current file;

[0027] Perform a function activation operation on the multiple activity indicators to obtain an activity factor of the current file.

[0028] In the above solution, the determining of the activity parameter value of the current file based on the activity factor and the differential parameter further includes:

[0029] Get the custom activity value of the current file;

[0030] An activity parameter value of the current file is determined based on the activity factor, the differential parameter, and the custom activity value.

[0031] In the above solution, the method further includes:

[0032] Update the activity parameter value of the current file according to a preset time interval.

[0033] According to another aspect of the present application, a data processing device is provided, characterized in that the data processing device includes:

[0034] an acquiring unit, configured to acquire an activity parameter value of a current file; the activity parameter value being related to at least two different activity indicators of the current file;

[0035] a comparing unit, configured to compare the activity parameter value with a threshold;

[0036] The determining unit is configured to determine a processing method for the current file according to the comparison result, where the processing method represents loading the current file into a cache system or deleting the current file from the cache system.

[0037] According to a third aspect of the present application, an electronic device is provided, characterized in that the electronic device includes: a processor and a memory for storing a computer program that can be run on the processor,

[0038] Wherein, when the processor is used to run the computer program, it executes any method step of any of the above-mentioned data processing methods.

[0039] According to a fourth aspect of the present application, a computer-readable storage medium is provided, characterized in that it is used to store computer instructions, and when the computer instructions are executed by a processor, any method step in any of the above-mentioned data processing methods is executed.

[0040] The data processing method, apparatus, electronic device, and storage medium provided by embodiments of the present invention determine an activity parameter value for the current file based on at least two different activity indicators of the current file, and determine a processing method for the current file based on a comparison between the activity parameter value and a threshold. This allows for rapid and accurate determination of whether the current file is cold data or hot data, thereby improving file preloading efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a schematic diagram of the data processing method in this application. Figure 1 ;

[0042] Figure 2 This is a schematic diagram of the Alluxio system architecture used in this application.

[0043] Figure 3 This is a schematic diagram of the data processing method in this application. Figure 2 ;

[0044] Figure 4 This is a schematic diagram of the structural composition of the data storage device in this application;

[0045] Figure 5 This is a schematic diagram of the structural composition of the electronic device in this application. DETAILED DESCRIPTION

[0046] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0047] Figure 1 This is a schematic diagram of the data processing method in this application. Figure 1 , this method can be applied to electronic devices installed with an open source distributed cache system (Alluxio), including but not limited to servers and personal computers (PCs). Figure 1 As shown, the method includes:

[0048] Step 101: Obtain an activity parameter value of a current file; the activity parameter value is related to at least two different activity indicators of the current file;

[0049] In this application, the Alluxio system's master server (Alluxio Master node) periodically sends activity instructions to the electronic device to scan files in the underlying storage system. Based on the activity instructions, the electronic device scans the files in the underlying storage system to obtain an activity parameter value for the current file. The activity parameter is related to at least two different activity indicators for the current file.

[0050] Of course, the master server (Alluxio Master node) in the Alluxio system can also periodically scan files in different cache layers of the Alluxio system to obtain the activity parameter values ​​of the files.

[0051] Here, the activity metrics of the current file include, but are not limited to: the number of sub-files stored in the cache system (Alluxio) in the same directory as the current file, the number of other files loaded into the cache system simultaneously with the current file, the historical access count of the current file, the generation time of the current file, and the temperature attribute of the current file.

[0052] In this application, when the electronic device scans the underlying files based on the activity instruction of the current file, it can also determine the activity factor of the current file and the differential parameter of the activity of the current file over time. Based on this activity factor and this differential parameter, the activity parameter value of the current file can be determined.

[0053] In this application, when the electronic device determines the activity factor of the current file, it can obtain multiple activity metrics related to the current file in the cache system and perform a function activation operation on these multiple activity metrics to obtain the activity factor of the current file.

[0054] Here, the multiple activity metrics include, but are not limited to, at least two of the number of sub-files stored in the cache system in the same directory as the current file, the number of other files loaded into the cache system simultaneously with the current file, the historical access count of the current file, the generation time of the current file, and the temperature attribute of the current file.

[0055] This application defines the activity of the current file based on at least two activity metrics of the current file, which can reduce the misjudgment probability of evicting files from the cache system and also improve the efficiency of preloading files into the cache system.

[0056] In this application, the electronic device can also obtain the custom activity value of the current file; based on this activity factor, this differential parameter, and this custom activity value, the activity parameter value of the current file is determined.

[0057] Here, the file activity ζ is determined by the following formula (1):

[0058] ζ t = α t + βd t (ζ) + m (1);

[0059] Where, α t is the file activity factor, which needs to be calculated based on parameters such as the own situation and access time of the current file, d t (ζ) is the difference term of the file activity over time, β is the difference term coefficient, and m is the user-defined activity.

[0060] Here, the current file's own situation includes but is not limited to the number of times the file is accessed, the file generation time, the time the file is loaded into the cache system, the number of other files loaded into the cache system at the same time, and the number of files loaded into the cache system in the same directory.

[0061] In this application, the differential term d is determined by the following formula (2): t (ζ):

[0062] d t (ζ) = ζ (t-1) -ζ (t-2) (two);

[0063] Here, t-1 represents the file from the current moment to the next moment, and t-2 represents the file from the next moment to the next moment.

[0064] Here, by introducing the time-varying differential of file activity, we can increase the rate of change of file activity over time. When file activity shows a significant decline, introducing the differential term will accelerate the decline. User-defined activity allows users to manually set the activity level of a file. When the user sets m to greater than the threshold Γ, the file will be permanently identified as hot data.

[0065] Here, α is defined by the following formula (3): t :

[0066] (three);

[0067] Among them, α t for The result after the sum is transformed by the sigmoid function. After the sigmoid function transformation, α can be guaranteed t The value of is in the range of (0, 1), which makes it easy for users to customize the activity threshold.

[0068] Here, for , which can be defined as:

[0069] (Four);

[0070] in, This reflects the impact of other files in the same directory on the file activity. a1 is the proportional coefficient used for user customization. The proportion of the file activity factor, f cache1 is the number of files in the same directory as the file that are loaded into the cache system, f total1 The total number of files in the same directory as the file.

[0071] (five);

[0072] in, This reflects the impact of other files loaded into the cache system at the same time as the file on the file activity. Is the proportional coefficient, used for user customization The proportion in the file activity factor, The number of files uploaded to the underlying storage system at the same time as this file and loaded into the cache system. The total number of files uploaded to the underlying storage system at the same time as this file.

[0073] (six);

[0074] in, It reflects the impact of the number of times a file is accessed on the file activity. Is the proportional coefficient, used for user customization The proportion in the file activity factor, The number of times the file has been accessed historically.

[0075] (seven);

[0076] in, This reflects the impact of file creation time on file activity. Is the proportional coefficient, used for user customization The proportion in the file activity factor, The time when the file is uploaded to the underlying storage system, also known as the file generation time.

[0077] (eight);

[0078] in, Reflects the temperature attribute of the file. Among them, a5 is the proportional coefficient for user customization The weight of the file activity factor. R is a random number between 0 and 1, b is the set offset value, and t is the time interval between the last time the file was evicted from the cache and the current time. For files that have never been loaded, t is the time interval between the time the file was uploaded and the current time. μ is the time coefficient.

[0079] Introduction The purpose of R is to add a concept of activity that decreases over time. With the introduction of R, a file can be active or inactive at a given moment, but the probability of it being active decreases with time t. t represents the time since the file was evicted from the cache. This allows files to retain a temperature-like property after being evicted, which gradually decreases over time. A higher temperature increases the likelihood that a file will be reloaded into the cache.

[0080] This application defines file activity, which we can use to determine whether to load files into the cache system. File activity is not only related to the file's own properties, but also closely related to files in the same directory and uploaded in the same batch. At the same time, by introducing a model that gradually cools down over time, file activity is made more realistic in real-world scenarios.

[0081] Step 102: Compare the activity parameter value with a threshold;

[0082] Here, the threshold includes a first threshold and a second threshold, and the second threshold is greater than the first threshold;

[0083] In the present application, the electronic device may compare the activity parameter value of the file with the first threshold and the second threshold respectively.

[0084] Step 103 : determining a processing method for the current file according to the comparison result, wherein the processing method represents loading the current file into a cache system or deleting the current file from the cache system.

[0085] In one implementation of the present application, when the electronic device determines, based on the comparison result, that the activity parameter value is less than or equal to the first threshold, the electronic device may delete the current file from the cache system.

[0086] The Alluxio system can have multiple cache layers. If the current file's cache layer is not the lowest cache layer, the electronic device can move the current file from the current cache layer to the next cache layer. If the current file's cache layer is the lowest cache layer, the electronic device can directly delete the current file from the cache system.

[0087] In another implementation of the present application, when the electronic device determines, according to the comparison result, that the activity parameter value is between the first threshold value and the second threshold value, the electronic device may keep the storage state of the current file unchanged;

[0088] In a third implementation of the present application, when the electronic device determines, based on the comparison result, that the activity parameter value is greater than or equal to the second threshold, it may also add the current file to the cache system.

[0089] Here, adding the current file to the cache system may refer to preloading the file stored in the underlying storage system into the cache system.

[0090] Here, when the current file is already in the cache system, the electronic device may also move the current file from the current cache layer in the cache system to the upper cache layer of the cache system when the cache layer where the current file is located is not the highest cache layer.

[0091] Figure 2 This is a schematic diagram of the Alluxio system architecture in this application; Figure 2 As shown, Alluxio system 201 is configured with three cache layers: L0, L1, and L2. Beneath Alluxio system 201 is an underlying storage system 202, which can be object storage or an open source distributed file system (HDFS).

[0092] Here, the Alluxio system 201 periodically scans files stored in each cache layer and the underlying storage system 202, and calculates the file activity of each file. For example, taking the files in the L1 layer as an example, assuming that the eviction threshold τ is set to 0.45 and the loading threshold Γ is set to 0.55, the scan yields a file activity of 0.4 for file f1, 0.53 for f2, 0.6 for f3, and 0.65 for f4.

[0093] First, for file f1, since its activity is below the threshold τ, the cache system (i.e., Alluxio system) 201 decides to evict it from the L1 layer and copy it to the L2 layer. Since file f2's activity is between the eviction threshold and the load threshold, it is not moved. However, for files f3 and f4, both have activity levels greater than the threshold Γ. Therefore, the cache system (i.e., Alluxio system) 201 moves files f4 and f3 to the L0 layer.

[0094] Data in the L0 layer is only evicted because there is no higher cache type above it. For data that needs to be evicted from the L2 layer, the cache system (i.e., the Alluxio system) 201 directly deletes the data from the L2 layer, retaining only the data in the underlying storage system 202.

[0095] In the present application, the electronic device can also determine the current remaining storage capacity of the cache system; if the remaining storage capacity is less than the capacity threshold, the multiple activity parameter values ​​of the current multiple files can be sorted; and the target file with the highest activity parameter value is loaded into the cache system according to the sorting result.

[0096] Continue as Figure 2 As shown, for files f3 and f4, the activity levels of both files are greater than a threshold Γ. The cache system (i.e., the Alluxio system) 201 will prioritize moving the more active file f4 to the L0 layer. If there is free space in the L0 layer, file f3 will also be moved to the L0 layer. If there is no free space in the L0 layer, file f3 will not be moved.

[0097] In this application, after the electronic device completes processing of the current file (deleting from the cache or adding to the cache), it can also update the activity parameter value of the current file according to a preset time interval, so as to judge the activity of the file at the next moment.

[0098] The data processing method provided in this application determines an activity parameter value for a file based on at least two different activity indicators for the file. Based on the comparison between the activity parameter value and a threshold, the method determines whether the file should be evicted from the cache system or preloaded into the cache system. This facilitates rapid access to remote data and improves the cache hit rate of the distributed cache system, enhancing overall system performance.

[0099] Furthermore, because this solution not only determines whether a file is cold or hot based on its access count, the Alluxio system can also promptly determine the hotness or coldness of newly added files in the underlying storage system. This eliminates the need for the client to send a request to the Alluxio cluster, allowing the Alluxio system to access the underlying storage system and preload the newly added file. This allows newly generated data to take advantage of the cache the first time it is accessed, improving file preloading efficiency.

[0100] Furthermore, by using multiple activity metrics to determine file activity, even if a file has never been accessed, it can still be considered active and added to the cache. This prevents frequently accessed hot data from being evicted from the cache due to being replaced by a larger amount of data that only needs to be accessed once.

[0101] The analysis of hot and cold data in this application is more realistic than that used in traditional Alluxio systems. It considers not only the access status of the file itself but also the access status of other related files. Furthermore, a cooling model is introduced to make the evolution of hot and cold data more realistic. This allows for quick and accurate determination of whether a file is hot or cold, accelerating data preloading efficiency.

[0102] Figure 3 This is a schematic diagram of the data processing method in this application. Figure 2 ,like Figure 3 As shown, the method includes:

[0103] Step 301: Alluxio Master sends a command to scan underlying file activity.

[0104] Step 302, obtaining file activity;

[0105] Step 303: evict files whose activity is lower than a threshold value τ from the cache;

[0106] Step 304: Load files with activity levels higher than a threshold value Γ into the cache;

[0107] In step 305, the activity update module in the Alluxio system updates the file activity at time interval T.

[0108] Here, steps 302 to 305 may be repeated after step 305 .

[0109] In this application, file activity is defined for each file. The Alluxio Master periodically scans the file activity of files in the system. When the activity of a file loaded into the cache falls below a threshold value τ, an asynchronous eviction process is triggered to remove the file from the cache. When the activity exceeds a threshold value Γ, an asynchronous loading process is triggered to load some or all of the most active files into the system cache, based on the remaining cache capacity in the system.

[0110] Here, asynchronous eviction and asynchronous loading mean that the scanning process, the eviction process, and the loading process are all different processes, and the processes can be performed asynchronously. That is, after the scanning process finishes scanning the current file, it can execute the scanning of the next file without waiting for the eviction process or the loading process to complete the file eviction or preloading.

[0111] Through this application, the files that need to be cached in the cache system are determined by the file activity of each file. Therefore, even if a file has never been accessed, it may still be determined to be an active file and then loaded into the cache system.

[0112] Figure 4This is a schematic diagram of the structure of the data storage device in this application. Figure 4 As shown, the device includes:

[0113] An acquiring unit 401 is configured to acquire an activity parameter value of a current file; the activity parameter value is related to at least two different activity indicators of the current file;

[0114] A comparing unit 402 is configured to compare the activity parameter value with a threshold value;

[0115] The determining unit 403 is configured to determine a processing method for the current file according to the comparison result, where the processing method represents loading the current file into a cache system or deleting the current file from the cache system.

[0116] In a preferred embodiment, the threshold includes a first threshold and a second threshold, and the second threshold is greater than the first threshold;

[0117] The electronic device further includes: a deleting unit 404 and an adding unit 405;

[0118] The deleting unit 404 is configured to delete the current file from the cache system when the determining unit 403 determines, based on the comparison result, that the activity parameter value is less than or equal to the first threshold;

[0119] The adding unit 405 is configured to add the current file to the cache system when the determining unit 403 determines, based on the comparison result, that the activity parameter value is greater than or equal to the second threshold.

[0120] Here, the electronic device is further configured to keep the storage state of the current file unchanged when the determining unit 403 determines, based on the comparison result, that the activity parameter value is between the first threshold and the second threshold.

[0121] In a preferred solution, the deleting unit 404 is further configured to delete the current file from the current cache layer in the cache system, so as to move the current file to a next cache layer in the cache system.

[0122] In a preferred solution, the adding unit 405 is further configured to move the current file from the current cache layer of the cache system to an upper cache layer of the cache system.

[0123] In a preferred embodiment, the electronic device further comprises: a sorting unit 406;

[0124] Specifically, the determining unit 403 is further configured to determine the current remaining storage capacity of the cache system;

[0125] A sorting unit 406 is configured to sort the activity parameter values ​​of the multiple current files if the remaining storage capacity is less than the capacity threshold;

[0126] The adding unit 405 is further configured to load the target file with the highest activity parameter value into the cache system according to the sorting result.

[0127] In a preferred embodiment, the electronic device further comprises:

[0128] Scanning unit 407, configured to scan underlying files based on the file activity instruction;

[0129] The determining unit 403 is further configured to determine an activity factor of the current file and a differential parameter of the activity of the current file over time according to the scanning result; and determine an activity parameter value of the current file based on the activity factor and the differential parameter.

[0130] In a preferred embodiment, the electronic device further comprises: a computing unit 408;

[0131] Specifically, the acquisition unit 401 is further configured to acquire a plurality of activity indicators related to the current file in the cache system, the plurality of activity indicators including at least two of the following: a number of sub-files stored in the cache system in the same directory as the current file, a number of other files loaded into the cache system simultaneously with the current file, a number of historical accesses to the current file, a generation time of the current file, and a temperature attribute of the current file;

[0132] The calculation unit 408 is configured to perform a function activation operation on the multiple activity indicators to obtain an activity factor of the current file.

[0133] In a preferred solution, the acquisition unit 401 is further configured to acquire a custom activity value of the current file;

[0134] The calculation unit 408 is configured to obtain an activity parameter value of the current file based on the activity factor, the differential parameter, and the custom activity value.

[0135] In a preferred embodiment, the electronic device further comprises:

[0136] The updating unit 409 is configured to update the activity parameter value of the current file according to a preset time interval.

[0137] It should be noted that the data processing device provided in the above embodiment is merely illustrated by the division of the aforementioned program modules when performing data processing. In actual applications, the aforementioned processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the data processing device provided in the above embodiment and the data processing method embodiment provided above are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0138] Figure 5 This is a schematic diagram of the structure of the electronic device in this application, such as Figure 5 As shown, the electronic device 500 can be a computer device based on Windows, Linux or MacOS operating system, including but not limited to computers, digital broadcast terminals, information transceiver equipment, medical equipment, fitness equipment, personal digital assistants and other terminals or servers. Here, the server includes but is not limited to physical servers and cloud servers. Figure 5 The electronic device 500 shown includes: at least one processor 501, a memory 502, at least one network interface 504, and a user interface 503. The network interface 504 can communicate with an external database to access and operate data. The external database may not be a hardware component of the computer device, but a database system deployed on a computer device outside the device, and provide database services to the outside through the network. The network interface 504 can also communicate with a message queue entity to put messages generated by the hardware entity into the message queue entity, or consume messages from the message queue entity. The message queue may not be a hardware component of the computer device, but a message queue system deployed on a computer device outside the device, and provide message services to the outside through the network. The network interface 504 can also send network messages to the network interface of the system under test and receive response messages. The system under test may not be a hardware component of the computer device, but a network application system deployed on a computer device outside the device, and provide application services to the outside through the network.

[0139] The various components in the electronic device 500 are coupled together via a bus system 505. It is understood that the bus system 505 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 505 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 5 Various buses are labeled as bus system 505.

[0140] The user interface 503 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or a touch screen.

[0141] It is understood that memory 502 can be volatile memory or non-volatile memory, or can include both volatile and non-volatile memory. Non-volatile memory can include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM); magnetic surface memory can include magnetic disk storage or magnetic tape storage. Volatile memory can include random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memory 502 described in the embodiments of the present application is intended to include, but is not limited to, these and any other suitable types of memory.

[0142] The memory 502 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device 500. Examples of such data include: any computer program used to operate on the electronic device 500, such as an operating system 5021 and an application 5022; contact data; phone book data; messages; pictures; audio, etc. Among them, the operating system 5021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 5022 can include various applications, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program that implements the method of the embodiment of the present application can be included in the application 5022.

[0143] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in processor 501 or by software instructions. Processor 501 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc. Processor 501 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium located in memory 502. Processor 501 reads information from memory 502 and, in conjunction with its hardware, completes the steps of the above method.

[0144] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0145] In an exemplary embodiment, the present application also provides a computer-readable storage medium, such as a memory 502 including a computer program. The computer program can be executed by a processor 501 of an electronic device 500 to perform the steps of the aforementioned method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface mount storage, optical disk, or CD-ROM; or various devices including any one or any combination of the aforementioned memories, such as a mobile phone, computer, tablet device, personal digital assistant, etc.

[0146] A computer-readable storage medium stores computer instructions, which, when executed by a processor, executes any one of the method steps in any one of the above-mentioned system testing methods.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0148] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0150] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0151] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0152] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining an activity parameter value of a current file; the activity parameter value is related to at least two different activity indicators of the current file; Comparing the activity parameter value with a threshold; determining a processing method for the current file according to the comparison result, wherein the processing method represents loading the current file into a cache system or deleting the current file from the cache system; The threshold includes a first threshold and a second threshold, and the second threshold is greater than the first threshold; Determining a processing method for the current file based on the comparison result includes at least one of the following: Determining, based on the comparison result, that the activity parameter value is less than or equal to the first threshold, deleting the current file from the cache system; Determining, based on the comparison result, that the activity parameter value is between the first threshold value and the second threshold value, and maintaining the storage state of the current file unchanged; Determining, based on the comparison result, that the activity parameter value is greater than or equal to the second threshold, adding the current file to the cache system; The step of obtaining the activity parameter value of the current file includes: Scanning the underlying file based on the file activity instruction to determine the activity factor of the current file and a differential parameter of the activity of the current file over time; An activity parameter value of the current file is determined based on the activity factor and the differential parameter.

2. The method according to claim 1, wherein The current file has multiple; Adding the current file to the cache system includes: Determining the current remaining storage capacity of the cache system; If the remaining storage capacity is less than the capacity threshold, sorting multiple activity parameter values ​​of multiple current files; The target file with the highest activity parameter value is loaded into the cache system according to the sorting result.

3. The method according to claim 1, wherein Determining the activity factor of the current file includes: Obtaining multiple activity indicators related to the current file in the cache system, the multiple activity indicators including at least two of the following: a number of sub-files stored in the cache system in the same directory as the current file, a number of other files loaded into the cache system simultaneously with the current file, a number of historical accesses to the current file, a generation time of the current file, and a temperature attribute of the current file; Perform a function activation operation on the multiple activity indicators to obtain an activity factor of the current file.

4. The method according to claim 1, wherein The determining of the activity parameter value of the current file based on the activity factor and the differential parameter further includes: Get the custom activity value of the current file; An activity parameter value of the current file is determined based on the activity factor, the differential parameter, and the custom activity value.

5. The method according to any one of claims 1 to 4, wherein: The method further comprises: Update the activity parameter value of the current file according to a preset time interval.

6. A data processing device, characterized in that: The data processing device includes: an acquiring unit, configured to acquire an activity parameter value of a current file; the activity parameter value being related to at least two different activity indicators of the current file; a comparing unit, configured to compare the activity parameter value with a threshold; a determining unit, configured to determine a processing method for the current file according to the comparison result, wherein the processing method represents loading the current file into a cache system or deleting the current file from the cache system; The threshold includes a first threshold and a second threshold, and the second threshold is greater than the first threshold; a deleting unit, configured to delete the current file from the cache system when the determining unit determines, based on the comparison result, that the activity parameter value is less than or equal to the first threshold; an adding unit, configured to add the current file to the cache system when the determining unit determines, based on the comparison result, that the activity parameter value is greater than or equal to the second threshold; The device is further configured to keep the storage state of the current file unchanged when the determining unit determines, based on the comparison result, that the activity parameter value is between the first threshold and the second threshold; A scanning unit, configured to scan underlying files based on file activity instructions; The determining unit is further configured to determine an activity factor of the current file and a differential parameter of the activity of the current file over time according to the scanning result; and determine an activity parameter value of the current file based on the activity factor and the differential parameter.

7. An electronic device, characterized in that: The electronic device comprises: a processor and a memory for storing a computer program capable of running on the processor, Wherein, when the processor is used to run the computer program, it executes the method steps in any one of the data processing methods described in claims 1 to 5.

8. A computer-readable storage medium, characterized in that Used to store computer instructions, which, when executed by a processor, execute the method steps in any one of the data processing methods according to claims 1 to 5.

Images