Data processing method, electronic device, and storage medium

Abstract

This application discloses a data processing method, an electronic device, and a storage medium. The data processing method includes: in response to a data write request for a first disk, determining the target data to be written to the first disk and the first position information of the write pointer in the first disk; if the first position information is a preset endpoint position, moving the write pointer to a preset start position; if the first position information is not a preset endpoint position, moving the write pointer in a first preset direction by a first preset distance; determining whether a target cache fragment exists in a cache fragment of the first disk according to second position information; and in response to the existence of a target cache fragment in a cache fragment of the first disk, replacing historical data in the target cache fragment with the target data. This application can achieve both cleaning up aging data and saving resources.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, electronic device, and storage medium. Background Technology

[0002] In CDN (Content Delivery Network) caching scenarios, large amounts of data are systematically stored on disks. Over time, some of this data may become invalid; however, this invalid data still occupies disk storage space, reducing disk utilization efficiency. To address this, the business layer needs to periodically execute pre-configured aging policies to clean up invalid data on disk and free up more storage space. However, this increases the resource consumption and workload of the business layer. Summary of the Invention

[0003] The purpose of this application is to provide a data processing method, electronic device, and storage medium to solve the problem of how to balance cleaning up aging data and saving resources.

[0004] To solve the above-mentioned technical problems, the embodiments of this application are implemented as follows: On one hand, embodiments of this application provide a data processing method, including: In response to a data write request to the first disk, the target data to be written to the first disk is determined, and the first position information of the write pointer in the first disk is determined; If the first position information is a preset end point position, then the write pointer is moved to a preset start point position; if the first position information is not the preset end point position, then the write pointer is moved in a first preset direction, and the moving distance is a first preset distance. Based on the second location information, it is determined whether a target cache fragment exists in the cache fragments of the first disk; the second location information indicates the position of the write pointer after the move; the position of the target cache fragment in the first disk corresponds to the second location information; In response to the presence of the target cache fragment in the cache fragment of the first disk, the historical data in the target cache fragment is replaced with the target data.

[0005] On the other hand, embodiments of this application provide a data processing apparatus, including: The determining unit is configured to, in response to a data write request for the first disk, determine the target data to be written in the first disk and determine the first position information of the write pointer in the first disk; The moving unit is configured to move the write pointer to a preset starting point if the first position information is a preset endpoint position; and to move the write pointer in a first preset direction and the moving distance is a first preset distance if the first position information is not the preset endpoint position. The determining unit is further configured to determine whether a target cache fragment exists in the cache fragments of the first disk based on the second location information; the second location information indicates the position of the write pointer after the move; the position of the target cache fragment in the first disk corresponds to the second location information; The replacement unit is configured to replace historical data in the target cache fragment with the target data in response to the existence of the target cache fragment in the cache fragment of the first disk.

[0006] In another aspect, embodiments of this application provide an electronic device, including a processor and a memory electrically connected to the processor, the memory storing a computer program, and the processor being used to call and execute the computer program from the memory to implement the above-described data processing method.

[0007] In another aspect, embodiments of this application provide a computer-readable storage medium for storing a computer program that can be executed by a processor to implement the above-described data processing method.

[0008] In another aspect, embodiments of this application provide a computer program product, including a computer program, which is executed by a processor to implement the above-described data processing method.

[0009] The embodiments of this application adopt the following technical solutions: In response to a data write request to the first disk, the target data to be written to the first disk is determined, and the first position information of the write pointer in the first disk is determined; if the first position information is a preset endpoint position, the write pointer is moved to a preset start position; if the first position information is not a preset endpoint position, the write pointer is moved in a first preset direction, and the moving distance is a first preset distance; based on the second position information, it is determined whether a target cache fragment exists in the cache fragment of the first disk; the second position information indicates the position of the write pointer after the move; the position of the target cache fragment in the first disk corresponds to the second position information; in response to the existence of a target cache fragment in the cache fragment of the first disk, the historical data in the target cache fragment is replaced with the target data.

[0010] The above-described technical solutions adopted in the embodiments of this application can achieve at least the following beneficial effects: After determining the target data to be written and the first position information, if the first position information is a preset endpoint position, it means that the number of cache fragments in the first disk has reached the maximum value. At this time, the data with the longest writing time in the entire first disk is the historical data in the first cache fragment of the first disk. By moving the write pointer to the preset starting position and determining the first cache fragment as the target cache fragment based on the moved position, and by replacing the historical data in the target cache fragment with the target data, the historical data with the longest writing time in the first disk can be overwritten by the target data to achieve the clearing of aging data. If the first position information is not the preset endpoint position, the write pointer is moved in the first preset direction and the moving distance is the first preset distance. In this way, each time new data is written, the write pointer will be moved to form an infinite loop of movement. Thus, each data write can automatically overwrite the historical data with the longest writing time in the first disk to achieve the clearing of aging data. The business layer does not need to configure and periodically execute the aging policy, which reduces the resource consumption and workload of the business layer in handling aging problems. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in one or more embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic flowchart of a data processing method according to an embodiment of this application; Figure 2 This is an interactive schematic diagram of a first disk according to an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a first disk according to an embodiment of this application; Figure 4 This is a schematic diagram of the write pointer movement process in a data processing method according to an embodiment of this application; Figure 5 This is a schematic block diagram of a data processing apparatus according to an embodiment of this application; Figure 6 This is a schematic block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0013] This application provides a data processing method, an electronic device, and a storage medium.

[0014] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0015] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein.

[0016] The data processing method provided in this application can be executed by an electronic device or by software installed in an electronic device. Specifically, the electronic device can be a terminal device or a server device. The terminal device can include smartphones, laptops, smart wearable devices, vehicle terminals, etc., and the server device can include an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0017] Figure 1 This is a schematic flowchart of a data processing method according to an embodiment of this application, such as... Figure 1 As shown, the method includes: S102, in response to a data write request for the first disk, determines the target data to be written in the first disk and determines the first position information of the write pointer in the first disk.

[0018] A hard disk is a hardware device used to store data, employing magnetic recording technology to read and write information on rotating platters. It is a common non-volatile storage medium in computer systems, retaining data even after power is lost.

[0019] The first disk in this embodiment can be any disk.

[0020] For example, the disk type of the first disk can be HDD (Hard Disk Drive), SSD (Solid State Drive), SSHD (Solid State Hybrid Drive), etc.

[0021] The first disk may include multiple cache fragments.

[0022] A disk cache is a region in memory used to temporarily store data read from or about to be written to the disk. Caching reduces the number of direct disk accesses, thus improving performance. A disk cache can be divided into multiple independent shards, each managing a portion of the cached data; these shards are what we refer to as cache shards.

[0023] It is important to note that if the disk cache of the first disk is not yet full of data, a data write operation will trigger the creation of a new cache shard, thus increasing the number of cache shards in the first disk; if the disk cache of the first disk is already full of data, a data write will not cause the number of cache shards in the first disk to change.

[0024] A data write request can be used to request that target data be written to the disk cache of the first disk.

[0025] The target data can be any data set. The target data can be determined by the data write request.

[0026] It should be noted that when there are multiple target data, steps S102-S108 in the data processing method provided in this application embodiment can be executed separately for each target data, and the processing flow for each target data is the same.

[0027] A write pointer is a pointer or marker used to indicate the next location where data is written.

[0028] Determining the first position information of the write pointer in the first disk can be achieved by reading the position information of the write pointer in the first disk. This first position information corresponds to the cache fragment of the previous data written before step S102.

[0029] The number of cache fragments for the previous data write before step S102 can be one or more. If the number of cache fragments for the previous data write is one, the first location information can correspond to the cache fragment for the previous data write; if the number of cache fragments for the previous data write is multiple, the first location information can correspond to the last cache fragment among the multiple cache fragments for the previous data write.

[0030] For example, if the disk cache of the first disk is not yet full of data, and the first disk includes K cache fragments that have been written to, then there is a correspondence between the first position information of the write pointer and the Kth cache fragment. Here, K is an integer greater than or equal to 0.

[0031] Alternatively, the disk cache of the first disk is full of data. The first disk includes N cache fragments that have been written to, and before executing step S102, the cache fragment that was last written to was the i-th cache fragment among the N cache fragments. In this case, there is a correspondence between the first position information of the write pointer and the i-th cache fragment. Here, N is an integer greater than 1, and i is an integer greater than or equal to 1.

[0032] In one specific implementation, the data write request includes a file identifier; in response to the data write request for the first disk, determining the target data to be written in the first disk includes: determining the target file corresponding to the data write request based on the file identifier; determining whether the total data volume of the target file is greater than a preset data volume threshold; if the total data volume is greater than the preset data volume threshold, then splitting the target file to obtain multiple file data, and determining each file data in the multiple file data as a target data; the data volume of each file data is less than or equal to the preset data volume threshold; if the total data volume is less than or equal to the preset data volume threshold, then determining the data content of the target file as the target data.

[0033] When a data write request includes a file identifier, the data write request can be used to request that the data content of a target file be written to the disk cache of a first disk, and the target file can be determined based on the file identifier.

[0034] A file identifier is a string, number, or other form of identification information used to uniquely identify a file.

[0035] The file identifier included in the data write request can be used to uniquely identify the corresponding file, which is the target file corresponding to the data write request.

[0036] Considering the limited storage capacity of cache shards, a preset data volume threshold can be pre-configured based on the storage capacity of the cache shards. This preset data volume threshold is used to determine whether a file can be written to the cache shard.

[0037] The system determines whether the total data size of the target file exceeds a preset data size threshold. If the total data size exceeds the preset data size threshold, it means the target file exceeds the storage capacity of a cache shard, and cannot be directly written to a cache shard. Therefore, the target file needs to be split into multiple file data, and each file data is designated as a target data. The data size of each file data is less than or equal to the preset data size threshold. In this way, for any given file data, the data size will not exceed the storage capacity of a cache shard, and the file data can be directly written to a cache shard. If the total data size is less than or equal to the preset data size threshold, it means the target file does not exceed the storage capacity of a cache shard, and the target file can be directly written to a cache shard. Therefore, the data content of the target file can be designated as the target data.

[0038] In this implementation, by comparing the preset data volume threshold configured for the storage capacity of the cache shard with the total data volume of the target file, it can be determined whether the target file exceeds the storage capacity of a cache shard. In this way, if the target file is too large to be directly placed into a cache shard, it will be split and then processed. If the size of the target file is appropriate, it can be processed directly without splitting, thereby avoiding the situation where the storage capacity of the cache shard is insufficient, resulting in data writing failure.

[0039] S104, if the first position information is a preset end position, then the write pointer is moved to a preset start position; if the first position information is not a preset end position, then the write pointer is moved in a first preset direction, and the moving distance is a first preset distance.

[0040] The preset starting position can be a custom-configured write pointer position that corresponds to the first cache fragment of the first disk.

[0041] The preset endpoint position can be another write pointer position configured in a custom way, which corresponds to the last cached fragment of the first disk.

[0042] It should be noted that the last cache fragment of the first disk mentioned above refers to the last cache fragment among multiple cache fragments of the first disk when the disk cache of the first disk is full, that is, when the number of cache fragments in the first disk reaches a predetermined maximum value.

[0043] The first preset direction can be from a preset start position to a preset end position. The value of the first preset distance can be customized, and the first preset distance can be used to represent the distance between the write pointer positions corresponding to two adjacent cache fragments.

[0044] If there is a correspondence between the first position information and the last cache fragment in the first disk, the first position information is used as the starting point for the write pointer movement. The write pointer is moved to the preset starting point position, and then the position of the moved write pointer corresponds to the first cache fragment in the first disk.

[0045] If there is a correspondence between the first position information and the i-th cache fragment in the first disk, the write pointer is moved in a first preset direction with the first position information as the starting point, and the moving distance is the first preset distance. Then, the position of the moved write pointer corresponds to the (i+1)-th cache fragment in the first disk. i is an integer greater than or equal to 1.

[0046] For example, the first disk includes a fragmented data area, which contains one or more cache fragments. The fragmented data area can be designed as an infinite circular queue with its ends linked. During system initialization, a write pointer `write_pos` can be set, serving as the cursor for data writing. When data needs to be written to the first disk, writing begins from the position of `write_pos`. Each time data is written to a cache fragment, the position of the write pointer `write_pos` moves forward one step. Each step is a fixed distance. After the write pointer reaches a preset endpoint, the next move will automatically jump to the preset starting position, ensuring that newly written data overwrites previously written historical data, thus achieving automatic aging of cached data. All data writes to cache fragments follow a first-in, first-out (FIFO) principle. This automatic rollback allows for the cleanup of aging data. The aforementioned forward movement of the write pointer `write_pos` occurs from the preset starting position to the preset ending position.

[0047] For example, the preset start position can be represented by 1, and the preset end position can be represented by 500. That is, when the first position information is the preset start position, the value of the write pointer is 1, and when the first position information is the preset end position, the value of the write pointer is 500.

[0048] Assuming that each data write operation writes a target data to the first disk, and this target data requires a cache slice, and the cache slices corresponding to two adjacent data writes are different, then the value of the write pointer is incremented by 1 each time a data write is performed. When the value of the write pointer reaches 500, the value of the write pointer will no longer be incremented by 1 for the next data write, but will become 1.

[0049] In one specific implementation, the preset endpoint position is determined by the effective capacity ratio threshold of the first disk, and the effective capacity ratio threshold is determined by the disk type of the first disk.

[0050] For example, the disk type of the first disk can be HDD, SSD, etc.

[0051] Given the lower read / write efficiency of sectors on the inner tracks of HDDs, writing data to these tracks can lead to a significant performance degradation. In contrast, SSDs do not exhibit this performance inconsistency between the inner and outer tracks. Based on this reasoning, different effective capacity ratio thresholds can be configured for different disk types. For example, if the first disk type is HDD, the effective capacity ratio threshold can be configured to 80%; if the first disk type is SSD, the effective capacity ratio threshold can be configured to 99%.

[0052] The preset endpoint position can be a custom-configured write pointer position that corresponds to the last valid cached fragment of the first disk.

[0053] Based on the theoretical maximum value S1 of the number of cache fragments in the first disk and the effective capacity ratio threshold p1, the maximum effective cache fragment value S2 in the first disk is determined. The theoretical maximum value S1 can be a predetermined fixed value, and the effective capacity ratio threshold p1 can be determined by the disk type of the first disk. Based on this, the maximum effective cache fragment value S2 in the first disk can also be a fixed value.

[0054] The last valid cache fragment of the first disk mentioned above refers to the last cache fragment among the multiple cache fragments of the first disk when the number of cache fragments in the first disk reaches S2.

[0055] In this implementation, the effective capacity ratio threshold is determined by the disk type of the first disk, and then the preset endpoint position is determined. This limits the movement range of the write pointer, which indirectly limits the range of cache fragments that can be written to the target data, thereby avoiding the performance degradation of the first disk caused by data write operations.

[0056] S106, Based on the second location information, determine whether the target cache fragment exists in the cache fragment of the first disk; the second location information indicates the position of the write pointer after the move; the position of the target cache fragment in the first disk corresponds to the second location information.

[0057] The second position information may be the initial preset position. Alternatively, the second position information may also be the position after moving the write pointer a first preset distance in the first preset direction, starting from the first position information.

[0058] After step S104 is executed, the following two situations may occur: (a1) The first disk has just been used, and the currently established cache fragments have not yet filled the disk cache of the first disk. That is, the number of cache fragments in the first disk has not yet reached the predetermined maximum value.

[0059] In case (a1), the target cache shard does not exist in the first disk, so a new cache shard needs to be created and the target data is written into the newly created cache shard.

[0060] (a2) The first disk has been filled at least once, and the currently established cache fragments have filled the disk cache of the first disk. That is, the number of cache fragments in the first disk has reached the predetermined maximum value.

[0061] In case (a2), the target cache fragment exists in the first disk.

[0062] Determining whether a target cache fragment exists in the cache fragments of the first disk based on the second location information can be achieved by querying whether a cache fragment exists in the cache fragments of the first disk based on the second location information, and the location of the cache fragment in the first disk corresponds to the second location information.

[0063] If the query result indicates that there is a cache shard whose position on the first disk corresponds to the second position information, then the cache shard can be used as the target cache shard; if the query result indicates that there is no cache shard whose position on the first disk corresponds to the second position information, then it can be determined that there is no target cache shard.

[0064] In one specific implementation, the maximum number of cached shards in the first disk is determined based on the total capacity of the first disk, the effective capacity ratio threshold of the first disk, and the pre-configured average data volume of the shards.

[0065] The total capacity of the first disk refers to the theoretical maximum storage capacity of the first disk. The effective capacity ratio threshold of the first disk can be found in the corresponding description of the aforementioned implementation method. The average data size per shard can be the average of the maximum data size that each cache shard can store.

[0066] First, the available storage space of the first disk can be calculated, specifically by referring to the following formula (1): Available storage space = Total capacity of the first disk × Effective capacity ratio threshold of the first disk − Metadata occupied space (1) The space occupied by metadata can be calculated using the following formula (2): Metadata space usage = Maximum number of cached fragments in the first disk × size of metadata object (2) In the above formula (2), the size of the metadata object refers to the size of the metadata object allocated to each cache shard in the first disk. For example, the size of a metadata object can be 12 bytes.

[0067] Next, the maximum number of cache fragments in the first disk can be calculated, as shown in the following formula (3): The maximum number of cached shards in the first disk = available storage space / (average data size of shards + size of metadata objects) (3) Combining the above formulas (1) to (3), we can obtain the following formula (4): The maximum number of cached fragments in the first disk = ((total capacity of the first disk × effective capacity ratio threshold of the first disk) - (maximum number of cached fragments in the first disk × size of metadata objects)) / (average data size of fragments + size of metadata objects) (4) Considering that both sides of the equal sign in the above formula (4) contain "the maximum number of cache fragments in the first disk", the above formula (4) can be transformed to obtain the following formula (5): The maximum number of cached fragments in the first disk = (total capacity of the first disk × effective capacity ratio threshold of the first disk) / (average data size of fragments + 2 × size of metadata objects) (5) That is, referring to the above formula (5), the maximum number of cache fragments in the first disk can be calculated based on the total capacity of the first disk, the effective capacity ratio threshold of the first disk, the pre-configured average data volume of the fragments, and the size of the metadata object allocated to each cache in the first disk.

[0068] In addition, considering that the size of the metadata object (e.g., 12 bytes) may be much smaller than the average data size of the fragment (e.g., 128MB, 1GB, etc.), the above formula (5) can be simplified to obtain the following formula (6): The maximum number of cached fragments in the first disk = (total capacity of the first disk × effective capacity ratio threshold of the first disk) / average data size of fragments (6) That is, referring to the above formula (6), the maximum number of cached fragments in the first disk can be calculated based on the total capacity of the first disk, the effective capacity ratio threshold of the first disk, and the pre-configured average data volume of the fragments.

[0069] Since the size of the metadata object may be much smaller than the average data size of the shard, after simplifying the formula, the maximum number of cached shards in the first disk calculated using three parameters can be approximated as the same value as the maximum number of cached shards in the first disk calculated using four parameters.

[0070] In addition, given that the maximum number of cache fragments in the first disk has been determined, the space occupied by the metadata can be determined by referring to the aforementioned formula (2) based on the maximum number of cache fragments in the first disk and the size of the metadata object allocated to each cache in the first disk.

[0071] In this implementation, the maximum number of cache shards in the first disk is predetermined by some easily obtainable disk parameters. This indirectly determines the maximum capacity of the storage space allocated to the metadata of the cache shards in the first disk. In this way, no matter how data is written, the storage space required for the metadata will not exceed the maximum capacity. Therefore, there is no need to dynamically expand the storage space for the metadata, saving memory overhead and computing resources.

[0072] To more fully illustrate the above technical effects, the following example uses a CDN caching scenario: CDN file caching mechanisms may suffer from high memory consumption, requiring additional overhead to manage metadata. In a CDN file caching system, metadata can be used to record a lot of information about a file, including but not limited to: file size, owner, file permissions, file creation time, etc. Each metadata element is typically 128 bytes. Since each file must have corresponding metadata, it is possible that the space occupied by metadata may be exhausted before the disk is full, preventing the storage of new files on the disk. Furthermore, managing metadata is an essential task for CDN file caching systems, potentially employing efficient data structures such as B+ trees to manage file system space allocation for fast search and insertion operations. However, maintaining such data structures also requires additional memory and CPU resources, undoubtedly increasing the system's memory overhead.

[0073] Based on this implementation, the maximum capacity of the storage space allocated to the metadata of the cache shards in the first disk can be determined by the maximum number of cache shards in the first disk. This avoids the problem in the CDN caching scenario mentioned above where the space occupied by the metadata is exhausted but the disk is not full. Moreover, it does not require dynamic space allocation for the metadata, saving memory overhead and overcoming the problem of large memory overhead caused by dynamic expansion of storage space for metadata in the CDN caching scenario mentioned above.

[0074] S108, in response to the existence of a target cache fragment in the cache fragment of the first disk, replace the historical data in the target cache fragment with the target data.

[0075] If a target cache fragment exists in the cache fragments of the first disk, the target cache fragment stores historical data. The data writing process for writing historical data to the target cache fragment is similar to that for writing target data, and can be referred to the corresponding descriptions of steps S102-S108 in the data processing method provided in the embodiments of this application.

[0076] Replace historical data in the target cache shard with target data. For example, if the target cache shard stores historical data x1 and the target data is x2, after the replacement is performed, the target cache shard will store target data x2.

[0077] In a caching scenario, if historical data in a target cache shard is overwritten before it expires, the expired historical data can be used as new target data and rewritten to the first disk using the data processing method provided in this application embodiment.

[0078] Furthermore, historical data in the target cache shard can be considered as the data with the longest write time in the entire first disk. By overwriting the data with the target data with the longest write time, aging data can be cleared, minimizing the deletion of data that has not yet expired, thus optimizing the utilization efficiency of the first disk. Moreover, by automatically rolling back and clearing aging data, decoupling from the business layer can be achieved, reducing the memory overhead and computing resources consumed by the business layer.

[0079] In one specific implementation, each cache shard in the first disk has a corresponding metadata object, which is used to store the metadata of the data content in the cache shard; the data processing method further includes: in response to the existence of a target cache shard in the first disk, generating metadata of the target data based on the target data and the second location information; and replacing the historical metadata in the metadata object corresponding to the target cache shard with the metadata of the target data.

[0080] Metadata is data that describes data, such as the offset of a cache fragment in the disk data area, the filename, size, and MD5 value of the cache fragment, etc. Metadata objects, such as Dir objects, are structures specifically designed to store and manage this metadata. Each cache shard in the first disk has a corresponding Dir object, which can be used to store metadata about the data content within the cache shard.

[0081] For example, the metadata area of ​​the first disk may include the following three parts: Header, Directory, and Ender.

[0082] The Header can be used to record all read and write control identifiers on the first disk, such as the available disk space, the position of the write pointer, the number of disk rollbacks, the metadata synchronization sequence number, and so on.

[0083] The Directory represents an index data structure for all cache shards on the first disk. Each cache shard is allocated a corresponding Dir object in the Directory, which stores information such as the cache shard's MD5 value, size, and offset. By using the recorded cache shard's MD5 value, the complete cache shard can be quickly matched, thus enabling rapid response to user access requests. The Ender field can be used to identify whether metadata has been maliciously corrupted or the write process has been interrupted. The Ender field contains a checksum field and a flag. When reading metadata, the system recalculates the checksum of the metadata and compares it with the checksum in the Ender field. It also checks whether the metadata validity flag is set, thereby detecting whether the metadata has been maliciously corrupted or the write process has been interrupted.

[0084] Metadata can be persistently stored in the header of each disk. When the file caching system is working, it loads the metadata from the disk into memory to improve the overall performance and responsiveness of the system.

[0085] When data in any cached shard is accessed, the corresponding Dir object can be efficiently found by traversing the Directory based on the MD5 value of the cached shard. Then, the storage location of the cached shard on the first disk can be obtained based on the information recorded in the Dir object. When the resource is hit, the accessed data is directly returned to the user to provide service.

[0086] To minimize the memory footprint of metadata and reduce the overhead of managing metadata nodes, a data structure can be pre-configured for the Directory. The Directory can include multiple Dir objects, each corresponding to a cache shard on the first disk. The size of the Dir object can be referenced in the description of the size of the metadata object allocated to each cache on the first disk in the aforementioned implementation.

[0087] For example, the size of a Dir object can be 12 bytes.

[0088] Because there is a correspondence between Dir objects and cache shards, Dir objects only need to store a small amount of necessary pointers and state information to match metadata with cache shards.

[0089] For example, the metadata stored in the Dir object includes, but is not limited to: the offset of the cache shard in the first disk, the filename, size, MD5 value of the cache shard, etc.

[0090] By using a fixed-size Dir object, the amount of content that can be stored can be estimated during system initialization based on the available disk space and average content size, and sufficient storage space can be allocated. This means that regardless of how many cache fragments are currently present in the system's disk cache, the space occupied by its Directory remains constant.

[0091] The process of providing sufficient storage space for metadata shards can be referred to in the aforementioned section on determining the space occupied by metadata based on the maximum number of cache shards in the first disk and the size of the metadata object allocated to each cache in the first disk.

[0092] In this way, sufficient storage space can be allocated for metadata during the initial disk initialization, and metadata information will be automatically synchronized during operation, eliminating the need for additional management operations. Whenever a data write operation triggers the creation of a new cache shard, a Dir object will be allocated for that cache shard in the Directory so that the newly written data can be correctly referenced and accessed subsequently.

[0093] While overwriting the historical data in the target cache shard with the new target data, the metadata in the metadata object corresponding to the target cache shard can also be overwritten with the metadata of the new target data.

[0094] Based on the target data and the second location information, the metadata of the target data is generated, which may include: determining the file name, data size, and MD5 value of the target data; determining the offset of the cache fragment in the first disk based on the second location information; and using the file name, data size, MD5 value, and offset of the cache fragment in the first disk as the metadata of the target data.

[0095] The historical metadata in the metadata object corresponding to the target cache shard is replaced by the metadata of the target data. For example, the target cache shard corresponds to Dir object 1, which stores historical metadata including: filename a1, data size b1, MD5 value c1, and the cache shard's offset d1 on the first disk. The target data's metadata includes: filename a2, data size b2, MD5 value c2, and the cache shard's offset d2 on the first disk. After replacing the historical metadata in the metadata object corresponding to the target cache shard with the target data's metadata, Dir object 1 will then store: filename a2, data size b2, MD5 value c2, and the cache shard's offset d2 on the first disk.

[0096] In this implementation, old metadata can be cleared by using new metadata to overwrite the historical metadata in the metadata object, saving metadata storage space. Considering that the structure of the metadata object is relatively simple, whether loading metadata from the first disk into memory or flushing metadata from memory to the first disk, full metadata reading and writing can be achieved without any conversion in between, which helps to improve the overall performance and response speed of the system.

[0097] In one specific implementation, there are multiple first disks arranged in a preset order, and the capacity of each first disk is a first preset capacity. Among the multiple first disks, the interval between the initial write positions of the write pointers of two adjacent first disks is determined by the number of first disks and the first preset capacity.

[0098] The first preset capacity can be determined based on the total capacity of the first disk and the effective capacity ratio threshold.

[0099] Considering that in scenarios involving large amounts of data writing, the target data to be written may be evenly distributed across multiple first disks, and that HDD disk utilization may affect service performance when it reaches a preset threshold, such as 80%, in order to avoid the write positions of all first disks reaching the inner disk at the same time, causing the read and write performance of all first disks to decrease simultaneously, thereby affecting the overall service capability of the device, the initial write positions of each first disk can be evenly distributed.

[0100] For example, in a multi-disk initialization scenario, there are N first disks, each with the same capacity, which can be represented by C. The interval between the initial write positions of each first disk can be (C / N), thus ensuring the overall stability of the device's capabilities. N is an integer greater than 1.

[0101] In addition, in single-disk initialization scenarios, such as adding a first disk, or the first disk's metadata being corrupted, the initial write position of the first disk can be randomly set.

[0102] It is important to emphasize that the initial write position is not the same concept as the aforementioned preset starting position.

[0103] For example, there are three first disks: First Disk 1, First Disk 2, and First Disk 3. The preset capacity can be represented by 90, so the preset start position can be represented by 1, and the preset end position can be represented by 90. Among the multiple first disks, the initial write position interval between the write pointers of two adjacent first disks can be 90 / 3 = 30.

[0104] Specifically, the initial write position of the first disk 1 can be represented by 5, the initial write position of the first disk 2 can be represented by 35, and the initial write position of the first disk 3 can be represented by 65.

[0105] During the process of writing various target data to the first disk 1, the position of the write pointer can be gradually increased from 5. After reaching 90, the next data write jumps to 1, and then continues to gradually increase from 1, and so on.

[0106] During the process of writing various target data to the first disk 2, the position of the write pointer can be gradually increased from 35 to 90. After that, the next data write jumps to 1, and then continues to gradually increase from 1, and so on.

[0107] During the process of writing various target data to the first disk 3, the position of the write pointer can be gradually increased from 65. After reaching 90, the next data write jumps to 1, and then continues to gradually increase from 1, and so on.

[0108] That is, the capacity of the first disk 1, the first disk 2 and the first disk 3 is 90, and the capacity of each first disk will not be different due to different initial write positions.

[0109] In this implementation, by dispersing the initial write positions of each first disk, each first disk can bring its data write position closer to the inner disk at different times during the data write process. This avoids the performance degradation of each first disk at the same time, and optimizes the service capabilities of the multiple first disks from the overall perspective.

[0110] In one specific implementation, the data processing method further includes: upon receiving a file access request, generating a corresponding first file identifier based on the first file name carried in the file access request; querying each metadata object in the first disk to see if a first metadata object corresponding to the first file identifier exists; if a first metadata object exists, reading the third position information of the write pointer from the first metadata object; and obtaining the data content in the cache fragment corresponding to the third position information based on the third position information.

[0111] Based on the first filename carried in the file access request, a corresponding first file identifier is generated. This can be achieved by using a preset hash algorithm to calculate and process the first filename to obtain a hash value corresponding to the first filename, and then using this hash value as the first file identifier.

[0112] In the first disk, each metadata object can store a file identifier, and a file identifier can correspond to one or more metadata objects.

[0113] For example, if the total amount of data in file 1 is too large, file 1 is pre-split into 3 target data before the data is written. These 3 target data are written to cache shard 1, cache shard 2 and cache shard 3 respectively. Then, the metadata object of cache shard 1 stores the file identifier of file 1, the metadata object of cache shard 2 stores the file identifier of file 1, and the metadata object of cache shard 3 stores the file identifier of file 1.

[0114] When each metadata object includes a file identifier, the existence of a first metadata object corresponding to the first file identifier can be queried among the metadata objects in the first disk based on the first file identifier.

[0115] In each metadata object of the first disk, a write pointer position can be stored. This position is used to determine the location of the corresponding cache slice on the first disk. Based on this, the third position information of the write pointer can be read from the first metadata object, and the data content in the cache slice corresponding to the third position information can be obtained.

[0116] In this implementation, during file access, the file name can be used to determine the corresponding file identifier, and the file identifier can be used to determine one or more corresponding metadata objects. The position of the write pointer can be read from the metadata object, and then the data content in the corresponding cache segment can be found. This allows for quick and accurate retrieval of the data content of the file to be accessed, thus improving data access efficiency.

[0117] In this embodiment, after determining the target data to be written and the first position information, if the first position information is a preset endpoint position, it means that the number of cache fragments in the first disk has reached the maximum value. At this time, the data with the longest writing time in the entire first disk is the historical data in the first cache fragment in the first disk. By moving the write pointer to the preset starting position and determining the first cache fragment as the target cache fragment based on the moved position, and by replacing the historical data in the target cache fragment with the target data, the historical data with the longest writing time in the first disk can be overwritten by the target data to achieve the clearing of aging data. If the first position information is not the preset endpoint position, the write pointer is moved in the first preset direction and the moving distance is the first preset distance. In this way, each time new data is written, the write pointer will be triggered to move, so as to form an infinite loop of movement. Thus, each data write can automatically overwrite the historical data with the longest writing time in the first disk to achieve the clearing of aging data. The business layer does not need to configure and periodically execute the aging strategy, which reduces the resource consumption and workload of the business layer in handling aging problems.

[0118] Figure 2 This is an interactive schematic diagram of a first disk according to an embodiment of this application.

[0119] like Figure 2 As shown, the first disk 204 may include a disk data area 210 and a disk header metadata 208. The first disk may periodically synchronize the disk header metadata 208 to the kernel 202; the metadata synchronized to the kernel 202 is the metadata 206. The kernel 202 may write new resources from the page cache to the first disk 204; this data writing process can be referred to the corresponding descriptions of steps S102-S108 above. The first disk 204 may also send the data content of files that users wish to access to the page cache, thereby providing services to users.

[0120] Figure 3 This is a schematic diagram of the structure of a first disk according to an embodiment of this application.

[0121] like Figure 3 As shown, the first disk includes: unallocated space 302, metadata area 304, fragmented data area 306, and reserved space 308.

[0122] The metadata area 304 includes: the beginning 310, the metadata directory 312, and the end 314.

[0123] The metadata directory 312 includes multiple directory objects 316. The directory objects 316 can be found in the descriptions of the Dir objects and the corresponding metadata objects mentioned above.

[0124] The fragmented data area 306 includes a single fragment 318, which can be referred to in the corresponding description of the cache fragments described above. The single fragment 318 may include a fragment header 320 and entity data 322.

[0125] Figure 4 This is a schematic diagram of the movement process of the write pointer in a data processing method according to an embodiment of this application.

[0126] like Figure 4 As shown, if the first disk does not include cache fragments, that is, if no data is written to the disk cache of the first disk, the write pointer position can be position 402.

[0127] After repeatedly executing the data processing method provided in the embodiments of this application to write multiple target data to the first disk, the position of the write pointer can be position 404.

[0128] After repeatedly executing the data processing method provided in the embodiments of this application, thereby writing multiple target data to the first disk, the write pointer position can be position 406. This position 406 can be a preset endpoint position.

[0129] After executing the data processing method provided in this embodiment of the application and writing target data to the first disk, the write pointer can be positioned at position 408. Position 408 can be a preset starting position. That is, the write pointer moves from position 406 to position 408.

[0130] The process of moving the write pointer can be referred to the corresponding explanation of step S104 above.

[0131] Since the technical concepts are similar, please refer to the corresponding descriptions of the foregoing method embodiments.

[0132] In summary, specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing can be advantageous.

[0133] The above are the data processing methods provided in the embodiments of this application. Based on the same idea, the embodiments of this application also provide a data processing device.

[0134] Figure 5 This is a schematic block diagram of a data processing apparatus according to an embodiment of this application, such as... Figure 5 As shown, the data processing apparatus 500 includes: The determining unit 502 is configured to, in response to a data write request for the first disk, determine the target data to be written in the first disk and determine the first position information of the write pointer in the first disk; The moving unit 504 is configured to move the write pointer to a preset starting point if the first position information is a preset endpoint position; and to move the write pointer in a first preset direction and move the distance by a first preset distance if the first position information is not the preset endpoint position. The determining unit 502 is further configured to determine whether a target cache fragment exists in the cache fragments of the first disk based on the second location information; the second location information indicates the position of the write pointer after the move; the position of the target cache fragment in the first disk corresponds to the second location information; Replacement unit 506 is used to replace historical data in the target cache fragment with the target data in response to the existence of the target cache fragment in the cache fragment of the first disk.

[0135] In one embodiment, each cache shard in the first disk has a corresponding metadata object, the metadata object being used to store metadata of the data content in the cache shard; the data processing device 500 further includes: a generation unit; The generation unit is configured to, in response to the existence of the target cache fragment in the first disk, generate metadata of the target data based on the target data and the second location information; The replacement unit 506 is further configured to replace the historical metadata in the metadata object corresponding to the target cache shard with the metadata of the target data.

[0136] In one embodiment, the maximum number of cache shards in the first disk is determined based on the total capacity of the first disk, the effective capacity ratio threshold of the first disk, and the pre-configured average data volume of the shards.

[0137] In one embodiment, there are multiple first disks arranged in a preset order, and the capacity of each first disk is a first preset capacity. Among the multiple first disks, the interval between the initial write positions of the write pointers in two adjacent first disks is determined by the number of first disks and the first preset capacity.

[0138] In one embodiment, the data write request includes a file identifier; when the determining unit 502 determines the target data to be written in the first disk in response to the data write request for the first disk, it performs the following steps: Based on the file identifier, determine the target file corresponding to the data write request; Determine whether the total data volume of the target file is greater than a preset data volume threshold; If the total data volume is greater than the preset data volume threshold, the target file is split into multiple file data, and each file data is determined as one target data; the data volume of each file data is less than or equal to the preset data volume threshold. If the total data volume is less than or equal to the preset data volume threshold, then the data content of the target file is determined as the target data.

[0139] In one embodiment, the data processing device 500 further includes: a generation unit, a query unit, a reading unit, and an acquisition unit; The generation unit is used to generate a corresponding first file identifier based on the first file name carried in the file access request when a file access request is received; The query unit is used to query whether a first metadata object corresponding to the first file identifier exists in each metadata object in the first disk according to the first file identifier; The reading unit is used to read the third position information of the write pointer from the first metadata object if the first metadata object exists. The acquisition unit is used to acquire the data content in the cache fragment corresponding to the third location information based on the third location information.

[0140] In one embodiment, the preset endpoint position is determined by the effective capacity ratio threshold of the first disk, and the effective capacity ratio threshold is determined by the disk type of the first disk.

[0141] In this embodiment, the data processing apparatus includes: a determining unit, configured to, in response to a data write request for a first disk, determine target data to be written in the first disk and determine first position information of a write pointer in the first disk; a moving unit, configured to, if the first position information is a preset endpoint position, move the write pointer to a preset starting position; if the first position information is not the preset endpoint position, move the write pointer in a first preset direction, and the moving distance is a first preset distance; the determining unit is further configured to, based on second position information, determine whether a target cache fragment exists in a cache fragment of the first disk; the second position information indicates the position of the moved write pointer; the position of the target cache fragment in the first disk corresponds to the second position information; and a replacing unit, configured to, in response to the existence of the target cache fragment in a cache fragment of the first disk, replace historical data in the target cache fragment with the target data.

[0142] Those skilled in the art will understand that Figure 5 The data processing device in the document can be used to implement the data processing method described above. The details described therein should be similar to those in the method section above. To avoid being too complicated, they will not be repeated here.

[0143] Based on the same idea, this application also provides an electronic device, such as... Figure 6As shown. Electronic devices can vary considerably due to differences in configuration or performance, and may include one or more processors 601 and memory 602. Memory 602 may store one or more application programs or data. Memory 602 may be temporary or persistent storage. The application programs stored in memory 602 may include one or more modules (not shown), each module may include a series of computer-executable instructions for the electronic device. Furthermore, processor 601 may be configured to communicate with memory 602 and execute the series of computer-executable instructions in memory 602 on the electronic device. The electronic device may also include one or more power supplies 603, one or more wired or wireless network interfaces 604, one or more input / output interfaces 605, and one or more keyboards 606.

[0144] Specifically, in this embodiment, the electronic device includes a memory and one or more programs, wherein one or more programs are stored in the memory, and one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for use in the electronic device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following: In response to a data write request to the first disk, the target data to be written to the first disk is determined, and the first position information of the write pointer in the first disk is determined; If the first position information is a preset end point position, then the write pointer is moved to a preset start point position; if the first position information is not the preset end point position, then the write pointer is moved in a first preset direction, and the moving distance is a first preset distance. Based on the second location information, it is determined whether a target cache fragment exists in the cache fragments of the first disk; the second location information indicates the position of the write pointer after the move; the position of the target cache fragment in the first disk corresponds to the second location information; In response to the presence of the target cache fragment in the cache fragment of the first disk, the historical data in the target cache fragment is replaced with the target data.

[0145] In this embodiment, after determining the target data to be written and the first position information, if the first position information is a preset endpoint position, it means that the number of cache fragments in the first disk has reached the maximum value. At this time, the data with the longest writing time in the entire first disk is the historical data in the first cache fragment in the first disk. By moving the write pointer to the preset starting position and determining the first cache fragment as the target cache fragment based on the moved position, and by replacing the historical data in the target cache fragment with the target data, the historical data with the longest writing time in the first disk can be overwritten by the target data to achieve the clearing of aging data. If the first position information is not the preset endpoint position, the write pointer is moved in the first preset direction and the moving distance is the first preset distance. In this way, each time new data is written, the write pointer will be triggered to move, so as to form an infinite loop of movement. Thus, each data write can automatically overwrite the historical data with the longest writing time in the first disk to achieve the clearing of aging data. The business layer does not need to configure and periodically execute the aging strategy, which reduces the resource consumption and workload of the business layer in handling aging problems.

[0146] This application also proposes a computer-readable storage medium that stores one or more computer programs, each computer program including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform various processes of the above-described data processing method embodiments, specifically for executing: In response to a data write request to the first disk, the target data to be written to the first disk is determined, and the first position information of the write pointer in the first disk is determined; If the first position information is a preset end point position, then the write pointer is moved to a preset start point position; if the first position information is not the preset end point position, then the write pointer is moved in a first preset direction, and the moving distance is a first preset distance. Based on the second location information, it is determined whether a target cache fragment exists in the cache fragments of the first disk; the second location information indicates the position of the write pointer after the move; the position of the target cache fragment in the first disk corresponds to the second location information; In response to the presence of the target cache fragment in the cache fragment of the first disk, the historical data in the target cache fragment is replaced with the target data.

[0147] In this embodiment, after determining the target data to be written and the first position information, if the first position information is a preset endpoint position, it means that the number of cache fragments in the first disk has reached the maximum value. At this time, the data with the longest writing time in the entire first disk is the historical data in the first cache fragment in the first disk. By moving the write pointer to the preset starting position and determining the first cache fragment as the target cache fragment based on the moved position, and by replacing the historical data in the target cache fragment with the target data, the historical data with the longest writing time in the first disk can be overwritten by the target data to achieve the clearing of aging data. If the first position information is not the preset endpoint position, the write pointer is moved in the first preset direction and the moving distance is the first preset distance. In this way, each time new data is written, the write pointer will be triggered to move, so as to form an infinite loop of movement. Thus, each data write can automatically overwrite the historical data with the longest writing time in the first disk to achieve the clearing of aging data. The business layer does not need to configure and periodically execute the aging strategy, which reduces the resource consumption and workload of the business layer in handling aging problems.

[0148] This application provides a computer program product, including a computer program, which is executed by a processor to implement the various processes of the data processing method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0149] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0150] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0151] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0153] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0155] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0156] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0157] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0158] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0159] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0160] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0161] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the scope of the claims of this application.

Claims

1. A data processing method, characterized in that, include: In response to a data write request to the first disk, the target data to be written to the first disk is determined, and the first position information of the write pointer in the first disk is determined; If the first position information is a preset end point position, then the write pointer is moved to a preset start point position; if the first position information is not the preset end point position, then the write pointer is moved in a first preset direction, and the moving distance is a first preset distance. Based on the second location information, it is determined whether the target cache fragment exists in the cache fragment of the first disk; the second location information indicates the position of the write pointer after the move; The location of the target cache fragment in the first disk corresponds to the second location information; In response to the presence of the target cache fragment in the cache fragment of the first disk, the historical data in the target cache fragment is replaced with the target data.

2. The method according to claim 1, characterized in that, Each cache shard in the first disk has a corresponding metadata object, which is used to store the metadata of the data content in the cache shard; The method further includes: In response to the existence of the target cache fragment in the first disk, metadata of the target data is generated based on the target data and the second location information; The historical metadata in the metadata object corresponding to the target cache shard is replaced by the metadata of the target data.

3. The method according to claim 2, characterized in that, The maximum number of cache shards in the first disk is determined based on the total capacity of the first disk, the effective capacity ratio threshold of the first disk, and the pre-configured average data volume of the shards.

4. The method according to claim 1, characterized in that, The number of first disks is multiple and the multiple first disks are arranged in a preset order. The capacity of each first disk is a first preset capacity. Among the multiple first disks, the interval between the initial write positions of the write pointers of two adjacent first disks is determined by the number of first disks and the first preset capacity.

5. The method according to claim 1, characterized in that, The data write request includes a file identifier; the step of determining the target data to be written in the first disk in response to the data write request for the first disk includes: Based on the file identifier, determine the target file corresponding to the data write request; Determine whether the total data volume of the target file is greater than a preset data volume threshold; If the total data volume is greater than the preset data volume threshold, the target file is split into multiple file data, and each file data is determined as one target data; the data volume of each file data is less than or equal to the preset data volume threshold. If the total data volume is less than or equal to the preset data volume threshold, then the data content of the target file is determined as the target data.

6. The method according to claim 1, characterized in that, The method further includes: Upon receiving a file access request, a corresponding first file identifier is generated based on the first file name carried in the file access request; Based on the first file identifier, query each metadata object in the first disk to see if there is a first metadata object corresponding to the first file identifier; If the first metadata object exists, then read the third position information of the write pointer from the first metadata object; Based on the third location information, obtain the data content in the cache fragment corresponding to the third location information.

7. The method according to claim 1, characterized in that, The preset endpoint position is determined by the effective capacity ratio threshold of the first disk, and the effective capacity ratio threshold is determined by the disk type of the first disk.

8. An electronic device, characterized in that, The device includes a processor and a memory electrically connected to the processor, the memory storing a computer program, and the processor being configured to call and execute the computer program from the memory to implement the data processing method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The storage medium is used to store a computer program that can be executed by a processor to implement the data processing method as described in any one of claims 1-7.

10. A computer program product, characterized in that, Includes a computer program, which is executed by a processor to implement the data processing method as described in any one of claims 1-7.

Images