Distributed file processing method and device, equipment and storage medium

Abstract

The invention provides a distributed file processing method and device, equipment and a storage medium, relates to the technical field of computers, can be applied to various scenes such as cloud technology, artificial intelligence, intelligent traffic and auxiliary driving, and the method is applied to a distributed storage system and comprises the steps that in response to data back-brushing triggering for file fragments in a cache pool, the file fragments in the cache pool are cached; obtaining fragment information and fragment data volume of the file fragments, wherein the fragment information comprises a fragment sequence representing that the file fragments correspond to the file fragments in the to-be-processed file to which the file fragments belong; determining a data volume interval to which the file data volume of the to-be-processed file belongs based on the fragment information and the fragment data volume; if the belonging data volume interval is a small file interval, data processing is conducted on the to-be-processed file according to a preset multi-level back-flushing strategy, the small file interval comprises a plurality of sub-intervals, and the multi-level back-flushing strategy is used for indicating the mapping relation between the sub-intervals and various data operations corresponding to the data back-flushing trigger event. The access efficiency of the small file can be remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a distributed file processing method, apparatus, device and storage medium. Background Art

[0002] Distributed storage systems usually use file system services to manage shard storage to achieve concurrent reading and writing. To speed up file reading, a cache pool is built on the basis of the storage pool to provide cache services. Data consistency is regularly transmitted between the cache pool and the storage pool. The data in the cache pool is sent to the storage pool and the synchronized data in the cache pool is deleted. The reading and writing of file data first accesses the cache pool, and then accesses the storage pool if the file is not hit. However, if the cache pool is not hit and data access needs to be obtained from the storage pool, the query operation volume will increase, especially in the multi-file access scenario with high traffic. The access consumption of files that do not hit the cache is very high, which reduces the efficiency of file reading. Summary of the invention

[0003] The present application provides a distributed file processing method, apparatus, device and storage medium, which can significantly improve the timeliness and reliability of distributed file processing.

[0004] On the one hand, the present application provides a distributed file processing method, which is applied to a distributed storage system, wherein the distributed storage system is provided with a storage pool and a cache pool corresponding to the storage pool, and the method comprises:

[0005] In response to a triggering event for data flushing of a file shard in the cache pool, obtaining shard information and shard data volume of the file shard, wherein the shard information includes a shard order corresponding to the file shard in the to-be-processed file, and the data flushing refers to synchronizing the file shard in the cache pool to the storage pool;

[0006] Determine the data volume interval to which the file data volume of the to-be-processed file belongs based on the shard information and the shard data volume;

[0007] If the data volume interval is a small file interval, data processing is performed on the to-be-processed file according to a preset multi-level flushing strategy, the small file interval includes multiple sub-intervals, and the multi-level flushing strategy is at least used to indicate the mapping relationship between the multiple sub-intervals and the multiple data operations corresponding to the triggering event of the data flushing. On the other hand, a distributed file processing device is provided, which is applied to a distributed storage system, the distributed storage system is provided with a storage pool and a cache pool corresponding to the storage pool, and the device includes:

[0008] A first acquisition module: in response to a triggering event of data flushing for a file shard in the cache pool, acquires shard information and shard data volume of the file shard, wherein the shard information includes a shard order corresponding to the file shard in the to-be-processed file, and the data flushing refers to synchronizing the file shard in the cache pool to the storage pool;

[0009] An interval determination module: used for determining the data volume interval to which the file data volume of the to-be-processed file belongs based on the fragment information and the fragment data volume;

[0010] Flush response module: used to process the file to be processed according to a preset multi-level flush strategy if the data volume interval is a small file interval. The small file interval includes multiple sub-intervals, and the multi-level flush strategy is at least used to indicate the mapping relationship between the multiple sub-intervals and the multiple data operations corresponding to the triggering event of the data flush.

[0011] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the distributed file processing method as described above.

[0012] On the other hand, a computer-readable storage medium is provided, in which at least one instruction or at least one program is stored. The at least one instruction or the at least one program is loaded and executed by a processor to implement the distributed file processing method as described above.

[0013] On the other hand, a server is provided, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the distributed file processing method as described above.

[0014] On the other hand, a terminal is provided, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the distributed file processing method as described above.

[0015] On the other hand, a computer program product or a computer program is provided, which includes computer instructions, and when the computer instructions are executed by a processor, the distributed file processing method as described above is implemented.

[0016] The distributed file processing method, apparatus, device, storage medium, server, terminal, computer program and computer program product provided by this application have the following technical effects:

[0017] The present application obtains the shard information and shard data volume of the file shard in response to the trigger event of data flushing for the file shard in the cache pool, the shard information includes the shard order corresponding to the file shard in the to-be-processed file, and data flushing refers to synchronizing the file shard in the cache pool to the storage pool; based on the shard information and the shard data volume, the data volume interval to which the file data volume of the to-be-processed file belongs is determined; if the data volume interval to which it belongs is a small file interval, data processing is performed on the to-be-processed file according to a preset multi-level flushing strategy, the small file interval includes multiple sub-intervals, and the multi-level flushing strategy is at least used to indicate the mapping relationship between the multiple sub-intervals and the multiple data operations corresponding to the trigger event of data flushing; in this way, through the shard information and the shard order, the data volume interval to which the file data volume of the to-be-processed file belongs is determined; if the data volume interval to which it belongs is a small file interval, data processing is performed on the to-be-processed file according to the preset multi-level flushing strategy, the small file interval includes multiple sub-intervals, and the multi-level flushing strategy is at least used to indicate the mapping relationship between the multiple sub-intervals and the multiple data operations corresponding to the trigger event of data flushing The data volume of the file is determined by the slice data volume, which overcomes the defect that the distributed storage system cannot effectively perceive the file size, and uses the original slice information in the system as the information basis for judging the data volume. There is no need to change the data storage settings and add complex judgment components, which reduces the difficulty of system adaptation. In addition, through a multi-level flushing strategy, reasonable selection of data operations is made for the file slices of small files corresponding to different data volume intervals, so as to select data flushing or not based on the interval to which the file data volume belongs, realize hierarchical storage of small files, realize persistent storage of some small files in the cache pool, reduce multiple queries in the process of reading small files and reduce the corresponding number of data writes, effectively reduce file reading and writing consumption, and thus improve data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application;

[0020] Figure 2 It is a flowchart of a distributed file processing method provided by an embodiment of the present application;

[0021] Figure 3 It is a schematic diagram of the principle of data flushing in the prior art;

[0022] Figure 4 It is a schematic diagram of the principle of data elimination in the prior art;

[0023] Figure 5 It is a schematic diagram of data sharding storage provided by an embodiment of the present application;

[0024] Figure 6 It is a flowchart of another distributed file processing method provided by an embodiment of the present application;

[0025] Figure 7 It is a flowchart of another distributed file processing method provided by an embodiment of the present application;

[0026] Figure 8 It is a schematic diagram of data storage in a cache pool or storage pool provided in an embodiment of the present application;

[0027] Fig. 9 and Fig.10 This is a schematic diagram of the principle of data flushing provided by an embodiment of the present application;

[0028] Fig.11 It is a schematic diagram of a framework of a disk data reading and writing service in a cache pool provided by an embodiment of the present application;

[0029] Fig.12 This is a schematic diagram of the principle of data elimination provided by an embodiment of the present application;

[0030] Fig.13 It is a schematic diagram of a framework of a distributed file processing device provided by an embodiment of the present application;

[0031] Fig.14 It is a hardware structure block diagram of an electronic device of a distributed file processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server comprising a series of steps or submodules is not necessarily limited to those steps or submodules clearly listed, but may include other steps or submodules that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0035] Ceph: an open source distributed storage system that provides object, file, and block storage services.

[0036] OSD (Object Storage Device): Ceph's data storage service that manages the reading and writing of disk data;

[0037] MDS (Metadata Server): Cephfs metadata service (Ceph MetadataServer), used to manage the directory tree of the file system;

[0038] Message Queue (MQ): A "first-in, first-out" data structure, usually used to solve problems such as application decoupling, asynchronous messaging, and traffic peak shaving.

[0039] See also Figure 1 , Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application, such as Figure 1 As shown, the application environment may include at least a terminal 01 and a server 02. In practical applications, the terminal 01 and the server 02 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0040] The server 02 in the embodiment of the present application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0041] Specifically, cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or a local area network to achieve data computing, storage, processing, and sharing. Cloud technology can be applied to various fields, such as medical cloud, cloud Internet of Things, cloud security, cloud education, cloud conferencing, artificial intelligence cloud services, cloud applications, cloud calls, and cloud social networking. Cloud technology is based on the cloud computing business model application. It distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space, and information services as needed. The network that provides resources is called a "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, expanded at any time, and paid for by use. As a basic capability provider for cloud computing, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service)) platform will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0042] According to the logical function division, the PaaS (Platform as a Service) layer can be deployed on the IaaS layer, and the SaaS (Software as a Service) layer can be deployed on the PaaS layer, or SaaS can be directly deployed on IaaS. PaaS is a platform for software operation, such as databases, web containers, etc. SaaS is a variety of business software, such as web portals, SMS mass senders, etc. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0043] Specifically, the server 02 mentioned above may include a physical device, which may specifically include a network communication submodule, a processor, a memory, etc., and may also include software running in the physical device, which may specifically include an application program, etc.

[0044] Specifically, terminal 01 may include physical devices such as smart phones, desktop computers, tablet computers, laptops, digital assistants, augmented reality (AR) / virtual reality (VR) devices, intelligent voice interaction devices, smart home appliances, smart wearable devices, and vehicle-mounted terminal devices, and may also include software running in physical devices, such as applications.

[0045] In an embodiment of the present application, terminal 01 is used to run a client to receive a data read operation or a data write operation from a data read initiator to generate a corresponding file access instruction or a file write instruction, and forward it to server 02; a distributed storage system having a cache pool and a storage pool can be deployed on server 02, for performing corresponding file search, reading and forwarding operations, or performing write operations in response to file access instructions or file write instructions; and is also used to, when a data flush trigger is detected, perform a data volume interval judgment on the to-be-processed file to which the corresponding file shard belongs, and determine the corresponding data operation in a preset multi-level flush strategy based on the judgment result, and then perform data processing on the file shard corresponding to the to-be-processed file. If the data volume interval belongs to a small file interval, it is determined not to perform data synchronization oriented to the storage pool, so that the small files are retained in the cache pool to facilitate the reading of small files in the cache pool.

[0046] Furthermore, it is understandable that Figure 1 What is shown is merely an application environment of a distributed file processing method, and the application environment may include more or fewer nodes, and this application does not impose any limitation thereto.

[0047] The application environment involved in the embodiment of the present application, or the terminal 01 and the server 02 in the application environment, etc., can be a distributed system formed by connecting a client and multiple nodes (any form of computing devices in the access network, such as a server, a user terminal) through network communication. The distributed system can be a blockchain system, which can provide the above-mentioned distributed file processing services and data storage services, etc.

[0048] Specifically, the distributed storage system is provided with a storage pool and a cache pool corresponding to the storage pool. Taking the Ceph system as an example, refer to Figure 3 , including SSD (Solid State Disk) cache pool and HDD (Hard Disk Drive) storage pool, to store metadata and data in separate pools. The client directly accesses the SSD cache pool, and the reading and writing of file data will first access the SSD cache pool, and then access the HDD storage pool if there is no hit. However, when the cache pool misses and needs to go to the HDD pool to obtain data access, the performance is worse than directly reading the storage pool without building a cache pool, especially when accessing a large number of small files. Since the data volume of small files themselves is small, if the cache pool does not hit, and then goes to the HDD pool to access, the access consumption of small files will be relatively high. In addition, the current data cache pool cannot identify small files, and the flushing and deletion of file data are indistinguishable.

[0049] In an example, refer to Figure 3 and Figure 4, Cephfs file data in the cache pool and storage pool are stored in 4M slices. The data of large files may occupy multiple 4M objects, and small files generally only occupy part of a 4M object. After the client performs a write IO operation on a large or small file, some 4M data blocks corresponding to the file in the cache pool are modified, which will change the object mark, such as marking it as dirty data, to indicate that the data has not been flushed back to the storage pool. The "dirty data" here refers to the data in the cache pool that has not been synchronized and stored in the storage pool. Existing file systems will perform regular flush operations on dirty data, and after flushing, change the object mark to indicate that it has been flushed, such as marking it as clean data, and subsequently delete (eliminate) the clean data in the cache pool to free up cache space. The "clean data" here refers to the data in the cache pool that has been successfully synchronized and stored in the storage pool. However, the data size of small files is generally in the KB range, corresponding to at most one data block in the cache pool. If the small file does not hit the cache pool when accessed due to cache pool flushing and elimination, then compared with the solution where no cache pool is built and only one access to the storage pool is required, file data access in this way requires one more operation of writing data to the cache pool and one more operation of reading the cache pool, which increases the file access overhead ratio. Instead of being accelerated, small file requests are slowed down.

[0050] In view of this, in order to solve at least one of the above problems, this application introduces a distributed file processing method based on the above application environment, which is applied to a distributed storage system, including a cache pool and a storage pool corresponding to the cache pool. There is a data flushing processing requirement between the storage pool and the cache pool. The embodiment of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. Please refer to Figure 2 , Figure 2 It is a flowchart of a distributed file processing method provided by an embodiment of the present application. This specification provides method operation steps such as the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the actual system or server product is executed, it can be executed in the order of the method shown in the embodiment or the figure or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, as Figure 2 As shown, the method may include the following steps S201-S205:

[0051] S201: In response to a triggering event for flushing data of a file shard in a cache pool, obtain shard information and shard data volume of the file shard.

[0052] Specifically, each file to be processed may correspond to one or more file shards, each file shard corresponds to a data block, the size of the data block is set based on actual needs, such as the 4M data block in the aforementioned Ceph, each file shard corresponds to a file object, and each file to be processed may correspond to a file object set including one or more file objects. The shard information includes a shard order that represents the file shards in the corresponding file to be processed, and the shard order refers to the file shard's shard number of the corresponding file to be processed. Specifically, the shard information may be a shard number. Data flushing refers to synchronizing the file shards in the cache pool to the storage pool, such as copying and storing the shard data of each file shard corresponding to the file to be processed to the storage pool.

[0053] Specifically, the shard information can be obtained based on the object name of the file object corresponding to the file shard, and the object name can include a file identifier and shard information, and the file identifier is a unique identification number of the to-be-processed file to which the file shard belongs. Exemplarily, the object name can be {inode number}.{shard number}, {inode number} is the file identifier, and {shard number} is the shard information.

[0054] Specifically, the shard data volume refers to the data size of the file shard, and the shard data volume of each file shard is less than or equal to a preset data block size, such as the aforementioned 4M.

[0055] For example, reference Figure 5 In the file mapping diagram, the file data volume of a file is 22M. Based on the setting of 4M as a data block, it is divided into 6 file slices (slices 0-5). Correspondingly, the file logical address will also be divided into 6 file objects according to the data block size. In the storage pool and cache pool, the file logical address corresponds to different data blocks and corresponding file objects in the cache pool. The slice data volume of file slices 0-4 is 4M, the data volume of file slice 5 is 2M, the file slice with slice number 00000000 is the first slice of the file, and the slice number 00000005 represents the sixth slice. It can be understood that the data volume of a small file is generally smaller than the data block size, such as KB level, such as 128KB, 512KB, and only contains one file slice, that is, it corresponds to only one data block in the cache pool or storage pool, and its file logical address corresponds to only one file object.

[0056] In some embodiments, reference Figure 6 The triggering event of data flushing is generated based on the following S301-S307:

[0057] S301: Obtain a queue of information to be flushed back, where the queue of information to be flushed back is used to store placement group information of placement groups to be flushed back corresponding to a cache pool, and each placement group includes a group of file objects corresponding to file slices.

[0058] Specifically, a placement group (Pg) includes a group of file objects, which are used to manage the status of the group of file objects, such as changing the object tag after writing or flushing a file object. Multiple data read and write services can also be run in the cache pool or storage pool, and each data read and write service can correspond to multiple placement groups. The placement group information includes information used to characterize the attributes of the placement group, and can include identification information of the placement group, which is placed in the queue of information to be flushed to wait for subsequent flushing processing operations. In some embodiments, the queue of information to be flushed has priority information, and the queue priority of the placement group information can be determined according to the data volume of the file slices that have not been flushed back. It can be understood that the higher the queue priority, the larger the amount of file slice data that has not been flushed back to storage corresponding to the placement group information, and vice versa. If the total data volume of dirty data corresponding to the placement group information is 100MB, it is added to the queue of information to be flushed back with the second priority. If the data volume is 400MB, it is added to the queue of information to be flushed back with the first priority. It can be understood that the data flushing is triggered first from the placement group information of the queue of information to be flushed back with a high priority.

[0059] S303: Obtain object information of each file object corresponding to the placement group information to be flushed.

[0060] Specifically, for each placement group information in the queue of information to be flushed back, the object information of each file object included in the placement group information is listed, and the object information includes an object tag for indicating the fragment data state of the file fragment and the duration of the fragment data state. The fragment data state may include not being flushed back to the storage pool and being flushed back to the storage pool. The object tag of the former may be, for example, "dirty data", and the object tag of the latter may be, for example, "clean data". The duration refers to the duration of the file fragment being in the current fragment data state, such as the duration of being marked as "dirty data".

[0061] S305: If the object tag indicates that the data status of the file segment is not flushed back to the storage pool, and the duration exceeds a preset status duration, determine to generate a trigger event for flushing data corresponding to the file segment.

[0062] Specifically, the preset state duration refers to the upper limit of the time that the file segment that needs to perform data flushing operation is in the state of not flushing back to the storage pool. It can be set based on actual needs, such as based on the capacity of the cache pool. The larger the capacity, the longer the preset state duration, and vice versa. Combined with the data state of the segment and the duration, it is judged whether to generate a trigger event for data flushing. When the object tag of the file object indicates that the file segment is data that has not been flushed back to the storage pool and the duration of this state exceeds the preset state duration, it indicates that it meets the data flushing conditions, and a trigger event for data flushing corresponding to the file segment is generated. The above operation is performed for each file object in the placement group information to realize the flushing judgment of a single file segment. Exemplarily, it is judged whether the file segment is "dirty data" through the object tag. If it is and the duration of being in the dirty state exceeds the preset state duration (such as 600s), a trigger event for data flushing corresponding to the file segment is generated, and then S203 is executed.

[0063] S307: If the object mark indicates that the fragment data status of the file fragment is not flushed back to the storage pool and the duration does not exceed the preset status duration, or the object mark indicates that the fragment data status of the file fragment is flushed back to the storage pool, it is determined that a trigger event for data flushing corresponding to the file fragment is not generated.

[0064] For example, if the file shard is "dirty data" and the duration of being in the dirty state is less than the preset state duration (such as 600s), or the file shard is "clean data", data flushing is not triggered. By setting a flushing queue and combining the object information of the file object to judge the state of the shard, a separate data flushing trigger for each file shard can be achieved, which can realize fine-grained data management at the shard level and facilitate the persistent storage of subsequent small files.

[0065] In some embodiments, the acquisition of group information placed in the queue for information to be flushed can be triggered based on the amount of stored data in the cache pool reaching the upper limit of the amount of data to be flushed. When the amount of stored data reaches the upper limit of the amount of data to be flushed, after the data is flushed, the flushed file fragments will be marked as "clean data".

[0066] S203: Determine the data volume interval to which the file data volume of the to-be-processed file belongs based on the fragment information and the fragment data volume.

[0067] It can be understood that if the files to be processed need to be stored in a hierarchical manner, the file size must first be sensed. By judging the order of the file shards in the files to be processed and the amount of data in a single shard, it can be determined whether the files to be processed exceed a single data block, and then its data volume range can be determined.

[0068] Specifically, based on the data volume interval to which the file data volume belongs, the to-be-processed file can be classified into small files and large files, corresponding to the small file interval and the large file interval, respectively. A small file refers to a file whose file data volume is less than the data block size, and a large file refers to a file whose file data volume is greater than or equal to the data block size. It can be understood that a small file includes only one file fragment, and a large file may include two or more file fragments. Exemplarily, the data block size may be 4M, and accordingly, the small file interval is 0 to 4M, and the large file interval is greater than 4M.

[0069] In some embodiments, S203 may include:

[0070] S401: If the fragment information indicates that the fragment order of the file fragment is the first fragment, and the fragment data volume is less than or equal to the data block size, determine that the data volume interval to which the file data volume of the to-be-processed file belongs is a small file interval;

[0071] S403: If the shard data volume is equal to the data block size, or the shard information indicates that the shards of the file shards are sorted as other shards after the first shard, determine that the data volume interval to which the file data volume of the to-be-processed file belongs is a large file interval.

[0072] Specifically, the data block size refers to the size of the data block in the distributed storage system, and the data block is the basic unit of file storage and transmission in the distributed storage system. The first shard indicates that it corresponds to the first data block of the file to be processed. If the data size of the first shard is smaller than the data block size, it indicates that the file data volume of the file to be processed is smaller than the data volume of a single data block, and thus belongs to the small file interval from 0 to the data block size. If the file to be processed corresponds to more than one file shard, its file data volume must be larger than the data block size, and it corresponds to a large file. If the shard data volume of the first shard is equal to the data block size, it indicates that the file to be processed belongs to the large file interval. Exemplarily, refer to Figure 8 , for file fragment 00000005, it indicates that its fragment ranking is 6, and the corresponding file to be processed is a large file.

[0073] S205: If the data volume interval is a small file interval, data processing is performed on the to-be-processed file according to a preset multi-level flushing strategy.

[0074] Specifically, the small file interval includes multiple sub-intervals, and the multi-level flushing strategy is at least used to indicate the mapping relationship between the multiple sub-intervals and the triggering events of the data flushing, and the data operations corresponding to different data volume intervals are different. According to the sub-interval to which the file data volume of the to-be-processed file belongs, the target data operation matched in the multi-level flushing strategy is determined, and then the file segment of the to-be-processed file is processed based on the target data operation. The multiple sub-intervals at least include the first-level file interval, the second-level file interval and the third-level file interval, in which the file data volume increases in sequence. The first-level file interval is the interval with the smallest file data volume preset in the distributed storage system, that is, based on the data volume, the small files can also be at least divided into the first-level small files corresponding to the first-level file interval, the second-level small files corresponding to the second-level file interval, and the third-level small files corresponding to the third-level small files. The file data volumes of the first-level small files, the second-level small files and the third-level small files increase in sequence. The multiple data processing operations at least include the operation of not executing data flushing corresponding to the first-level file interval, the operation of probabilistic data flushing corresponding to the second-level file interval, and the operation of data flushing corresponding to the third-level file interval.

[0075] It can be understood that more than one second-level file interval can be set, that is, multiple second-level file intervals with increasing file data volumes can be set, and different second-level file intervals correspond to different data flushing probabilities. As the file data volume increases, the data flushing probability corresponding to the second-level file interval increases, and vice versa.

[0076] Accordingly, in some embodiments, S401 may include S4011: if the shard information indicates that the shard order of the file shard is the first shard, and the shard data volume is less than or equal to the first data volume, determine that the data volume interval is the first-level file interval. The first data volume is less than the size of a single data block in the distributed storage system and is the upper limit of the first-level file interval. The data block here is the smallest storage unit in the distributed storage system, such as the aforementioned 4M; illustratively, refer to Figure 8 , the first data volume is 128k, then the to-be-processed files whose shard data volume of the first shard (00000000) is less than or equal to 128k belong to the small file interval. The interval judgment of the to-be-processed files is performed by combining the original shard information and shard data volume in the distributed storage system, without changing the data storage settings and adding complex judgment components, and can be simply adapted to various distributed storage systems, overcoming the defect that the distributed storage system cannot effectively perceive the file size.

[0077] Further, S203 may also include S4012: if the shard information indicates that the shard order of the file shard is the first shard, and the shard data volume is greater than the first data volume and less than or equal to the second data volume, determine that the data volume interval is a second-level file interval. Specifically, the second data volume is less than the size of a single data block and greater than the first data volume. For example, refer to Figure 8 , the second data volume can be 512k. If the data volume of the first shard is greater than 128k and less than or equal to 512k, it belongs to the second-level file interval. In this way, the smallest file and secondary small files are distinguished, the file classification granularity is refined, and the hierarchical storage of small files of different levels is realized, which effectively utilizes the cache pool space and avoids excessive cache occupancy.

[0078] It can be understood that the file can also be divided into multiple second-level small files with increasing file data volume, that is, multiple second-level file intervals can be set, such as including a first sub-level interval, a second sub-level interval, etc. Exemplarily, the first sub-level interval can be a shard data volume greater than 128k and less than or equal to 218k, the second sub-level interval can be a shard data volume greater than 128k and less than or equal to 512k, etc.

[0079] Furthermore, S203 may also include S4013: if the sharding information indicates that the sharding order of the file sharding is the first sharding, and the sharding data volume of the first sharding is greater than the second data volume, determine that the data volume interval to which it belongs is a third-level file interval. The third-level file interval is a file whose data volume is greater than the second data volume and less than the data block size. Exemplarily, the third-level file interval may be, for example, greater than 512k and less than 4M. In this way, combined with the data volume partitioning setting based on the data block, hierarchical perception of small files and large files is realized, which is beneficial to the subsequent hierarchical flushing processing.

[0080] Accordingly, S205 may include S2051: if the data volume interval is a first-level file interval, then a data flushing operation on the file segment corresponding to the to-be-processed file is not performed.

[0081] Specifically, based on the correspondence between the data volume interval and the multiple data operations corresponding to the triggering events of data flushing in the multi-level flushing strategy, the target flushing strategy corresponding to the data volume interval to which the to-be-processed file belongs is determined. The target flushing strategy corresponding to the first-level file interval is not to flush the data of the to-be-processed file, that is, not to copy it to the storage pool. For example, refer to Fig. 9, for the file shard of ino.00000000, if it is determined that the file to be processed is the first-level small file among the small files, no data flushing operation will be performed, and the current shard mark of the file shard will be maintained, such as "dirty data". In this way, the first-level small files can be stored in the cache pool permanently, and can be directly hit when accessed without re-searching the storage pool, and the repeated transmission of small files between the cache pool and the storage pool can be avoided, thereby improving the access efficiency of small files and reducing the processing consumption of small files.

[0082] Further, S205 may include S2052-S2054:

[0083] S2052: If the data volume interval is a second-level file interval, generate flush indication information based on attribute information of the file to be processed.

[0084] Specifically, the flush information generation refers to mapping the attribute information to a numerical value to obtain a corresponding attribute string, and determining the flush indication information in combination with the attribute string. In some embodiments, S2052 may specifically include:

[0085] S501: Perform a hash operation on the attribute information of the file to be processed to obtain an attribute string;

[0086] S503: extracting indication information from the attribute character string to obtain refresh indication information.

[0087] Specifically, the attribute information is mapped into a numerical attribute string through a hash operation, and the attribute strings corresponding to the same attribute information are consistent, so that the flush indication information corresponding to the same to-be-processed file is consistent, thereby unifying the data operations corresponding to the triggering events of its data flush. In one example, the hash operation is a hash operation, and accordingly, the attribute string is a hash value. The flush indication information can be a partial field of the attribute string itself, such as the last digit, or it can also be a partial field in the base string obtained after base conversion of the attribute string, such as the last digit.

[0088] In one embodiment, S503 specifically includes: performing binary conversion on the attribute string to obtain a binary string; determining the value of the specified bit sequence in the binary string as the flush indication information. That is, the attribute string is converted into a string composed of binary digits, and then a digit of its fixed bit sequence is selected as the flush indication information. Exemplarily, the specified bit sequence can be the last digit, and the flush indication information is 1 to indicate a flush is confirmed, and if it is 0, it indicates no flush. In this way, the binary conversion is used to achieve equal probability generation of the two types of flush indication information, optimize the file storage structure and achieve efficient generation of flush responses.

[0089] S2053: If the flush indication information indicates that flushing is confirmed, a data flush operation is performed on the file segments corresponding to the to-be-processed file, so as to synchronously store the file segments in the storage pool.

[0090] S2054: If the flush indication information indicates not to flush, the data flush operation on the file segment corresponding to the to-be-processed file is not performed.

[0091] Specifically, in the case of determining to flush, the second-level small files are processed in the same way as large files, stored in the storage pool and the object tag is modified, such as changing it to "net data". If there is no flush, it is consistent with the processing method of the first-level small files, no data flush is performed, and the current object tag is maintained. A probabilistic flush mechanism is adopted for the files to be processed in the second-level file interval, and the representation meaning of the flush indication information is combined to determine whether to store it in the storage pool, reduce the flush traffic of the second-level small files, effectively utilize the persistence capability and storage capacity of the cache pool, and avoid occupying too much cache pool space to prevent the cache limitation of subsequent large files.

[0092] It can be understood that when there are multiple second-level file intervals, the probability of performing data flushing for the to-be-processed file increases with the increase of the file data volume corresponding to the file interval, that is, the larger the file data volume corresponding to the second-level file interval, the greater the probability of performing data flushing. Exemplarily, the flushing indication information is the last digit of the quaternary string corresponding to the attribute string. For the first sub-level interval exemplified above, if the flushing indication information is 0 or 1, data flushing is performed, if it is 2 or 3, data flushing is not performed. For the second sub-level interval, if the flushing indication information is 0, 1 or 2, data flushing is performed, if it is 3, data flushing is not performed.

[0093] Further, S205 may also include S2055: if the data volume interval belongs to a third-level file interval, a data flushing operation is performed on the file to be processed to synchronously store the file fragments to the storage pool. The data processing method corresponding to the third-level file interval is the same as that of the large file, and it is synchronized to the storage pool to reduce the data volume pressure of the cache pool. In this way, multi-level data flushing and hierarchical storage for small files are implemented to minimize the amount of read operations on small files and improve file query efficiency. For example, refer to Fig. 9, the to-be-processed file to which the file shard ino.00000000 belongs is 1M, and belongs to the third-level file interval. In response to the triggering event of data flushing, the data is directly flushed to copy it to the storage pool. After the storage is successful, the shard mark of the file shard is modified to indicate that the storage pool has been flushed, such as modified to "net data". It can be understood that compared with the first-level small files and the second-level small files, the data volume of the third-level small files is larger. If the cache pool is not hit and the storage pool is searched again, the redundant average overhead is small, and the indifferent data flushing for the third-level files and large files is conducive to the space release of the cache pool and the rationality of optimizing file storage.

[0094] In the embodiment of the present application, the method further includes S207: if the data volume interval is a large file interval, a data flushing operation is performed on the to-be-processed file to synchronously store the file fragments in the storage pool. The average overhead of re-searching the storage pool for a large file that misses the cache pool is small and occupies a large amount of memory. Flushing it back to the storage pool without distinction is conducive to freeing up space in the cache pool.

[0095] In an example, refer to Fig.10 , the client writes large or small files into the cache pool. For files below 4M, only one data block is stored, corresponding to one file shard. For data above 4M, sharding is performed in units of 4M and stored in more than one data block, corresponding to more than one file object. When the duration of dirty data in the cache pool exceeds the preset state duration, the data volume interval to which it belongs is determined through file perception, and a multi-level flushing strategy is implemented for the small file interval. The first-level small files are not flushed for persistent storage. For the second-level small files, probabilistic flushing is performed based on the flushing indication information. For the third-level small files and large file intervals, all flushing is performed.

[0096] In one embodiment, reference Fig.11, the disk data read and write service runs the flush mode selection module, the pg object acquisition module, the flush timing judgment module, the small file perception module, the hierarchical flush strategy module and the flush execution module. Specifically, the cache pool organizes the placement group information (pg) of the dirty data to be flushed in the queue of information to be flushed, and takes out the pg from the queue of information to be flushed with high priority for the flush process; the flush mode selection module obtains the proportion of dirty data in the current cache pool to set the flush speed; then, the pg object acquisition module lists the file objects inside the selected pg, and obtains its object information to input the flush time judgment module; the flush time judgment module determines whether the file segment corresponding to the file object is "dirty data" based on the object information, and whether the file segment is in the "dirty number" Whether the duration of the "data" state reaches the preset state duration (for example, 600s). If any of them is not reached, the data flushing operation will not be performed; if all are reached, the data volume interval to which the file shard belongs is determined through the small file perception module, and the result is sent to the hierarchical flushing strategy module to determine the target flushing strategy, and the judgment process of whether to flush described in the previous text is executed. If the judgment result is to determine to flush, it will be transferred to the flushing execution module to copy the shard data of the file shard from the cache pool to the storage pool; if it is determined not to flush, it is determined that the data operation corresponding to the triggering event of the data flush is not to flush the data. In this way, the flushing flow of small files can be reduced through a multi-level flushing strategy, so that the dirty small files can be kept in the cache pool in a dirty state, which helps small files to persist in the cache pool for a long time.

[0097] It is understandable that the space capacity of the cache pool is limited, so the data stored in it needs to be cleaned up regularly. Figure 7 The method may further include S601-S603:

[0098] S601: In response to the amount of data stored in the cache pool satisfying a data cleanup condition, filter out file shards whose shard data status is that they have been flushed back to the storage pool;

[0099] S603: Delete the file fragments that have been flushed back to the storage pool from the cache pool.

[0100] Specifically, the amount of stored data refers to the total amount of data occupied in the cache pool. If the amount of stored data reaches the upper limit of the amount of eliminated data, it is determined that the data cleaning conditions are met, and file shards are screened for each stored file to be processed. In some cases, all file shards that have been flushed back to the storage pool can be deleted. In other cases, file shards that are marked as having been flushed back to the storage pool and whose duration of this state exceeds the specified duration can be deleted. It can be understood that for files that have been flushed back, regular cleaning is performed in the above manner to free up cache pool space, and for the first-level small files and some second-level small files among the small files, since they are always in a state of not being flushed back to the storage pool, that is, no data cleaning operation will be performed, so that they can be persistently stored in the cache pool, so that each access can directly hit the cache pool without redundant queries and writes, thereby optimizing their access performance.

[0101] In an example, refer to Fig.12 , a queue of information to be eliminated is constructed in the cache pool, in which the placement group information to be eliminated is stored, and the priority of the placement group information is set based on the amount of data to be eliminated; the elimination mode selection module is used to set the data cleaning speed according to the amount of data to be eliminated, and then obtain the file objects corresponding to the placement group information taken out by the object acquisition module in the pg, and then determine whether the file object is "clean data" through the elimination judgment module. If so, it will be transferred to the elimination judgment module for data clearing, if not, the data cleaning operation will not be performed. The object mark of the first-level small files and some second-level small files that do not perform data flushing is always "dirty data", which will be persisted in the cache pool. Subsequent access to the file will hit in the cache pool. For the deleted "clean data" (another part of the second-level small files and large files), it will not hit in the cache pool and needs to be further accessed in the storage pool. In this way, the performance of small file access in the tiered storage cache is improved.

[0102] The technical solution of the present application senses the data volume range of small files in the cache pool and executes a multi-level flushing strategy accordingly, so that the first-level small files and some second-level small files are persisted in the cache pool, providing them with better access services, so that the distributed file system can be better applied to scenarios where small files are frequently accessed, such as AI scenarios with a large number of small files.

[0103] The present application embodiment also provides a distributed file processing device 13, such as Fig.13 As shown, Fig.13 A schematic diagram of the structure of a distributed file processing device provided in an embodiment of the present application is shown, and the device may include the following modules.

[0104] The first acquisition module 10 is used to obtain the shard information and shard data volume of the file shard in response to the trigger event of data flushing for the file shard in the cache pool, wherein the shard information includes the shard order corresponding to the file shard in the to-be-processed file, and the data flushing refers to synchronizing the file shard in the cache pool to the storage pool;

[0105] The interval determination module 20 is used to determine the data volume interval to which the file data volume of the to-be-processed file belongs based on the fragment information and the fragment data volume;

[0106] The flush response module 30 is used to process the file to be processed according to a preset multi-level flush strategy if the data volume interval is a small file interval. The small file interval includes multiple sub-intervals. The multi-level flush strategy is at least used to indicate the mapping relationship between multiple sub-intervals and multiple data operations corresponding to the triggering events of the data flush.

[0107] In some embodiments, the multiple sub-intervals include at least a first-level file interval, a second-level file interval, and a third-level file interval in which the file data volume increases successively, and the multiple data processing operations include at least an operation of not performing data flushing corresponding to the first-level file interval, an operation of probabilistic data flushing corresponding to the second-level file interval, and an operation of data flushing corresponding to the third-level file interval.

[0108] In some embodiments, the flush response module 30 is further used for not performing a data flush operation on the file segment corresponding to the to-be-processed file if the data volume interval to which it belongs is a first-level file interval.

[0109] In some embodiments, the flush response module 30 is also used to: if the data volume interval to which it belongs is a second-level file interval, generate flush indication information based on the attribute information of the file to be processed; if the flush indication information indicates that the flush is confirmed, perform a data flush operation on the file fragments corresponding to the file to be processed, so as to synchronously store the file fragments in the storage pool.

[0110] In some embodiments, the refresh response module 30 includes:

[0111] Hash operation submodule: used to perform hash operation on the attribute information of the file to be processed to obtain an attribute string. The attribute strings corresponding to the same attribute information are consistent.

[0112] Information extraction submodule: used to extract indication information from the attribute character string to obtain the refresh indication information.

[0113] In some embodiments, the information extraction submodule includes:

[0114] Binary conversion unit: used to convert the attribute string into binary to obtain a binary string;

[0115] Indication determination unit: used to determine the value of the specified bit sequence in the binary string as the flush indication information.

[0116] In some embodiments, the flush response module 30 is further used to: if the data volume interval belongs to a third-level file interval, perform a data flush operation on the file to be processed to synchronously store the file fragments in the storage pool.

[0117] In some embodiments, the device also includes a large file flushing module: if the data volume interval to which it belongs is a large file interval, a data flushing operation is performed on the file to be processed, so as to synchronously store the file fragments in the storage pool.

[0118] In some embodiments, the interval determination module 20 includes a first interval sub-module: if the shard information indicates that the shard order of the file shard is the first shard and the shard data volume is smaller than the data block size, determine that the data volume interval to which the file data volume of the file to be processed belongs is a small file interval, and the data block size refers to the size of the data block in the distributed storage system, and the data block is the basic unit of file storage and transmission in the distributed storage system.

[0119] In some embodiments, the interval determination module 20 includes a second interval submodule: used to determine that the data volume interval to which the file data volume of the file to be processed belongs is a large file interval if the shard data volume is equal to the data block size, or the shard information indicates that the shards of the file shard are sorted as other shards after the first shard.

[0120] In some embodiments, the apparatus further comprises:

[0121] Queue acquisition module: used to acquire the queue of information to be flushed back, which is used to store the placement group information of the placement group to be flushed back corresponding to the cache pool, and each placement group includes a group of file objects corresponding to the file slices;

[0122] Object information acquisition module: used to acquire object information of each file object corresponding to the placement group information to be flushed, the object information including an object tag for indicating the fragment data state of the file fragment and the duration of the fragment data state;

[0123] Flush trigger module: used to determine the trigger event for generating a data flush corresponding to a file shard if the object tag indicates that the shard data status of the file shard is not flushed to the storage pool and the duration exceeds the preset status duration.

[0124] In some embodiments, the apparatus further comprises:

[0125] The shard screening module is used to screen out the file shards whose shard data status has been flushed back to the storage pool in response to the amount of data stored in the cache pool meeting the data cleaning condition;

[0126] Fragment deletion module: used to delete file fragments that have been flushed back to the storage pool from the cache pool.

[0127] It should be noted that the above device embodiment and method embodiment are based on the same implementation method.

[0128] An embodiment of the present application provides a distributed file processing device, which can be a terminal or a server, including a processor and a memory, in which at least one instruction or at least one program is stored. The at least one instruction or the at least one program is loaded and executed by the processor to implement the distributed file processing method provided in the above method embodiment.

[0129] The memory can be used to store software programs and modules. The processor executes various functional applications and distributed file processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store the operating system, application programs required for the function, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0130] The method embodiments provided in the embodiments of the present application can be executed in electronic devices such as mobile terminals, computer terminals, servers or similar computing devices. Fig.14 1 is a hardware structure block diagram of an electronic device of a distributed file processing method provided in an embodiment of the present application. Fig.14As shown, the electronic device 900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 910 (the processor 910 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 930 for storing data, and one or more storage media 920 (such as one or more mass storage devices) for storing application programs 923 or data 922. Among them, the memory 930 and the storage medium 920 can be short-term storage or permanent storage. The program stored in the storage medium 920 may include one or more modules, each of which may include a series of instruction operations in the electronic device. Furthermore, the central processing unit 910 can be configured to communicate with the storage medium 920 and execute a series of instruction operations in the storage medium 920 on the electronic device 900. The electronic device 900 may also include one or more power supplies 960, one or more wired or wireless network interfaces 950, one or more input and output interfaces 940, and / or, one or more operating systems 921, such as Windows Server TM , Mac OS X TM , Unix TM , LinuxTM, FreeBSDTM, etc.

[0131] The input / output interface 940 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the electronic device 900. In one example, the input / output interface 940 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In one example, the input / output interface 940 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0132] It can be understood by those skilled in the art that Fig.14 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.14 More or fewer components as shown, or with Fig.14 Different configurations are shown.

[0133] An embodiment of the present application also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to a distributed file processing method in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the distributed file processing method provided by the above method embodiment.

[0134] Optionally, in this embodiment, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0135] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various optional implementations.

[0136] It can be seen from the embodiments of the distributed file processing method, device, equipment, server, terminal, storage medium and program product provided by the present application that the present application obtains the shard information and shard data volume of the file shard in response to the trigger event of data flushing for the file shard in the cache pool, the shard information includes the shard order corresponding to the file shard in the corresponding to-be-processed file, and data flushing refers to synchronizing the file shards in the cache pool to the storage pool; determining the data volume interval to which the file data volume of the to-be-processed file belongs based on the shard information and the shard data volume; if the data volume interval belongs to a small file interval, data processing is performed on the to-be-processed file according to a preset multi-level flushing strategy, the small file interval includes multiple sub-intervals, and the multi-level flushing strategy is at least used to indicate the multiple sub-intervals and the corresponding to the trigger event of the data flushing. The mapping relationship between multiple data operations; in this way, the data volume of the file is determined by the shard information and the shard data volume, which overcomes the defect that the distributed storage system cannot effectively perceive the file size, and uses the original shard information in the system as the information basis for judging the data volume, without changing the data storage settings and adding complex judgment components, reducing the difficulty of system adaptation; and, through a multi-level flushing strategy, reasonable selection of data operations is made for the file shards of small files corresponding to different data volume intervals, so as to select data flushing or not based on the interval to which the file data volume belongs, to achieve hierarchical storage of small files, and to achieve persistent storage of some small files in the cache pool, thereby reducing multiple queries in the process of reading small files and reducing the corresponding number of data writes, effectively reducing file reading and writing consumption, and thus improving data processing efficiency.

[0138] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of the present application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0139] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0140] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0141] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A distributed file processing method, applied to a distributed storage system, wherein the distributed storage system is provided with a storage pool and a cache pool corresponding to the storage pool, characterized in that: The method comprises: In response to a triggering event for data flushing of a file shard in the cache pool, obtaining shard information and shard data volume of the file shard, wherein the shard information includes a shard order corresponding to the file shard in the to-be-processed file, and the data flushing refers to synchronizing the file shard in the cache pool to the storage pool; Determine the data volume interval to which the file data volume of the to-be-processed file belongs based on the shard information and the shard data volume; If the data volume interval is a small file interval, data processing is performed on the file to be processed according to a preset multi-level flushing strategy, the small file interval includes multiple sub-intervals, and the multi-level flushing strategy is at least used to indicate the mapping relationship between the multiple sub-intervals and the multiple data operations corresponding to the triggering events of the data flushing.

2. The method according to claim 1, characterized in that The multiple sub-intervals include at least a first-level file interval, a second-level file interval, and a third-level file interval in which the file data volume increases successively, and the multiple data processing operations include at least an operation of not performing data flushing corresponding to the first-level file interval, an operation of probabilistic data flushing corresponding to the second-level file interval, and an operation of data flushing corresponding to the third-level file interval.

3. The method according to claim 2, characterized in that If the data volume interval to which the file data volume of the to-be-processed file belongs is a small file interval, processing the data of the to-be-processed file according to a preset multi-level flushing strategy includes: If the data volume interval to which the file data volume of the to-be-processed file belongs is the first-level file interval, the data flushing operation on the file slice corresponding to the to-be-processed file is not performed.

4. The method according to claim 2, characterized in that: If the data volume interval to which the file data volume of the to-be-processed file belongs is a small file interval, processing the data of the to-be-processed file according to a preset multi-level flushing strategy includes: If the data volume interval to which the file data volume of the to-be-processed file belongs is the second-level file interval, generating flush indication information based on the attribute information of the to-be-processed file; If the flush indication information indicates that flushing is confirmed, a data flush operation is performed on the file slices corresponding to the to-be-processed file, so as to synchronously store the file slices in the storage pool.

5. The method according to claim 4, characterized in that The generating of the flush indication information based on the attribute information of the to-be-processed file comprises: Performing a hash operation on the attribute information of the to-be-processed file to obtain an attribute string, wherein the attribute strings corresponding to the same attribute information are consistent; Indication information is extracted from the attribute character string to obtain the refresh indication information.

6. The method according to claim 5, characterized in that The extracting the indication information from the attribute character string to obtain the refresh indication information comprises: Performing binary conversion on the attribute character string to obtain a binary string; The value of the specified bit sequence in the binary string is determined as the flush indication information.

7. The method according to claim 2, characterized in that If the data volume interval is a small file interval, processing the to-be-processed file according to a preset multi-level flushing strategy includes: If the data volume interval is a third-level file interval, a data flush operation is performed on the to-be-processed file to synchronously store the file fragments in the storage pool.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: If the data volume interval is a large file interval, a data flush operation is performed on the to-be-processed file to synchronously store the file fragments in the storage pool.

9. The method according to any one of claims 1 to 7, characterized in that: The step of determining the data volume interval to which the file data volume of the to-be-processed file belongs based on the shard information and the shard data volume comprises: If the shard information indicates that the shard order of the file shard is the first shard, and the shard data volume is smaller than the data block size, it is determined that the data volume interval to which the file data volume of the file to be processed belongs is a small file interval, and the data block size refers to the size of the data block in the distributed storage system, and the data block is the basic unit of file storage and transmission in the distributed storage system.

10. The method according to claim 9, characterized in that The method further comprises: If the shard data volume is equal to the data block size, or the shard information indicates that the shards of the file shards are sorted as other shards after the first shard, it is determined that the data volume interval to which the file data volume of the to-be-processed file belongs is a large file interval.

11. The method according to any one of claims 1 to 7, characterized in that: The triggering event of data flushing is generated based on the following method: Acquire a queue of information to be flushed back, the queue of information to be flushed back is used to store placement group information of the placement groups to be flushed back corresponding to the cache pool, each placement group includes a group of file objects corresponding to the file slices; Obtaining object information of each file object corresponding to the placement group information to be flushed, the object information including an object tag for indicating a fragment data state of the file fragment and a duration of the fragment data state; If the object tag indicates that the fragment data status of the file fragment is not flushed back to the storage pool, and the duration exceeds the preset status duration, it is determined to generate a trigger event for flushing the data corresponding to the file fragment.

12. The method according to any one of claims 1 to 6, characterized in that The method further comprises: In response to the amount of data stored in the cache pool satisfying a data cleanup condition, filtering out file shards whose shard data status is that they have been flushed back to the storage pool; The file fragments that have been flushed back to the storage pool are deleted from the cache pool.

13. A distributed file processing device, applied to a distributed storage system, wherein the distributed storage system is provided with a storage pool and a cache pool corresponding to the storage pool, characterized in that: The device comprises: A first acquisition module: in response to a triggering event of data flushing for a file shard in the cache pool, acquires shard information and shard data volume of the file shard, wherein the shard information includes a shard order corresponding to the file shard in the to-be-processed file, and the data flushing refers to synchronizing the file shard in the cache pool to the storage pool; An interval determination module: used for determining the data volume interval to which the file data volume of the to-be-processed file belongs based on the fragment information and the fragment data volume; Flush response module: used to process the file to be processed according to a preset multi-level flush strategy if the data volume interval is a small file interval. The small file interval includes multiple sub-intervals, and the multi-level flush strategy is at least used to indicate the mapping relationship between the multiple sub-intervals and the multiple data operations corresponding to the triggering event of the data flush.

14. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the distributed file processing method according to any one of claims 1 to 12.

15. A computer device, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the distributed file processing method according to any one of claims 1 to 12.

Images