A distributed file caching method, system, device and storage medium
By using a distributed file caching method and optimizing the data file caching strategy with a multi-task prediction model, the problem of repeated data transmission in autonomous driving simulation tasks is solved, thereby improving task efficiency and success rate.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGZHOU WERIDE TECH LTD CO
- Filing Date
- 2022-07-26
- Publication Date
- 2026-05-15
AI Technical Summary
In autonomous driving simulation tasks, training data is repeatedly transmitted from the data center to the graphics card cluster, which slows down the simulation task or even causes it to fail.
By using a distributed file caching method, it is determined whether a data file is cached locally. A pre-built multi-task prediction model is used to predict caching strategies, including cache location and lifespan, thereby optimizing the data file processing flow.
It improves the efficiency and success rate of simulation tasks, reduces redundant data transmission, and lowers network and hard drive wear and tear.
Smart Images
Figure CN115269522B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data storage technology, and in particular to a distributed file caching method, system, device and storage medium. Background Technology
[0002] In the field of autonomous driving, simulation is a crucial component. The data required for these simulations grows exponentially with increasing vehicle mileage, the number of vehicles, and the iterative development of onboard hardware. Most companies establish or utilize internet data centers to store this data. However, simulations often rely on dedicated GPU clusters, leading to the repeated transfer of large amounts of training data between the data center and the GPU cluster. This slows down the simulation process and can even cause it to fail due to data transmission issues. Therefore, for large-scale simulations, rapidly implementing file data transfer, exchange, and utilization to improve simulation speed is a critical technical challenge that needs to be addressed by those skilled in the art. Summary of the Invention
[0003] This application provides a distributed file caching method, system, device, and storage medium to improve the technical problem in the prior art where a large amount of training data is repeatedly and multiple times transmitted from the data center to the graphics card cluster during large-scale simulation tasks, thereby affecting the speed of the simulation task and even causing the simulation task to fail due to data transmission problems.
[0004] In view of this, the first aspect of this application provides a distributed file caching method, comprising:
[0005] Upon receiving a simulation task, it is determined whether the data file required for the simulation task is cached locally. If so, the locally cached data file is used to perform the simulation task; otherwise, the data file is downloaded from the data center to the local machine to perform the simulation task.
[0006] The target attributes of the data file are updated. A preset multi-task prediction model is used to predict whether the data file should be cached locally, the hard drive on which the data file is cached, and the cache lifespan based on the target attributes, so as to obtain the caching strategy result of the data file. The target attributes include file attributes, network attributes, graphics card cluster attributes, and simulation task attributes.
[0007] The data file is processed according to the caching strategy results of the data file.
[0008] Optionally, the pre-set multi-task prediction model consists of a parallel discrimination model, a file importance model, and a file lifetime model, and the caching strategy results include cache discrimination results, cache hard disk location results, and cache lifetime results.
[0009] The discrimination model is used to predict whether the data file should be cached locally based on the target attributes of the data file, and to obtain a caching discrimination result.
[0010] The file importance model is used to predict the hard drive where the data file is cached based on the target attributes of the data file, and to obtain the cache hard drive location result. The hard drive includes solid-state drives and hard disk drives.
[0011] The file lifetime model is used to predict the local cache lifetime of the data file based on the target attributes of the data file, and obtain the cache lifetime result.
[0012] Optionally, processing the data file according to the caching strategy result of the data file includes:
[0013] If the cache determination result indicates that the data file should not be cached locally, then the data file is deleted locally after the simulation task ends.
[0014] If the cache determination result indicates that the data file should be cached locally, then the data file is cached to the corresponding hard drive according to the cache hard drive location result, and the file lifespan of the data file is set according to the cache lifespan result.
[0015] Optionally, the configuration process of the pre-set multi-task prediction model is as follows:
[0016] A multi-task learning network is constructed, which consists of three parallel sub-convolutional neural networks;
[0017] Obtain training samples, which include target attributes of multiple files and corresponding caching strategy labels. The caching strategy labels include three sub-labels: cache label, cache disk location label, and cache lifetime label.
[0018] The training samples are input into the multi-task learning network to perform multi-task learning, and the sub-prediction results output by each sub-convolutional neural network are obtained. The network parameters are shared among the sub-convolutional neural networks.
[0019] The network parameters of the multi-task learning network are adjusted based on the sub-prediction results and corresponding sub-labels of each sub-convolutional neural network until the multi-task learning network converges, thus obtaining a trained pre-set multi-task prediction model.
[0020] Optionally, the file attributes include file size, file owner, file access frequency, and / or file creation time;
[0021] The network attributes include network speed, network packet loss rate, and / or network latency;
[0022] The graphics card cluster attributes include the graphics card cluster address, hard disk read / write speed, total hard disk capacity, used hard disk capacity, and / or the health status of the graphics card cluster;
[0023] The simulation task attributes include simulation task type and / or simulation task priority.
[0024] A second aspect of this application provides a distributed file caching system, comprising:
[0025] The judgment module is used to determine whether the data file required by the simulation task is cached locally when a simulation task is received. If so, the locally cached data file is used to perform the simulation task; otherwise, the data file is downloaded from the data center to the local machine to perform the simulation task.
[0026] The caching strategy prediction module is used to update the target attributes of the data file. By using a preset multi-task prediction model, it predicts whether the data file should be cached locally, the hard drive on which the data file is cached, and the cache lifespan based on the target attributes, and obtains the caching strategy result of the data file. The target attributes include file attributes, network attributes, graphics card cluster attributes, and simulation task attributes.
[0027] The processing module is used to process the data file according to the caching strategy result of the data file.
[0028] Optionally, the pre-set multi-task prediction model consists of a parallel discrimination model, a file importance model, and a file lifetime model, and the caching strategy results include cache discrimination results, cache hard disk location results, and cache lifetime results.
[0029] The discrimination model is used to predict whether the data file should be cached locally based on the target attributes of the data file, and to obtain a caching discrimination result.
[0030] The file importance model is used to predict the hard drive where the data file is cached based on the target attributes of the data file, and to obtain the cache hard drive location result. The hard drive includes solid-state drives and hard disk drives.
[0031] The file lifetime model is used to predict the local cache lifetime of the data file based on the target attributes of the data file, and obtain the cache lifetime result.
[0032] Optionally, the processing module is specifically used for:
[0033] If the cache determination result indicates that the data file should not be cached locally, then the data file is deleted locally after the simulation task ends.
[0034] If the cache determination result indicates that the data file should be cached locally, then the data file is cached to the corresponding hard drive according to the cache hard drive location result, and the file lifespan of the data file is set according to the cache lifespan result.
[0035] A third aspect of this application provides a distributed file caching device, the device including a processor and a memory;
[0036] The memory is used to store program code and transmit the program code to the processor;
[0037] The processor is configured to execute any one of the distributed file caching methods described in the first aspect according to the instructions in the program code.
[0038] A fourth aspect of this application provides a computer-readable storage medium for storing program code that, when executed by a processor, implements the distributed file caching method described in any of the first aspects.
[0039] As can be seen from the above technical solutions, this application has the following advantages:
[0040] This application provides a distributed file caching method, including: upon receiving a simulation task, determining whether the data file required by the simulation task is cached locally; if so, using the locally cached data file for the simulation task; otherwise, downloading the data file from the data center to the local machine for the simulation task; updating the target attributes of the data file, and using a pre-set multi-task prediction model to predict whether the data file should be cached locally, the hard drive on which the data file is cached, and the cache lifespan based on the target attributes to obtain the caching strategy result for the data file. The target attributes include file attributes, network attributes, GPU cluster attributes, and simulation task attributes; and processing the data file according to the caching strategy result.
[0041] In this application, if the data files required for the simulation task are cached locally, the data is directly obtained from the local machine for the simulation task. If the data files are not cached locally, they are downloaded from the data center for the simulation task. During the simulation, the file attributes, network attributes, GPU cluster attributes, and simulation task attributes of the data files are updated. A pre-set multi-task prediction model predicts whether the data file should be cached locally, as well as the hard drive space and cache lifespan, based on the target attributes of the data file. This yields the caching strategy result for the data file, which is then used to cache the data file. By considering multiple dimensions of information, such as the file attributes, network attributes, GPU cluster attributes, and simulation task attributes, to obtain the caching strategy result, the hit rate of locally cached files can be improved. This avoids repeated and multiple data file transfers, reduces network and hard drive wear and tear, and thus improves the efficiency and completion rate of the simulation task. This addresses the technical problem in existing technologies where a large amount of training data is repeatedly and multiple times transferred from the data center to the GPU cluster during large-scale simulation tasks, affecting the speed of the simulation task and even causing the simulation task to fail due to data transmission issues. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0043] Figure 1 A flowchart illustrating a distributed file caching method provided in this application embodiment;
[0044] Figure 2 A schematic diagram of the network structure of a pre-built multi-task prediction model provided in an embodiment of this application;
[0045] Figure 3 This is a schematic diagram of a distributed file caching system provided in an embodiment of this application. Detailed Implementation
[0046] This application provides a distributed file caching method, system, device, and storage medium to improve the technical problem in the prior art where a large amount of training data is repeatedly and multiple times transmitted from the data center to the graphics card cluster during large-scale simulation tasks, thereby affecting the speed of the simulation task and even causing the simulation task to fail due to data transmission problems.
[0047] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.
[0048] This application takes into account that the data relied upon by simulation tasks increases exponentially with the increase in vehicle mileage, the number of vehicles, and the iteration of in-vehicle hardware. Most enterprises establish or use internet data centers to store this data, but during simulation, a separate GPU cluster is often used to support it. This means that a large amount of training data is repeatedly transferred from the data center to the GPU cluster. Network speed, network fluctuations, and I / O bottlenecks in the data center and GPU cluster hard drives can all reduce the speed of simulation tasks and cause simulation tasks to fail due to data transmission corruption. Therefore, for large-scale simulation tasks, how to quickly and intelligently realize the transmission, exchange, and use of file data is of great significance for improving model training efficiency and effectiveness, and reducing fluctuations and task failures caused by network or disk I / O.
[0049] To improve the above issues, please refer to Figure 1 This application provides a distributed file caching method, including:
[0050] Step 101: Upon receiving a simulation task, determine whether the data file required for the simulation task is cached locally. If so, use the locally cached data file to perform the simulation task; otherwise, download the data file from the data center to the local machine to perform the simulation task.
[0051] Upon receiving a simulation task, the system first checks whether the data file required for the simulation task is cached locally. If the data file is cached locally, the system directly uses the cached data file for the simulation task. If the data file is not cached locally, the system downloads the data file from the data center to the local machine. The first data file downloaded from the data center can be cached on a faster solid-state drive for use by the current simulation task.
[0052] Step 102: Update the target attributes of the data file. Based on the target attributes, the preset multi-task prediction model predicts whether the data file should be cached locally, the hard drive where the data file is cached, and the cache lifespan, to obtain the caching strategy result of the data file. The target attributes include file attributes, network attributes, graphics card cluster attributes, and simulation task attributes.
[0053] While performing simulation tasks using data files, the target attributes of these data files are updated asynchronously. These attributes include file attributes, network attributes, GPU cluster attributes, and simulation task attributes. File attributes may include file size, file owner, file access frequency, and / or file creation time, etc.; network attributes may include network speed, network packet loss rate, and / or network latency, etc.; GPU cluster attributes may include GPU cluster address, hard drive read / write speed, total hard drive capacity, used hard drive capacity, and / or GPU cluster health status, etc.; simulation task attributes may include simulation task type, required dataset size, and / or simulation task priority, etc. The updated file attributes, network attributes, GPU cluster attributes, and simulation task attributes are then input into a pre-built multi-task prediction model for caching strategy prediction. Specifically, the model predicts whether the data file should be cached locally, the cached hard drive, and the cache lifetime, thus obtaining the caching strategy result for the data file.
[0054] The pre-built multi-task prediction model in this embodiment consists of a parallel discriminant model, a file importance model, and a file lifetime model. The specific network structure can be found in [reference needed]. Figure 2 The discriminant model is used to predict whether a data file should be cached locally based on its target attributes, and obtains the cache discrimination result; the file importance model is used to predict the hard drive where the data file will be cached based on its target attributes, and obtains the cache hard drive location result, which includes solid-state drives and mechanical hard drives; the file lifetime model is used to predict the cache lifetime of the data file locally based on its target attributes, and obtains the cache lifetime result.
[0055] After inputting the target attributes of the data file into the pre-set multi-task prediction model, the discrimination model determines whether the data file should be cached locally based on the target attributes, thus obtaining the cache discrimination result. The file importance model, assuming that the data file should be cached locally, predicts whether the data file should be cached on a slower mechanical hard drive or a faster solid-state drive, thus obtaining the cache hard drive location result. At the same time, the file lifetime model predicts the local cache lifetime of the data file based on the target attributes of the data file, that is, when the data file will be deleted from the local cache, thus obtaining the cache lifetime result. In other words, the caching strategy result includes the cache discrimination result, the cache hard drive location result, and the cache lifetime result.
[0056] Furthermore, the configuration process of the pre-set multi-task prediction model in this embodiment is as follows:
[0057] Construct a multi-task learning network, which consists of three parallel sub-convolutional neural networks;
[0058] Obtain training samples, which include target attributes of multiple files and corresponding caching policy labels. The caching policy labels include three sub-labels: cache label, cache disk location label, and cache lifetime label.
[0059] The training samples are input into the multi-task learning network to perform multi-task learning, and the sub-prediction results output by each sub-convolutional neural network are obtained. The network parameters are shared among the sub-convolutional neural networks.
[0060] Adjust the network parameters of the multi-task learning network based on the sub-prediction results and corresponding sub-labels of each sub-convolutional neural network until the multi-task learning network converges, thus obtaining a trained pre-set multi-task prediction model.
[0061] This application processes three tasks in parallel: determining whether a file should be cached locally, extracting file importance to select the caching hard drive, and predicting file lifespan. It utilizes a multi-task learning method from deep learning, sharing input data and underlying features to allow different tasks to correlate and influence each other, thereby obtaining a better file caching strategy. This helps determine whether each data file should be cached locally, on which hard drive, and for what duration. The multi-task learning network constructed in this embodiment includes three parallel sub-convolutional neural networks. Each sub-convolutional neural network can use existing network structures, such as residual networks or lightweight networks. Training samples are input into the multi-task learning network for multi-task learning. The first sub-convolutional neural network extracts features from the target attributes and determines whether the training sample should be cached locally, obtaining a first sub-prediction result. The second sub-convolutional neural network extracts features from the target attributes. When the first sub-convolutional neural network determines that the training sample should be cached locally, it determines whether the training sample should be cached on a mechanical hard drive or a solid-state drive, obtaining a second prediction result. The third sub-convolutional neural network... The neural network extracts features from the target attributes, predicts the cache lifetime of the training sample locally, and obtains the third sub-prediction result. Then, it calculates the loss value based on each sub-prediction result and the corresponding sub-label, and updates the network parameters in reverse through the loss value until the multi-task learning network converges (e.g., the number of iterations reaches the maximum number of iterations, or the training error is lower than the preset error threshold, etc.). The trained multi-task learning network is then used as the preset multi-task prediction model. The trained first sub-convolutional neural network is the discriminant model, the trained second sub-convolutional neural network is the file importance model, and the third sub-convolutional neural network is the file lifetime model.
[0062] In obtaining the caching strategy for data files, the embodiments of this application consider the file attributes, network attributes, graphics card cluster attributes, and simulation task attributes of the data files. These attributes are all input into a pre-set multi-task prediction model for prediction. The same input is passed through three different parallel deep learning models to obtain three different sets of outputs. This intelligently, diversifiedly, and more flexibly generates the caching strategy for data files, thereby improving the efficiency and success rate of simulation tasks. Compared with traditional caching methods such as Least Recently Used, Least Recently Used, and First-In-First-Out (FIFO) algorithms, the embodiments of this application consider multi-dimensional and more comprehensive information, resulting in a more comprehensive and reliable caching strategy.
[0063] Step 103: Process the data file according to the data file caching strategy results.
[0064] If the cache determination result indicates that the data file should not be cached locally, then the local data file will be deleted after the simulation task ends; if the cache determination result indicates that the data file should be cached locally, then the data file will be cached to the corresponding hard drive according to the cache hard drive location result, and the file lifespan of the data file will be set according to the cache lifespan result.
[0065] In this embodiment, if the data file required for the simulation task is cached locally, the data is directly obtained from the local cache for the simulation task. If it is not cached locally, it is downloaded from the data center to the local cache for the simulation task. During the simulation, the file attributes, network attributes, GPU cluster attributes, and simulation task attributes of the data file are updated. A pre-set multi-task prediction model predicts whether the data file should be cached locally, as well as the hard disk and cache lifespan, based on the target attributes of the data file. This yields the caching strategy result for the data file, which is then used to cache the data file. By considering multiple dimensions of information such as the file attributes, network attributes, GPU cluster attributes, and simulation task attributes of the data file to obtain the caching strategy result, the hit rate of the local cached file can be improved. This avoids repeated and multiple transmissions of data files, reduces network and hard disk wear, and improves the efficiency and completion of the simulation task. This addresses the technical problem in existing technologies where a large amount of training data is repeatedly and multiple times transmitted from the data center to the GPU cluster during large-scale simulation tasks, affecting the speed of the simulation task and even causing the simulation task to fail due to data transmission issues.
[0066] The above is an embodiment of a distributed file caching method provided by this application. The following is an embodiment of a distributed file caching system provided by this application.
[0067] Please refer to Figure 3 This application provides a distributed file caching system, comprising:
[0068] The judgment module is used to determine whether the data file required for the simulation task is cached locally when a simulation task is received. If so, the local cached data file is used for the simulation task; otherwise, the data file is downloaded from the data center to the local machine for the simulation task.
[0069] The caching strategy prediction module is used to update the target attributes of data files. It uses a pre-set multi-task prediction model to predict whether the data file should be cached locally, the hard drive on which the data file is cached, and the cache lifespan based on the target attributes, thus obtaining the caching strategy result for the data file. The target attributes include file attributes, network attributes, graphics card cluster attributes, and simulation task attributes.
[0070] The processing module is used to process the data file according to the caching strategy of the data file.
[0071] As a further improvement, the pre-built multi-task prediction model consists of a parallel discrimination model, a file importance model, and a file lifetime model. The caching strategy results include cache discrimination results, cache hard disk location results, and cache lifetime results.
[0072] The discriminant model is used to predict whether a data file should be cached locally based on its target attributes, thus obtaining a caching discrimination result.
[0073] The file importance model is used to predict the hard drive where the data file is cached based on the target attributes of the data file, and obtain the cache hard drive location result, including solid-state drives and hard disk drives;
[0074] The file lifetime model is used to predict the local cache lifetime of a data file based on its target attributes, thus obtaining the cache lifetime result.
[0075] As a further improvement, the processing module is specifically used for:
[0076] If the cache determination result indicates that the data file should not be cached locally, then the local data file will be deleted after the simulation task ends.
[0077] If the cache determination result indicates that the data file should be cached locally, then the data file is cached to the corresponding hard drive according to the cache hard drive location result, and the file lifespan of the data file is set according to the cache lifespan result.
[0078] As a further improvement, file attributes include file size, file owner, file access frequency, and / or file creation time;
[0079] Network attributes include network speed, network packet loss rate, and / or network latency;
[0080] The attributes of a graphics card cluster include the cluster address, hard drive read / write speed, total hard drive capacity, used hard drive capacity, and / or the health status of the graphics card cluster.
[0081] Simulation task attributes include simulation task type and / or simulation task priority.
[0082] In this embodiment, if the data file required for the simulation task is cached locally, the data is directly obtained from the local cache for the simulation task. If it is not cached locally, it is downloaded from the data center to the local cache for the simulation task. During the simulation, the file attributes, network attributes, GPU cluster attributes, and simulation task attributes of the data file are updated. A pre-set multi-task prediction model predicts whether the data file should be cached locally, as well as the hard disk and cache lifespan, based on the target attributes of the data file. This yields the caching strategy result for the data file, which is then used to cache the data file. By considering multiple dimensions of information such as the file attributes, network attributes, GPU cluster attributes, and simulation task attributes of the data file to obtain the caching strategy result, the hit rate of the local cached file can be improved. This avoids repeated and multiple transmissions of data files, reduces network and hard disk wear, and improves the efficiency and completion of the simulation task. This addresses the technical problem in existing technologies where a large amount of training data is repeatedly and multiple times transmitted from the data center to the GPU cluster during large-scale simulation tasks, affecting the speed of the simulation task and even causing the simulation task to fail due to data transmission issues.
[0083] This application embodiment also provides a distributed file caching device, the device including a processor and a memory;
[0084] The memory is used to store program code and transfer the program code to the processor;
[0085] The processor is used to execute the distributed file caching method in the foregoing method embodiments according to the instructions in the program code.
[0086] This application also provides a computer-readable storage medium, characterized in that the computer-readable storage medium is used to store program code, which, when executed by a processor, implements the distributed file caching method in the aforementioned method embodiments.
[0087] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system and modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0088] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0089] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0090] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.
[0091] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0092] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.
[0093] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for executing all or part of the steps of the methods described in the various embodiments of this application through a computer device (which may be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.
[0094] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A distributed file caching method, characterized in that, include: Upon receiving a simulation task, it is determined whether the data file required for the simulation task is cached locally. If so, the locally cached data file is used to perform the simulation task; otherwise, the data file is downloaded from the data center to the local machine to perform the simulation task. The target attributes of the data file are updated. A pre-set multi-task prediction model is used to predict whether the data file should be cached locally, the hard drive on which the data file is cached, and the cache lifetime based on the target attributes, thereby obtaining the caching strategy result for the data file. The target attributes include file attributes, network attributes, graphics card cluster attributes, and simulation task attributes. The pre-set multi-task prediction model consists of a parallel discrimination model, a file importance model, and a file lifetime model. The caching strategy result includes cache discrimination result, cache hard drive location result, and cache lifetime result. The discrimination model is used to predict whether the data file should be cached locally based on the target attributes of the data file, and to obtain a caching discrimination result. The file importance model is used to predict the hard drive where the data file is cached based on the target attributes of the data file, and to obtain the cache hard drive location result. The hard drive includes solid-state drives and hard disk drives. The file lifetime model is used to predict the local cache lifetime of the data file based on the target attributes of the data file, and obtain the cache lifetime result. The data file is processed according to the caching strategy results of the data file.
2. The distributed file caching method according to claim 1, characterized in that, The step of processing the data file according to the caching strategy result of the data file includes: If the cache determination result indicates that the data file should not be cached locally, then the data file is deleted locally after the simulation task ends. If the cache determination result indicates that the data file should be cached locally, then the data file is cached to the corresponding hard drive according to the cache hard drive location result, and the file lifespan of the data file is set according to the cache lifespan result.
3. The distributed file caching method according to claim 1, characterized in that, The configuration process of the pre-set multi-task prediction model is as follows: A multi-task learning network is constructed, which consists of three parallel sub-convolutional neural networks; Obtain training samples, which include target attributes of multiple files and corresponding caching strategy labels. The caching strategy labels include three sub-labels: cache label, cache disk location label, and cache lifetime label. The training samples are input into the multi-task learning network to perform multi-task learning, and the sub-prediction results output by each sub-convolutional neural network are obtained. The network parameters are shared among the sub-convolutional neural networks. The network parameters of the multi-task learning network are adjusted based on the sub-prediction results and corresponding sub-labels of each sub-convolutional neural network until the multi-task learning network converges, thus obtaining a trained pre-set multi-task prediction model.
4. The distributed file caching method according to claim 1, characterized in that, The file attributes include file size, file owner, file access frequency, and / or file creation time; The network attributes include network speed, network packet loss rate, and / or network latency; The graphics card cluster attributes include the graphics card cluster address, hard disk read / write speed, total hard disk capacity, used hard disk capacity, and / or the health status of the graphics card cluster; The simulation task attributes include simulation task type and / or simulation task priority.
5. A distributed file caching system, characterized in that, include: The judgment module is used to determine whether the data file required by the simulation task is cached locally when a simulation task is received. If so, the locally cached data file is used to perform the simulation task; otherwise, the data file is downloaded from the data center to the local machine to perform the simulation task. The caching strategy prediction module is used to update the target attributes of the data file. Based on the target attributes, it predicts whether the data file should be cached locally, the hard drive where the data file is cached, and the cache lifetime using a pre-set multi-task prediction model, thus obtaining the caching strategy result for the data file. The target attributes include file attributes, network attributes, GPU cluster attributes, and simulation task attributes. The pre-set multi-task prediction model consists of a parallel discrimination model, a file importance model, and a file lifetime model. The caching strategy result includes a cache discrimination result, a cache hard drive location result, and a cache lifetime result. The discrimination model is used to predict whether the data file should be cached locally based on the target attributes of the data file, and to obtain a caching discrimination result. The file importance model is used to predict the hard drive where the data file is cached based on the target attributes of the data file, and to obtain the cache hard drive location result. The hard drive includes solid-state drives and hard disk drives. The file lifetime model is used to predict the local cache lifetime of the data file based on the target attributes of the data file, and obtain the cache lifetime result. The processing module is used to process the data file according to the caching strategy result of the data file.
6. The distributed file caching system according to claim 5, characterized in that, The processing module is specifically used for: If the cache determination result indicates that the data file should not be cached locally, then the data file is deleted locally after the simulation task ends. If the cache determination result indicates that the data file should be cached locally, then the data file is cached to the corresponding hard drive according to the cache hard drive location result, and the file lifespan of the data file is set according to the cache lifespan result.
7. A distributed file caching device, characterized in that, The device includes a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the distributed file caching method according to any one of claims 1-4 according to the instructions in the program code.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code, which, when executed by a processor, implements the distributed file caching method according to any one of claims 1-4.