Cache resource creation method and device, electronic equipment and storage medium

Abstract

The invention discloses a cache resource creation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a target resource creation task initiated by a client to a target cluster; querying configuration information of each node in the target cluster according to the target resource creation task, and selecting a target node meeting the target resource creation task from each node according to the configuration information; determining a target scheduling strategy corresponding to the target node; sending the target resource creating task and the target node to the proxy terminal according to the target scheduling strategy, so that the proxy terminal creates a target resource meeting the target resource creating task based on the target node, and the target node is used for executing a distributed model training task according to the target resource. According to the method provided by the invention, the target node meeting the resource creation task is automatically selected through the obtained resource creation task, and the distributed resources for service model training are automatically created for the user. And the model training IO reading efficiency is improved.

Description

technical field [0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for creating a cache resource. Background technique [0002] With the development of AI, the demand for data and computing power is increasing sharply. There are some distributed and parallel model training methods for business, but in the actual environment, due to the low efficiency of data reading, there is a bandwidth bottleneck. At the same time, the current solutions in the industry are relatively scattered, and cannot combine caching and data in parallel to realize automatic management. For example, the current distributed multi-machine multi-card training solutions in the industry are all without cache. Large-scale multi-machine parallel training will have the problem of storage bandwidth bottlenecks. The IO reading efficiency of model training is low, resulting in calculation speeds that cannot meet e...

AI Technical Summary

Problems solved by technology

[0002] With the development of AI, the demand for data and computing power is increasing sharply. There are some distributed and parallel model training methods for business, but in the actual environment, due to the low efficiency of data reading, there is a bandwidth bottleneck.

At the same time, the current solutions in the industry are relatively scattered, and cannot combine cache and data in parallel to realize automatic management

For example, the current distributed multi-machine and multi-card training solutions in the industry are all without cache. Large-scale multi-machine parallel training will have the problem of storage bandwidth bottlenecks. The IO reading efficiency of model training is low, resulting in calculation speeds that cannot meet expectations. Effect

Images