Unlock instant, AI-driven research and patent intelligence for your innovation.

Data caching method, system and device in AI cluster, and computer medium

A data caching and clustering technology, applied in the field of AI clusters, can solve problems affecting the performance of AI cluster data caching and achieve the effect of improving data caching performance

Active Publication Date: 2021-11-02
INSPUR SUZHOU INTELLIGENT TECH CO LTD
View PDF4 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

One of the basic functions of the artificial intelligence platform is the operation of files, including the local download cache of the data set, the reading of files during the training process, and other file operations. These all depend on the storage resources of the cluster, and the AI ​​cluster has very high storage requirements. High, with frequent IO operations, which makes storage resources the bottleneck of data caching in the AI ​​cluster, affecting the data caching performance of the AI ​​cluster

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data caching method, system and device in AI cluster, and computer medium
  • Data caching method, system and device in AI cluster, and computer medium
  • Data caching method, system and device in AI cluster, and computer medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

[0053] see figure 1 , figure 1 It is a flowchart of a data caching method in an AI cluster provided by the embodiment of this application.

[0054] A method for caching data in an AI cluster provided in an embodiment of the present application may include the following steps:

[0055] Step S101: Determine the target data set to be cached.

[0056] In practical applications, because there are multiple data sets in the AI ​​cluster, and the user may only cache one or a few of them,...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data caching method, system and device in an AI cluster, and a computer medium. The method comprises the steps of determining a to-be-cached target data set; obtaining a weight value of the target data set on each cluster node in the AI cluster; determining a target cluster node for caching the target data set; obtaining a target shortest path from other cluster nodes in the AI cluster to the target cluster node and a previous node of the target cluster node in the target shortest path, wherein the other cluster nodes comprise nodes except the target cluster node in the AI cluster; and determining a cache path for caching the target data set to the target cluster node based on the weight value, the target shortest path and the preceding node, so as to cache the target data set to the target cluster node according to the cache path. According to the method, the cache path can be matched with the storage capacity of the AI cluster, caching of the target data set based on the cache path is equivalent to caching of the data set based on the storage performance of the AI cluster, and the data caching performance of the AI cluster can be improved.

Description

technical field [0001] The present application relates to the technical field of AI clusters, and more specifically, to a data caching method, system, device, and computer medium in an AI cluster. Background technique [0002] With the vigorous development of artificial intelligence (AI) related industries, more and more researchers in scientific research enterprises and universities have higher and higher requirements for computing power. The construction of AI cluster platform effectively solves the problem of enterprises and scientific research. University requirements for computing power. One of the basic functions of the artificial intelligence platform is the operation of files, including the local download cache of the data set, the reading of files during the training process, and other file operations. These all depend on the storage resources of the cluster, and the AI ​​cluster has very high storage requirements. High, there are frequent IO operations, which make...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/2455
CPCG06F16/24552H04L67/568G06N20/00G06F12/0813G06F12/0875G06F12/0893G06F12/0824
Inventor 姬贵阳
Owner INSPUR SUZHOU INTELLIGENT TECH CO LTD