Method and apparatus for analyzing algorithm program data

Abstract

The invention discloses a method and device for analyzing algorithm program data. The method includes: a background program receives data asynchronously sent when an algorithm program runs to an anchor point, and saves it in a data pool, wherein the data has an identifier; and acquires data with a specified identifier from the data pool for analysis and processing. The device includes: a receiving unit for receiving data asynchronously sent when an algorithm program runs to an anchor point, wherein the data has an identifier; a data pool for saving the data received by the receiving unit; a processing unit for using Analyzing and processing the data with the specified identifier obtained from the data pool. The method and device can not only compare algorithms running on different machines by analyzing and comparing intermediate data, but also check algorithm errors in real time. The tested algorithm program has no perception of the data analysis process, does not affect the business, and saves large-scale Cluster test algorithm debugging and analysis time.

Description

technical field [0001] The present invention relates to the field of distributed computing, and more particularly, to a method for analyzing algorithm program data. Background technique [0002] In large-scale cluster computing scenarios, common cluster scheduling systems include: SLURM, PBS, LSF, and SGE, etc. The interaction logic of these systems is: writing job scripts, submitting jobs, waiting for job queues, and viewing jobs, all of which are not provided. The operation of the debugger, the only way to troubleshoot the problem is to view the log after the job is executed. [0003] In the process of deploying large-scale algorithms on a computing cluster, once an error occurs, it is impossible to accurately troubleshoot the problem in real time. For example, in a cluster operating system such as SLURM, it is necessary to troubleshoot the problem according to the log. Some algorithm problems cannot be reproduced by running a single instance on a single computing node, a...

AI Technical Summary

Problems solved by technology

Some algorithm problems cannot be reproduced by running a single instance on a single computing node. They must be run on a large scale on a cluster. This problem is very common in large-scale parallel computing. At this time, if you use the algorithm log to troubleshoot problems, it is not a Easy things, there are many variables and complex logic in the algorithm, and the log cannot fully track the change of a certain variable in a specific process

[0004] In large-scale computing systems, the evaluation of algorithms is often carried out by analyzing the result data sets. This evaluation method can only make conclusive and statistical evaluations of algorithms. To evaluate a certain sub-process or sub-algorithm in an algorithm, a separate Targeted output of the result data set has two disadvantages: 1) The more sub-processes are evaluated, the more result data sets will be generated, which is inconvenient to manage; 2) If you only want to evaluate a few sub-algorithms, you need not modify the code In the case of the need to run a complete algorithm, wasting unnecessary waiting time

Images