Method and device for optimizing data storage space in matrix inversion process

Abstract

Embodiments of the present invention provide a method and device for optimizing data storage space during matrix inversion. Wherein, the method for optimizing the data storage space in the process of matrix inversion includes: for the i-th round of block, obtain After the first auxiliary matrix, the second auxiliary matrix, the third auxiliary matrix and the transition matrix of the i-th round of blocking, clear the block files generated by the shuffling operation in the i-th round of blocking; store the transition matrix of the i-th round For the data cache, the data cache for storing the transition matrix of the (i-2) round of block is cleared. The method and device for optimizing data storage space in the matrix inversion process provided by the embodiments of the present invention can reduce the occupation of data storage space in the current round of block process, and greatly reduce the total data storage space occupied by each round of block process.

Description

technical field [0001] The embodiments of the present invention relate to the field of computer technology, and in particular, to a method and device for optimizing data storage space in a matrix inversion process. Background technique [0002] The Spark framework is an open source cluster computing framework based on memory computing that integrates batch processing and real-time processing, and can process large data sets in parallel. The Spark framework uses elastic distributed data sets for iterative calculations, which improves computational efficiency. [0003] As the scale of data continues to expand, the requirements for the amount of disk space used for calculations are also getting higher and higher. If the storage space is insufficient during calculation, the Resilient Distributed Datasets (RDD) cache will fail, the shuffle operation will fail, and eventually the calculation task will fail. [0004] When calculating the inverse matrix of a square matrix, the met...

AI Technical Summary

Problems solved by technology

In the later stage of the matrix block inversion algorithm, as the number of iterations increases, the calculation time of each iteration cannot be determined, and the disk increment cannot be determined, so that precise control of disk capacity cannot be achieved, and sufficient disk space cannot be guaranteed.

If the disk space is insufficient, the cached intermediate data will be eliminated, which will cause repeated calculations and affect computing performance

Images