Data processing device, data processing method, and recording medium

Abstract

The invention relates to a data processing apparatus, a data processing method, and a recording medium, wherein feature vectors can be quantized efficiently. A data processing device according to an embodiment includes a sub-vector group generating unit, a codebook generating unit, and a converting unit. The sub-vector group generating unit generates, from a feature vector set of N number of D-dimensional feature vectors, M number of sub-vector groups (where M<D holds true). Each of the M number of sub-vector groups includes N number of dimension-variable sub-vectors obtained from the N numberof D-dimensional feature vectors. For each of the M number of sub-vector groups, the codebook generating unit performs clustering of the N number of dimension-variable sub-vectors, and generates a codebook in which the representative vector of each cluster is associated with an index. The converting unit performs product quantization using the codebook and converts each of the N number of D-dimensional feature vectors into a compressed code made of a combination of M number of indexes.

Description

[0001] This application benefits from the priority of Japanese Patent Application No. 2018-024700 filed on February 15, 2018, and incorporates the entire contents thereof. technical field [0002] Embodiments of the present invention relate to a data processing device, a data processing method, and a recording medium. Background technique [0003] With the advent of the era of big data, the necessity of retaining a large number of feature vectors used as examples in, for example, pattern recognition and the like has increased. Accompanying this, hardware costs such as a memory and a hard disk drive for retaining feature vectors increase. As one of the solutions to this problem, there is known a method of reducing the memory size of feature vectors by direct product quantization. Direct product quantization is a technique of dividing a feature vector into a plurality of sub-vectors, referring to a codebook, and replacing each sub-vector with an index of a representative vect...

AI Technical Summary

Problems solved by technology

However, in this method, the degree of dispersion varies greatly between different sub-vectors, and the number of clusters may be excessive or too small due to sub-vectors when generating a codebook.

Furthermore, if the codebook generated in this way is used to perform direct product quantization of feature vectors, there is a concern that the quantization efficiency will decrease.

Images