Vector data processing method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a vector data processing method and device, equipment and a storage medium. The method comprises the following steps: receiving newly added vector data, and placing the newly added vector data in a cache; if the newly added vector data in the cache meets a preset quantity, extracting a training sample from the newly added vector data, and training a classifier according to the training sample to obtain a target classifier; based on a target classifier, classifying the newly-added vector data according to the similarity of newly-added vector features; and establishing a feature classification index according to a classification result, and performing vector data retrieval according to the feature classification index. According to the scheme, the newly added vector data can be trained in a targeted manner, the training time is saved under the condition that the vector data is continuously increased, the training complexity is reduced, the problem that the sampling and training time is increased when the data set is continuously expanded is solved, and the processing efficiency is improved.

Description

technical field [0001] The embodiments of the present application relate to the technical field of data processing, and in particular, to a vector data processing method, device, device, and storage medium. Background technique [0002] With the development and landing of artificial intelligence technology, more and more previously "undescribable" data are described and calculated in the form of feature vector data, such as text, voice, appearance, etc. When these massive feature data gather When they are together, how to quickly and accurately find the same or similar data has become an urgent problem to be solved. [0003] When the size of the data set continues to expand, if the classifier is not retrained, the classification accuracy of the data set will decrease; if the reclassifier is trained, the resampling and training time of each computing node will become shorter and shorter. Long, when the data set becomes larger, the sampling and data loading time will increase...

AI Technical Summary

Problems solved by technology

[0003] When the size of the data set continues to expand, if the classifier is not retrained, the classification accuracy of the data set will decrease; if the reclassifier is trained, the resampling and training time of each computing node will become shorter and shorter. Long, when the data set becomes larger, the sampling and data loading time will increase non-linearly, and the search time will also increase with the increase of the overall data size

Images