Training data set generation method and device

Abstract

The invention discloses a method and device for generating a training data set. The method includes: grabbing the text of a webpage; detecting whether the text contains a picture; if so, acquiring the picture; identifying the picture to obtain the picture Text data; generate a training data set according to the text data in the text and the text data in the picture; if not, generate a training data set according to the text data in the text. The invention can improve the richness and completeness of training data.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a method and device for generating a training data set. Background technique [0002] Deep learning enables machine learning to achieve many applications and expands the scope of artificial intelligence. Its motivation is to establish and simulate the neural network of the human brain for analysis and learning. It imitates the mechanism of the human brain to explain data, such as images, sounds and text. One of the core issues in deep learning is training data. Deep learning requires a large amount of data. It can be said that the amount of training data plays a key role in the intelligence of artificial intelligence. [0003] In the existing technology, training data is generally divided into two types: image data and text data, and these two types of data are used in different directions of artificial intelligence. For example, text data is used in natural language processing ap...

AI Technical Summary

Problems solved by technology

The collection of training data usually uses crawler tools to grab webpage data from webpages, and webpages often contain text and related pictures at the same time. If only the text data or picture data in them are simply used, it will not only waste data It will also make the training data information missing and affect the model training effect

Images