Image recognition device, image recognition method, and object recognition model

Abstract

To provide an image recognition technique for recognizing an unknown image with high accuracy.SOLUTION: An object recognition unit 20 recognizes an object in an input image using an object recognition model. A recognition accuracy determination unit 40 determines the accuracy of recognizing the object in the input image. A supervised image conversion unit 60 labels an input image with object recognition accuracy which is below a predetermined threshold on the basis of a feature quantity of the input image, to be converted into a supervised image. A transfer learning unit 80 performs transfer learning on the object recognition model using the supervised image as training data to update the object recognition model.SELECTED DRAWING: Figure 1

Description

[[Technical Field]]

[0001] The present invention relates to image recognition technology. [[Background Art]]

[0002] In an apparatus that recognizes objects from images, images with low recognition accuracy are classified as unknown images that are difficult to recognize. Manual annotation work is performed on unknown images, in which a human identifies and labels objects in the images.

[0003] Patent Document 1 discloses a system that acquires an unknown image for which a trained model has not yet been created, selects a trained model of a known image with imaging conditions similar to the acquired unknown image from among trained models, performs image analysis on the unknown image using the selected trained model, and provides the result of the image analysis. [[Prior Art Documents]] [[Patent Documents]]

[0004] [[Patent Document 1]] International Publication No. WO 2019 / 003355 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0005] Manual annotation requires a large amount of time and cost, and thus has the problem of being unsuitable for practical use.

[0006] The present invention has been made in view of such circumstances, and an object of the present invention is to provide an image recognition technology capable of recognizing unknown images with high accuracy. [[Means for Solving the Problem]]

[0007] To solve the above problems, an image recognition device according to one aspect of the present invention includes: an object recognition unit that recognizes objects in an input image using an object recognition model; a recognition accuracy determination unit that determines the recognition accuracy of objects in the input image; a supervised image conversion unit that labels input images whose object recognition accuracy falls below a predetermined threshold based on the features of the input image and converts them into supervised images; and a transfer learning unit that transfers the object recognition model using the supervised images as training data and updates the object recognition model.

[0008] Another aspect of the present invention is an image recognition method. This method includes the steps of: recognizing an object in an input image using an object recognition model; determining the accuracy of object recognition in the input image; labeling input images whose object recognition accuracy falls below a predetermined threshold based on the features of the input image and converting them into supervised images; and updating the object recognition model by transfer learning using the supervised images as training data.

[0009] Another aspect of the present invention is an object recognition model. This object recognition model causes a computer to recognize objects in an input image, and is updated by transfer learning using supervised images labeled based on the features of the input image as training data, for input images in which the object recognition accuracy falls below a predetermined threshold.

[0010] Furthermore, any combination of the above components, as well as conversions of the expression of the present invention between methods, apparatus, systems, recording media, computer programs, etc., are also valid embodiments of the present invention. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide an image recognition technology that can recognize unknown images with high accuracy. [Brief explanation of the drawing]

[0012] [Figure 1] This is a diagram showing the configuration of an image recognition device according to an embodiment. [Figure 2] Figures 2(a) to 2(d) show examples of images input to the object recognition unit in Figure 1. [Figure 3] Figures 3(a) to 3(d) illustrate the results of the object recognition unit in Figure 1 recognizing objects in the images in Figures 2(a) to 2(d) based on the trained object recognition model. [Figure 4] This diagram illustrates supervised images, which are unidentified images that have been labeled. [Figure 5] This is a flowchart illustrating the image recognition procedure using the image recognition device shown in Figure 1. [Modes for carrying out the invention]

[0013] Figure 1 is a diagram showing the configuration of an image recognition device 100 according to an embodiment. The image recognition device 100 includes an input unit 10, an object recognition unit 20, an object recognition model storage unit 30, a recognition accuracy determination unit 40, an output unit 50, a supervised image conversion unit 60, a feature quantity-label database 70, a transfer learning unit 80, and a pre-trained dataset storage unit 90.

[0014] The input unit 10 acquires an image in which an object should be recognized and supplies it to the object recognition unit 20.

[0015] The object recognition model storage unit 30 stores an object recognition model with excellent recognition accuracy that has been pre-trained using a pre-training dataset. The pre-training dataset storage unit 90 stores the pre-training dataset used to train the object recognition model.

[0016] The object recognition unit 20 recognizes objects in the input image using a trained object recognition model stored in the object recognition model storage unit 30. The object recognition unit 20 supplies the object recognition result in the input image to the recognition accuracy determination unit 40. The recognition result includes at least one candidate object class and its recognition accuracy.

[0017] When the object recognition accuracy of an object obtained by the object recognition unit 20 is equal to or higher than a predetermined threshold, the recognition result is supplied to the output unit 50. The output unit 50 outputs the recognition result of the input image.

[0018] The recognition accuracy determination unit 40 determines the recognition accuracy of an object in the input image, and supplies low recognition accuracy images whose object recognition accuracy does not reach a predetermined threshold as unknown images to the supervised image conversion unit 60.

[0019] The supervised image conversion unit 60 assigns a label to an unknown image supplied from the recognition accuracy determination unit 40 based on the feature amount of the unknown image, converts the unknown image into a supervised image, and supplies the supervised image to the transfer learning unit 80. As an example of the feature amount used for assigning a label to an unknown image, the feature amount of an intermediate layer in the latter stage of a neural network, which is an intermediate output obtained when an image is input to a trained object recognition model stored in the object recognition model storage unit 30, may be used.

[0020] The feature-label database 70 is a database that accumulates pairs of image feature amounts and labels. The supervised image conversion unit 60 refers to the feature-label database 70, acquires a label corresponding to the feature amount that is most similar to the feature amount of the unknown image, assigns the acquired label to the unknown image, and converts the unknown image into a supervised image. As an example of the feature-label database, a database may be used in which the feature amount of an intermediate layer in the latter stage of a neural network, which is an intermediate output obtained when an image is input to a trained object recognition model stored in advance in the object recognition model storage unit 30, is associated with the label of the image and accumulated.

[0021] The transfer learning unit 80 adds the supervised image supplied from the supervised image conversion unit 60 to the pre-training dataset stored in the pre-training dataset storage unit 90 to construct a new dataset, performs transfer learning on the object recognition model using the new dataset, and stores the updated object recognition model in the object recognition model storage unit 30.

[0022] Here, the pre-training dataset storage unit 90 stores, if available, a dataset such as big data used for pre-training the object recognition model. If the dataset used for pre-training the object recognition model is not available, the object recognition unit 20 may store known input images in which objects have been recognized with high recognition accuracy as the pre-training dataset. In this case, the transfer learning unit 80 adds supervised images with labeled unknown images to the pre-training dataset of known images stored in the pre-training dataset storage unit 90 to construct a new dataset, and then transfer-learns the object recognition model using the new dataset.

[0023] In transfer learning, for example, a new neural network is generated by replacing the final output layer of a pre-trained neural network model with a new layer and retraining the parameters of the new layer using a new dataset as training data.

[0024] By using the updated object recognition model, the object recognition unit 20 can recognize objects with high accuracy even when an image containing objects that previously had low recognition accuracy is input. This makes it possible to recognize objects of unknown classes.

[0025] Figures 2(a) to 2(d) show examples of images input to the object recognition unit 20. Figure 2(a) is an image of a person, Figure 2(b) is an image of a motorcycle, Figure 2(c) is an image of a car, and Figure 2(d) is an image of a bicycle. There are other types of input images as well; for example, there may be a total of 10 types.

[0026] The initial object recognition model was trained using images of three types of subjects—people, motorcycles, and cars—as a pre-set dataset, since a large number of these three types of images were available.

[0027] Figures 3(a) to 3(d) illustrate the results of object recognition performed by the object recognition unit 20 based on the trained object recognition model in Figures 2(a) to 2(d).

[0028] The initial object recognition model is pre-trained on three classes: people, motorcycles, and cars. As a result, as shown in Figure 3(a), the image in Figure 2(a) is labeled "person," as shown in Figure 3(b), the image in Figure 2(b) is labeled "motorcycle," and as shown in Figure 3(c), the image in Figure 2(c) is labeled "car." However, since the initial object recognition model is not trained on the bicycle class, the recognition result for the image in Figure 2(d) is inaccurate, and the label remains unknown, as shown in Figure 3(d).

[0029] The supervised image transformation unit 60 extracts the features of the image in Figure 2(d) and, referring to the feature-label database 70, obtains the label corresponding to the feature that is most similar to the features of the image in Figure 2(d). In this case, the obtained label is "bicycle". As shown in Figure 4, the supervised image transformation unit 60 attaches the label "bicycle" obtained from the feature-label database 70 to the image in Figure 2(d) and transforms it into a supervised image.

[0030] The transfer learning unit 80 uses the supervised images in Figure 4 as new training data to transfer-learn the object recognition model and generate a new object recognition model. As a result, the new object recognition model can distinguish between four classes: people, motorcycles, cars, and bicycles. In this way, whenever an unknown image with low recognition accuracy is detected, it is converted into a labeled supervised image, and the object recognition model is transferred-learned using the supervised image as new training data, enabling it to distinguish between all 10 classes.

[0031] Here, if an object recognition model is trained using only supervised images labeled for unknown classes as training data, it may become unable to correctly recognize images of known classes that it was previously able to distinguish. Therefore, it is preferable to train the object recognition model using a new dataset as training data, which includes supervised images of unknown classes in addition to supervised images of known classes. For example, when performing transfer training using images of bicycles, a new dataset is created by adding supervised images of bicycles to a pre-trained dataset of supervised images of people, motorcycles, and cars, and the object recognition model is trained using this dataset containing four types of supervised images as training data. This makes it possible to ultimately generate an object recognition model that can correctly recognize all types of images.

[0032] Figure 5 is a flowchart illustrating the image recognition procedure performed by the image recognition device 100.

[0033] The object recognition unit 20 inputs the image to be recognized into a trained object recognition model and recognizes the object in the image (S10).

[0034] If the object recognition accuracy is above a predetermined threshold (N in S20), the recognition result label is output (S30), the process returns to step S10, and another image is input.

[0035] If the object recognition accuracy is below a predetermined threshold, for example, if the probability of the first candidate label being correct is less than 50% (Y in S20), the input image is treated as an unknown image and feature extraction of the image is performed (S40).

[0036] The supervised image conversion unit 60 obtains labels that match the features of the unknown image from the feature-label database 70 (S50), and then assigns the obtained labels to the unknown image to generate a supervised image (S60).

[0037] The procedure from step S10 to step S60 is repeated until a predetermined number of supervised images of objects of the same class, for example 30 images, are accumulated (N in S70). When a predetermined number, in this case 30 supervised images of the same class, has been accumulated (Y in S70), the transfer learning unit 80 adds the predetermined number of supervised images to the dataset used for pre-training to generate a new dataset, and transfer-learns the trained object recognition model using the new dataset (S80). The transfer learning unit 80 generates a new object recognition model that has undergone transfer learning and stores it in the object recognition model storage unit 30 (S90).

[0038] In the above explanation, transfer learning was performed after a predetermined number of supervised images of objects of the same class had been accumulated. However, transfer learning may also be performed each time a supervised image is generated.

[0039] The various processes of the image recognition device 100 described above can, of course, be implemented using hardware such as a CPU and memory, but can also be implemented using firmware stored in ROM (read-only memory) or flash memory, or software on a computer. The firmware program and software program can be recorded on a recording medium readable by a computer and provided, transmitted and received with a server via a wired or wireless network, or transmitted and received as data broadcasting on terrestrial or satellite digital broadcasting.

[0040] In conventional configurations, it was necessary to manually annotate unknown images and add them as training data during transfer learning. In this embodiment, the image recognition device 100 detects unknown images with low recognition accuracy based on the recognition results of a trained object recognition model, automatically converts the unknown images into supervised images, adds the converted supervised images as new training data, and retrains a portion of the trained object recognition model. As a result, unknown images can be recognized with high accuracy without manual intervention.

[0041] The present invention has been described above based on embodiments. The embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of their components and processing processes, and that such modifications also fall within the scope of the present invention. [Explanation of symbols]

[0042] 10 Input unit, 20 Object recognition unit, 30 Object recognition model storage unit, 40 Recognition accuracy determination unit, 50 Output unit, 60 Supervised image conversion unit, 70 Feature-label database, 80 Transfer learning unit, 90 Pre-trained dataset storage unit, 100 Image recognition device.

Claims

1. An object recognition unit that recognizes objects in an input image using an object recognition model, A recognition accuracy determination unit that determines the recognition accuracy of an object in the input image, A supervised image conversion unit that, for input images in which the recognition accuracy of the object falls below a predetermined threshold, labels the input image based on its feature quantities and converts it into a supervised image; An image recognition device characterized by including a transfer learning unit that transfers the object recognition model using the supervised images as training data and updates the object recognition model.

2. It further includes a database that stores pairs of image features and labels, The supervised image conversion unit refers to the database to obtain a label corresponding to the feature that is most similar to the feature of the input image, and attaches the obtained label to the input image to convert it into a supervised image, as described in claim 1.

3. The image recognition device according to claim 1 or 2, characterized in that the transfer learning unit constructs a new dataset by adding the supervised images to the dataset used for pretraining the object recognition model, and performs transfer learning of the object recognition model using the new dataset as training data.

4. The steps involve using an object recognition model to recognize objects in the input image, The steps include determining the accuracy of object recognition in the input image, For input images in which the recognition accuracy of the object falls below a predetermined threshold, the steps include labeling the input image based on its features and converting it into a supervised image, An image recognition method characterized by including the step of updating the object recognition model by performing transfer learning using the supervised images as training data.

Images