Image recognition device, image recognition method, and recording medium
By combining the object recognition unit, the recognition accuracy determination unit, and the transfer learning unit, unknown images are automatically converted into trained images, solving the problem of low recognition accuracy of unknown images and achieving efficient and high-precision image recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- JVC KENWOOD CORP
- Filing Date
- 2021-10-07
- Publication Date
- 2026-06-09
Smart Images
Figure CN116324876B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to image recognition technology. Background Technology
[0002] In devices that identify objects from images, images with low recognition accuracy are classified as unknown images that are difficult to identify. For unknown images, a manual indexing process is performed, in which objects within the image are identified and labeled by human hands.
[0003] Patent document 1 discloses the following system: acquiring an unknown image for which no learned model has been created, selecting a learned model of a known image whose shooting conditions are similar to those of the acquired unknown image from the learned models, performing image analysis on the unknown image using the selected learned model, and providing the result of the image analysis.
[0004] Existing technical documents
[0005] Patent documents
[0006] Patent Document 1: International Publication No. 2019 / 003355 Summary of the Invention
[0007] Manual indexing is time-consuming and costly, making it impractical for many users.
[0008] The present invention was made in view of the following situation, and its object is to provide an image recognition technology capable of recognizing unknown images with high accuracy.
[0009] To address the aforementioned issues, an image recognition apparatus according to a certain embodiment includes: an object recognition unit that uses an object recognition model to recognize objects in an input image; a recognition accuracy determination unit that determines the recognition accuracy of objects in the input image; a trained image conversion unit that, for input images where the object recognition accuracy is less than a predetermined threshold, adds labels based on the feature values of the input image to convert it into a trained image; and a transfer learning unit that uses the trained image as training data to perform transfer learning on the object recognition model to update the object recognition model.
[0010] Another embodiment of this method is an image recognition method. This method includes the following steps: using an object recognition model to identify objects in an input image; determining the recognition accuracy of the objects in the input image; for input images where the object recognition accuracy is less than a predetermined threshold, adding labels based on the feature values of the input image to convert it into a trained image; and using the trained image as training data to perform transfer learning on the object recognition model to update the object recognition model.
[0011] Another embodiment of this method uses a non-temporary computer-readable recording medium that stores an object recognition model. This object recognition model enables a computer to identify objects in an input image, using trained images as training data and updating it through transfer learning. The trained images are obtained by adding labels to the input images whose object recognition accuracy is less than a predetermined threshold, based on the feature values of the input images.
[0012] Furthermore, any combination of the above-mentioned constituent elements, or the manifestation of this embodiment in a way that is converted between methods, apparatus, systems, recording media, computer programs, etc., is also effective as a way of this embodiment.
[0013] According to this embodiment, an image recognition technology capable of highly accurate identification of unknown images can be provided. Attached Figure Description
[0014] Figure 1 This is a structural diagram of the image recognition device in the embodiment.
[0015] Figure 2 (a) to (d) represent input to Figure 1 An example of an image from the object recognition unit.
[0016] Figure 3 (a) to (d) are explanations Figure 1 The object recognition unit identifies objects based on a learned object recognition model. Figure 2 The resulting graphs of the objects in images (a) to (d).
[0017] Figure 4 This is a diagram illustrating the training images for which labels have been added to unknown images.
[0018] Figure 5 This is an explanation Figure 1 A flowchart of the image recognition steps of an image recognition device. Detailed Implementation
[0019] Figure 1 This is a structural diagram of the image recognition device 100 in the embodiment. The image recognition device 100 includes an input unit 10, an object recognition unit 20, an object recognition model storage unit 30, a recognition accuracy determination unit 40, an output unit 50, a trained image conversion unit 60, a feature quantity-label database 70, a transfer learning unit 80, and a pre-learning dataset storage unit 90.
[0020] The input unit 10 acquires an image of the object to be identified and provides the image to the object recognition unit 20.
[0021] The object recognition model storage unit 30 stores an object recognition model with excellent recognition accuracy that has been pre-learned using a pre-learning dataset. The pre-learning dataset storage unit 90 stores the pre-learning dataset used in the learning of the object recognition model.
[0022] The object recognition unit 20 uses the learned object recognition model stored in the object recognition model storage unit 30 to recognize objects in the input image. The object recognition unit 20 provides the recognition result of the objects in the input image to the recognition accuracy determination unit 40.
[0023] If the object recognition unit 20 achieves an object recognition accuracy of at least a predetermined threshold, the recognition result is provided to the output unit 50. The recognition result includes at least one candidate object category and its recognition accuracy. The output unit 50 outputs the recognition result of the input image.
[0024] The recognition accuracy determination unit 40 determines the recognition accuracy of objects in the input image and provides low-recognition-accuracy images (where the object recognition accuracy is less than a predetermined threshold) as unknown images to the trained image conversion unit 60.
[0025] For an unknown image provided by the recognition accuracy determination unit 40, the trained image conversion unit 60 adds labels based on the feature values of the unknown image and converts it into a trained image, and then provides it to the transfer learning unit 80. As an example of the feature values used to add labels to the unknown image, the intermediate output of the neural network, i.e., the feature values of the intermediate layer of the later stage of the neural network, can also be used when an image is input to the learned object recognition model stored in the object recognition model storage unit 30.
[0026] The feature-label database 70 is a database that stores pairs of image features and labels. The trained image conversion unit 60 refers to the feature-label database 70, obtains the label corresponding to the feature most similar to the feature of the unknown image, adds the obtained label to the unknown image, and converts it into a trained image. As an example of the feature-label database, a database that associates the features of the intermediate output (i.e., the intermediate layer of the later stage of the neural network) with the label of the image when an image is input to the learned object recognition model pre-stored in the object recognition model storage unit 30.
[0027] The transfer learning unit 80 adds the trained images supplied by the trained image conversion unit 60 to the pre-learning dataset stored in the pre-learning dataset storage unit 90 to form a new dataset. The object recognition model is then transferred using the new dataset, and the updated object recognition model is saved to the object recognition model storage unit 30.
[0028] Here, in the pre-learning dataset storage unit 90, if a large dataset or similar dataset used in the pre-learning of the object recognition model can be utilized, this dataset is saved as pre-learning data. If the dataset used in the pre-learning of the object recognition model cannot be utilized, known input images from which the object recognition unit 20 has identified objects with high accuracy can be stored as pre-learning datasets. In this case, the transfer learning unit 80 forms a new dataset by adding trained images labeled with unknown images to the known image pre-learning dataset stored in the pre-learning dataset storage unit 90, and performs transfer learning on the object recognition model using the new dataset.
[0029] In transfer learning, as an example, the final output layer of a learned model's neural network is replaced with a new layer, and the new dataset is used as training data to learn the parameters of the new layer again, thereby generating a new neural network.
[0030] By using an updated object recognition model, the object recognition unit 20 can recognize objects with high accuracy even when the input image contains objects with low recognition accuracy. Therefore, it can recognize objects of unknown categories.
[0031] Figure 2 Figures (a) to (d) are examples of images input to the object recognition unit 20. Figure 2 (a) is an image of a person. Figure 2 (b) is an image of a motorcycle. Figure 2 (c) is an image of a car. Figure 2 (d) is an image of a bicycle. The input image may also include other types, for example, a total of ten types.
[0032] The initial object recognition model could make extensive use of images of people, motorcycles, and cars, so these three types of images were used as a pre-set dataset for learning.
[0033] Figure 3 (a) to (d) illustrate how the object recognition unit 20 identifies objects based on the learned object recognition model. Figure 2 The resulting graphs of the objects in images (a) to (d).
[0034] The initial object recognition model had been pre-learned for three categories: people, motorcycles, and cars. Therefore, the recognition result, such as... Figure 3 As shown in (a) Figure 2 Identify labels such as "person" in image (a), such as Figure 3 As shown in (b) Figure 2 Identify tags such as "motorcycle" in image (b), such as Figure 3 As shown in (c) Figure 2In image (c), labels such as "car" are identified. However, because the initial object recognition model did not learn categories such as bicycles, ... Figure 3 As shown in (d), Figure 2 The recognition results for image (d) are of low accuracy and the labels are unclear.
[0035] The trained image conversion unit extracts 60 Figure 2 The feature values of image (d) are obtained by referring to the feature value-label database 70. Figure 2 The label corresponds to the feature most similar to the feature quantity of image (d). In this case, the obtained label is "bicycle". Figure 4 As shown, the trained image conversion unit 60 adds labels such as "bicycle" obtained from the feature-label database 70 to the image. Figure 2 The image of (d) is converted into a trained image.
[0036] Transfer Learning Department 80 uses Figure 4 The trained images are used as new training data to enable the object recognition model to perform transfer learning, generating a new object recognition model. This new model can then recognize four categories: people, motorcycles, cars, and bicycles. Thus, whenever an unknown image with low recognition accuracy is detected, it is converted into an labeled trained image, which is then used as new training data for the object recognition model to perform transfer learning, thereby enabling it to recognize all ten categories.
[0037] Here, if only trained images labeled with unknown categories are used as training data for the object recognition model to perform transfer learning, it is possible that the model may fail to correctly recognize images of known categories that can be recognized so far. Therefore, it is more preferable to use a new dataset, in which trained images of unknown categories are appended to the trained images of known categories, as training data for the object recognition model to perform transfer learning. For example, when using images of bicycles for transfer learning, a new dataset is created by appending trained images of bicycles to a pre-learning dataset containing trained images of people, motorcycles, and cars, and this dataset containing the four types of trained images is used as training data for the object recognition model to perform transfer learning. This allows the generation of an object recognition model that can ultimately correctly recognize all types of images.
[0038] Figure 5 This is a flowchart illustrating the image recognition steps of the image recognition device 100.
[0039] The object recognition unit 20 inputs the image of the object to be recognized into the learned object recognition model and recognizes the object in the image (S10).
[0040] If the object recognition accuracy is above the specified threshold (No in S20), output the label of the recognition result (S30), return to step S10, and input other images.
[0041] If the object recognition accuracy is less than a specified threshold, for example, if the probability of the correct answer as the first candidate label is less than 50% ("Yes" in S20), the input image is treated as an unknown image for feature extraction (S40).
[0042] The trained image conversion unit 60 obtains labels that match the features of the unknown image from the feature-label database 70 (S50), and generates a trained image by adding the obtained labels to the unknown image (S60).
[0043] The process from step S10 to step S60 is repeated until a predetermined number of training images of the same category of objects have been accumulated, for example, 30 images (S70 "No"). If the predetermined number of training images of the same category has been accumulated (here, 30 images) (S70 "Yes"), the transfer learning unit 80 adds the predetermined number of training images to the dataset used in pre-learning to generate a new dataset, and uses the new dataset to perform transfer learning on the learned object recognition model (S80). The transfer learning unit 80 generates a new object recognition model that has undergone transfer learning and saves it in the object recognition model storage unit 30 (S90).
[0044] In the above description, transfer learning was performed after accumulating a specified number of training images of objects of the same category, but transfer learning can also be performed each time a training image is generated.
[0045] The various processing methods described above by the image recognition device 100 can be implemented using hardware such as a CPU and memory, or using software such as firmware stored in ROM (Read-Only Memory) or flash memory, or a computer. The firmware or software program can be provided by recording it on a readable recording medium such as a computer, or it can be transmitted and received from a server via wired or wireless networks, or it can be transmitted and received as data broadcast via terrestrial wave or satellite digital broadcasting.
[0046] In conventional structures, unknown images require manual indexing and are added as training data during transfer learning. In the image recognition device 100 of this embodiment, based on the recognition results of the learned object recognition model, unknown images with low recognition accuracy are detected, and the unknown images are automatically converted into trained images. These converted trained images are then added as new training data, and the learned object recognition model is relearned. Therefore, unknown images can be recognized with high accuracy without manual intervention.
[0047] The present invention has been described above based on the embodiments. Those skilled in the art should understand that the embodiments are illustrative, and the combination of the constituent elements and processing procedures can have various modifications, and such modifications are also within the scope of the present invention.
[0048] Industrial availability
[0049] This invention can be used in image recognition technology.
[0050] Symbol Explanation
[0051] 10. Input Section
[0052] 20. Object recognition unit
[0053] 30 Object Recognition Model Storage Unit
[0054] 40. Recognition Accuracy Judgment Department
[0055] 50 Output Unit
[0056] 60 trained image conversion units
[0057] 70 Feature Quantities - Label Database
[0058] 80 Transfer Learning Department
[0059] 90 Pre-learning dataset storage department
[0060] 100 Image recognition device.
Claims
1. An image recognition device, characterized in that, include: The object recognition model storage unit stores object recognition models that enable computers to recognize objects in input images. The object recognition unit uses the object recognition model to recognize the objects in the input image; The recognition accuracy determination unit determines the recognition accuracy of objects in the input image; The trained image conversion unit adds labels to the input image based on the feature quantity of the input image for the input image whose object recognition accuracy is less than a specified threshold, thereby converting it into a trained image. The feature quantity is the intermediate output after the input image is input into the object recognition model, and is the feature quantity of the intermediate layer of the subsequent stage of the object recognition model. The transfer learning unit uses the trained images as training data to perform transfer learning on the object recognition model, thereby updating the object recognition model. as well as A database that stores pairs of image features and labels. The trained image conversion unit refers to the database, obtains the label corresponding to the feature value most similar to the feature value of the input image, and adds the obtained label to the input image to convert it into the trained image. The transfer learning unit appends the trained images to the pre-learning dataset used for the object recognition model to form a new dataset, and uses the new dataset as training data to perform transfer learning on the object recognition model.
2. The image recognition device as described in claim 1, characterized in that, When the number of trained images with the same label exceeds a certain limit, the transfer learning unit appends the trained images to the pre-learning dataset used for the object recognition model to form the new dataset.
3. An image recognition method, characterized in that, Includes the following steps: The objects in the input image are identified using an object recognition model that enables a computer to recognize objects in the input image; Determine the accuracy of object recognition in the input image; The conversion step involves adding labels to the input image based on the feature values of the input image for an input image whose object recognition accuracy is less than a specified threshold, thereby converting it into a trained image. The feature values are the intermediate outputs after the input image is input into the object recognition model, and are the feature values of the intermediate layers of the subsequent levels of the object recognition model. The update step involves using the trained images as training data to perform transfer learning on the object recognition model, thereby updating the object recognition model. as well as Store pairs of image features and labels in the database. In the conversion step, referring to the database, the label corresponding to the feature most similar to the feature of the input image is obtained, and the obtained label is added to the input image to convert it into the trained image; In the update step, the trained images are appended to the pre-learning dataset used for the object recognition model to form a new dataset, and the new dataset is used as training data to perform transfer learning on the object recognition model.