Device and method for classifying digital image with image classifier, training image classifier, and determining image dataset for training
The method trains image classifiers using synthetic digital images and pre-trained models to automate the training process, addressing the inefficiency of human-annotated data and ensuring accurate class representation.
Patent Information
- Application Number
- JP2024114091
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-07-18
- Filing Date
- 2024-07-17
- Publication Date
- 2025-01-30
AI Technical Summary
Existing image classification technologies require human-annotated digital images of real-world objects for training, which is time-consuming and resource-intensive.
A method for training image classifiers using synthetic digital images without human annotation, utilizing sensors to capture images and employing pre-trained models to select and label synthetic images based on class representation, thereby creating image datasets for training.
Enables efficient and automated training of image classifiers without the need for human intervention, ensuring accurate representation of classes in the training data.
Smart Images
Figure 2025015476000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an apparatus and method for classifying digital images using an image classifier, training the image classifier, and determining an image data set for training. [Background technology]
[0002] Training the image classifier uses annotated training data that includes digital images of real-world objects. Summary of the Invention [Means for solving the problem]
[0003] A method for classifying digital images with an image classifier includes providing classes representing objects, determining a synthetic digital image including the object according to the classes, training the image classifier to predict a class representing the object according to the synthetic digital image, and determining the class with the image classifier according to the digital image. The image classifier is trained without digital images of real-world objects. The image classifier is trained without requiring human annotation of the synthetic digital image.
[0004] For classifying or using digital images of real-world objects, the method may comprise a step of capturing the digital images by a sensor, in particular a camera, a radar sensor, a LiDAR sensor, an ultrasonic sensor, an infrared sensor or a thermal imaging sensor, or a step of operating a technical system, in particular a robot, preferably a vehicle, a household appliance, a power tool, a personal assistant system or an access control system, depending on the class.
[0005] The method may include determining a plurality of synthetic digital images in a class or classes, and selecting a synthetic digital image from the plurality of synthetic digital images for training an image classifier.
[0006] The step of selecting the synthetic digital image may include determining, by a pre-trained image classifier, that the synthetic digital image contains objects that are represented by the classes provided to determine the synthetic digital image, ensuring that the synthetic digital images used in training adequately represent the objects of the classes.
[0007] A method for training an image classifier to classify digital images with the image classifier comprises the steps of providing classes representing objects, determining a composite digital image containing the object according to the classes, and training the image classifier to predict the class representing the object according to the composite digital image, which means that the training does not require human annotation or digital image input.
[0008] The method may include determining a plurality of synthetic digital images in a class or classes, and selecting a synthetic digital image from the plurality of synthetic digital images for training an image classifier.
[0009] The step of selecting the synthetic digital image may include determining, by a pre-trained image classifier, that the synthetic digital image contains objects that are represented by the classes provided to determine the synthetic digital image, ensuring that the synthetic digital images used in training adequately represent the objects of the classes.
[0010] A method for determining an image dataset for training an image classifier may comprise the steps of providing classes representing objects, determining according to the classes a synthetic digital image containing the object and labelled with the class, and adding the synthetic digital image to the image dataset, which means that determining the image dataset does not require human annotation or digital image input.
[0011] The method may include determining a plurality of synthetic digital images for a class or classes, and selecting a synthetic digital image for the image dataset from the plurality of synthetic digital images.
[0012] The step of selecting the synthetic digital image may include determining, by a pre-trained image classifier, that the synthetic digital image contains objects that are represented by the classes provided to determine the synthetic digital image, ensuring that the synthetic digital images in the image dataset adequately represent the objects of the classes.
[0013] An apparatus comprising at least one processor and at least one memory configured to store instructions, the at least one processor is configured to execute the instructions, when executed by the at least one processor, to cause the apparatus to perform the method. The apparatus has advantages corresponding to the advantages of the method.
[0014] A computer program comprising computer readable instructions for causing a computer to carry out the method when executed by a computer provides advantages corresponding to those of the method.
[0015] Further advantageous embodiments can be derived from the following description and the drawings. [Brief description of the drawings]
[0016] [Figure 1] 1 shows a schematic diagram of an apparatus; [Diagram 2] 1 illustrates the architecture diagrammatically. [Diagram 3] 2 shows a schematic flow chart of a method for determining an image dataset. [Figure 4] 1 shows a schematic flow chart of a method for training an image classifier. [Diagram 5] 1 shows a schematic flow chart of a method for classifying an image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0017] 1 generally illustrates an apparatus 100. The apparatus 100 comprises at least one processor 102 and at least one memory 104 configured to store instructions that, when executed by the at least one processor 102, cause the apparatus 100 to perform a method.
[0018] At least one processor 102 is configured to execute instructions.
[0019] The instructions according to one embodiment of the device 100 are for causing the device 100 to perform a method for classifying a digital image using an image classifier.
[0020] The instructions for one embodiment of the apparatus 100 are for causing the apparatus 100 to perform a method for training an image classifier.
[0021] The instructions of one embodiment of the apparatus 100 are for causing the apparatus 100 to perform a method for determining an image dataset for training an image classifier.
[0022] The device 100 may be configured to capture digital images with a sensor 106. The sensor 106 may be, for example, a camera, a radar sensor, a LiDAR sensor, an ultrasonic sensor, an infrared sensor, or a thermal imaging sensor.
[0023] Depending on the class, the device 100 may be configured to operate an actuator 108 of a technical system 110. The technical system 110 is for example a robot, preferably a vehicle, a household appliance, a power tool, a personal assistant system or an access control system.
[0024] An example for operating the technical system 110 is to output a class, e.g. a text, on a human machine interface (not shown). In the case of a home appliance, e.g. a refrigerator, the class may represent, e.g., the name of a produce, e.g. an apple, captured by a sensor 106 in the appliance. In the case of a home appliance, e.g. a cooker, the class may represent, e.g., the name of a cooking state of a food in the cooker, e.g. raw or cooked, captured by a sensor 106 in the cooker.
[0025] An example for operating the technical system 110 is to move the technical system 110 on a trajectory to avoid collision with an object of a class indicating that the object is an obstacle, or on a trajectory to move over an object of a class indicating that the object is suitable for moving over.
[0026] One example for operating the technical system 110 is to open access to objects of a class that indicate that the object has access rights, or not open access otherwise.
[0027] In this example, the technical system 110 comprises a sensor 106 and an actuator 108. The device 100 may comprise the sensor 106 or the actuator 108 instead of the technical system 110. The technical system 110 may comprise a device.
[0028] The technical system 110 may comprise a trained image classifier or an image classifier having parameters and architecture of a trained classifier. The technical system 110 may be configured to capture a digital image with the sensor 106, determine a class for the captured digital image with the trained image classifier, and operate the actuator 108 in response to the class determined for the captured digital image by the trained image classifier.
[0029] Training an image classifier may be separate from using the trained image classifier.
[0030] 2 illustrates a schematic of an architecture 200. The architecture 200 comprises an image classifier 202 and a generative model 204.
[0031] The generative model 204 is configured to determine a synthetic digital image 208 including the object according to a class 206 representing the object.
[0032] The generative model 204 may be a model trained on a large amount of web-scraped data. The model may be a model of the family of diffusion probability models, which are a class of latent variable models that can model image data distributions by learning to denoise images corrupted by Gaussian noise. The generative model 204 may be a text-image model, such as a stable diffusion model, e.g., DALL-E.
[0033] The image classifier 202 is configured to determine a class 206 for a composite digital image 208 .
[0034] The architecture 200 may include a pre-trained image classifier 212.
[0035] The pre-trained image classifier 212 is trained to determine whether the objects contained in the composite digital image are objects represented by the classes 206 provided to determine the composite digital image 208 .
[0036] The pre-trained image classifier 212 may be a visual-linguistic (VL) model such as Contrastive Language Image Pre-Training (CLIP).
[0037] The architecture 200 may comprise a selector 214 configured to select the synthetic digital image 208 for the image dataset 216 in response to an output 218 of the pre-trained image classifier 212. The selector 214 is configured, for example, to select the synthetic digital image 210 for the image dataset 216 if the output 218 indicates that the synthetic digital image 208 includes an object represented by the class 206 provided for determining the synthetic digital image 208, or to not select the synthetic digital image 208 for the image dataset 216 otherwise.
[0038] According to one example, a class is a text, e.g., a string, describing an object in natural language. Examples of text are words, e.g., cat, dog, limousine, scooter. The text may be a phrase, a sentence, or a part of a sentence, e.g., "inside the limousine", "red scooter", "scooter in winter", "scooter at night", "scooter in accident". A class may be a numeric or alphanumeric encoding of the text.
[0039] Training may use a training pipeline that relies on a common optimizer, e.g., Adam, and may involve training multiple instances of an image classifier until convergence and selecting the best image classifier from the multiple instances.
[0040] FIG. 3 shows a flow chart of a method for determining an image dataset 216 for training an image classifier 202.
[0041] The method for determining the image dataset 216 includes step 302 .
[0042] In step 302, a class 206 is provided that represents an object.
[0043] The method for determining the image dataset 216 includes a step 304 .
[0044] In step 304, a synthetic digital image 208 containing the object and labeled with a class 206 is determined according to the class.
[0045] The method for determining the image dataset 216 optionally includes step 306 .
[0046] In step 306, the synthetic digital image 208 is either selected for the image dataset 216 or not. In this example, the synthetic digital image 208 is either selected for the image dataset 216 or not depending on the output 218 of the pre-trained image classifier 212. If the synthetic digital image 208 is selected for the image dataset 216, step 308 is performed. Otherwise, step 302 is performed.
[0047] The synthetic digital image 208 is selected for the image dataset 216, for example, if the output 218 indicates that the synthetic digital image 208 includes an object represented by the class 206 provided for determining the synthetic digital image 208. Otherwise, the synthetic digital image 208 may not be selected for the image dataset 216.
[0048] In step 308 , the composite digital image 208 is added to the image dataset 216 .
[0049] Then, step 302 is executed.
[0050] According to one example, a number of synthetic digital images 208 in a class or classes are determined and either selected or not selected for training an image classifier.
[0051] FIG. 4 shows a flow chart of a method for training the image classifier 202 and classifying digital images with the image classifier 202 .
[0052] The method for training the image classifier 202 includes step 402 .
[0053] Step 402 includes training the image classifier 202 with the image dataset 216. The image dataset 216 may be determined using a method for determining the image dataset 216. The method for training the image classifier 202 may include a method for determining the image dataset 216.
[0054] FIG. 5 shows a flow chart of a method for classifying a digital image by the image classifier 202.
[0055] The method for classifying a digital image may include a method for determining the image dataset 216 and / or a method for training the image classifier 202 .
[0056] The method for classifying a digital image includes step 502 .
[0057] Step 502 involves capturing a digital image with the sensor 106 .
[0058] The method for classifying a digital image includes a step 504 .
[0059] Step 504 involves determining a class by the image classifier 202 as a function of the digital image.
[0060] The method for classifying a digital image includes a step 506 .
[0061] Step 506 includes operating the technology system 110 according to the class.
Claims
1. A method for classifying a digital image by an image classifier (202), comprising: Providing (302) a class (206) representing an object; determining (304) a synthetic digital image (208) containing said object according to said class (206); training (402) the image classifier (202) to predict the class (206) representing the object in response to the synthetic digital image (208); determining (504) a class by the image classifier (202) in response to the digital image; The method according to claim 1, further comprising:
2. - capturing (502) said digital image by a sensor (106), in particular a camera, a radar sensor, a LiDAR sensor, an ultrasonic sensor, an infrared sensor or a thermal imaging sensor, or operating (506) a technical system (110), in particular a robot, preferably a vehicle, a household appliance, a power tool, a personal assistant system or an access control system, depending on said class; The method of claim 1 , comprising:
3. determining (304) a plurality of composite digital images (208) in said class (206) or classes; selecting (306) the synthetic digital image from the plurality of synthetic digital images (208) for training the image classifier (202); 3. The method according to claim 1 or 2, characterized in that it comprises:
4. The step of selecting (306) the composite digital image (208) comprises: determining by a pre-trained image classifier (212) that the object contained in the synthetic digital image is the object represented by the class provided to determine the synthetic digital image (208); The method of claim 3, comprising:
5. 1. A method for training an image classifier (202) to classify a digital image by said image classifier, comprising: Providing (302) a class (206) representing an object; determining (304) a synthetic digital image (208) containing said object according to said class (206); training (402) the image classifier (202) to predict the class (206) representing the object in response to the synthetic digital image (208); The method according to claim 1, further comprising:
6. determining (304) a plurality of composite digital images (208) in said class or classes; selecting (306) the synthetic digital image (208) from the plurality of synthetic digital images (208) for training the image classifier (202); The method of claim 5, comprising:
7. The step of selecting (306) the composite digital image (208) comprises: determining by a pre-trained image classifier (212) that the object contained in the synthetic digital image (208) is the object represented by the class provided for determining the synthetic digital image (208); The method of claim 6, comprising:
8. 1. A method for determining an image dataset (216) for training an image classifier (202), comprising: Providing (302) a class (206) representing an object; determining (304) a synthetic digital image (208) containing said object and labelled with said class (206) in response to said class (206); adding (308) the composite digital image (208) to the image dataset (216); The method according to claim 1, further comprising:
9. determining (304) a plurality of composite digital images (208) in said class (206) or classes; selecting (306) the composite digital image (208) for the image dataset (216) from the plurality of composite digital images (208); 9. The method of claim 8, comprising:
10. The step of selecting (306) the composite digital image comprises: determining by a pre-trained image classifier that the object contained in the synthetic digital image is the object represented by the class provided to determine the synthetic digital image; 10. The method of claim 9, comprising:
11. At least one processor (102); At least one memory (104) configured to store instructions; An apparatus (100) comprising: The at least one processor (102) is configured to execute the instructions, which, when executed by the at least one processor (102), cause the apparatus (100) to perform the method of any one of claims 1 to 10.
1. An apparatus (100).
12. A computer program comprising computer readable instructions which, when executed by a computer, cause the computer to carry out the method according to any one of claims 1 to 10.