Apparatus and methods for classifying image, training image classifier, and determining data set
By generating synthetic digital images and verifying them using pre-trained image classifiers, the problem in the existing technology that training image classifiers requires real-world object images and human annotations is solved, and efficient and low-cost image classifier training is achieved.
Patent Information
- Application Number
- CN202410964374.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-07-18
- Filing Date
- 2024-07-18
- Publication Date
- 2025-01-21
AI Technical Summary
The prior art requires digital images and human annotations of real-world objects when training image classifiers, and there is a problem of high data acquisition and labeling costs.
By generating synthetic digital images, image classifiers are trained based on the object's category, avoiding reliance on real-world object images and human annotations. The method involves generating synthetic digital images using a generative model and validating the accuracy of the synthetic digital images through a pre-trained image classifier.
It achieves training of image classifiers without the need for real-world object images and human annotations, reduces the cost of data acquisition and annotation, and improves training efficiency.
Smart Images
Figure CN119339123A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an apparatus and method for classifying digital images using an image classifier, training the image classifier, and determining an image data set for training. Background Art
[0002] The training of image classifiers uses annotated training data consisting of digital images of real-world objects. Summary of the invention
[0003] A method for classifying a digital image with an image classifier, wherein the method includes providing a class representing an object, determining a synthetic digital image including the object based on the class, training the image classifier to predict a class representing the object based on the synthetic digital image, and determining the class based on the digital image with the image classifier. The image classifier is trained without the digital image of the real world object. The image classifier is trained without requiring human annotation of the synthetic digital image.
[0004] In order to classify or use the digital image of the real-world object, the method may comprise capturing the digital image with a sensor, in particular a camera, a radar sensor, a lidar sensor, an ultrasound sensor, an infrared sensor or a thermal image sensor, or depending on the category an actuating technical system, in particular a robot, preferably a vehicle, a household appliance, a power tool, a personal assistance system or an access control system.
[0005] The method may include determining a plurality of synthetic digital images for the class or classes and selecting a synthetic digital image from the plurality of synthetic digital images for training the image classifier.
[0006] Selecting the synthetic digital image may include determining with a pre-trained image classifier that the synthetic digital image includes an object represented by a class provided for determining the synthetic digital image. This confirms that the synthetic digital image used in training represents the class of objects well.
[0007] A method for training an image classifier for classifying digital images with the image classifier comprises providing a class representing an object, determining a synthetic digital image comprising the object depending on the class, and training the image classifier to predict the class representing the object depending on the synthetic digital image. This means that neither human annotation nor digital image input is required for training.
[0008] The method may include determining a plurality of synthetic digital images for the class or classes and selecting a synthetic digital image from the plurality of synthetic digital images for training the image classifier.
[0009] Selecting the synthetic digital image may include determining with a pre-trained image classifier that the synthetic digital image includes an object represented by a class provided for determining the synthetic digital image. This verifies during training that the synthetic digital image used in training represents the class of objects well.
[0010] A method for determining an image dataset for training an image classifier may include providing a category representing an object, determining a synthetic digital image including the object and labeled with the category depending on the category, and adding the synthetic digital image to the image dataset. This means that determining the image dataset requires neither human annotation nor digital image input.
[0011] The method may include determining a plurality of composite digital images for the category or categories and selecting a composite digital image for the image dataset from the plurality of composite digital images.
[0012] Selecting the synthetic digital image may include determining with a pre-trained image classifier that the synthetic digital image includes an object represented by a category provided for determining the synthetic digital image. This confirms that the synthetic digital image in the image dataset represents the object of that category well.
[0013] A device, wherein the device comprises at least one processor and at least one memory configured to store instructions, which, when executed by the at least one processor, causes the device to perform the method, and wherein the at least one processor is configured to execute the instructions. The device has advantages corresponding to those of the method.
[0014] A computer program comprising computer readable instructions which, when executed by a computer, cause the computer to perform the method, provides advantages corresponding to those of the method. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Further advantageous embodiments can be derived from the following description and the accompanying drawings. In the drawings:
[0016] Figure 1 A device is schematically depicted,
[0017] Figure 2 Schematically depicts an architecture,
[0018] Figure 3 schematically depicts a flow chart of a method for determining an image data set,
[0019] Figure 4 schematically depicts a flow chart of a method for training an image classifier,
[0020] Figure 5A flow chart of a method for classifying an image is schematically depicted. DETAILED DESCRIPTION
[0021] Figure 1 The device 100 is schematically depicted. The device 100 comprises at least one processor 102 and at least one memory 104, the memory 104 being configured to store instructions which, when executed by the at least one processor 102, cause the device 100 to perform a method.
[0022] The at least one processor 102 is configured to execute instructions.
[0023] The instructions according to one embodiment of the device 100 cause the device 100 to perform a method of classifying a digital image using an image classifier.
[0024] The instructions according to one embodiment of the device 100 cause the device 100 to perform a method for training an image classifier.
[0025] The instructions according to one embodiment of the device 100 cause the device 100 to perform a method for determining an image data set for training an image classifier.
[0026] The device 100 may be configured to capture digital images with a sensor 106. The sensor 106 is, for example, a camera, a radar sensor, a lidar sensor, an ultrasonic sensor, an infrared sensor, or a thermal image sensor.
[0027] The device 100 can be configured to actuate an actuator 108 of a technical system 110 depending on the category. The technical system 110 is, for example, a robot, preferably a vehicle, a domestic appliance, a power tool, a personal assistance system or an access control system.
[0028] An example for actuating the technical system 110 is to output a category, such as a text, on a human-machine interface (not depicted). For a household appliance (such as a refrigerator), the category may, for example, represent the name of a product (such as an apple) captured by the sensor 106 inside the household appliance. For a household appliance (such as a cooker), the category may, for example, represent the name of the cooking state of food in the cooker (such as raw or cooked) captured by the sensor 106 inside the cooker.
[0029] Examples for actuating the technical system 110 are moving the technical system 110 on a trajectory to avoid a collision with an object of a class indicating that the object is an obstacle, or moving the technical system 110 on a trajectory to pass over an object of a class indicating that the object is suitable for passing over it.
[0030] An example for actuating the technical system 110 is to open access for an object of a class indicating that the object has access rights, or to not open access otherwise.
[0031] In this example, the technical system 110 includes the sensor 106 and the actuator 110. The device 100 may include the sensor 106 or the actuator 110 instead of the technical system 110. The technical system 110 may include the device.
[0032] The technical system 110 may include a trained image classifier or an image classifier having parameters and architecture of a trained classifier. The technical system 110 may be configured to capture a digital image with the sensor 106, determine a category of the captured digital image with the trained image classifier, and actuate the actuator 108 depending on the category determined for the captured digital image with the trained image classifier.
[0033] The training of an image classifier may be separated from the use of the trained image classifier.
[0034] Figure 2 The architecture 200 is schematically depicted. The architecture 200 includes an image classifier 202 and a generative model 204.
[0035] Generative model 204 is configured to determine, depending on category 206 representing an object, a composite digital image 208 that includes the object.
[0036] The generative model 204 may be a model trained on a large amount of web crawled data. The model may be a model of the diffusion probability model family, a class of latent variable models that can model image data distribution by learning to denoise images corrupted by Gaussian noise. The generative model 204 may be a text-to-image model, such as a stable diffusion model, such as DALL-E.
[0037] Image classifier 202 is configured to determine a class 210 for composite digital image 208 .
[0038] The architecture 200 may include a pre-trained image classifier 212 .
[0039] The pre-trained image classifier 212 is trained for determining whether a composite digital image includes an object that is represented by the category 206 provided for determining the composite digital image 208 .
[0040] The pre-trained image classifier 212 may be a visual-language (VL) model, such as contrastive-language image pre-training (CLIP).
[0041] The architecture 200 may include a selector 214 configured to select the synthetic digital image 208 for the image dataset 216 depending on an output 218 of the pre-trained image classifier 212. The selector 214 is configured, for example, to select the synthetic digital image 208 for the image dataset 216 when the output 218 indicates that the synthetic digital image 208 includes an object represented by the category 206 provided for determining the synthetic digital image 208, and not to select the synthetic digital image 210 for the image dataset 216 otherwise.
[0042] According to an example, a category is a text, such as a string, that describes an object in natural language. Examples of text are words, such as cat, dog, limousine, scooter. The text may be a phrase, a sentence, or a part of a sentence, such as "inside a limousine", "red scooter", "scooter in winter", "scooter at night", "scooter in accident". The category may be a numeric or alphanumeric encoding of the text.
[0043] The training may use a training pipeline that relies on a general optimizer (e.g., Adam). The training may train multiple instances of the image classifier until convergence and select the best image classifier from the multiple instances.
[0044] Figure 3 A flow chart of a method for determining an image data set 216 for training an image classifier 202 is depicted.
[0045] The method for determining the image data set 216 includes step 302 .
[0046] In step 302, a category 206 representing an object is provided.
[0047] The method for determining the image data set 216 includes step 304 .
[0048] In step 304, a composite digital image 208 including the object and labeled with the category 206 is determined depending on the category.
[0049] The method for determining the image data set 216 optionally includes step 306 .
[0050] In step 306, the synthetic digital image 208 is either selected for the image data set 216 or not. In this example, the synthetic digital image 208 is either selected for the image data set 216 or not, depending on the output 218 of the pre-trained image classifier 212. In the event that the synthetic digital image 208 is selected for the image data set 216, step 308 is performed. Otherwise, step 302 is performed.
[0051] For example, when output 218 indicates that composite digital image 208 includes an object represented by category 206 provided for determining composite digital image 208, composite digital image 208 is selected for image dataset 216. Otherwise, composite digital image 208 may not be selected for image dataset 216.
[0052] In step 308 , the composite digital image 208 is added to the image dataset 216 .
[0053] Afterwards, execute step 302.
[0054] According to an example, a plurality of synthesized digital images 208 of the class or classes are determined and selected to be used for training the image classifier or not to be used for training the image classifier.
[0055] Figure 4 A flow chart of a method for training the image classifier 202 to classify digital images with the image classifier 202 is depicted.
[0056] The method for training the image classifier 202 includes step 402 .
[0057] Step 402 includes training the image classifier 202 with the image data set 216. The image data set 216 may be determined using a method for determining the image data set 216. The method for training the image classifier 202 may include a method for determining the image data set 216.
[0058] Figure 5 A flow chart of a method for classifying a digital image with the image classifier 202 is depicted.
[0059] The method for classifying a digital image may include a method for determining the image dataset 216 and / or a method for training the image classifier 202 .
[0060] The method for classifying a digital image includes step 502 .
[0061] Step 502 includes capturing a digital image with the sensor 106 .
[0062] The method for classifying a digital image includes step 504 .
[0063] Step 504 includes determining a class depending on the digital image using the image classifier 202. The method for classifying a digital image includes step 506.
[0064] Step 506 includes actuating the technical system 110 depending on the category.
Claims
1. A method for classifying a digital image using an image classifier (202), characterized in that The method includes providing (302) a class (206) representing an object, determining (304) a composite digital image (208) including the object based on the class (206), training (402) an image classifier (202) to predict the class (206) representing the object based on the composite digital image (208), and determining (504) the class based on the digital image using the image classifier (202).
2. The method according to claim 1, characterized in that The method comprises capturing (502) the digital image with a sensor (106), in particular a camera, a radar sensor, a lidar sensor, an ultrasound sensor, an infrared sensor or a thermal image sensor, or, depending on the category, actuating (506) a technical system (110), in particular a robot, preferably a vehicle, a household appliance, a power tool, a personal assistance system or an access control system.
3. The method according to claim 1 or 2, characterized in that: The method includes determining (304) a plurality of synthetic digital images (208) for the class (206) or classes and selecting (306) a synthetic digital image from the plurality of synthetic digital images (208) for training an image classifier (202).
4. The method according to claim 3, characterized in that Selecting (306) the composite digital image (208) includes determining, using a pre-trained image classifier (212), that the composite digital image includes an object that is represented by a class provided for determining the composite digital image (208).
5. A method for training an image classifier to classify digital images using the image classifier (202), characterized in that The method includes providing (302) a class (206) representing an object, determining (304) a composite digital image (208) including the object based on the class (206), and training (402) an image classifier (202) to predict the class (206) representing the object based on the composite digital image (208).
6. The method according to claim 5, characterized in that The method includes determining (304) a plurality of synthetic digital images (208) for the class or classes and selecting (306) a synthetic digital image (208) from the plurality of synthetic digital images (208) for training an image classifier (202).
7. The method according to claim 6, characterized in that Selecting (306) the composite digital image (208) includes determining, using a pre-trained image classifier (212), that the composite digital image (208) includes an object that is represented by a category provided for determining the composite digital image (208).
8. A method for determining an image data set (216) for training an image classifier (202), characterized in that The method includes providing (302) a category (206) representing an object, determining (304) a composite digital image (208) including the object and labeled with the category (206) based on the category (206), and adding (308) the composite digital image (208) to an image dataset (216).
9. The method according to claim 8, characterized in that The method includes determining (304) a plurality of synthesized digital images (208) for the category (106) or categories, and selecting (306) a synthesized digital image (208) from the plurality of synthesized digital images for an image data set (216).
10. The method according to claim 9, characterized in that Selecting (306) the composite digital image includes determining, using a pre-trained image classifier, that the composite digital image includes an object that is represented by a class provided for determining the composite digital image.
11. A device (100), characterized in that The device (100) comprises at least one processor (102) and at least one memory (104), the memory (104) being configured to store instructions which, when executed by the at least one processor (102), cause the device (100) to perform a method according to one of claims 1 to 10, wherein the at least one processor (102) is configured to execute the instructions.
12. A computer program, characterized in that The computer program comprises computer readable instructions which, when executed by a computer, cause the computer to perform the method according to one of claims 1 to 10 .