Distorted image data generation system and method

Abstract

The present invention provides a distorted image data generation system comprising: an augmented data generation unit for generating, in a system for generating image data to be used as training data of an artificial intelligence model, various types of distorted images of an original image by using a generative model; and a classification model training unit for training a classification model by using the original image and the distorted image together.

Description

System and method for generating distorted image data

[0001] The present invention relates to a distorted image data generation system and method capable of generating waste image data used in an artificial intelligence-based waste classification device.

[0002] A large amount of waste is produced every day from homes, factories, restaurants, etc., and many people separate and throw away food waste among general waste, recyclable waste, and food waste.

[0003] However, recyclable waste is still being discharged mixed with general waste, and manpower is being directly invested in separating it.

[0004] In addition, companies that use PET or glass bottles are importing recyclable waste from overseas because recyclable waste is not properly separated and is disposed of as general waste.

[0005] In Korea, various inventions are being made to solve these problems, and in particular, methods, devices, and systems for separating PET bottles are being applied for and registered as patents.

[0006] For example, Korean Patent Publication No. 10-2023-0084017, "Recyclable waste sorting device, method, and program using an artificial intelligence model," relates to a recyclable waste sorting device, which captures images of waste on a conveyor through at least one camera in a preset sorting area, analyzes the captured images based on an artificial intelligence model to determine the type of each waste on the conveyor, and analyzes the captured images using an artificial intelligence model to determine the type of waste for each waste on the conveyor.

[0007] For these AI-based recyclable waste sorting devices, the performance of the AI ​​model trained using images collected from a vision camera as training data is crucial.

[0008] To improve the performance of artificial intelligence models used in waste sorting devices, it is essential to secure training data used in the artificial intelligence models.

[0009] The technical problem to be achieved by the present invention is to provide a distorted image data generation system and method capable of generating waste image data used in an artificial intelligence-based waste classification device.

[0010] In order to solve such a problem, a distorted image data generation system according to an embodiment of the present invention comprises: an augmented data generation unit that generates various types of distorted images for an original image using a generation model in a system that generates image data used as training data for an artificial intelligence classification model; and a classification model learning unit that learns a classification model using the original image and the distorted image together.

[0011] The above original image may include misclassified images of target objects that the robot arm was unable to classify based on the artificial intelligence model.

[0012] The distorted image may include a distorted or broken form of the misclassified image.

[0013] The above generative model may use one or more of a variational autoencoder (VAE), a generative adversarial network (GAN), and a diffusion model.

[0014] The above generation model can generate the distorted image in preset stages according to the degree of distortion.

[0015] The above classification model can be repeatedly trained with the above generation model and the above classification model until the classification error rate of the target object becomes below a preset standard.

[0016] Among the various combinations of generative models and classification models, the combination with the smallest number of iterative executions can be selected.

[0017] The robot arm may further include a misclassification image determination unit that collects misclassified images of target objects that the robot arm was unable to classify.

[0018] A method for generating distorted image data according to an embodiment of the present invention includes a misclassification image determination step of collecting misclassified images of target objects that a robot arm was unable to classify; a data augmentation step of generating various types of distorted images for the misclassified images using a generative model; and a classification model learning step of learning a classification model using the misclassified images and the distorted images together.

[0019] The above distorted image includes a distorted or broken form of the above misclassified image.

[0020] The above generative model may use one or more of a variational autoencoder (VAE), a generative adversarial network (GAN), and a diffusion model.

[0021] The above generation model can generate the distorted image in preset stages according to the degree of distortion.

[0022] The above classification model can be repeatedly trained with the above generation model and the above classification model until the classification error rate of the target object becomes below a preset standard.

[0023] Among the various combinations of generative models and classification models, the combination with the smallest number of iterative executions can be selected.

[0024] In addition to the technical problems of the present invention mentioned above, other features and advantages of the present invention are described below or may be clearly understood by a person skilled in the art to which the present invention pertains from such description and explanation.

[0025] According to the present invention as described above, the following effects are achieved.

[0026] The present invention can reduce the possibility of misclassification of a target object by generating various types of distorted images of the target object to be classified and training the classification model.

[0027] The present invention can generate a distorted image of an appropriate level by presetting a classification error rate of a target object and repeatedly training a generation model and a classification model until the classification error rate of the target object becomes lower than a preset standard.

[0028] The present invention not only reduces the possibility of misclassification of a target object by repeatedly training a generation model and a classification model until the classification error rate of the target object becomes below a preset standard, thereby generating a distorted image of an appropriate level, but also eliminates the need for a separate image labeling task for a new target object.

[0029] Through the method for generating distorted image data according to the present invention, the optimal combination of generation models and classification models can be selected by selecting the combination with the smallest number of cyclic repetitions among various combinations of generation models and classification models.

[0030] In addition, other features and advantages of the present invention may be newly discovered through the embodiments of the present invention.

[0031] FIG. 1 is a schematic diagram of a waste classification system using an artificial intelligence-based robot arm applied to a distorted image data generation system according to one embodiment of the present invention.

[0032] FIG. 2 is a schematic diagram of a distorted image data generation system according to one embodiment of the present invention.

[0033] FIG. 3 is a drawing illustrating a process for generating a distorted image according to one embodiment of the present invention.

[0034] FIG. 4 is a drawing illustrating a process for generating a distorted image according to another embodiment of the present invention.

[0035] FIG. 5 is a flowchart illustrating a process for generating distorted image data according to one embodiment of the present invention.

[0036] FIG. 6 is a diagram illustrating a process of generating distorted image data through cyclic iteration according to one embodiment of the present invention until the classification error rate of the target object becomes lower than a preset standard.

[0037] When adding reference numbers to components of each drawing in this specification, it should be noted that identical components are given the same numbers as much as possible even if they are shown in different drawings.

[0038] Meanwhile, the meanings of the terms described in this specification should be understood as follows.

[0039] Singular expressions should be understood to include plural expressions unless the context clearly defines otherwise, and the scope of rights should not be limited by these terms.

[0040] The terms "include" or "have" should be understood as not excluding in advance the presence or addition of one or more other features, numbers, steps, operations, components or combinations thereof.

[0041] As used herein, the term "waste" includes plastics, PET bottles, glass bottles, glass, paper, Styrofoam, general waste, and industrial waste.

[0042] As used herein, the term "original image" refers to an image collected rather than generated by a generative model. Specifically, the original image may be an image collected by a sensor or an image input from an external source.

[0043] The target object used in this specification refers to an object to be classified among various types of objects such as waste.

[0044] Hereinafter, preferred embodiments of the present invention designed to solve the above problems will be described in detail with reference to the attached drawings.

[0045] FIG. 1 is a schematic diagram of a waste classification system using an artificial intelligence-based robot arm applied to a distorted image data generation system according to one embodiment of the present invention, FIG. 2 is a schematic diagram of a distorted image data generation system according to one embodiment of the present invention, FIG. 3 is a diagram explaining a process of generating a distorted image according to one embodiment of the present invention, and FIG. 4 is a diagram explaining a process of generating a distorted image according to another embodiment of the present invention.

[0046] Referring to FIG. 1, a waste classification system using an artificial intelligence-based robot arm may include a distorted image data generation system (1000) according to the present invention, a vision camera (2000), and a robot arm (3000).

[0047] Objects such as waste moving on a conveyor belt (4000) can be collected through a vision camera (2000), and the target objects to be classified can be selected through a distorted image data generation system (1000) according to the present invention, and then the target objects can be classified through a robot arm (3000).

[0048] The distorted image data generation system (1000) according to the present invention generates image data used as training data for an artificial intelligence classification model, and can select a target object to be classified from waste through the artificial intelligence model.

[0049] Typically, when waste such as plastic, PET bottles, and glass bottles arrive at a waste sorting facility, they often arrive in distorted forms, such as crushed or broken, rather than in their entirety.

[0050] However, in the case of artificial intelligence classification models, there was a problem that there was a high possibility of misclassification of the target object to be selected because it was trained in the form of a complete image of an object such as waste.

[0051] The distorted image data generation system according to the present invention can reduce the possibility of misclassification of a target object by generating various types of distorted images of the target object to be classified and training the classification model.

[0052] Referring to FIG. 2, a distorted image data generation system (1000) according to an embodiment of the present invention includes a misclassified image determination unit (100), an augmented data generation unit (200), and a classification model training unit (300).

[0053] A misclassified image determination unit (100) according to one embodiment of the present invention can determine a misclassified image of a target object that was not classified by a robot arm (3000) among waste materials (10) collected through a vision camera (2000).

[0054] That is, a misclassified image of a target object refers to an image of a target object that is misclassified because the distorted image data generation system (1000) fails to select the target object as the target object even though it is the target object, or when the distorted image data generation system (1000) normally sends a signal of the target object to the robot arm (3000) but the robot arm (3000) fails to classify the target object.

[0055] The augmented data generation unit (200) can generate various types of distorted images of the original image using the generation model.

[0056] At this time, the original image may be an input image of waste such as plastic, PET bottles, glass bottles, etc., or, depending on the embodiment, may be a misclassified image of a target object that the robot arm could not classify based on an artificial intelligence model.

[0057] The generation model that can be used in the augmented data generation unit (200) may utilize one or more of a variational auto encoder (VAE), a generative adversarial network (GAN), and a diffusion model, but is not limited to the examples described above, and an artificial intelligence model capable of image generation or image transformation may be used.

[0058] A generative adversarial network (GAN) consists of a generator and a discriminator. The generator's role is to take a random vector as input and generate images that are as natural as possible, tailored to the user's needs. The generator can be implemented as a neural network, with the number of output nodes equal to the pixel size of the output image. The discriminator's role is to determine whether a given input image is natural. If the input image is natural, it outputs a value close to 1, and if not, it outputs a value close to 0, so it can be viewed as a regression model. GAN training occurs through adversarial interactions between the generator and the discriminator.

[0059] Various generative models that have been developed and become known since GAN can be used. For example, DCGAN (Deep Convolutional Generative Adversarial Network) that improves image generation performance by introducing a convolutional neural network (CNN) to the initial model of GAN; WGAN (Wasserstein GAN) that increases learning stability by changing the loss function of the original GAN ​​to the Wasserstein distance; CycleGAN that enables style conversion between two domains; Pix2Pix that performs conversion between images; StyleGAN that can control the style of the generated image; BigGAN that generates high-resolution images using a large-scale image dataset; SRGAN (Super-Resolution GAN) that converts low-resolution images into high-resolution images; StarGAN that performs image conversion between multiple domains with a single model; ProGAN (Progressive Growing GAN) that generates images by gradually increasing the resolution; Conditional GAN ​​that receives conditional information as input and generates images with desired characteristics, etc. can be used as the generative models of the present invention.

[0060] Variational autoencoders (VAEs) are used as generative models by introducing probability distributions into the latent space. By constraining latent variables to follow probability distributions, new points can be sampled from the latent space even after training to generate images. A variational autoencoder can be composed of an encoder and a decoder, which are neural networks. The encoder calculates a latent variable z from the input image x. The size of the latent variable can be set to be smaller than the size of the input image and follows a Gaussian probability distribution q(z|x). The decoder can output the input x again from the latent variable z. The difference between the output value and the input value is measured as an error.

[0061] Various generative models developed and known after VAE can be used. For example, Conditional VAE (CVAE) that generates images with desired characteristics using conditional variables; Beta-VAE that promotes separation of latent spaces by adding weights to the Evidence Lower BOund (ELBO) loss function of VAE, thereby increasing independence between latent variables and obtaining a more meaningful latent space; Sequential VAE (SVAE) that learns the temporal structure of the latent space to perform generation and transformation of sequence data; Hierarchical VAE (HVAE) that effectively models complex data structures by hierarchically organizing multiple latent spaces; Adversarial Autoencoder (AAE) that combines VAE and GAN to further refine the latent space and achieve better image generation and transformation performance; Variational U-Net that combines the U-Net structure with VAE to perform image segmentation and image transformation tasks; VAEGAN that combines the advantages of VAE and GAN to simultaneously improve the quality and diversity of image generation; Disentangled VAE that learns the variables of the latent space to have independent characteristics to separate various characteristics of data.

[0062] The diffusion model is a deep generative model that generates data. It utilizes a forward process, which gradually adds noise to data to turn it into complete noise, and a reverse process, which gradually restores the noise to create data. The diffusion model parameterizes the reverse process, trains it as a deep learning model, and generates data from noise through the learned reverse transformation. Therefore, the diffusion model can be thought of as a parameterized Markov chain trained to generate an image after a finite amount of time.

[0063] Referring to FIG. 3, a distorted image data generation system (1000) according to one embodiment of the present invention can generate a distorted image in preset stages according to the degree of distortion by utilizing a generation model.

[0064] For example, a distorted image (11 to 13) according to the degree of distortion can be generated using a generative model from an original image of a target object (10) in its complete form.

[0065] A distorted image is a distorted or broken version of the original image.

[0066] Here, the preset stages are preset stages ranging from a shape similar to the original image to a degree of shape deformation. The more preset stages there are, the lower the classification error rate of the target object, but it may take a lot of time and energy for computing.

[0067] As will be described later, the distorted image data generation system (1000) according to the present invention can generate a distorted image of an appropriate stage by presetting a classification error rate of a target object and repeatedly training a generation model and a classification model until the classification error rate of the target object becomes lower than a preset standard.

[0068] Referring to FIG. 4, a distorted image data generation system (1000) according to another embodiment of the present invention can generate a generated image (11) in a complete form and distorted images (12, 13) in preset steps according to the degree of distortion using a generation model from an original image in a distorted form of a target object (10).

[0069] The classification model training unit (300) can train the classification model by using the original image and the distorted image generated by the generation model together.

[0070] At this time, the classification model can be a deep learning or neural network. The neural network models can include various types of models such as CNN (Convolution Neural Network), R-CNN (Region with Convolution Neural Network), RPN (Region Proposal Network), RNN (Recurrent Neural Network), S-DNN (Stacking-based deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restrcted Boltzman Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, Classification Network, etc., but are not limited thereto.

[0071] Hereinafter, with reference to FIGS. 5 and 6, a process of generating a distorted image through cyclic iteration execution of a distorted image data generating system (1000) according to one embodiment of the present invention will be described in more detail.

[0072] FIG. 5 is a flowchart illustrating a process of generating distorted image data according to one embodiment of the present invention, and FIG. 6 is a diagram illustrating a process of generating distorted image data until the classification error rate of a target object becomes lower than a preset standard through cyclic iteration according to one embodiment of the present invention.

[0073] Referring to FIGS. 5 and 6, a distorted image data generation system (1000) according to one embodiment of the present invention can be iterated through a misclassified image determination step (S100) of collecting misclassified images of target objects that a robot arm has failed to classify, a data augmentation step (S300) of generating various types of distorted images for the misclassified images using a generation model, and a classification model learning step (S500) of learning a classification model using both the misclassified images and the distorted images.

[0074] At this time, whether or not to terminate the cyclic repetition execution (S700) can be determined by repeatedly training the generation model and the classification model until the classification error rate of the target object becomes lower than a preset standard through the classification model, thereby generating a distorted image in stages according to the degree of distortion.

[0075] As an example, referring to FIG. 3, in the first cycle, a misclassified image (10) of a target object that the robot arm failed to classify is collected through a misclassified image determination unit (100), and a first-stage distorted image (11) for the misclassified image can be generated through an augmented data generation unit (200). The misclassified image (10) and the first-stage distorted image (11) can be used together to train a classification model learning unit (300). The target object is classified again using the classification model learned in this way using the vision camera (2000) and the robot arm (3000), and if the classification error rate is higher than a preset classification error rate, the second cycle is executed.

[0076] In the second loop, a second-stage distorted image (12) can be generated through an augmented data generation unit (200) using the misclassified image (10) and the first-stage distorted image (11). The misclassified image (10), the first-stage distorted image (11), and the second-stage distorted image (12) can be used together to train a classification model learning unit (300). Through the classification model learned in this way, the target object is again classified using the vision camera (2000) and the robot arm (3000), and if the classification error rate is higher than the preset classification error rate, the third loop is executed, and if the classification error rate is lower than the preset classification error rate, the loop repetition execution is terminated.

[0077] The present invention not only reduces the possibility of misclassification of a target object by repeatedly training a generation model and a classification model until the classification error rate of the target object becomes below a preset standard, thereby generating a distorted image of an appropriate level, but also eliminates the need for a separate image labeling task for a new target object.

[0078] By selecting a combination with the smallest number of cyclic iterations among various combinations of generation models and classification models through a method for generating distorted image data according to an embodiment of the present invention, an optimal combination of generation models and classification models can be selected.

[0079] The systems or devices described above may be implemented as hardware components, or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0080] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0081] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., singly or in combination. The medium may continuously store a computer-executable program, or may temporarily store it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording media or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0082] It will be apparent to those skilled in the art that the present invention described above is not limited to the above-described embodiments and the attached drawings, and that various substitutions, modifications, and changes are possible within the scope that does not depart from the technical spirit of the present invention.

[0083] The present invention relates to a system for generating image data used as training data for an artificial intelligence model, wherein a generative model is used to generate various types of distorted images for an original image, and the original image and the distorted image are used together to train a classification model to generate distorted image data.

Claims

1. In a distorted image data generation system that generates image data used as training data for an artificial intelligence classification model, An augmented data generation unit that generates various types of distorted images of the original image using a generative model; and A distorted image data generation system including a classification model learning unit that learns a classification model using the original image and the distorted image together.

2. In paragraph 1, A distorted image data generation system characterized in that the above original image includes a misclassified image of a target object that the robot arm failed to classify based on an artificial intelligence model.

3. In paragraph 2, A distorted image data generation system, characterized in that the distorted image includes a distorted or damaged form of the misclassified image.

4. In paragraph 3, The above generative model is a distorted image data generation system that uses at least one of a variational autoencoder (VAE), a generative adversarial network (GAN), and a diffusion model.

5. In paragraph 4, A distorted image data generation system characterized in that the above generation model generates the distorted image in preset stages according to the degree of distortion.

6. In paragraph 5, A distorted image data generation system that repeatedly trains the generation model and the classification model until the classification error rate of the target object becomes below a preset standard through the classification model.

7. In paragraph 6, A distorted image data generation system characterized by selecting a combination with the smallest number of cyclic iterations among various combinations of generation models and classification models.

8. In paragraph 2, A distorted image data generation system further comprising a misclassification image determination unit that collects misclassified images of target objects that the robot arm fails to classify.

Images