How to generate images for learning

Abstract

To embody a discriminating target detection model capable of greatly increasing training images used as training data for the discriminating target detection model, thereby efficiently detecting and accurately discriminating animal species even at night.SOLUTION: A training image generation method according to the present disclosure is a method for generating training images in training a prescribed discriminating target detection model, including: a first step for obtaining a visible light image in daytime of a discriminating target; a second step for determining a type of the discriminating target in the visible light image; a third step for generating, from the obtained visible light image of the discriminating target, a converted infrared image corresponding to the visible light image, based on a result of machine learning using an infrared image in night time against the determined discriminating target as teacher data; and a fourth step for padding a training image using the converted infrared image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a technique for generating learning images for training a predetermined target detection model. [Background technology]

[0002] In recent years, damage to agricultural crops caused by wild birds and animals has become a serious problem, and there is an urgent need to create a detection model for targets (objects, animals, etc.) that can efficiently detect and accurately distinguish animal species in order to capture specific pests. In training such a detection model, the video and images required for model learning are generally collected, and in order to improve accuracy, so-called "padding processing" is performed, that is, processing is performed to add noise to the input data, distort the data (enlarge, reduce, deform, change the angle, etc.), or convert the data to grayscale (black and white).

[0003] Regarding this padding process, for example, Patent Document 1 describes the use of an augmented training dataset when training a deep neural network (DNN) to recognize cats and dogs in images. To create such an augmented training dataset, a style transfer network is first trained to perform style transfer using a photographed image as a style target image and converting any realistic input image into a converted image. The converted image has the content of the input image, maintains the realism quality of the input image, and is in a style that matches the style of the style target image. Then, the trained style transfer network is used to convert the training images of the original training dataset into converted training images, and each converted training image is labeled with the training label of the corresponding training image of the original training dataset, thereby obtaining a desired augmented training dataset. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-32821 A Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, it is said that detection of targets using a target detection model usually requires more than tens of thousands of pieces of training data (input images such as teacher data). However, in reality, the frequency with which pests are photographed by surveillance cameras etc. (i.e., the frequency with which pests appear) is very low. As a result, there is a shortage of useful training data, and even if the conventional padding process is performed, it is currently not possible to sufficiently improve the accuracy of the target detection model.

[0006] In addition, the above-mentioned conventional technology (Patent Document 1) also uses a grayscale image as an example, but when converting an image (infrared image) taken at night with an infrared light and a camera into a grayscale image, an accurate black and white image may not be obtained due to differences in wavelength and reflectance between visible light and infrared light. In addition, infrared images tend to have a lot of unique noise that is not included in daytime images (visible light images). Furthermore, in infrared images, infrared light is reflected, especially by the surface of an animal's eyes, and an image that should originally be a mixture of black and white may appear to be all white.

[0007] Therefore, the present invention has been made to solve the above-mentioned problems, and aims to provide a method for generating learning images that can significantly increase the number of learning images used as learning data in training a target detection model, thereby realizing a target detection model that can efficiently detect animal species even at night and accurately (with high precision) distinguish them. [Means for solving the problem]

[0008] In order to solve the above problems, the present invention employs the following configuration. That is, an example of the present disclosure is a method for generating learning images in training a predetermined judgment target detection model, the learning image generating method including a first step of acquiring a daytime visible light image that includes or may include an image of the judgment target, a second step of determining the type of the judgment target in the visible light image, a third step of generating a converted infrared image corresponding to the visible light image from the acquired visible light image of the judgment target based on a result of machine learning using a nighttime infrared image of the judgment target as training data, and a fourth step of padding the learning image using the converted infrared image.

[0009] In addition, a converted image generation model that generates and outputs a converted infrared image corresponding to the visible light image of each of a plurality of objects to be judged from the visible light image of each object can be created in advance by machine learning using nighttime infrared images of each object to be judged as training data, and this model can be used in the third step. Effect of the Invention

[0010] In the learning image generating method according to the present disclosure, the converted infrared image generated from the visible light image of the object to be determined is used as a learning image for the augmentation process, so that by combining it with other augmentation processes, the number of learning images as learning data can be significantly increased. This can improve the determination accuracy of the object detection model. In addition, since the visible light image of the object to be determined is converted into an infrared image, noise can be suppressed compared to when an actually captured infrared image is converted into a grayscale image, and even if the object to be determined is a pest (animal), reflection of infrared rays by the eyes is prevented, and as a result, an accurate black-and-white image for learning can be obtained. In this respect, too, the determination accuracy of the object detection model can be improved, and it becomes possible to efficiently detect and accurately (precisely) distinguish animal species even at night. [Brief description of the drawings]

[0011] [Figure 1]1 is a plan view illustrating an example of a hardware configuration for executing a learning image generating method according to an embodiment of the present disclosure. FIG. [Diagram 2] 1 is a flow diagram showing an example of a procedure for carrying out a learning image generating method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. The same elements are given the same reference numerals and duplicated explanations will be omitted. Furthermore, the shape and dimensional ratios are merely examples and are not limited to those shown in the drawings.

[0013] 1 is a plan view showing a schematic example of a hardware configuration for executing a learning image generating method according to an embodiment of the present disclosure. A control and arithmetic device 1 used in this embodiment is connected to, for example, an imaging device 2 for capturing a visible light image of a determination target, and / or an external storage device 3 for electromagnetically recording the visible light image captured by the imaging device 2. Here, the control and arithmetic device 1 includes a control and arithmetic section 11, a communication interface (I / F) section 12, a storage section 13, an input section 14, and an output section 15, and each section is connected to each other via a bus line 16 so as to be able to communicate with each other.

[0014] The control and calculation unit 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like, and controls each component and performs various calculations in response to information processing. The communication I / F unit 12 is, for example, a communication module for communicating with other components, i.e., "units" and "devices", via wired or wireless communication. The communication method used by the communication I / F unit 12 for communication is not particularly limited and is arbitrary, and examples of such methods include a LAN (Local Area Network) and a USB (Universal Serial Bus), and an appropriate communication line equivalent to the bus line 16 can also be applied. Both the imaging device 2 and / or the external storage device 3 are provided so as to be able to communicate with the control and calculation unit 11 and the like via the communication I / F unit 12.

[0015] The storage unit 13 is an auxiliary storage device such as a hard disk drive (HDD) or a solid state drive (SSD), and stores various calculation programs executed by the control and calculation unit 11, a control program for controlling communication with the imaging device 2 and / or the external storage device 3, images (including videos) output from the imaging device 2 and / or the external storage device 3, a database including various calculation parameters required for image processing, a database or library of teacher data for machine learning, and various models such as a machine-learned converted image generation model. In this way, the calculation programs, control programs, various models, and the like stored in the storage unit 13 are executed by the control and calculation unit 11 to generate learning images for training a predetermined judgment target detection model, and thus a learned judgment target detection model can be constructed.

[0016] The input unit 14 is an interface device for accepting various input operations from a user who uses the learning image generating method according to the present disclosure, and may be realized, for example, by a mouse, a keyboard, a touch panel, an audio microphone, etc. The output unit 15 is an interface device for notifying the user, etc. of various information by display, voice, printing, etc., and may be realized, for example, by a display, a speaker, a printer, etc.

[0017] Next, Fig. 2 is a flow diagram showing an example of a procedure when implementing a learning image generating method according to an embodiment of the present disclosure. Note that each process described below is merely an example, and each process may be modified as much as possible within the scope of the technical idea of ​​the present disclosure. Also, the process described below may appropriately omit, replace, or add steps.

[0018] First, when the learning and training of the judgment target detection model for vermin (animals) is started (step S101), the control and calculation unit 11 of the control and calculation device 1 acquires (reads) video data as daytime visible light images from, for example, an imaging device 2 installed outdoors in an area where the judgment target may appear and / or an external storage device 3 that stores videos captured in advance by the imaging device 2, and appropriately stores the data in the storage unit 13 (step S102: first step). Note that the type of the imaging device 2 is not particularly limited, and examples include an image sensor camera connected to the Jetson Nano to enable high-speed / high-sensitivity object detection.

[0019] Next, the control and calculation unit 11 converts the video data of the visible light image into still images using an image processing library such as OpenCV, and extracts still images of the visible light image in which the target may be captured as a result of object detection (thinning process: step S103).The control and calculation unit 11 then determines whether the target is present in the extracted still images of the visible light image, and the type of the target, and adds appropriate annotations to the still images (step S104: second step).

[0020] Next, the control and calculation unit 11 outputs the visible light image in which the object to be determined has been determined to a converted image generating step (step S206: third step). In this converted image generating step (step S206), a converted infrared image corresponding to the visible light image of the object to be determined previously is generated from the visible light image of the object to be determined based on the result of machine learning (for example, 10,000 times or more as the number of learning times) using nighttime infrared images of the object to be determined (e.g., images separately obtained for each object to be determined) as teacher data. Note that the machine learning here is not particularly limited, but feature extraction by GANs (Generative Adversarial Networks) or the like can be used.

[0021] Alternatively, prior to the above processes, the control and calculation unit 11 may create in advance, as a converted image generating step (step S206), a converted image generation model that generates a converted infrared image corresponding to a visible light image of the object to be determined from the visible light image of the object to be determined by machine learning using a nighttime infrared image of the object to be determined as training data. Even in this case, it is possible to generate a converted infrared image corresponding to the visible light image of the object to be determined from the previously acquired visible light image of the object to be determined by using such a converted image generation model.

[0022] Furthermore, the control calculation unit 11 outputs the generated converted infrared image and stores it appropriately in the storage unit 13 (step S107). Then, the control calculation unit 11 adds the obtained converted infrared image to a learning database or library as a process of augmenting the learning data, and performs other processes such as adding noise, distorting (enlarging, reducing, deforming, changing the angle, etc.), and grayscaling the converted infrared image obtained in addition to the previous learning, and further adds the augmented image to the learning database or library (step S108: fourth step). Using the learning image obtained in this way, the training of the judgment target detection model is completed, and a learned judgment target detection model is created (step S109).

[0023] According to the learning image generating method according to the present embodiment including the above steps, the control arithmetic device 1 that executes the method, and the learned judgment target detection model trained using the learning image generated thereby, a converted infrared image, which is a nighttime image, is generated from a visible light image of a judgment target such as a vermin (animal) based on machine learning, and the converted infrared image is used as a learning image for the padding process. Therefore, by combining with other padding processes, the number of learning images as learning data can be significantly increased, and as a result, the judgment accuracy of the judgment target detection model can be improved. In addition, since the visible light image, which is a daytime image of the judgment target, is converted into an infrared image, noise is suppressed and infrared reflection by the eyes of the judgment target vermin (animal) is prevented compared to the case where an actually captured infrared image at night is converted into a grayscale image, and as a result, an accurate black-and-white image for learning can be obtained. Therefore, the judgment accuracy of the judgment target detection model can be further improved, and it becomes possible to efficiently detect and accurately (precisely) distinguish animal species even at night.

[0024] The above embodiment has been described in detail as an example of the present disclosure, but the above description merely shows an example of the present disclosure in every respect, and it goes without saying that various improvements and modifications can be made without departing from the scope of the present disclosure. In addition, the above embodiment may be partially replaced or appropriately combined. For example, the conversion image generation model by machine learning may be created in advance by another control device (computer) or an appropriate open system, and instead of machine learning, a model of a deep learning method such as CNN (Convolutional Neural Network) may be used. [Explanation of symbols]

[0025] 1...control and calculation device, 2...imaging device, 3...external storage device, 11...control and calculation section, 12...communication interface (I / F) section, 13...storage section, 14...input section, 15...output section, 16...bus line, S101 to S105, S107 to S109, S206...steps.

Claims

[Claim 1] 1. A method for generating learning images for training a predetermined target detection model, comprising: A first step of acquiring a daytime visible light image of a subject to be determined; a second step of determining a type of the object in the visible light image; a third step of generating a converted infrared image corresponding to the visible light image from the obtained visible light image of the object based on a result of machine learning using a nighttime infrared image of the object as training data; a fourth step of augmenting the learning image using the converted infrared image; A method for generating images for learning, comprising:

Images