Information processing device, computer program, and information processing method

Abstract

Provided are an information processing device, a computer program, and an information processing method that can be expected to support machine learning that handles images.　An information processing device according to the present embodiment comprises a processing unit that: uses first training data in which an object image and a first type of feature image of the object image are associated with one another to train, by means of first machine learning, an image generating model that generates an image corresponding to an input image, the first type of feature image serving as an input and the object image serving as a correct answer image; and uses second training data in which the object image and a second type of feature image, different from the first type of feature image, of the object image are associated with one another to train, by means of second machine learning, the image generating model that has been subjected to the first machine learning, the second type of feature image serving as an input and the object image serving as a correct answer image.

Description

Information processing device, computer program, and information processing method

[0001] The present disclosure relates to an information processing device, a computer program, and an information processing method for performing machine learning that handles images.

[0002] Patent Document 1 proposes a data cleansing method that performs data cleansing while verifying the validity of a training image set by dividing the image set, overtraining a convolutional neural network (CNN) using a portion of the divided image set, and deleting and reclassifying images based on the image classification results using the overtrained CNN, while changing the configuration of the training images. It determines whether the image set from which images have been deleted and reclassified satisfies an acceptance criterion, and repeats re-division of the image set until the acceptance criterion is satisfied.

[0003] Japanese Patent Application Laid-Open No. 2021-157497

[0004] In recent years, machine learning models that handle images, known as artificial intelligence (AI), have been widely used, such as detection models that detect specific objects from images and segmentation models that classify objects captured in each pixel of an image. To generate such machine learning models, a large amount of training data is required, for example, which associates input images with correct answer data. However, while it is relatively easy to collect images that can be used as input images, annotation work to create correct answer data for these images requires specialized knowledge from an operator and requires a great deal of time and expense. In particular, because objects such as plants do not have a specific shape and their leaves grow randomly, manual annotation is prone to errors, and noise is often included in data that should be correct.

[0005] The present disclosure has been made in consideration of the above circumstances, and its purpose is to provide an information processing device, a computer program, and an information processing method that are expected to support machine learning that handles images.

[0006] An information processing device according to one embodiment includes a processing unit, and the processing unit uses first learning data in which an object image and a first type of feature image of the object image are associated to train an image generation model that generates an image corresponding to an input image through a first machine learning process in which the first type of feature image is used as an input and the object image is used as a correct image, and the image generation model that has undergone the first machine learning process is trained through a second machine learning process in which second learning data in which an object image and a second type of feature image different from the first type of feature image of the object image are associated to train the image generation model that has undergone the first machine learning process to train the image generation model that has undergone the first machine learning process through a second machine learning process in which the second type of feature image is used as an input and the object image is used as a correct image.

[0007] In one embodiment, it is expected to support machine learning that deals with images.

[0008] FIG. 1 is a schematic diagram for explaining an overview of an information processing system according to the present embodiment. FIG. 2 is a block diagram showing an example of a configuration of an information processing device according to the present embodiment. FIG. 3 is a schematic diagram for explaining an example of a data augmentation technique. FIG. 4 is a schematic diagram for explaining an example of a data augmentation technique. FIG. 5 is a flowchart showing the procedure of a target domain training process performed by the information processing device according to the present embodiment. FIG. 6 is a flowchart showing the procedure of a local feature training process performed by the information processing device according to the present embodiment. FIG. 7 is a flowchart showing the procedure of a training data generation process performed by the information processing device according to the present embodiment. FIG. 8 is a flowchart showing the procedure of a detection model generation process performed by the information processing device according to the present embodiment. FIG. 9 is a schematic diagram for explaining the combined use of prompt input in image generation. FIG. 10 is a schematic diagram for explaining an overview of an information processing system according to embodiment 2. FIG. 11 is a block diagram showing an example of a configuration of an information processing device according to embodiment 2. FIG. 12 is a flowchart showing an example of a keypoint detection process performed by the information processing device according to embodiment 2.

[0009] Specific examples of information processing systems according to embodiments of the present invention will be described below with reference to the drawings. Note that the present invention is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope of the claims.

[0010] [First Embodiment] <System Overview> Fig. 1 is a schematic diagram illustrating an overview of an information processing system according to this embodiment. The information processing system according to this embodiment aims to generate a detection model 102 that detects features of agricultural crops, plants, etc. from images of the crops, plants, etc. captured by, for example, a camera, and ultimately realize automated or semi-automated management of the agricultural crops, etc. using this detection model 102. In order to generate a more accurate detection model 102 through machine learning, the information processing system according to this embodiment generates a large amount of training data in an image generation model 101 for machine learning the detection model 102. To achieve this, the information processing system according to this embodiment performs the following processes (1) to (4).

[0011] (1) Target Domain Training The target domain training process is a process of training the image generation model 101 to learn a first type of feature of an object for which an image is to be generated. In this example, the object is, for example, a crop such as grapes, and target domain training is performed using a large number of images captured by a camera in a grape farm or the like. In this example, the first type of feature is the edge of the image. In the target domain training, the image generation model 101 is trained using training data in which images of the crops obtained by photography are associated with edge images obtained by extracting the edges of the images. The edge images are generated by performing existing image processing for edge extraction on the images of the crops captured by the camera. The image generation model 101 is a training model that accepts at least an input image and generates at least an image in accordance with the input image. In the target domain training, so-called supervised machine learning is performed using training data in which edge images are used as input images and images of the crops are associated as correct images, so that the image generation model 101 generates an image of the crop in accordance with the input edge image.

[0012] (2) Local Feature Training Local feature training is a process of training the image generation model 101 to learn a second type of feature of the object for which an image is to be generated. The object is the same as the object in the target domain training, and in this example, it is a crop such as grapes. In this example, the second type of feature is a keypoint representing the shape of the crop stem, for example, a combination of points indicating the positions of the nodes on the stem and lines connecting the points corresponding to the stem. In local feature training, the image generation model 101 is trained using training data that associates images of the crop obtained by photography with keypoint images that indicate keypoint images of the crop captured in the images. The keypoint images are images obtained by, for example, having a worker familiar with the crop annotate images of the crop photographed by a camera. In local feature training, so-called supervised machine learning is performed using training data that associates the keypoint images as input images with images of the crop as correct images, thereby allowing the image generation model 101 to generate images of the crop in response to the input keypoint images. It is also possible to perform further detailed follow-up learning of local features using learning data in which key point images indicating the size or position of the fruit, etc., are associated as correct images.

[0013] In this embodiment, the edge images included in the training data used in the target domain training are images automatically created by performing image processing on the crop images. Furthermore, the keypoint images included in the training data used in the local feature training are images obtained by an operator annotating the crop images. However, keypoint images may also be generated from crop images using a learning model that detects keypoints from images. In the information processing system according to this embodiment, the number of training data used in the local feature training is smaller than the number of training data used in the target domain training.

[0014] (3) Data Generation In data generation, training data for generating the detection model 102 is generated using the image generation model 101 that has learned the features of the object through the above-mentioned target domain training and local feature training. Keypoint images created in advance using an appropriate method are input to the image generation model 101, and a generated image of the object output by the image generation model 101 is obtained, thereby generating training data in which the image of the object and the keypoint image are associated with each other. The input keypoint image may be, for example, an image created by hand by an operator, or may be an image automatically created based on random numbers, or may be an image of keypoints extracted from an image of a crop other than the target crop, or may be an image created by various other methods.

[0015] (4) Detection Model Learning In detection model learning, a detection model 102 is generated by performing machine learning using learning data generated by a data generation process using the image generation model 101. The detection model 102 is a learning model that receives, for example, an image of a crop as an input and detects keypoints of the crop. In the detection model learning process, so-called supervised machine learning is performed using learning data in which an image of a crop is used as an input image and keypoint images are associated as correct images, thereby performing machine learning to receive an image of an object as an input and detect keypoints of the object. Note that the information output by the detection model 102 may be, for example, a keypoint image, or may be, for example, information on the coordinates of the keypoints.

[0016] In the information processing system according to this embodiment, it is expected that the detection model 102 generated by the processes (1) to (4) above can be used to develop devices for automating work in farms where agricultural crops are grown.

[0017] <Device Configuration> Fig. 2 is a block diagram showing an example configuration of an information processing device according to this embodiment. The information processing device 1 according to this embodiment is configured to include a processing unit 11, a memory unit (storage) 12, a communication unit (transceiver) 13, a display unit 14, and an operation unit 15. Note that in this embodiment, the processing is described as being performed by a single information processing device 1, but the processing may also be performed in a distributed manner by multiple information processing devices. The information processing device 1 can be realized by installing a computer program according to this embodiment in various general-purpose devices, such as a personal computer, a server computer, a smartphone, or a tablet terminal device.

[0018] The processing unit 11 is configured using an arithmetic processing device such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), or a quantum processor, and storage devices such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The processing unit 11 reads and executes a program 12a stored in the storage unit 12, thereby performing various processes such as a process of learning an image generation model 101 and a process of learning a detection model 102.

[0019] The storage unit 12 is configured using a large-capacity storage device such as a hard disk, for example. The storage unit 12 stores various programs executed by the processing unit 11 and various data required for the processing of the processing unit 11. In this embodiment, the storage unit 12 stores the program 12a executed by the processing unit 11, and is also provided with a learning data storage unit 12b that stores learning data for machine learning, and a model information storage unit 12c that stores information about a learning model.

[0020] In this embodiment, the program (computer program, program product) 12a is provided in a form recorded on a recording medium 99 such as a memory card or an optical disc, and the information processing device 1 reads the program 12a from the recording medium 99 and stores it in the storage unit 12. However, the program 12a may also be written to the storage unit 12, for example, during the manufacturing stage of the information processing device 1. Alternatively, the program 12a may be distributed by another remote server device or the like and acquired by the information processing device 1 via communication. For example, the program 12a may be read from the recording medium 99 by a writing device and written to the storage unit 12 of the information processing device 1. The program 12a may be provided in a form distributed via a network or in a form recorded on the recording medium 99.

[0021] The training data storage unit 12b stores training data used in the training process of the image generation model 101 and training data used in the training process of the detection model 102. The training process of the image generation model 101 includes two training processes: the above-mentioned target domain training and local feature training. The training data used in the target domain training is data in which an edge image of the object is associated as an input image with an image of the object as a ground truth image. The training data used in the local feature training is data in which a keypoint image of the object is associated as an input image with an image of the object as a ground truth image. The training data used in the training process of the detection model 102 is data generated using the image generation model 101, and is data in which an image of the object is associated as an input image with keypoints of the object (such as keypoint images or keypoint coordinates) as ground truth values.

[0022] The model information storage unit 12c stores information about learning models such as the image generation model 101 and the detection model 102. The information about the learning models may include, for example, information indicating the structure of the learning models and information about parameters of the learning models determined by machine learning processing.

[0023] The image generation model 101 is a learning model that accepts at least an image as input and generates and outputs an image corresponding to the input image. In this embodiment, the image generation model 101 is, for example, a learning model such as img2img or ControlNet, which adds image input to Stable Diffusion. The image generation model 101 according to this embodiment is a learning model that has undergone target domain training and local feature training based on a pre-trained learning model that accepts text information (so-called prompts) and images as input and outputs an image corresponding to these inputs. However, the image generation model 101 according to this embodiment does not necessarily need to have input and output of text information. When performing target domain training and local feature training in this embodiment, the input text information may be null or may be specific text (e.g., "Please generate an image of agricultural crops in the daytime").

[0024] The detection model 102 is a learning model that receives an image as input and outputs a detection result of keypoints of the target crop depicted in the image. The keypoint detection result may be, for example, a keypoint image, or may be, for example, coordinate information of the keypoints, or may be information in other forms. The detection model 102 may employ, for example, a learning model configured as a CNN (Convolutional Neural Network). Furthermore, as with the image generation model 101, the detection model 102 may employ a pre-trained learning model.

[0025] The communication unit 13 communicates with various devices via a network N including a mobile phone communication network, a wireless local area network (LAN), the Internet, etc. The communication unit 13 transmits data provided by the processing unit 11 to other devices, and provides data received from other devices to the processing unit 11. In this embodiment, the communication unit 13 communicates with a server device via the network N, for example, and can acquire information about a pre-trained image generation model.

[0026] The display unit 14 is configured using a liquid crystal display or the like, and displays various images, characters, etc. based on the processing of the processing unit 11. In the present embodiment, the display unit 14 displays various information such as, for example, an image of the target crop photographed by a camera, an edge image extracted from the image of the crop, a keypoint image annotated to the image of the crop, information about the progress of machine learning, and information about the results of machine learning.

[0027] The operation unit 15 accepts user operations and notifies the processing unit 11 of the accepted operations. For example, the operation unit 15 may be input devices such as a mouse and a keyboard, and these input devices may be configured to be detachable from the information processing device 1. Furthermore, for example, the operation unit 15 may accept user operations via an input device such as a mechanical button or a touch panel provided on the surface of the display unit 14.

[0028] The storage unit 12 may be an external storage device connected to the information processing device 1. The information processing device 1 may be a multi-computer including multiple computers, or may be a virtual machine virtually constructed by software. The information processing device 1 is not limited to the above configuration, and may not include, for example, the display unit 14 and the operation unit 15.

[0029] In the information processing device 1 according to this embodiment, the processing unit 11 reads and executes the program 12a stored in the storage unit 12, whereby a target domain tracking / learning processing unit 11a, a local feature tracking / learning processing unit 11b, a data generation processing unit 11c, a detection model learning processing unit 11d, a display processing unit 11e, and the like are realized as software functional units in the processing unit 11. Note that in the figure, functional units related to machine learning processing of the image generation model 101 and the detection model 102 are shown as functional units of the processing unit 11, and functional units related to other processing are not shown.

[0030] The target domain training processing unit 11a performs target domain training on the image generation model 101. The target domain training processing unit 11a downloads and acquires information on a pre-trained training model, for example, by communicating with a predetermined server device, and uses the acquired training model as the image generation model 101. The target domain training processing unit 11a generates multiple edge images by performing image processing for edge extraction, for example, by the Canny method or the HED (Holistically-Nested Edge Detection) method, on multiple images of target crops captured in advance by a camera, and stores the images of the crops and the edge images extracted from these images as training data in the training data storage unit 12b in association with each other. The target domain training processing unit 11a performs target domain training on the image generation model 101 by performing so-called supervised machine learning using the edge images included in the training data as input images and the images of the crops as correct images. The target domain training processing unit 11a stores information such as parameters of the image generation model 101 obtained by the target domain training in the model information storage unit 12c.

[0031] The local feature tracking learning processing unit 11b performs local feature tracking for the image generation model 101 that has undergone target domain tracking learning by the target domain tracking learning processing unit 11a. The local feature tracking learning processing unit 11b acquires an image of the target crop and a keypoint image, which is an annotation result for the image, and stores the two images in association with each other as learning data in the learning data storage unit 12b. The local feature tracking learning processing unit 11b may generate keypoint images by receiving input of the positions of keypoints in the crop image (annotation work) from a user, or may acquire the results of annotation work that has been performed in advance. The local feature tracking learning processing unit 11b performs so-called supervised machine learning on the image generation model 101 using the keypoint image included in the learning data as an input image and the crop image as a ground truth image, thereby performing local feature tracking. The local feature tracking learning processing unit 11b stores information such as parameters of the image generation model 101 obtained by the local feature tracking learning in the model information storage unit 12c.

[0032] In this embodiment, the number of training data used in the local feature training by the local feature training processing unit 11b is smaller than the number of training data used in the target domain training by the target domain training processing unit 11a. For example, several thousand to several tens of thousands of training data are used in the target domain training, while several tens to several hundreds of training data are used in the local feature training. In the information processing system according to this embodiment, by performing target domain training in advance, it is possible to sufficiently improve the accuracy of image generation by the image generation model 101 even if the number of training data used in the local feature training is reduced.

[0033] The data generation processing unit 11c performs processing to generate training data for machine learning of the detection model 102 using the image generation model 101 for which target domain training has been performed by the target domain training processing unit 11a and local feature training has been performed by the local feature training processing unit 11b. The data generation processing unit 11c acquires keypoint images generated in advance by a user, for example, inputs the acquired keypoint images to the image generation model 101, and acquires generated images output by the image generation model 101. The data generation processing unit 11c associates the keypoint images input to the image generation model 101 with the generated images output by the image generation model 101 to generate training data. The data generation processing unit 11c acquires multiple keypoint images, performs image generation using the image generation model 101 multiple times, generates multiple pieces of training data, and stores the data in the training data storage unit 12b.

[0034] The data generation processing unit 11c may also generate keypoint images to be input to the image generation model 101. The data generation processing unit 11c can generate keypoint images by, for example, receiving drawing input of the keypoint images from a user. Furthermore, for example, the data generation processing unit 11c may generate keypoint images based on some algorithm or random numbers, and in this case, the user may be able to set constraints and the like regarding the generation of the keypoint images in advance.

[0035] Furthermore, for example, the data generation processing unit 11c may generate keypoint images from images of a different type of agricultural crop from the type of agricultural crop for which an image is to be generated using the image generation model 101. For example, when an image of grapes is to be generated using the image generation model 101, keypoint images generated from images of another agricultural crop, such as tomatoes or cucumbers, can be input to the image generation model 101, and an image of grapes can be generated from the keypoint images of tomatoes, cucumbers, etc. The data generation processing unit 11c can generate keypoint images of another agricultural crop, for example, using a detection model 102 that has been machine-learned in advance for another agricultural crop, such as tomatoes or cucumbers.

[0036] In addition, in this embodiment, the data generation processing unit 11c performs so-called augmentation processing to expand the training data. The data expansion processing is a process of increasing the number of training data by performing various image processing operations on the original image, such as rotation, movement, enlargement, reduction, inversion, color change, blurring, or noise addition. The information processing system according to this embodiment can employ two types of data expansion methods. Figures 3 and 4 are schematic diagrams illustrating examples of data expansion methods.

[0037] 3 shows an example of the first data augmentation technique. In the first data augmentation technique, the data generation processing unit 11c performs data augmentation on a keypoint image created in advance by a user or the like, and generates multiple keypoint images from one keypoint image. The data generation processing unit 11c inputs these multiple keypoint images to the image generation model 101, respectively, to obtain multiple object images. The data generation processing unit 11c uses pairs of keypoint images and object images corresponding to input and output to the image generation model 101 as training data, and stores the multiple training data in the training data storage unit 12b.

[0038] 4 shows an example of the second data augmentation technique. In the second data augmentation technique, the data generation processing unit 11c inputs keypoint images created in advance by a user or the like to the image generation model 101 and acquires object images output by the image generation model 101. The data generation processing unit 11c performs data augmentation on these object images to generate multiple object images from one object image. At this time, the data generation processing unit 11c also performs similar data augmentation on the original keypoint images to generate keypoint images corresponding to each object image. The data generation processing unit 11c uses pairs of corresponding keypoint images and object images as training data and stores the multiple training data in the training data storage unit 12b.

[0039] The data generation processing unit 11c may generate even more training data by using the first data extension method and the second data extension method in combination.

[0040] The detection model learning processing unit 11d of the processing unit 11 performs machine learning processing of the detection model 102 using the learning data generated by the data generation processing unit 11c. The detection model learning processing unit 11d acquires information about the detection model 102 in its initial state, created in advance by, for example, a designer, and acquires the learning data stored in the learning data storage unit 12b. The detection model learning processing unit 11d performs so-called supervised machine learning using an image of the target crop included in the learning data as an input image and a keypoint image as a correct answer image, thereby generating a detection model 102 that accepts an image of the target crop as an input and outputs information about keypoints of the crop. The detection model learning processing unit 11d stores information such as parameters of the detection model 102 obtained by machine learning in the model information storage unit 12c.

[0041] The display processing unit 11e performs processing to display various characters and images on the display unit 14. In the present embodiment, the display processing unit 11e displays, for example, images such as object images and keypoint images included in the learning data stored in the learning data storage unit 12b on the display unit 14. Furthermore, for example, the display processing unit 11e displays, on the display unit 14, information such as the progress or processing results of the machine learning processing of the image generation model 101 or the detection model 102. The display processing unit 11e may also display various other information on the display unit 14.

[0042] <Flowchart> Figure 5 is a flowchart showing the steps of the target domain training process performed by the information processing device 1 according to this embodiment. The target domain training processing unit 11a of the processing unit 11 of the information processing device 1 according to this embodiment acquires multiple images of target crops, for example, by reading them from the storage unit 12 (step S1). The target domain training processing unit 11a performs image processing, for example, the Canny method or the HED method, on each image read in step S1 to extract edges (step S2). The target domain training processing unit 11a generates training data that associates the images of the crops acquired in step S1 with edge images obtained by extracting edges from these images in step S2 (step S3). The target domain training processing unit 11a stores the multiple training data generated in step S3 in the training data storage unit 12b (step S4).

[0043] After a sufficient amount of training data is obtained through steps S1 to S4, the target domain training processing unit 11a performs machine learning processing using this training data. The target domain training processing unit 11a acquires information about the image generation model 101, which has been pre-trained, for example, by reading information stored in the storage unit 12 or by communicating with a server device that provides the training model (step S5). The target domain training processing unit 11a reads training data for the target domain training from the training data storage unit 12b (step S6). The target domain training processing unit 11a performs supervised machine learning using the edge image included in the training data read in step S6 as input and the image of the target agricultural product as the correct image (step S7). The target domain training processing unit 11a stores information such as parameters of the image generation model 101 obtained by the machine learning in step S7 in the model information storage unit 12c (step S8), and then terminates the target domain training processing.

[0044] 6 is a flowchart showing the procedure of the local feature training process performed by the information processing device 1 according to this embodiment. The local feature training processing unit 11b of the processing unit 11 of the information processing device 1 according to this embodiment acquires multiple images of target crops, for example, by reading them from the storage unit 12 (step S21). The local feature training processing unit 11b acquires annotation information in which keypoint information is associated with the crop images read in step S21, for example, by reading pre-created information from the storage unit 12 (step S22). Note that the local feature training processing unit 11b may acquire the annotation information by, for example, displaying the crop images acquired in step S21 on the display unit 14 and receiving input of keypoint positions from a user. The local feature training processing unit 11b generates training data in which the crop images acquired in step S21 are associated with the annotation information (i.e., keypoint images) acquired in step S22 (step S23). The local feature tracking learning processing unit 11b stores the plurality of learning data generated in step S23 in the learning data storage unit 12b (step S24).

[0045] After a sufficient amount of training data is obtained through steps S21 to S24, the local feature tracking learning processing unit 11b performs machine learning processing using this training data. The local feature tracking learning processing unit 11b acquires information such as parameters of the image generation model 101, which has been subjected to target domain training by the target domain training processing unit 11a, by reading it from the model information storage unit 12c (step S25). The local feature tracking learning processing unit 11b reads training data for local feature training from the training data storage unit 12b (step S26). The local feature tracking learning processing unit 11b performs supervised machine learning using the keypoint images included in the training data read in step S26 as input and the target crop image as the correct image (step S27). The local feature tracking learning processing unit 11b stores information such as parameters of the image generation model 101 obtained by the machine learning in step S27 in the model information storage unit 12c (step S28), thereby terminating the local feature training processing.

[0046] 7 is a flowchart showing the procedure of the learning data generation process performed by the information processing device 1 according to this embodiment. The data generation processing unit 11c of the processing unit 11 of the information processing device 1 according to this embodiment acquires an input keypoint image by, for example, reading a keypoint image created in advance by a user or the like from the storage unit 12 (step S41). The keypoint image may be, for example, an image created by a user through annotation work from an image of a target crop (e.g., grapes), an image drawn by the user at their own discretion, an image generated by the information processing device 1 or the like based on random numbers, or an image generated by extracting keypoints from an image of a crop (e.g., tomatoes) different from the target crop (e.g., grapes).

[0047] The data generation processing unit 11c performs data expansion to generate multiple keypoint images by performing image processing such as rotation, movement, enlargement, reduction, inversion, color change, blurring, or noise addition on the keypoint image acquired in step S41 (step S42).

[0048] The data generation processing unit 11c inputs the multiple keypoint images generated by data expansion in step S42 to the image generation model 101 that has undergone the above-mentioned target domain training and local feature training (step S43). The data generation processing unit 11c acquires images of the object output by the image generation model 101 in response to the input of the multiple keypoint images (step S44). The data generation processing unit 11c generates training data that associates the keypoint images input to the image generation model 101 with the images of the object acquired from the image generation model 101 (step S45). The data generation processing unit 11c stores the training data generated in step S45 in the training data storage unit 12b (step S46), and ends the training data generation process.

[0049] In this flowchart, the data generation processing unit 11c performs data augmentation of the keypoint image using the first data augmentation method, but the present invention is not limited to this. The data generation processing unit 11c may perform data augmentation of the object image using the second data augmentation method. The data generation processing unit 11c may further increase the amount of training data by using both the first and second data augmentation methods. The data generation processing unit 11c may also generate training data without performing data augmentation.

[0050] 8 is a flowchart showing the procedure of the detection model generation process performed by the information processing device 1 according to this embodiment. The detection model learning processing unit 11d of the processing unit 11 according to this embodiment acquires information about the detection model 102, which has been created in advance by a user or the like and has initial values ​​set for its parameters, by reading it from the storage unit 12 (step S61). However, the detection model 102 acquired in step S61 may be one that has already undergone some kind of learning. Furthermore, in step S61, the detection model learning processing unit 11d may generate the detection model 102 by setting initial values ​​for its parameters, rather than acquiring information about the detection model 102 created in advance.

[0051] The detection model learning processing unit 11d acquires the learning data generated by the above-described learning data generation process and stored in the learning data storage unit 12b (step S62). The detection model learning processing unit 11d performs supervised machine learning using the target image included in the acquired learning data as an input and the keypoint image as a ground truth image (step S63). The detection model learning processing unit 11d stores information such as parameters of the detection model 102 obtained by the machine learning in step S63 in the model information storage unit 12c (step S64), and ends the detection model generation process.

[0052] In the information processing system according to this embodiment, it is expected that the detection model 102 generated by the information processing device 1 will be used to automate, for example, the growth management of agricultural crops. For example, a camera can be installed in a farm or the like where agricultural crops are grown, and images of the crops taken by the camera can be input into the detection model 102 to obtain key point information, and the positions of the nodes and the length of the stems of the crops can be determined based on the key point information. By continuously performing these determinations on the crops and recording the determination results in a database or the like, it is expected that the growth of the crops can be managed.

[0053] <Combined Use of Prompt Input> The image generation model 101 according to this embodiment accepts input of text information (prompt) and images (keypoint images), and generates images (object images) according to these inputs. In the information processing system according to this embodiment, when using the image generation model 101 that has performed target domain training and local feature training to generate training data for machine learning of the detection model 102, it is possible to increase the number of images to be generated, change the type of images, etc., by appropriately changing the text information input to the image generation model 101.

[0054] FIG. 9 is a schematic diagram illustrating the combined use of prompt input in image generation. In this diagram, for example, eight types of text information 1 to 8 are combined with one keypoint image. By inputting a set of keypoint images and text information into the image generation model 101, the image generation model 101 generates an image corresponding to the input. By inputting eight sets of keypoint images and text information into the image generation model 101, the image generation model 101 generates eight types of object images. By inputting sets of text information with different contents with the same keypoint image into the image generation model 101 in this way, multiple target images can be obtained, thereby achieving data expansion of the training data.

[0055] The text information may include various phrases relating to the state of the crop, such as "Please generate an image with many (or few) leaves," "Please generate an image with buds," "Please generate an image with flowers in bloom," "Please generate an image with fruit," or "Please generate an image in a withered state."

[0056] The text information may include various phrases related to the environment surrounding the crops, such as "Please generate an image during the daytime (or nighttime, etc.)" or "Please generate an image in rainy weather," etc. This allows the image generation model 101 to be used to increase the number of nighttime images, for example, when there are many daytime images and few nighttime images of actual photographs of crops.

[0057] The above text information is merely an example, and is not limiting. Any text information may be paired with a keypoint image and input to the image generation model 101 .

[0058] <Other Application Examples> In the present embodiment, an example has been described in which an image of an object is acquired as a crop, and a detection model 102 is generated to detect key points indicating the positions of the crop's branches, nodes, etc. However, the application of the present disclosure is not limited to crops and key points. For example, the present disclosure can be applied to generating a detection model for detecting the skeleton of a person, animal, etc., when the object is a human or animal. For example, the present disclosure can be applied to generating a detection model for detecting the location of the object's main parts when the object is a moving body such as a vehicle. For example, the present disclosure can be applied to generating a detection model for detecting obstacles on a road (such as automobiles, motorcycles, pedestrians, or objects) when the object is a road. For example, the present disclosure can be applied to generating a detection model for detecting obstacles on the object (such as ships or floating objects) when the object is an ocean or river. For example, the present disclosure can be applied to generating a detection model for detecting candidate regions of lesions (such as nodules, masses, infiltrates, or pneumothorax) in an X-ray image of an object. For example, the present disclosure can be applied to generating a detection model for detecting diseased or pest-infested regions of a plant when the object is a plant.

[0059] For example, the present disclosure can be applied to fields such as fruit thinning or leaf thinning of plants. For example, the present disclosure can be applied to the field of appearance inspection of industrial products for foreign matter, scratches, defects, surface imperfections, abrasions, dents, burrs, chips, stains, unevenness, or color unevenness. For example, the present disclosure can be applied to the field of biometric authentication using fingerprints, irises, faces, ears, palm prints, finger veins, knuckles, gait, voiceprints, or signatures. For example, the present disclosure can be applied to the field of autonomous driving of automobiles, such as learning from road videos during abnormal weather conditions.

[0060] <Summary> In the information processing system according to the present embodiment having the above configuration, the information processing device 1 trains an image generation model 101 that generates an image corresponding to an input image by first machine learning (target domain training) using first training data in which an object image and a first type of feature image (e.g., edge image) are associated with each other, with the first type of feature image as an input and the object image as a correct answer image. The information processing device 1 then trains the image generation model 101 that has undergone the first machine learning by second machine learning (local feature training) using second training data in which the object image and a second type of feature image (e.g., keypoint image) different from the first type of feature image are associated with each other, with the second type of feature image as an input and the object image as a correct answer image. This allows the information processing system according to the present embodiment to generate an image generation model 101 that generates an object image corresponding to the second type of feature image, and to generate training data that combines the object image and the second type of feature image using this image generation model 101. Therefore, the information processing system according to this embodiment is expected to provide sufficient training data for machine learning to generate a detection model 102 that detects the second type of feature image from an object image, and to generate a highly accurate detection model 102. The information processing system according to this embodiment is expected to support machine learning that handles images.

[0061] Furthermore, in the information processing system according to this embodiment, the information processing device 1 inputs a plurality of second type feature images (e.g., keypoint images) to the image generation model 101 that has undergone second machine learning (additional learning of local features), acquires object images generated by the image generation model 101, and generates third learning data in which the second type feature images and the object images are associated with each other. As a result, the information processing system according to this embodiment performs machine learning using the generated third learning data, thereby generating a detection model 102 that accepts object images as input and outputs second type feature images.

[0062] Furthermore, in the information processing system according to this embodiment, the information processing device 1 uses the generated third learning data to generate a detection model 102 that accepts an object image as an input and outputs a second type of feature image through third machine learning (detection model learning) in which an object image is input and a second type of feature image is used as a ground truth image. As a result, the information processing system according to this embodiment can use the generated detection model 102 to obtain the second type of feature image from an image of the object photographed by a camera or the like, and is expected to realize various processes such as management or automation based on the second type of feature of the object.

[0063] In the information processing system according to this embodiment, the object image is an image of a crop, the first type of feature image is an edge image, and the second type of feature image is a keypoint image. The edge image, which is the first type of feature image, is an image generated by image processing such as the Canny method or the HED method, and the keypoint image, which is the second type of feature image, is an image generated by annotation work by a user or the like. As a result, the information processing system according to this embodiment first performs a first machine learning process using the first type of feature image that can be automatically generated by image processing of the object image, thereby allowing the image generation model 101 to learn the features of the object in advance, and then performs a second machine learning process using the second type of feature image that requires annotation work, thereby allowing the image generation model 101 to learn more detailed and local features. By first performing the first machine learning process using the first type of feature image to learn the features of the object in advance, it is expected that the number of second type of feature images used in the second machine learning process can be reduced.

[0064] Furthermore, in the information processing system according to this embodiment, the target image is an image of a first agricultural crop (e.g., grapes), and the information processing device 1 acquires a second type of feature image generated based on an image of a second agricultural crop (e.g., tomatoes) different from the first agricultural crop, inputs the acquired second type of feature image to the image generation model 101, and acquires an image of the first agricultural crop output by the image generation model 101. As a result, the information processing system according to this embodiment is expected to generate the second type of feature image by substituting the image of the second agricultural crop when, for example, the amount of images of the first agricultural crop obtained is small, thereby increasing the amount of third learning data to be used in the third machine learning.

[0065] Furthermore, in the information processing system according to this embodiment, the information processing device 1 acquires a second type of feature image and generates a plurality of second type of feature images by performing data expansion based on the acquired second type of feature image. The information processing device 1 inputs each of the generated plurality of second type of feature images to the image generation model 101 and acquires an object image output by the image generation model 101. As a result, the information processing system according to this embodiment is expected to generate a large number of object images from a small number of second type of feature images and increase the amount of third learning data used in the third machine learning.

[0066] Furthermore, in the information processing system according to this embodiment, the information processing device 1 acquires a second type of feature image, inputs the acquired second type of feature image to the image generation model 101, acquires an object image output by the image generation model 101, and generates a plurality of object images by performing data expansion based on the acquired object image. As a result, the information processing system according to this embodiment is expected to generate many object images from a small number of second type of feature images, thereby increasing the amount of third learning data used in the third machine learning.

[0067] Furthermore, in the information processing system according to this embodiment, the first machine learning using the first learning data is performed first to allow the image generation model 101 to learn the characteristics of the object in advance, so that the number of data items in the second learning data used in the second machine learning can be made smaller than the number of data items in the first learning data.

[0068] Furthermore, in the information processing system according to this embodiment, a pre-trained learning model that accepts text information and images as input and outputs text information and images according to the input is used as the image generation model 101, and first and second machine learning are performed. As a result, the information processing system according to this embodiment is expected to obtain the image generation model 101 that generates highly accurate object images through the first and second machine learning. Furthermore, the information processing system according to this embodiment is expected to generate a larger amount of third learning data by appropriately adjusting the text information input to the image generation model 101 (combining so-called prompt engineering).

[0069] In this embodiment, the first type of feature image is an edge image, but this is not limited thereto, and any image may be used as the first type of feature image. Similarly, in this embodiment, the second type of feature image is a keypoint image, but this is not limited thereto, and any image may be used as the second type of feature image. Furthermore, in this embodiment, the first crop to be the target is grapes, but this is not limited thereto, and the first crop may be any crop. Similarly, in this embodiment, the second crop different from the first crop is tomatoes, but this is not limited thereto, and the second crop may be any crop.

[0070] [Embodiment 2] An information processing system according to embodiment 2 is a system that monitors agricultural crops and the like, using the detection model 102 generated by the information processing system according to embodiment 1 described above.

[0071] 10 is a schematic diagram for explaining an overview of an information processing system according to embodiment 2. The information processing system according to embodiment 2 is configured to include one or more cameras 203 installed, for example, in a farm or vegetable garden where agricultural crops are grown, and an information processing device 201 that monitors the agricultural crops based on images captured by the cameras 203. The information processing device 201 and the cameras 203 can send and receive data via wired or wireless communication. The cameras 203 are installed in positions where they can capture images of the agricultural crops, and transmit the captured images of the agricultural crops to the information processing device 201.

[0072] The information processing device 201 according to the second embodiment includes a detection model 102 that the information processing device 1 according to the first embodiment has generated in advance through machine learning. That is, the information processing device 201 stores in advance information such as the structure and internal parameters of the detection model 102, which is a machine-learned learning model. The information processing device 201 inputs a captured image acquired from a camera 203 to the detection model 102, and acquires a keypoint image output by the detection model 102 in response to the input. The information processing device 201 determines the growth level or presence or absence of abnormalities of the crops photographed by the camera 203 based on the acquired keypoint image, and displays the determination result to provide it to the user.

[0073] 11 is a block diagram showing an example of the configuration of an information processing device 201 according to embodiment 2. The information processing device 201 according to embodiment 2 is configured to include a processing unit 211, a memory unit (storage) 212, a communication unit (transceiver) 213, a display unit 214, an operation unit 215, etc. The processing unit 211 is configured using an arithmetic processing device such as a CPU, an MPU, a GPU, or a quantum processor, and storage devices such as a ROM and a RAM. The processing unit 211 reads and executes a program 212a stored in the storage unit 212 to perform various processes such as detection of key points of agricultural products based on captured images acquired from a camera 203.

[0074] The storage unit 212 is configured using a large-capacity storage device such as a hard disk. In this embodiment, the storage unit 212 stores a program 212a executed by the processing unit 211, and is also provided with an image data storage unit 212b that stores image data and a model information storage unit 212c that stores information related to the learning model. In the second embodiment, the program (computer program, program product) 212a may be provided in the form of distribution via a network or recorded on a recording medium 98.

[0075] The image data storage unit 212b stores image data such as images of agricultural products captured by the camera 203 and keypoint images detected based on the captured images. The learning data storage unit 212c stores information about the detection model 102 that has undergone machine learning in advance, such as information indicating the structure of the learning model and information about parameters of the learning model determined by the machine learning process.

[0076] The communication unit 213 communicates with the camera 203 via a wired or wireless network, and acquires images of agricultural products captured by the camera 203. The communication unit 213 may also communicate with the information processing device 1, and acquire information about the machine-learned detection model 102 from the information processing device 1.

[0077] The display unit 214 is configured using a liquid crystal display or the like, and displays information such as, for example, photographed images of agricultural products acquired by the camera 203, keypoint images detected based on the photographed images, and the results of determining the growth degree of the agricultural products based on the keypoint images. The operation unit 215 accepts user operations via input devices such as a mouse, keyboard, or touch panel, and notifies the processing unit 211 of the accepted operations.

[0078] In addition, in the information processing device 201 according to the second embodiment, the processing unit 211 reads and executes the program 212a stored in the memory unit 212, whereby the captured image acquisition unit 211a, the detection processing unit 211b, the display processing unit 211c, etc. are realized in the processing unit 11 as software functional units.

[0079] The captured image acquisition unit 211a communicates with the camera 203 via the communication unit 213 and performs processing to acquire captured images of agricultural products captured by the camera 203. The captured image acquisition unit 211a stores the captured image data acquired from the camera 203 in the image data storage unit 212b.

[0080] The detection processing unit 211b performs processing to detect key points of agricultural products captured in the captured images acquired by the captured image acquisition unit 211a from the camera 203. The detection processing unit 211b configures the detection model 102 based on the information stored in the model information storage unit 212c, inputs the captured images acquired by the captured image acquisition unit 211a to the detection model 102, and acquires key point images output by the detection model 102 in response to this. The detection processing unit 211b may store the acquired key point images in the image data storage unit 212b in association with the original captured images.

[0081] The detection processing unit 211b can also repeatedly detect key points of the crop at a predetermined cycle, such as once an hour or once a day, calculate the size of the crop, such as its length or height, based on the detected key points, and store the change in size at the predetermined cycle as the growth rate. The detection processing unit 211b can also detect whether the crop is bent or curved, for example, based on the detected key points, and determine whether there is an abnormality depending on the degree of the bent or curved state. The detection processing unit 211b may also perform various other determination processes based on the results of detection of key points of the crop.

[0082] The display processing unit 211c displays, for example, the captured image acquired by the captured image acquisition unit 211a from the camera 203, the keypoint image detected by the detection processing unit 211b based on the captured image, or various information such as the growth level of the crop or the presence or absence of abnormalities determined based on the keypoint image, on the display unit 214. The display processing unit 211c may also display various other characters, images, etc. on the display unit 214.

[0083] 12 is a flowchart showing an example of the procedure of keypoint detection processing performed by the information processing device 201 according to Embodiment 2. The captured image acquisition unit 211a of the processing unit 211 of the information processing device 201 according to Embodiment 2 communicates with the camera 203 via the communication unit 213 and acquires captured images of agricultural products captured by the camera 203 (step S81). The captured image acquisition unit 211a stores the captured images acquired in step S81 in the image data storage unit 212b together with information such as the date and time of capture (step S82).

[0084] Next, the detection processing unit 211b of the processing unit 211 inputs the captured image acquired in step S81 to the detection model 102, whose information such as parameters is stored in advance in the model information storage unit 212c (step S83). The detection processing unit 211b acquires a keypoint image output by the detection model 102 in response to the input of the captured image in step S83 (step S84). The detection processing unit 211b stores the keypoint image acquired in step S84 in the image data storage unit 212b in association with the captured image stored in step S82 (step S85).

[0085] The detection processing unit 211b also determines, for example, whether or not there is an abnormality in the crops based on the keypoint image acquired in step S84 (step S86). The display processing unit 211c of the processing unit 211 notifies the user of the determination result of whether or not there is an abnormality in the crops by displaying, on the display unit 214, for example, the photographed image acquired in step S81 and the keypoint image acquired in step S84, together with information on the determination result of whether or not there is an abnormality, and then ends the processing. Note that the information processing device 201 according to the second embodiment can repeatedly perform the processing of this flowchart at a predetermined cycle, for example, once a day.

[0086] In the information processing system according to the second embodiment having the above configuration, the information processing device 201 acquires an image (object image) of a target agricultural product captured by the camera 203, inputs the acquired image to the detection model 102, acquires a keypoint image (a second type of feature image) output by the detection model 102 in response to the acquired image, and displays (outputs) the acquired keypoint image on the display unit 214. The detection model 102 is a learning model obtained by machine learning using learning data (third learning data) that associates an object image generated by the image generation model 101, which generates an image of the object when a keypoint image is input, with the keypoint image, and outputs a keypoint image of the object when the image of the object is input. The image generation model 101 is generated by machine learning using learning data (first learning data) that associates an object image with an edge image (a first type of feature image) and learning data (second learning data) that associates the object image with the keypoint image. As a result, the information processing system according to the second embodiment is expected to use the detection model 102 generated by machine learning using a larger amount of training data (third training data) to accurately detect key points from images of the target agricultural products.

[0087] The other configurations of the information processing system according to the second embodiment are the same as those of the information processing system according to the first embodiment, so the same reference numerals are used for the same parts and detailed description thereof will be omitted.

[0088] The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0089] The matters described in each embodiment can be combined with each other. Furthermore, the independent claims and dependent claims described in the claims can be combined with each other in any and all combinations, regardless of the reference format. Furthermore, the claims use a format in which a claim references two or more other claims (multiple claim format), but this is not limited to this. A multiple claim (multi-multi claim) that references at least one other multiple claim may also be used.

[0090] 1 Information processing device (computer) 11 Processing unit 11a Target domain tracking learning processing unit 11b Local feature tracking learning processing unit 11c Data generation processing unit 11d Detection model learning processing unit 11e Display processing unit 12 Storage unit 12a Program (computer program) 12b Learning data storage unit 12c Model information storage unit 13 Communication unit 14 Display unit 15 Operation unit 98, 99 Recording medium 101 Image generation model 102 Detection model 201 Information processing device (computer) 203 Camera 211 Processing unit 211a Captured image acquisition unit 211b Detection processing unit 211c Display processing unit 212 Storage unit 212a Program (computer program) 212b Image data storage unit 212c Model information storage unit 213 Communication unit 214 Display unit 215 Operation unit

Claims

1. An information processing device comprising: a processing unit, which uses first learning data in which an object image and a first type of feature image of the object image are associated with each other to train an image generation model that generates an image different from the input image in response to the input image, by a first machine learning method in which the first type of feature image is used as an input and the object image is used as a correct image; and which trains the image generation model that has undergone the first machine learning method, by using second learning data in which an object image and a second type of feature image different from the first type of feature image of the object image are associated with each other, by using the second type of feature image as an input and the object image is used as a correct image.

2. The information processing device of claim 1, wherein the processing unit inputs a plurality of second types of feature images to the image generation model that performed the second machine learning, acquires a plurality of object images corresponding to the second types of feature images generated by the image generation model, and generates third learning data that corresponds to the input second types of feature images and the acquired object images.

3. The information processing device according to claim 2, wherein the processing unit uses the generated third learning data to generate a learning model that accepts an object image as input and outputs a second type of feature image by a third machine learning process in which the object image is input and the second type of feature image is a correct answer image.

4. The information processing device according to claim 2, wherein the object image is an image of a first crop, and the processing unit acquires the second type of feature image generated based on an image of a second crop of a different type from the first crop, and inputs the acquired second type of feature image into the image generation model to acquire the object image generated by the image generation model.

5. The information processing device according to claim 2, wherein the processing unit acquires the second type of feature image, performs data expansion based on the acquired second type of feature image to generate a plurality of the second type of feature images, and inputs the generated plurality of the second type of feature images to the image generation model to acquire a plurality of the object images generated by the image generation model.

6. The information processing device according to claim 2, wherein the processing unit acquires the second type of feature image, inputs the acquired second type of feature image into the image generation model to acquire the object image generated by the image generation model, and performs data expansion based on the acquired object image to generate a plurality of the object images.

7. The information processing device described in claim 2, wherein the image generation model is a model that accepts text information and images as input and generates an image based on the input text information and image, and the processing unit acquires the second type of characteristic image, acquires multiple types of text information, inputs the acquired second type of characteristic image and the multiple types of text information to the image generation model, and acquires multiple object images generated by the image generation model.

8. A computer program causing a computer to execute a process of: training an image generation model that generates an image different from an input image in response to an input image, using first training data in which an object image and a first type of feature image of the object image are associated with each other, by a first machine learning method in which the first type of feature image is used as an input and the object image is used as a correct image; and training the image generation model that has undergone the first machine learning, using second training data in which an object image and a second type of feature image different from the first type of feature image of the object image are associated with each other, by a second machine learning method in which the second type of feature image is used as an input and the object image is used as a correct image.

9. An information processing method in which an information processing device uses first learning data in which an object image and a first type of feature image of the object image are associated with each other to train an image generation model that generates an image different from the input image in response to the input image, by a first machine learning method in which the first type of feature image is used as an input and the object image is used as a correct image, and the image generation model that has undergone the first machine learning is trained by using second learning data in which an object image and a second type of feature image different from the first type of feature image of the object image are associated with each other, by a second machine learning method in which the second type of feature image is used as an input and the object image is used as a correct image.

10. The information processing method according to claim 9, wherein the object image is an image of an agricultural crop, the first type of feature image is an edge image of the object image, and the second type of feature image is a key point image of the object image.

11. The information processing method according to claim 10, wherein the first type of characteristic image is an image generated by image processing of the object image, and the second type of characteristic image is an image generated by annotation work on the object image.

12. The information processing method according to claim 9, wherein the number of data in the second learning data is smaller than the number of data in the first learning data.

13. A computer program that causes a computer to execute a process of acquiring a second type of feature image, inputting the acquired second type of feature image into an image generation model that has been trained using first learning data in which an object image and a first type of feature image of the object image correspond to each other, and second learning data in which an object image and a second type of feature image of the object image correspond to each other, and that outputs an object image when the second type of feature image is input, and outputs the object image.

14. An information processing method in which an information processing device acquires a second type of feature image, and is trained using first learning data in which an object image and a first type of feature image of the object image correspond to each other, and second learning data in which an object image and a second type of feature image of the object image correspond to each other, and when the second type of feature image is input, the acquired second type of feature image is input to an image generation model that outputs an object image, and outputs a target image.

15. An information processing device comprising a processing unit, the processing unit acquiring a second type of feature image, and learning from first learning data in which an object image and a first type of feature image of the object image correspond to each other, and second learning data in which an object image and a second type of feature image of the object image correspond to each other, the information processing device inputting the acquired second type of feature image into an image generation model that outputs an object image when the second type of feature image is input, and outputting a target image.

16. A computer program that causes a computer to execute a process of acquiring an object image, learning with third learning data that associates an object image generated by an image generation model that generates an object image when a second type of feature image is input with the second type of feature image, and outputting the second type of feature image by inputting the acquired object image into the learning model that outputs the second type of feature image when an object image is input.

17. The computer program according to claim 16, wherein the image generation model is trained using first learning data in which an object image corresponds to a first type of feature image of the object image, and second learning data in which an object image corresponds to a second type of feature image of the object image.

18. An information processing method in which an information processing device acquires an object image, and is trained using third learning data that associates an object image generated by an image generation model that generates an object image when a second type of feature image is input with the second type of feature image, and outputs the second type of feature image by inputting the acquired object image into the learning model that outputs the second type of feature image when an object image is input.

19. An information processing device comprising a processing unit, the processing unit acquiring an object image, and trained with third learning data that associates an object image generated by an image generation model that generates an object image when a second type of feature image is input with the second type of feature image, and outputs the second type of feature image by inputting the acquired object image into the learning model that outputs the second type of feature image when an object image is input.

20. A computer program that causes a computer to execute the following processes: acquiring an object image; generating a first type of feature image of the object image; storing multiple combinations of the acquired object image and the generated first type of feature image as first learning data; acquiring a second type of feature image of the object image; and storing multiple combinations of the acquired second type of feature image and the object image as second learning data.

21. A computer program as described in claim 20, wherein an image generation model that has been trained using the first learning data and the second learning data and that outputs an object image when a second type of feature image is input thereto stores multiple combinations of the target image obtained by inputting the acquired second type of feature image and the second type of feature image as third learning data.

22. An information processing method in which an information processing device acquires an object image, generates a first type of feature image of the object image, stores multiple combinations of the acquired object image and the generated first type of feature image as first learning data, acquires a second type of feature image of the object image, and stores multiple combinations of the acquired second type of feature image and the object image as second learning data.

23. An information processing device comprising a processing unit, which acquires an object image, generates a first type of feature image of the object image, stores multiple combinations of the acquired object image and the generated first type of feature image as first learning data, acquires a second type of feature image of the object image, and stores multiple combinations of the acquired second type of feature image and the object image as second learning data.

Images