Information processing apparatus, information processing method, and recording medium

Abstract

The present technology pertains to an information processing device, an information processing method, and a recording medium capable of easily acquiring an image suitable for an AI case. An information processing apparatus according to the present technology includes a selection unit that selects a training image for training a learning model from among a pre-reserved image group, in accordance with a use case of the learning model that accepts an image as an input. This technology can be applied, for example, to a dataset generation device that generates a dataset comprising a large number of training images.

Description

Technical Field

[0001] The present technology relates to an information processing device, an information processing method, and a recording medium, and more specifically, to an information processing device, an information processing method, and a recording medium that can easily acquire an image suitable for use cases of AI. Background Art

[0002] In recent years, for the purpose of use such as learning of artificial intelligence (AI), it is necessary to prepare a data set including a large number of images. For example, Patent Document 1 describes a data management system that classifies raw data collected from a data source and generates a data set.

[0003] Reference List

[0004] Patent Literature

[0005] Patent document 1: JP 2021-068181A. Summary of the invention

[0006] Technical issues

[0007] The data management system described in Patent Document 1 requires that a large number of images for AI learning be collected by the users themselves through a method for imaging actual scenes, searching for appropriate images from images posted on the Internet, or using a data set posted on a website.

[0008] With these methods, there are cases where effort is expended to collect a large amount of images or the collected images are not suitable for use cases of AI.

[0009] The present technology has been created in view of such circumstances, and is configured to easily acquire images suitable for use cases of AI.

[0010] Technical solution to the problem

[0011] An information processing device according to one aspect of the present technology includes: a selection unit that selects a learning image used for learning of a learning model from a pre-saved image group according to a use case of the learning model using the image as an input.

[0012] An information processing method according to one aspect of the present technology includes: an information processing device selecting a learning image used for learning a learning model from a pre-saved image group according to a use case of the learning model using an image as an input.

[0013] A recording medium according to one aspect of the present technology records a program for executing a process for selecting a learning image used for learning a learning model from a pre-saved image group according to a use case of the learning model using an image as an input.

[0014] In one aspect of the present technology, according to a use case of a learning model that uses an image as input, a learning image for learning the learning model is selected from a pre-saved image group. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 : is a diagram showing a configuration example of an AI learning system according to an embodiment of the present technology.

[0016] Figure 2 The diagram describes the process of generating a data set by a data set generating device.

[0017] Figure 3 : is a diagram showing an example of an input interface for each setting and an example of information input in each setting.

[0018] Figure 4 is used to describe Figure 2 Detailed illustration of the data set generation performed in step S5 in FIG.

[0019] Figure 5 is a diagram showing an example of a table used to select an image suitable for a use case.

[0020] Figure 6 is a diagram describing the process after a data set is generated.

[0021] Figure 7 is a diagram illustrating an example of an output interface displayed on a GUI and an example of displayed information.

[0022] Figure 8 is a diagram showing a first display example of the input GUI.

[0023] Fig. 9 is a diagram illustrating a second display example of the input GUI.

[0024] Fig.10 is a diagram illustrating a third display example of the input GUI.

[0025] Fig.11 is a diagram illustrating a fourth display example of the input GUI.

[0026] Fig.12 is a diagram illustrating a fifth display example of the input GUI.

[0027] Fig.13 is a diagram showing a first display example of an output GUI.

[0028] Fig.14 : is a diagram showing a display example of a learning image list screen.

[0029] Fig.15is a diagram showing a second display example of the output GUI.

[0030] Fig.16 is a diagram illustrating a third display example of the output GUI.

[0031] Fig.17 is a diagram illustrating a fourth display example of the output GUI.

[0032] Fig.18 is a block diagram showing a configuration example of a data set generating device.

[0033] Fig.19 is a diagram illustrating an example of camera simulation.

[0034] Fig. 20 is a diagram showing an example of an image output by an AI engine.

[0035] Fig.21 is a flowchart for describing the processing performed by the data set generating device.

[0036] Fig. 22 is a diagram showing another display example of the input GUI.

[0037] Fig.23 is a block diagram showing a configuration example of hardware of a computer. DETAILED DESCRIPTION

[0038] Embodiments for implementing the present technology will be described below. The description will be made in the following order.

[0039] 1. Overview of AI Learning System

[0040] 2. About GUI

[0041] 3. Configuration and behavior of the dataset generation device

[0042] 4. Modifications

[0043] <1. Overview of AI Learning System>

[0044] Figure 1 : is a diagram showing a configuration example of an AI learning system according to an embodiment of the present technology.

[0045] like Figure 1 As shown, the AI ​​learning system includes a data set generating device 1 and a learning device 2.

[0046] The data set generation device 1 is an information processing device that displays a graphical user interface (GUI) for inputting a use case of AI and generates a data set including a plurality of learning images according to the use case. The learning image is an image used for learning of AI. For example, a data set is generated by selecting an image suitable for the use case as a learning image from an image group pre-stored by the data set generation device 1.

[0047] In the data set generating device 1, images generated using CG, images captured in a real shooting manner, and metadata corresponding to each image are registered in a database. The metadata corresponding to each image includes information indicating the type of object imaged in the image or the type of background, a depth map corresponding to the image, a segmentation result of the image, etc. The images registered in the database may include still images or moving images.

[0048] The data set generating device 1 provides the generated data set to the learning device 2 .

[0049] The learning device 2 performs learning and generates an AI engine including a learning model (AI) using the data set provided from the data set generating device 1. The learning device 2 can relearn the AI ​​using the data set provided from the data set generating device 1.

[0050] Note that the learning device 2 may have a configuration including the data set generating device 1. In this case, when the user uses the GUI to input a use case, the learning device 2 may generate a data set and learn AI.

[0051] Reference Figure 2 The process of generating a data set by the data set generating device 1 is described.

[0052] In step S1 , a user uses a GUI displayed on the dataset generating apparatus 1 to input various settings to generate a dataset.

[0053] In step S2 to step S4 , the dataset generating apparatus 1 receives input of general settings, use cases, and user settings via the GUI.

[0054] In step S5, the data set generating device 1 generates a data set. By generating the data set, images according to the general settings, use cases, and user settings input via the GUI are selected from the image group registered in the database as learning images, and an image data set and a metadata data set are generated. The image data set is a data set including the plurality of learning images, and the metadata data set is a data set including metadata corresponding to each of the plurality of learning images. This will be referred to later. Figure 4 Describe the details of dataset generation.

[0055] In step S6 , the dataset generating device 1 displays a preview of the learning image on the GUI.

[0056] In step S7 , the user views the preview display of the learning image on the GUI, and determines whether the image dataset generated by the dataset generating apparatus 1 is a desired dataset.

[0057] In the case where it is determined in step S7 that the image dataset is not the desired dataset, the process returns to step S1, and the user further inputs or changes settings using the GUI. For example, the user can input an additional image as an image to be added to the image dataset, or input a 3DCG scene.

[0058] In step S8, the data set generation device 1 receives input of an additional image via the GUI. Here, for example, an option indicating whether to replace the additional image with an image in the database is input together with the additional image.

[0059] In step S9 , the data set generating device 1 determines whether to replace the additional image with the image in the database based on the options.

[0060] In the case where it is determined in step S8 to replace the additional image with the image in the database, in the data set generation in step S5, the data set generation device 1 selects an image to be added to the image data set from the image group stored in the database based on the additional image. Specifically, the data set generation device 1 searches for an image (similar image) similar to the additional image from the image group stored in the database, and adds the similar image to the image data set.

[0061] On the other hand, in the event that it is determined in step S8 not to replace the additional image with the image in the database, in step S6 , the data set generating apparatus 1 adds the additional image to the image data set as it is, and displays a preview of the learning image.

[0062] In step S10, the data set generating device 1 receives an input of a 3DCG scene via a GUI. When a 3DCG scene is input, for example, a 3DCG scene file including a 3D model of computer graphics (CG) (CG model) and rendering settings are input to the data set generating device 1. Here, the 3D model of CG indicates a model of a three-dimensional object and a surrounding environment formed in a virtual space.

[0063] In step S11, the dataset generation device 1 generates a rendered image by performing rendering using the 3DCG scene file, and adds the rendered image to the image dataset. Thereafter, in step S6, the dataset generation device 1 displays a preview of the learning image.

[0064] Note that the user can enter general settings, use cases, user settings, additional images, and 3DCG scenes in any order.

[0065] When the user views the preview display of the learning image updated each time each setting is input as described above and determines that the image data set is the desired data set, the user presses the camera simulation execution button on the GUI. Figure 6 Describes the flow after the camera simulation execution button is pressed.

[0066] Figure 3 : is a diagram showing an example of an input interface for each setting and an example of information input in each setting.

[0067] like Figure 3 As shown, general settings are input using an input interface such as a text box, a drop-down menu, or an icon. When inputting general settings, information about the camera used for camera simulation (camera information), the number of learning images to be output, the resolution of the learning images to be output, the format of the images to be output, which of the real shot images and the CG images is desired as the learning image, whether to perform enhancement, etc. are input.

[0068] Use an input interface such as a text box, a drop-down menu, or an icon to input the use case. In the input of the use case, for example, enter the type of the use case such as person recognition or noise reduction.

[0069] The user settings are entered using an input interface such as a text box, a drop-down menu, an icon, or a slider bar. In the user settings input, the conditions that the user desires for the learning image are entered, such as metadata (such as the type of object or background), statistics of the image (such as brightness or frequency).

[0070] The additional images are entered using drag and drop or an input interface such as a text box, drop-down menu, or icon. In the input of the additional images, an input is entered indicating the image to be added to the data set and an option to replace the additional image with a similar image in the database.

[0071] The input of the 3DCG scene is input using drag and drop or an input interface such as a text box, a drop-down menu or an icon. In the input of the 3DCG scene, a 3DCG scene file, settings of a renderer, whether to perform enhancement due to movement of a virtual camera or movement of an object, etc. are input.

[0072] Reference Figure 4 Described in Figure 2 Details of the dataset generation performed in step S5 in FIG.

[0073] In dataset generation, such as Figure 4 As shown, for example, any one of the three processes in step S31 to step S33 is performed according to the type of setting input via the GUI. In each of the three processes in step S31 to step S33, it is assumed that common settings are commonly input.

[0074] In the case where the use case and the general settings are input, in step S31, the data set generation device 1 selects the number of images suitable for the use case input in the general settings as learning images from the image group registered in the database. For example, the data set generation device 1 selects the images suitable for the use case based on a table in which each image registered in the database, a score for the use case, metadata, statistics, etc. are registered. The score for the use case indicates the degree to which each image registered in the database is suitable as a learning image for AI for a specific use case.

[0075] Figure 5 is a diagram showing an example of a table used to select an image suitable for a use case.

[0076] exist Figure 5 In the example in , the ID, image file, score for use case, object, and background (scene) of each image registered in the database are registered in the table.

[0077] In the table, the expected use cases are listed, and the scores for each use case are pre-registered. Figure 4 In the example in , noise reduction (NR), person recognition, object recognition, and depth estimation are cited as use cases. When the score for the use case is higher, the image is more suitable as a learning image for AI for the use case.

[0078] exist Figure 5 , 8 is assigned as the score of NR, 7 as the score of person recognition, 4 as the score of object recognition, and 6 as the score of depth estimation for the image assigned with ID 001. In the table, it is registered that a dog and a person are imaged as objects in the image assigned with ID 001, and it is registered that a room is imaged as a background.

[0079] exist Figure 5 In the table, 5 is assigned as the score of NR, 6 as the score of person recognition, 5 as the score of object recognition, and 7 as the score of depth estimation for the image assigned with ID 002. In the table, it is registered that a person, a car, and a bicycle are imaged as objects in the image assigned with ID 002, and it is registered that a town is imaged as a background.

[0080] exist Figure 5 In the table, 4 is assigned as the score of NR, 6 as the score of person recognition, 1 as the score of object recognition, and 3 as the score of depth estimation for the image assigned with ID 003. In the table, it is registered that a person is imaged as an object in the image assigned with ID 003, and it is registered that a river is imaged as a background.

[0081] exist Figure 5, 3 is assigned as the score of NR, 2 as the score of person recognition, 4 as the score of object recognition, and 5 as the score of depth estimation for the image assigned with ID 004. In the table, it is registered that a car and a sign are imaged as objects in the image assigned with ID 004, and it is registered that a forest is imaged as a background.

[0082] For example, the data set generating apparatus 1 selects as many images as those input in the general setting as learning images from among the images registered in the database in descending order of the scores for the use cases input via the GUI.

[0083] Back to Figure 4 , in the case where the user settings and the general settings are input, in step S32, for example, the data set generating device 1 selects learning images by referring to the metadata registered in the database. Specifically, the data set generating device 1 selects as many images corresponding to the user's expectations input in the user settings as those input in the general settings from the image group registered in the database based on the above-mentioned table as learning images.

[0084] In the case where the additional image and the general settings are input, in step S33, for example, the data set generation device 1 searches for an image similar to the additional image from the image group registered in the database, and adds the searched image to the image data set. For example, in the case where the number of learning images included in the data set exceeds the number input in the general settings, some images originally included in the data set are excluded from the data set by adding images similar to the additional image, so that the number of learning images becomes the same as the number input in the general settings. For example, the images to be excluded from the data set can be determined based on the score of each learning image for the use case, so that the images are excluded from the data set in ascending order of the scores for the use case.

[0085] Next, we will refer to Figure 6 Describe the process after generating the dataset.

[0086] In step S41 , the dataset generating apparatus 1 receives a press of a camera simulation execution button via the GUI.

[0087] When the camera simulation execution button is pressed, the data set generating apparatus 1 executes the processing in step S42 and step S46 surrounded by the dotted line.

[0088] In step S42, the dataset generation device 1 performs camera simulation. In the camera simulation, processing based on camera information for camera simulation is performed on the images, additional images, and rendered images included in the image dataset, and a simulated image dataset is generated.

[0089] The data set generating device 1 generates an image that reproduces an image captured by a camera indicated by the camera information, for example, by processing based on the camera information. The images included in the simulated image data set include images included in the image data set, additional images, and rendered images, which include noise generated on the image by imaging performed by the camera to be reproduced, etc. Note that the camera that is the reproduction target in the camera simulation is set to, for example, a camera that captures an image to be input to the AI ​​generated by the learning device 2.

[0090] In order to accurately reproduce an image captured by a camera to be reproduced, it is desirable that images included in an image data set to be a target of processing, additional images, and rendered images are ideal images. An ideal image is an image that does not include noise or the like.

[0091] In step S43 , the data set generating device 1 stores the simulated image data set.

[0092] In step S44 , the data set generating device 1 performs image analysis on the simulated image data set and obtains statistics of the entire simulated image data set.

[0093] In step S45 , the data set generating device 1 stores the statistics of the simulated image data set.

[0094] In step S46, the dataset generation device 1 performs metadata processing on the additional image and the rendered image. Specifically, the dataset generation device 1 performs object recognition and the like on the additional image and the rendered image, and acquires metadata corresponding to each of the additional image and the rendered image.

[0095] In step S47 , the data set generating apparatus 1 stores the metadata set generated in the data set generation in step S5 and the metadata acquired in step S46 as a single metadata set.

[0096] In step S48, the dataset generating device 1 displays the output dataset on the GUI. The output dataset includes the simulated image dataset, the statistics of the simulated image dataset, and the metadata dataset.

[0097] In step S49 , the user views the display of the output data set on the GUI, and determines whether the output data set is a desired data set.

[0098] If it is determined in step S49 that the output data set is not the expected data set, return Figure 2 In step S1 in , the user further inputs or changes settings using the GUI.

[0099] On the other hand, in the case where it is determined in step S49 that the output dataset is the desired dataset, in step S50, the user operates the learning device 2 to learn AI. To learn AI, the output dataset output from the dataset generating device 1 via the GUI is used.

[0100] Figure 7 is a diagram illustrating an example of an output interface displayed on a GUI and an example of displayed information.

[0101] like Figure 7 As shown, the preview display of the learning image is performed using an output interface such as an image or text. In the preview display of the learning image, a data set including an image selected as the learning image, an estimated time until the camera simulation process ends, etc. are displayed.

[0102] The output data set is displayed using an output interface such as an image, text, or graphic. In the display of the output data set, a data set including images selected as learning images (simulated images), metadata corresponding to each learning image, analysis results of each learning image, statistics of the entire image data set, and information about the input settings, etc. are displayed.

[0103] <2. About GUI>

[0104] Reference Figures 8 to 17 The GUI displayed by the dataset generation device 1 is described. In the dataset generation device 1, an input GUI for a user to input a use case and the like and an output GUI for a user to confirm an output dataset are displayed. For example, the input GUI is displayed before executing a camera simulation, and the output GUI is displayed before outputting an output dataset to the learning device 2 and after executing the camera simulation.

[0105] About input GUI

[0106] Figure 8 is a diagram showing a first display example of the input GUI.

[0107] like Figure 8 As shown, the input GUI includes an input area A1 and a preview area A2. In the input area A1, a screen including input means for inputting various settings is displayed, and in the preview area A2, a preview display of a learning image is performed.

[0108] Five tabs T1 to T5 are displayed on the upper side of the input area A1. When each of the tabs T1 to T5 is selected, a screen for inputting any one of general settings, use cases, user settings, additional images, and 3DCG scenes is displayed in the input area A1. Figure 8, the tag T1 is indicated in white, which indicates that the tag T1 is selected from the tags T1 to T5. In this case, in the input area A1, a general setting input screen is displayed, which is a screen including an input device for inputting general settings.

[0109] In the upper left portion of the general setting input screen, an input box B1 for inputting the number of learning images to be output is displayed. Figure 8 In the example in , 1000 learning images are input and output.

[0110] Below the input box B1, an input box B2 is displayed for inputting information about an image sensor provided in a camera to be reproduced in the camera simulation. For example, the model of the image sensor and the characteristics of the image sensor are input as the information about the image sensor. The data set generating device 1 can simulate noise etc. generated when an image is acquired by the image sensor based on the information about the image sensor. Figure 8 In the example in , the model number "IMX290" was entered.

[0111] On the lower side of the input box B2, an input box B3 for inputting information about a lens provided in a camera to be reproduced in the camera simulation is displayed. For example, the kind (type) of the lens is input as the information about the lens. Figure 8 In the example shown, the type "Wide-angle lens" is entered.

[0112] Below the input box B3, a check box C1 for selecting whether to input detailed settings is displayed. When detailed settings are selected, for example, an input device for inputting data of point spread function (PSF) or distortion measured for a camera to be reproduced is displayed on the general settings input screen.

[0113] Note that information about the image sensor, information about the lens, and detailed settings are included in the camera information used for camera simulation. Information about camera settings or imaging conditions can be input as the camera information.

[0114] On the lower side of the check box C1, an input box B4 for inputting the settings of enhancement is displayed. Input what is changed by enhancement (for example, change the noise amount or brightness) as the settings of enhancement. Figure 8 In the example in , the input creates a dark image and a bright image by changing the brightness of the image. In the case where it is not necessary to perform enhancement, the user can input, for example, a setting of not inputting enhancement or not performing enhancement as a setting.

[0115] Below the input box B4, an input box B5 for inputting the format (data format) of the learning image to be output is displayed. Figure 8 In the example in , the format ".exr" is entered.

[0116] Below the input box B5, an input box B6 for inputting the resolution of the learning image to be output is displayed. Figure 8 In the example in , a learning image with an output width of 4000 pixels and a height of 3000 pixels is input.

[0117] Fig. 9 is a diagram illustrating a second display example of the input GUI.

[0118] exist Fig. 9 , the tag T2 is indicated in white, which indicates that the tag T2 is selected from the tags T1 to T5. In this case, in the input area A1, a use case input screen is displayed, which is a screen including an input device for inputting a use case.

[0119] In the upper left portion of the use case input screen, an input box B11 for inputting a use case is displayed. Fig. 9 In the example above, the use case for AI input is noise reduction.

[0120] On the lower side of the input box B11, a list of expected use cases is displayed as icons and buttons. Fig. 9 In the example in FIG. 1 , an icon I1 indicating noise reduction and a button B12, an icon I2 indicating person recognition and a button B13, and an icon I3 indicating object recognition and a button B14 are displayed. Since noise reduction is entered as a use case in the input box B11, the icon I1 indicating noise reduction and the button B12 are highlighted and displayed as compared to the other icons and buttons. Fig. 9 The center is surrounded and indicated by a thick line.

[0121] The user can input the purpose (use case) of using AI by using the input box B11 or pressing the icon or box to perform input. When the use case is input using the input box B11, the input use case is reflected on the display of the icon and button, and when the use case is input using the icon or button, the input use case is reflected on the display of the input box B11.

[0122] When the common settings and use cases are entered, such as in Fig. 9 As shown on the right side of , in the preview area A2, a preview display is performed for displaying a list of learning images selected based on the common settings and the use case. In the preview display, a thumbnail image indicating each learning image is arranged and displayed. Fig. 9 In the example in , 4×3 (vertical×horizontal) thumbnail images are arranged and displayed in a tile-like shape.

[0123] When the number of selected learning images is greater than 12, the data set generating device 1 switches the thumbnail images displayed in the preview area A2 by receiving a predetermined operation of the user. Fig. 9In the example of the preview area A2 in , white and black circles shown on the lower side of the thumbnail images are displayed as information on the number of selected learning images.

[0124] In the lower left portion of the preview area A2, an input box B21 for presenting an estimated time until the processing of the camera simulation ends is displayed. Fig. 9 In the example in , one hour is displayed as the estimated time until the processing of the camera simulation ends.

[0125] In the lower right portion of the preview area A2, a camera simulation execution button B22 is displayed.

[0126] Note that in the preview area A2, a preview display of a simulated image can be performed. In the preview display of the simulated image, for example, a predetermined image on which a processing based on the input camera information is performed is displayed on the right side of the thumbnail image of the learning image. The predetermined one image may be one of the learning images included in the image data set, or may be one image determined in advance.

[0127] The user can check the preview display of the simulated image and confirm whether the processing performed on the image in the camera simulation is the desired processing.

[0128] Fig.10 is a diagram illustrating a third display example of the input GUI.

[0129] exist Fig.10 , the tag T3 is indicated in white, which indicates that the tag T3 is selected from the tags T1 to T5. In this case, in the input area A1, a user setting input screen is displayed, which is a screen including an input device for inputting user settings.

[0130] In the upper portion of the user setting input screen, an input box B31 for inputting the type of background of the learning image is displayed. Fig.10 In the example in , the input is a learning image in which the output images a town as the background.

[0131] On the lower side of the input box B31, a list of expected backgrounds is displayed as icons and buttons. Fig.10 In the example in FIG. 1 , icons and buttons indicating a town, a room, a forest, and a river are displayed, respectively. Since a town is input as a background in the input box B31, the icon and button indicating the town are highlighted and displayed as being in the middle compared to the other icons and buttons. Fig.10 The center is surrounded and indicated by a thick line.

[0132] The user can input the type of background desired as the background of the learning image by using the input box B31 or pressing an icon or button. In the case of inputting the type of background using the input box B31, the type of background input is reflected on the display of the icon and button, and in the case of inputting the type of background using the icon or button, the type of background input is reflected on the display of the input box B31.

[0133] On the lower side of the button indicating the type of background, an input box B32 for inputting the type of object of the learning image is displayed. Fig.10 In the example in , an input learning image is output that images a person and a bicycle as objects.

[0134] On the lower side of the input box B32, a list of expected objects is displayed as icons and buttons. Fig.10 In the example in FIG. 1 , icons and buttons indicating a person, a motor vehicle, a bicycle, and a dog are displayed. Since a person and a bicycle are input as objects in the input box B32, the icons and buttons indicating the person and the bicycle are highlighted and displayed as compared to the other icons and buttons. Fig.10 The center is surrounded and indicated by a thick line.

[0135] The user can input the type of object desired as the object of the learning image by using the input box B32 or pressing an icon or button. In the case of inputting the type of the object using the input box B32, the type of the input object is reflected on the display of the icon and button, and in the case of inputting the type of the object using the icon or button, the type of the input object is reflected on the display of the input box B32.

[0136] In the lower left portion of the user setting input screen, a slider bar SB1 for inputting the brightness of an image is displayed. The user can adjust the brightness of the learning image by moving the slider on the slider bar SB1. Fig.10 In the example in FIG. 1 , for example, when the slider of the slider bar SB1 is moved to the left by the user, the dataset generating device 1 selects an image darker than the image originally selected as the learning image as the learning image. The dataset generating device 1 can change the brightness of the learning image without changing the learning image according to the user's operation.

[0137] At the bottom center of the user setting input screen, a slider bar SB2 for inputting the frequency (spatial frequency) of the image is displayed. The user can adjust the frequency of the learning image by moving the slider on the slider bar SB2. Fig.10In the example in FIG. 1 , when the slider on the slider bar SB2 is moved to the left by the user, the data set generation device 1 selects, for example, an image having a flatter pattern than the image originally selected as the learning image (an image with little color change, etc.) as the learning image. The data set generation device 1 can change the frequency of the learning image without changing the learning image according to the user's operation.

[0138] In the lower right part of the user setting input screen, a slider bar SB3 for inputting the contrast of the image is displayed. The user can adjust the contrast of the learning image by moving the slider on the slider bar SB3. Fig.10 In the example in FIG. 1 , when the slider on the slider bar SB3 is moved to the left by the user, for example, the dataset generating device 1 selects an image having a lower contrast than an image originally selected as a learning image as a learning image. The dataset generating device 1 can change the contrast of the learning image without changing the learning image according to the user's operation.

[0139] When the common settings, use cases, and user settings are input, in the preview area A2 , a list of learning images selected based on the common settings, use cases, and user settings is displayed.

[0140] Fig.11 is a diagram illustrating a fourth display example of the input GUI.

[0141] exist Fig.11 , the tag T4 is indicated in white, which indicates that the tag T4 is selected from the tags T1 to T5. In this case, in the input area A1, an additional image input screen is displayed, which is a screen including an input device for inputting an additional image.

[0142] In the upper left portion of the additional image input screen, an input box B41 for inputting an additional image is displayed. In the input box B41, for example, a path of the additional image is input. Fig.11 In the example in , the path “C:￥Users￥Pictures￥dog.png” is input. Note that, similar to the images registered in the database, the additional images may include still images or moving images.

[0143] Below the input box B41, a check box C11 is displayed for selecting whether to search the database for similar images to the additional image. When searching for similar images is selected, the dataset generating device 1 searches the image group registered in the database for similar images to the additional image and adds the similar images to the image dataset.

[0144] When an additional image is input, a list of learning images including the additional image or images similar to the additional image is displayed in the preview area A2.

[0145] Fig.12 is a diagram illustrating a fifth display example of the input GUI.

[0146] exist Fig.12 , the tag T5 is indicated in white, which indicates that the tag T5 is selected from the tags T1 to T5. In this case, in the input area A1, a 3DCG scene input screen is displayed, which is a screen including an input device for inputting a 3DCG scene.

[0147] In the upper left part of the 3DCG scene input screen, an input box B51 for inputting a 3DCG scene file is displayed. In the input box B51, for example, the path of the 3DCG scene file is input. Fig.12 In the example in , the path "C:￥Users￥Documents￥animal.max" is entered.

[0148] Below the input box B51, an input box B52 for inputting a renderer for 3DCG scene rendering is displayed. Fig.12 In the example in , the renderer "S-renderer" was entered.

[0149] Below the input box B52, an input box B53 for inputting a virtual camera as a viewpoint of a rendered image among virtual cameras arranged in the virtual space is displayed. Fig.12 In the example in , the input generates a rendered image viewed from the viewpoint of Camera 001.

[0150] On the lower side of the input box B53, an input box B54 for inputting the settings of the enhancement is displayed. What is changed by the enhancement (for example, rotating the virtual camera) is input as the settings of the enhancement. Fig.12 In the example in , the input is to create multiple images by rotating the (virtual) camera when rendering. In the case where it is not necessary to perform enhancement, the user can input a setting such as not inputting enhancement or not performing enhancement.

[0151] When a 3DCG scene is input, a list of learning images including a rendering image generated based on the 3DCG scene file is displayed in the preview area A2. Note that the rendering image may include a still image or a moving image, similar to the images registered in the database.

[0152] About output GUI

[0153] For example, when the camera simulation execution button B22 is pressed on the input GUI and the process of the camera simulation ends, the output GUI is displayed.

[0154] Fig.13 is a diagram showing a first display example of an output GUI.

[0155] like Fig.13 As shown, the output GUI includes an output dataset display area A11. In the output dataset display area A11, the output dataset is displayed.

[0156] On the upper side of the output dataset display area A11, four tabs T11 to T14 are displayed. When each of the tabs T11 to T14 is selected, a screen for confirming any one of a list of simulated learning images, details of simulated learning images, statistics (analysis results) of simulated image datasets, and output settings is displayed in the output dataset display area A11. Fig.13 , the tag T11 is indicated in white, which indicates that the tag T11 is selected from the tags T11 to T14. In this case, in the output data set display area A11, a list of simulated learning images is displayed.

[0157] In the upper portion of the output data set display area A11, a list of simulation learning images is displayed. Specifically, thumbnail images indicating the simulation learning images are arranged and displayed. Fig.13 In the example in , a combination of three thumbnail images arranged in the depth direction is arranged and displayed in the horizontal direction. For example, multiple images, such as images of the same object type or images whose metadata or statistics (brightness, frequency, etc.) are close to each other, are arranged and displayed in the depth direction.

[0158] An input box B61 for inputting the type of metadata or the type of statistics (analysis data) of the learning image that the user wants to confirm is displayed on the lower side of the thumbnail image indicating the learning image. Fig.13 In the example in , the user input wants to confirm the depth map.

[0159] On the lower side of the input box B61, a list of metadata and statistics that can be displayed is displayed as icons and buttons. Fig.13 In the example in FIG. 1 , icons and buttons indicating a depth map and a segmentation result as metadata and frequency, color distribution, and brightness distribution as statistics are displayed. Since a depth map is input in the input box B61, the icon and button indicating the depth map are highlighted and displayed as being in the middle compared to other icons and buttons. Fig.13 The center is surrounded and indicated by a thick line.

[0160] The user can input the type of metadata or the type of statistics to be confirmed by using the input box B61 or pressing an icon or button. In the case of inputting the type of metadata or statistics using the input box B61, the type of metadata or statistics input is reflected on the display of the icon or button. In the case of inputting the type of metadata or statistics using the icon and button, the type of metadata or statistics input is reflected on the display of the input box B61.

[0161] On the lower side of the button indicating the type of metadata and statistics, a list of metadata and statistics of the type input using the input box B61 or the like is displayed. Specifically, images indicating metadata and statistics of the type input using the input box B61 or the like are arranged and displayed. The position of each of the images indicating metadata and statistics corresponds to the position of the simulated learning image displayed in the upper part of the output dataset display area A11. For example, an image indicating metadata corresponding to the learning image displayed on the first front side from the left in the upper part of the output dataset display area A11 is displayed on the first front side from the left in the lower part of the output dataset display area A11.

[0162] For example, when the user presses a thumbnail image displayed in the upper portion of the output data set display area A11, a pop-up window appears. Fig.14 The learning image list screen A12 shown in FIG. In the learning image list screen A12, a list of simulated learning images is displayed. Specifically, thumbnail images indicating simulated learning images are arranged and displayed in a tile-like shape. Fig.14 In the example in , 4×4 (vertical×horizontal) thumbnail images are arranged and displayed.

[0163] When the number of simulated learning images is greater than 16, the data set generating device 1 switches the thumbnail images displayed in the learning image list screen A12 by receiving a predetermined operation by the user. Fig.14 In the example of the learning image list screen A12 in , information on the number of simulation learning images is displayed as white and black circles indicated on the lower sides of the thumbnail images.

[0164] Fig.15 is a diagram showing a second display example of the output GUI.

[0165] exist Fig.15 , the tag T12 is indicated in white, which indicates that the tag T12 is selected from the tags T11 to T14. In this case, in the output data set display area A11, the details of the simulated learning image are displayed.

[0166] An input box B71 for inputting the type of metadata or statistic that the user wants to confirm is displayed at the upper left of the output data set display area A11. Fig.15 In the example in FIG. 4 , the input user wants to confirm the depth map, segmentation, frequency, color distribution, and brightness distribution.

[0167] On the right side of the input box B71, a list of metadata and statistics that can be displayed is displayed as icons and buttons. Fig.15 In the example in FIG. 1 , icons and buttons indicating depth map, segmentation, frequency, color distribution, and brightness distribution are displayed. Since the depth map, segmentation, frequency, color distribution, and brightness distribution are input using the input box B71, the icons and buttons indicating the depth map, segmentation, frequency, color distribution, and brightness distribution are highlighted and displayed in FIG. Fig.15 The center is surrounded and indicated by a thick line.

[0168] The user can input the type of metadata or the type of statistics to be confirmed by using the input box B71 or pressing an icon or button. In the case of inputting the type of metadata or statistics using the input box B71, the type of metadata or statistics input is reflected on the display of the icon or button. In the case of inputting the type of metadata or statistics using the icon and button, the type of metadata or statistics input is reflected on the display of the input box B71.

[0169] On the lower side of the input box B71, an image in which metadata indicating the type input using the input box B71 and the like is registered in association with the learning image and a table of graphs indicating statistics are displayed. Fig.15 In the example of the table in , the ID of the learning image, the thumbnail image of the learning image, the depth map, the image indicating the segmentation result, the graph indicating the frequency, the graph indicating the color distribution, and the histogram of the brightness are displayed as a list. Note that the ID of the learning image is not the ID assigned to each image in the database, but an ID newly assigned to the image selected as the learning image.

[0170] Note that in the table, learning images can be classified or searched based on IDs, etc.

[0171] Fig.16 is a diagram illustrating a third display example of the output GUI.

[0172] exist Fig.16 , the tag T13 is indicated in white, which indicates that the tag T13 is selected from the tags T11 to T14. In this case, in the output dataset display area A11, the statistics (analysis data) of the entire simulation image dataset are displayed.

[0173] In the upper left portion of the output dataset display area A11, an input box B81 is displayed for inputting the type of statistic of the entire image dataset that the user wants to confirm. Fig.16 In the example, the input user wants to confirm the color distribution and brightness distribution.

[0174] On the lower left side of the input box B81, a list of statistics that can be displayed is displayed as icons and buttons. Fig.16 In the example in FIG. 1 , icons and buttons indicating frequency, color distribution, and brightness distribution are displayed. Since color distribution and brightness distribution are input in input box B81, icons and buttons indicating color distribution and brightness distribution are highlighted and displayed in comparison with other icons and buttons. Fig.16 The center is surrounded and indicated by a thick line.

[0175] The user can input the type of statistic to be confirmed by using the input box B81 or pressing an icon or button to perform input. In the case where the type of statistic is input using the input box B81, the type of the input statistic is reflected on the display of the icon and button, and in the case where the type of statistic is input using the icon or button, the type of the input statistic is reflected on the display of the input box B81.

[0176] On the lower right side of the input box B81, a graph indicating statistics of the type input using the input box B81 etc. is displayed. Fig.16 In the example in , a graph indicating color distribution of a plurality of learning images included in a simulated image data set and a graph indicating brightness distribution of the plurality of learning images are displayed.

[0177] In the lower left portion of the output data set display area A11, a table indicating the type of object or background (scene) of each learning image is displayed. Fig.18 In the example of the table in , the type of the object of each learning image is indicated by three granularities including a large item, an intermediate item, and a small item. For example, the object of the learning image assigned with an ID of 001 is set to an animal in the large item, a dog in the intermediate item, and a papillon in the small item. The object of the learning image assigned with an ID of 002 is set to a vehicle in the large item and a motor vehicle in the intermediate item.

[0178] Note that in the table, learning images can be classified or searched based on IDs, etc.

[0179] In the lower right portion of the output data set display area A11, a frame B82 is displayed that visually indicates the distribution of the types of objects and backgrounds in the image data set. In the frame B82, for example, the size of the characters indicating the object is changed and displayed according to the number of learning images that image the same object. Fig.18 In the example of the frame B82 in FIG. 8 , when the number of learning images imaging the same object is larger, the size of the character indicating the object is displayed larger.

[0180] The user can press any one of the large items, middle items, and small items in the table in the lower left portion of the output dataset display area A11. In the case of pressing the portion of the large item in the table, the dataset generation device 1 performs display in the box B82 according to the number of learning images imaging animals, vehicles, etc., and in the case of pressing the portion of the middle item in the table, the dataset generation device 1 performs display in the box B82 according to the number of learning images imaging dogs, motor vehicles, etc. In this way, the user can specify the granularity of the type of the object displayed in the box B82 by pressing any one of the large items, middle items, and small items in the table.

[0181] By looking at the reference Figures 13 to 16 For each display on the output GUI described above, the user can confirm whether the output data set is the expected data set. In the case where it is determined that the output data set is the expected data set, the user uses the reference Fig.17 Describes the output GUI to enter output settings.

[0182] Fig.17 is a diagram illustrating a fourth display example of the output GUI.

[0183] exist Fig.17 , the tag T14 is indicated in white, which indicates that the tag T14 is selected from the tags T11 to T14. In this case, in the output data set display area A11, an input device for inputting output settings is displayed.

[0184] In the upper left portion of the output data set display area A11, an input box B91 is displayed for inputting the type of statistics (analysis data) that the user wants to include in the output data set. Fig.17 In the example in , the input output includes the output datasets of color distribution and brightness distribution.

[0185] On the lower left side of the input box B91, a list of statistics that can be output is displayed as icons and buttons. Fig.17 In the example in FIG. 1 , icons and buttons indicating frequency, color distribution, and brightness distribution are displayed. Since color distribution and brightness distribution are input in input box B91, icons and buttons indicating color distribution and brightness distribution are highlighted and displayed in comparison with other icons and buttons. Fig.17 The center is surrounded and indicated by a thick line.

[0186] The user can input the type of the output statistic by using the input box B91 or pressing an icon or button to perform input. In the case where the type of statistic is input using the input box B91, the type of the input statistic is reflected on the display of the icon and button, and in the case where the type of statistic is input using the icon or button, the type of the input statistic is reflected on the display of the input box B91.

[0187] Note that the output statistics can be either statistics for each learning image or statistics for the entire image dataset.

[0188] Below the button indicating the type of statistic, an input box B92 is displayed for inputting the type of metadata that the user wants to include in the output data set. Fig.17 In the example in , the input and output depth maps are used as metadata sets.

[0189] On the lower left side of the input box B92, a list of metadata that can be output is displayed as icons and buttons. Fig.17 In the example in FIG. 1 , icons and buttons indicating a depth map and a segmentation result are displayed. Since a depth map is input in the input box B92, the icon and button indicating the depth map are highlighted and displayed as being in the middle compared to other icons and buttons. Fig.17 The center is surrounded and indicated by a thick line.

[0190] The user can input the type of metadata to be output by using the input box B92 or pressing an icon or button to perform input. In the case of inputting the type of metadata using the input box B92, the type of metadata input is reflected on the display of the icon and button, and in the case of inputting the type of metadata using the icon or button, the type of metadata input is reflected on the display of the input box B92.

[0191] On the lower side of the button indicating the type of metadata, an input box B93 for inputting the path of the folder to which the output dataset is output is displayed. Fig.17 In the example in , enter the path "C:￥Users￥Documents".

[0192] When using reference Fig.17 After inputting output settings through the output GUI described above, for example, upon receiving a predetermined operation, the dataset generating apparatus 1 outputs an output dataset.

[0193] Note that in the input GUI and output GUI described above, the input box is implemented by a pull-down menu from which a desired menu can be selected, a text box in which text can be input, or a combo box from which a desired menu can be selected or text can be input.

[0194] As described above, the user can acquire a learning image suitable for learning AI for the use case, simply by inputting a use case of AI, etc., using the input GUI and the output GUI displayed by the data set generating device 1. The user can easily acquire a learning image suitable for learning AI with a simple operation without actually capturing an image or searching for an image posted on the Internet to find the image.

[0195] In the data set generating apparatus 1 , when only images that can be used without a license are registered in the database, the user can acquire a large number of learning images without worrying about the license.

[0196] <3. Configuration and behavior of the dataset generation device>

[0197] Configuration of the dataset generation device

[0198] Fig.18 is a block diagram showing a configuration example of the data set generating device 1 .

[0199] like Fig.18 As shown, the dataset generating device 1 includes an input / output I / F 11, an input information acquiring unit 12, a dataset generating unit 13, a dataset database 14, a rendering unit 15, a camera simulation executing unit 16, an image analyzing unit 17, a metadata processing unit 18, an output dataset storage unit 19, a display control unit 20 and a display unit 21.

[0200] The input / output I / F 11 is an interface for inputting data into the data set generating device 1 and outputting data from the data set generating device 1. The data set generating device 1 may include an input I / F and an output I / F, respectively. The input / output I / F 11 detects a user's operation on the input GUI or the output GUI, and provides information indicating the operation content to the input information acquiring unit 12. In addition, the input / output I / F 11 acquires an output data set from the output data set storage unit 19 through an unillustrated route, and outputs the output data set to the learning device 2.

[0201] The input information acquisition unit 12 acquires information about various settings input by the user based on the information provided from the input / output I / F 11. The input information acquisition unit 12 provides information about the general settings, use cases, user settings, and the additional image to the data set generation unit 13. The input information acquisition unit 12 provides information about the 3DCG scene to the rendering unit 15. In the case where a similar image to the additional image is not searched, the input information acquisition unit 12 provides the additional image to the camera simulation execution unit 16 and the metadata processing unit 18.

[0202] The data set generation unit 13 selects a learning image from the image group registered in the data set database 14 based on the information provided from the input information acquisition unit 12, and generates an image data set. The data set generation unit 13 functions as a selection unit that selects a learning image from the image group registered in the data set database 14. In addition, the data set generation unit 13 acquires metadata corresponding to the selected learning image from the data set database 14, and generates a metadata set.

[0203] In the case of searching for a similar image to the additional image, the data set generation unit 13 searches for a similar image to the additional image from the image group registered in the data set database 14 and adds the similar image to the image data set.

[0204] The data set generation unit 13 supplies the generated image data set to the camera simulation execution unit 16 , and supplies the metadata data set to the output data set storage unit 19 .

[0205] In the data set database 14 , images generated using CG, images captured in a real-life shooting manner, and metadata and statistics corresponding to each image are registered in advance.

[0206] The rendering unit 15 performs rendering based on the information on the 3DCG scene supplied from the input information acquisition unit 12 and generates a rendered image. The rendering unit 15 supplies the rendered image to the camera simulation execution unit 16 and the metadata processing unit 18.

[0207] The camera simulation execution unit 16 performs camera simulation on the additional image supplied from the input information acquisition unit 12, each of the learning images included in the image dataset supplied from the dataset generation unit 13, and the rendered image supplied from the rendering unit 15, and generates a simulated image dataset. The camera simulation execution unit 16 functions as a processing unit that performs processing on the additional image, the learning image included in the image dataset, and the rendered image based on the camera information.

[0208] Fig.19 is a diagram illustrating an example of camera simulation.

[0209] As described above, it is expected that the learning images, additional images, and rendered images included in the image dataset are ideal images. Fig.19 As shown, the camera simulation execution unit 16 generates a degraded image by adding degradation and noise generated on an image imaged by a camera to be reproduced to an ideal image.

[0210] Specifically, for example, as indicated by the following formula (1), the camera simulation execution unit 16 generates a degraded image I′ by applying a model that performs convolution of a degradation factor K with respect to an ideal image I and adds noise n.

[0211] [Formula 1]

[0212]

[0213] The AI ​​estimates the degradation factor and noise included in the degraded image by learning using the degraded image and the ideal image as learning data. Fig. 20When a captured image including the same degradation and noise as those included in the degraded image used during learning is input to an AI engine including AI, as indicated by arrow #1, the AI ​​engine outputs a reconstructed image with high image quality that is close to an ideal image, as indicated by arrow #2.

[0214] In this way, it is expected that the degradation and noise included in the degraded image used at the time of learning and the degradation and noise included in the captured image input to the AI ​​engine at the time of inference are the same degradation and noise. The camera simulation execution unit 16 can generate an image data set including a degraded image suitable for learning of AI by generating a degraded image including degradation and noise generated on the image by imaging by the camera to be reproduced, using the captured image captured by the camera to be reproduced as an input.

[0215] Note that the camera simulation performing unit 16 may generate a degraded image by applying a model corresponding to the lens system of a camera to be reproduced and a model corresponding to the sensor system to an ideal image.

[0216] The model corresponding to the lens system may be a model in which degradation such as blur, distortion, shadow, glare, ghosting, etc. caused by lens distortion, transmittance, optical filter, stray light, etc. is added to the ideal image. The model corresponding to the sensor system may be a model in which degradation caused by spectrum, color mixing, photoelectric conversion, etc. in the sensor is added to the ideal image. Furthermore, the model corresponding to the sensor system may be a model in which optical shot noise, dark current shot noise, random shot noise, pattern noise, white point noise, pixel value addition, etc. in the sensor are added to the ideal image.

[0217] The camera simulation performing unit 16 may generate a degraded image by performing application of a compression algorithm, conversion of a compression rate, compression at a variable bit rate, grayscale thinning, etc. In the case where the ideal image includes a moving image, the camera simulation performing unit 16 may generate a degraded image by thinning a frame.

[0218] The camera simulation execution unit 16 can generate a degraded image by applying a model that adds degradation to an ideal image in consideration of defects in the image captured by the sensor. In addition to defects of white, black, or random values, the defect of the pixel may be a defect of at least any one of pixels not used for the image (such as a pixel for image plane phase difference acquisition, a polarization pixel, an IR acquisition pixel, a UV acquisition pixel, a pixel for distance measurement, or a temperature pixel).

[0219] The camera simulation execution unit 16 may generate a degraded image by applying a model that takes into account other characteristics of the sensor. For example, the model may be a model that can acquire a degraded image by taking into account the color filter characteristics, color filter array, temperature characteristics, conversion efficiency, sensitivity (HDR rendering and gain characteristics), read order (rolling shutter distortion), etc. of the sensor.

[0220] The camera simulation performing unit 16 may generate a degraded image by applying a model that may acquire an image taking into account a camera corresponding to a multispectral image or a hyperspectral image.

[0221] The camera simulation execution unit 16 can generate a degraded image by performing a conversion for reproducing an imaging condition. The imaging condition is, for example, a condition such as illumination, saturation, or exposure. The illumination indicates, for example, the type of light source. For example, a conversion for reproducing a light source such as sunlight, tunnel lighting, or streetlights can be performed. In addition, a conversion for reproducing not only the type of light source but also the position of the light source or the direction in which the light source points can be performed. Degradation due to saturation is, for example, overexposure, etc., and indicates degradation of the maximum value of the color of the pixel value exceeding due to reflection from surrounding pixels. Degradation due to exposure is degradation caused under conditions such as shutter speed or aperture, and indicates underexposure, overexposure, etc. A conversion for reproducing the focus of the lens can be performed.

[0222] return Fig.18 , the camera simulation execution unit 16 provides the simulated image data set to the image analysis unit 17 and the output data set storage unit 19.

[0223] The image analysis unit 17 analyzes the image of the learning image included in the simulation image dataset supplied from the camera simulation execution unit 16 and acquires statistics of the entire image dataset. The image analysis unit 17 supplies the statistics of the entire image dataset to the output dataset storage unit 19.

[0224] The metadata processing unit 18 performs metadata processing on the additional image supplied from the input information acquisition unit 12 and the rendered image supplied from the rendering unit 15, and acquires metadata corresponding to the additional image and the rendered image. The metadata processing unit 18 supplies metadata corresponding to each of the additional image and the rendered image to the output dataset storage unit 19.

[0225] The output dataset storage unit 19 stores, as output datasets, the metadata dataset supplied from the dataset generation unit 13, the simulated image dataset supplied from the camera simulation execution unit 16, and the statistics of the simulated image dataset supplied from the image analysis unit 17. The output dataset storage unit 19 adds metadata corresponding to each of the additional image and the rendered image supplied from the metadata processing unit 18 to the metadata dataset, and stores the metadata dataset.

[0226] The display control unit 20 acquires information from each component of the data set generating device 1 through a route not shown, and generates an input GUI and an output GUI and displays the input GUI and the output GUI on the display unit 21 .

[0227] The display unit 21 includes, for example, a display, and displays an input GUI and an output GUI according to control of the display control unit 20. Note that the display unit 21 may be provided in an external device.

[0228] Behavior of the dataset generator

[0229] Here, we will refer to Fig.21 The flowchart in FIG. 1 describes the processing performed by the data set generating apparatus 1 having the above configuration. For example, when the input GUI is displayed on the display unit 21, the start Fig.21 in the processing.

[0230] In step S101 , the input information acquisition unit 12 receives a user's input of general settings.

[0231] In step S102, the input information acquisition unit 12 receives an input of a use case by the user. Note that in the case where the user does not assume a use case of AI generated by learning using the output data set, the processing in step S102 is skipped.

[0232] In step S103, the input information acquisition unit 12 receives the user's input of the user setting. Note that in the case where the user does not want to perform detailed settings, the processing in step S103 is skipped.

[0233] In step S104, the input information acquisition unit 12 receives the user's input for an additional image. Note that in the case where there is no image that the user wants to add to the image data set, the processing in step S104 is skipped.

[0234] In step S105, the input information acquisition unit 12 receives the user's input for the additional image. Note that in the case where the user does not wish to add the rendered image to the image data set, the processing in step S105 is skipped.

[0235] In step S106 , the input information acquisition unit 12 determines whether the camera simulation execution button is pressed.

[0236] In a case where it is determined in step S106 that the camera simulation execution button has not been pressed, the process returns to step S101 , and the subsequent processes are repeatedly executed.

[0237] When various settings are input in the process of step S101 to step S105, an image data set according to the input settings is generated, and a preview of the learning image is displayed on the input GUI. The user views the preview of the learning image and determines whether the image data set is the desired data set. In the case of determining whether the image data set is the desired data set, the camera simulation execution button is pressed by the user. In the case of determining that the camera simulation execution button is pressed in step S106, the process continues to step S107.

[0238] In step S107 , the camera simulation execution unit 16 executes camera simulation and generates a simulation learning data set.

[0239] In step S108 , the input / output I / F 11 outputs an output data set including a simulated learning data set.

[0240] According to the above-described processing, the user can acquire a learning image suitable for learning AI for the use case, etc., by simply inputting a use case of AI using the input GUI and the output GUI displayed by the data set generating device 1. The user can easily acquire a learning image suitable for learning AI with a simple operation without actually capturing an image or searching for an image posted on the Internet to find the image.

[0241] <4. Modification>

[0242] About input GUI

[0243] Fig. 22 is a diagram showing another display example of the input GUI.

[0244] like Fig. 22 As shown, the input GUI may include the input area A1 in addition to the preview area A2. Without displaying the preview area A2 as a part of the input GUI, for example, the camera simulation execution button B22 is displayed in the lower right portion of the input area A1.

[0245] About Computer

[0246] The above-described series of processing can be executed by hardware or software. When the series of processing is executed by software, a program constituting the software is installed from a program recording medium to a computer embedded in dedicated hardware, a general-purpose personal computer, or the like.

[0247] Fig.23 : is a block diagram showing a configuration example of hardware of a computer that executes the above-described series of processes by a program.

[0248] The CPU 501 , the ROM 502 , and the RAM 503 are connected to one another using a bus 504 .

[0249] An input / output interface 505 is further connected to the bus 504. An input unit 506 including a keyboard, a mouse, etc. and an output unit 507 including a display, a speaker, etc. are connected to the input / output interface 505. In addition, a storage unit 508 including a hard disk, a nonvolatile memory, etc., a communication unit 509 including a network interface, etc., and a drive 510 that drives a removable medium 511 are connected to the input / output interface 505.

[0250] In the computer configured as described above, for example, the CPU 501 performs the above-described series of processing by loading a program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing the program.

[0251] The program executed by the CPU 501 is recorded on the removable medium 511 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting and installed in the storage unit 508 , for example.

[0252] The program executed by the computer may be a program that executes a plurality of processing steps in time series in the order described herein, or may be a program that executes a plurality of processing steps in parallel or at necessary timing such as when a call is made.

[0253] Meanwhile, as used herein, a system refers to a collection of multiple components (devices, modules (components), etc.), and all constituent elements may or may not be located in the same housing. Therefore, multiple devices stored in separate housings and connected via a network constitute a system, and one device including multiple modules stored in a single housing is also a system.

[0254] The effects described herein are merely examples and are not intended to be limiting, and other effects may be obtained.

[0255] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications may be made without departing from the gist of the present technology.

[0256] For example, the present technology may be configured as cloud computing in which a plurality of devices share and collaboratively process one function via a network.

[0257] In addition, each step described in the above flowchart may be performed by one device or by a plurality of devices in a shared manner.

[0258] Furthermore, in the case where one step includes a plurality of processes, the plurality of processes included in one step may be executed by one device or may be executed by a plurality of devices in a shared manner.

[0259] Configuration combination example

[0260] The present technology may be configured as follows. (1)

[0262] An information processing device, comprising:

[0263] The selection unit selects a learning image for learning the learning model from a pre-saved image group according to a use case of the learning model using the image as an input. (2)

[0265] The information processing device according to (1) further includes:

[0266] A display control unit displays an input device for a user input use case. (3)

[0268] The information processing device according to (2), wherein:

[0269] The input means for inputting the use case includes any one of a pull-down menu, a text box, a combo box, and an icon. (4)

[0271] The information processing device according to (2) or (3), further comprising:

[0272] A processing unit performs processing on the learning image based on information about a camera that captures the image to be input into the learning model. (5)

[0274] The information processing device according to (4), wherein:

[0275] The modification processing unit performs modification processing by adding at least one of degradation and noise generated in an image imaged by the camera to the learning image. (6)

[0277] The information processing device according to (4) or (5), wherein:

[0278] The display control unit displays a list of images selected as learning images before performing the modification process on the learning images. (7)

[0280] The information processing device according to any one of (4) to (6), wherein:

[0281] The display control unit displays the image on which the processing is performed before the processing is performed on the learning image. (8)

[0283] The information processing device according to any one of (4) to (7), wherein:

[0284] The display control unit displays an input device for inputting information about the camera. (9)

[0286] The information processing device according to (8), wherein:

[0287] The information about the camera includes information about at least one of an image sensor and a lens provided in the camera. (10)

[0289] The information processing device according to (9), wherein:

[0290] The input means for inputting information about the camera includes an input means for inputting at least one of a model or characteristics of an image sensor and a type of a lens. (11)

[0292] The information processing device according to any one of (1) to (10), wherein:

[0293] The selection unit selects a learning image from the image group according to at least any one of a type of an object, a type of a background, brightness, frequency, and contrast input by a user. (12)

[0295] The information processing device according to any one of (1) to (11), wherein:

[0296] The selection unit adds an image selected from the image group based on an image input by a user or an image input by a user as a learning image. (13)

[0298] The information processing device according to any one of (1) to (12), wherein:

[0299] The selection unit adds an image generated based on the CG model input by the user as a learning image. (14)

[0301] The information processing device according to any one of (1) to (13), wherein:

[0302] The selection unit selects a learning image based on a table in which a degree to which each image included in the image group is suitable for learning a learning model for a predetermined case is registered. (15)

[0304] The information processing device according to any one of (1) to (14), further comprising:

[0305] an output unit that outputs the learning image to a learning device that learns the learning model; and

[0306] The display control unit displays a list of learning images before outputting the learning images. (16)

[0308] The information processing device according to (15), wherein:

[0309] The display control unit displays a list of at least one of metadata and statistics corresponding to the learning images before outputting the learning images. (17)

[0311] The information processing device according to (15) or (16), wherein:

[0312] The display control unit displays at least any one of the following items before outputting the learning image: statistics of a data set including multiple learning images, information indicating the type of object or background of each of the multiple learning images, and information indicating the distribution of the type of object or background in the data set. (18)

[0314] An information processing method performed by an information processing device, the method comprising:

[0315] Depending on the use case of the learning model using images as input, a learning image for learning the learning model is selected from a pre-saved set of images. (19)

[0317] A computer-readable recording medium recording a program for executing a process, the process comprising:

[0318] Depending on the use case of the learning model using images as input, a learning image for learning the learning model is selected from a pre-saved set of images.

[0319] Reference Mark List

[0320] 1 Dataset Generation Device

[0321] 2 Learning Devices

[0322] 11 Input / Output I / F

[0323] 12 Input information acquisition unit

[0324] 13 Dataset Generation Device

[0325] 14 Dataset Database

[0326] 15 Rendering Units

[0327] 16 Camera simulation execution units

[0328] 17 Image Analysis Unit

[0329] 18 Metadata Processing Unit

[0330] 19 Output data set storage unit

[0331] 20 Display control unit

[0332] 21 display units.

Claims

1. An information processing device, comprising: The selection unit selects a learning image for learning the learning model from a pre-saved image group according to a use case of the learning model using the image as an input.

2. The information processing device according to claim 1, further comprising: The display control unit displays an input device for a user to input the use case.

3. The information processing device according to claim 2, wherein: The input means for inputting the use case includes any one of a pull-down menu, a text box, a combo box and an icon.

4. The information processing device according to claim 2, further comprising: A processing unit performs processing on the learning image based on information about a camera that captures the image to be input into the learning model.

5. The information processing device according to claim 4, wherein: The modification processing unit performs the modification processing by adding at least one of degradation and noise generated in an image imaged by the camera to the learning image.

6. The information processing device according to claim 4, wherein: The display control unit displays a list of images selected as the learning images before the modification process is performed on the learning images.

7. The information processing device according to claim 4, wherein: The display control unit displays an image on which the processing process is performed before the processing process is performed on the learning image.

8. The information processing device according to claim 4, wherein: The display control unit displays an input device for inputting the information about the camera.

9. The information processing device according to claim 8, wherein: The information about the camera includes information about at least one of an image sensor and a lens provided in the camera.

10. The information processing device according to claim 9, wherein: The input device for inputting the information about the camera includes an input device for inputting at least one of a model or characteristics of the image sensor and a type of the lens.

11. The information processing device according to claim 1, wherein: The selection unit selects the learning image from the image group according to at least any one of a type of an object, a type of a background, brightness, frequency, and contrast input by a user.

12. The information processing device according to claim 1, wherein: The selection unit adds, as the learning image, an image selected from the image group based on an image input by a user or an image input by the user.

13. The information processing device according to claim 1, wherein: The selection unit adds an image generated based on a CG model input by a user as the learning image.

14. The information processing device according to claim 1, wherein: The selection unit selects the learning image based on a table in which is registered a degree to which each image included in the image group is suitable for learning the learning model for a predetermined use case.

15. The information processing device according to claim 1, further comprising: an output unit, outputting the learning image to a learning device for learning the learning model; as well as A display control unit displays a list of the learning images before outputting the learning images.

16. The information processing device according to claim 15, wherein: The display control unit displays a list of at least one of metadata and statistics corresponding to the learning images before outputting the learning images.

17. The information processing device according to claim 15, wherein: The display control unit displays at least any one of the following items before outputting the learning image: statistics of a data set including multiple learning images, information indicating the type of object or background of each of the multiple learning images, and information indicating the distribution of the type of the object or the background in the data set.

18. An information processing method performed by an information processing device, the method comprising: According to a use case of a learning model using images as input, a learning image for learning the learning model is selected from a pre-saved image group.

19. A computer-readable recording medium recording a program for executing a process, the process comprising: According to a use case of a learning model using images as input, a learning image for learning the learning model is selected from a pre-saved image group.

Images