Information processing device and its control method, image generation device, imaging device

Abstract

Create a dataset to be used for training the AI ​​model. [Solution] The information processing device comprises an image acquisition means for acquiring captured image data, an information acquisition means for acquiring date and time information and location information of the captured image data, an event information acquisition means for acquiring event information based on the date and time information and location information, and a creation means for acquiring multiple image data associated with the event information and creating a training dataset that can be used for machine learning of an AI model. The creation means creates a training dataset by adding annotation data containing correct answer information to each of the multiple image data.

Description

[Technical Field]

[0001] The present invention relates to a technique for creating training data used in machine learning. [Background Art]

[0002] In recent years, machine learning technology has been increasingly utilized in various fields. Among these, there is "supervised learning", in which machine learning is performed based on training data including correct answer data to generate an inference model. In order to obtain an inference model with high generalization performance in supervised learning, training data consisting of various inputs determined depending on the task to be solved and annotation data including high-quality correct answer information (GT: Ground Truth) corresponding to the inputs is required.

[0003] Generally, for high-quality training data, a public dataset created for purposes such as competitions is used, or if there is no training data suitable for the purpose, the user needs to create the dataset by themselves. To create a dataset by themselves, it is necessary to perform data grouping and add annotation data according to the user's purpose, which requires specialized knowledge and is a labor-intensive task. In order to reduce the work of data grouping, Patent Document 1 proposes a method of searching for event information from date and time information and position information when image data is captured, adding the event information to the image data, and performing grouping. [Prior Art Documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Unexamined Patent Application Publication No. 2014-11536 [Summary of the Invention] [Problem to be Solved by the Invention]

[0005] However, the technology described in Patent Document 1 only performs grouping of image data, and as such, it is difficult to use it as training data for training machine learning models. For example, training data requires annotation data to be added to the image data. In particular, training data used to perform training that is tailored to the user's purpose requires more detailed annotation data.

[0006] This invention has been made in view of these problems and aims to provide a technique for creating datasets used for training AI models. [Means for solving the problem]

[0007] To solve the above-mentioned problems, the information processing apparatus according to the present invention has the following configuration. That is, the information processing apparatus is Image acquisition means for acquiring captured image data, Information acquisition means for acquiring date and time information and location information of the captured image data, An event information acquisition means that acquires event information based on the aforementioned date and time information and location information, A means for acquiring multiple image data associated with the aforementioned event information and creating a training dataset that can be used for machine learning of an AI model, Equipped with, The creation means creates the training dataset by adding annotation data containing correct answer information to each of the multiple image data. [Effects of the Invention]

[0008] According to the present invention, a technique for creating datasets used for training AI models can be provided. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram shows the hardware configuration of an information processing device. [Figure 2]This block diagram shows the functional configuration of the client terminal. [Figure 3] This is a flowchart for creating a dataset. [Figure 4] This figure shows an example of the GUI screen of a dataset creation application. [Figure 5] This is a block diagram showing the functional configuration of the client terminal (Example 1). [Figure 6] This figure shows an example of the GUI screen of a dataset creation application (Modification 1). [Figure 7] This block diagram shows the functional configuration of the client terminal (Example 2). [Figure 8] This is a flowchart for creating a dataset (Modification 2). [Figure 9] This figure shows an example of a GUI screen for a schedule management application. [Figure 10] Block diagram showing the functional configuration of the client terminal (second embodiment). [Figure 11] Block diagram showing the functional configuration of the client terminal (third embodiment). [Figure 12] This figure shows an example of the GUI screen of a dataset creation application (third embodiment). [Modes for carrying out the invention]

[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] (First Embodiment) As a first embodiment of the information processing apparatus according to the present invention, a client terminal that creates a dataset used for training image-generating AI (AI model) using captured image data will be described below as an example.

[0012] <Device Configuration> FIG. 1 is a diagram showing the hardware configuration of the information processing apparatus. As an example of the hardware configuration, the information processing apparatus 10 includes a CPU 100, a bus 110, an input unit 120, a display unit 130, a communication unit 140, a first memory 150, and a second memory 160. The bus 110 interconnects each unit of the information processing apparatus 10.

[0013] The CPU 100 is a central processing unit that performs calculations and logical determinations for various types of processing. The CPU 100 also executes various programs to perform predetermined processing (for example, the processing of the control unit 200 in FIG. 2). The input unit 120 is constituted by a keyboard, a touch panel, or the like, and receives input from a user. The display unit 130 is constituted by a liquid crystal display or the like, displays various types of data and processing results, and presents information to the user. The communication unit 140 communicates with other devices via a wired connection or a wireless connection. Note that instructions from a user may be received from other devices via the communication unit 140, and processing results may be output to other devices.

[0014] The first memory 150 and the second memory 160 are memories that store a control program for executing processing described later and various types of data. Here, it is assumed that the first memory 150 mainly stores the control program and the second memory 160 mainly stores various types of data, but the present invention is not particularly limited to this configuration. The first memory 150 and the second memory 160 are constituted by ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), or the like, but other storage devices having the same function may also be used.

[0015] Figure 2 is a block diagram showing the functional configuration of a client terminal. The client terminal 20 is a terminal device on which a user views and operates the screen, and can be configured as, for example, a personal computer (PC), a smartphone, or a digital camera. The client terminal 20 includes a camera unit 210, a communication unit 140, a first memory 150, a second memory 160, a control unit 200, a GPS receiver unit 221, and a clock unit 231. However, the client terminal 20 shown in Figure 2 may be configured as a single device or as a system including multiple devices. Furthermore, the client terminal 20 is capable of accessing the internet / intranet via the communication unit 140 and connecting to a given search service, etc.

[0016] The camera unit 210 takes pictures and generates captured image data such as still images and videos. Here, the client terminal 20 may be a camera-equipped terminal with a built-in camera unit 210, or it may receive captured images by connecting to the camera unit 210 via the communication unit 140.

[0017] The GPS receiver 221 receives GPS signals from GPS satellites and obtains geographical location information. The clock unit 231 manages the date and time information (time) on the client terminal 20. Here, the date and time information may be obtained, for example, from NTP (Network Time Protocol) via the communication unit 140, or from the GPS signals received by the GPS receiver 221.

[0018] The control unit 200 includes a location information acquisition unit 220, a date and time information acquisition unit 230, an event information acquisition unit 240, a metadata setting unit 250, a dataset creation unit 260, and a learning unit 270. It is assumed that each function of the control unit 200 is implemented in software by the information processing device 10 executing various programs. However, some or all of the functions may be implemented by hardware such as an ASIC (Application-Specific Integrated Circuit).

[0019] The location information acquisition unit 220 acquires location information at the time the image was captured by the camera unit 210 via the GPS receiver unit 221. The date and time information acquisition unit 230 acquires date and time information at the time the image was captured by the camera unit 210. Here, the acquired location information and date and time information may be stored as data separate from the image data, or they may be stored as metadata attached to the image data. For example, Exif (Exchangeable image file format) can be used as metadata attached to the image data.

[0020] The event information acquisition unit 240 includes an event information search unit 241 and an event information determination unit 242, and acquires event information associated with an image based on location information and date / time information. The event information search unit 241 uses location information and date / time information as keywords to search for relevant event information via the network. The event information determination unit 242 performs a calculation process to determine the most matching information (phrase) from the search results and acquires the information of the obtained event.

[0021] Here, event information includes, but is not limited to, the event name, the names of each session within the event, the start times of each session, the main gender and race of participants, and whether it is held outdoors or indoors.

[0022] The metadata setting unit 250 stores information about an image as metadata, associating it with the image. Metadata includes, but is not limited to, information items such as date and time information, location information, event information, and camera setting information. As described above, metadata may be stored in the second memory 160 as captured image data including the metadata, or it may be stored as separate data that references the image.

[0023] The dataset creation unit 260 includes a grouping unit 261, a scraping unit 262, and an annotation data assignment unit 263, and creates a dataset 265 based on event information. As will be described in detail later, the dataset 265 is created by dividing (grouping) images into multiple groups based on grouping items specified by the user and assigning annotation data to them.

[0024] The event information acquisition unit 240 and the metadata setting unit 250 may be configured to operate simultaneously with image capture, or they may operate automatically when predetermined conditions are met (for example, when the number of captured images or the time exceeds a threshold). They may also be configured to operate when the user explicitly instructs them to create a dataset.

[0025] The learning unit 270 uses the dataset 265 created by the dataset creation unit 260 to perform machine learning and generate a machine learning model. Here, we assume an image generation AI model that generates new images based on input text and images as the machine learning model. However, application to image generation AI is just one example; it can also be applied to models for other tasks such as scene recognition, object detection, image enhancement, and translation.

[0026] <Device Operation> This embodiment describes an example of creating a dataset used for training (additional training) an image generation AI that generates new images based on input prompts (text or images). In particular, it describes a model in which event information related to an image of a car race taken by a user as a spectator is determined, and an image is collected by scraping based on that event information to create a dataset.

[0027] Figure 3 is a flowchart for creating a dataset. Multiple image data captured by the user are stored in the second memory 160, and processing S301 to S307 is performed on these image data to determine event information. Subsequently, a training dataset is created based on the event information of the image data selected by the user.

[0028] In S301, the control unit 200 acquires image data from the second memory 160. Here, it is assumed that image data of a car race ("Monaco Grand Prix") taken by the user as a spectator has been acquired.

[0029] In S302, the location information acquisition unit 220 acquires the location information of the image data acquired in S301 at the time of shooting. Here, it is assumed that the location information acquisition unit 220 acquired "longitude 43.74, latitude 7.43" from the GPS signal received by the GPS receiver unit 221 at the time of image shooting.

[0030] In S303, the date and time information acquisition unit 230 acquires the date and time the image data acquired in S301 was taken. Here, it is assumed that the date and time information acquisition unit 230 acquired "Date and time: May 26th, 1:30 PM," which was the information from the clock unit 231 at the time the image was taken.

[0031] In S304, the event information acquisition unit 240 determines a location information keyword based on the latitude and longitude information acquired as location information. The location information keyword is information (a string) related to location information, such as an address or facility name, derived from the latitude and longitude information. For example, the latitude and longitude information is searched via the network to determine the address or the name of the corresponding facility and acquired as the location information keyword. Here, for example, "longitude 43.739, latitude 7.428" is determined to have "Monaco" as the location information keyword.

[0032] In S305, the event information search unit 241 searches for corresponding event information via the network based on the date and time information obtained in S303 and the location information keyword obtained in S304. Event information refers to information (string) about an event that took place at that time and place. For example, it may be searched using a general search engine on the internet, or it may be searched using a generative AI (large-scale language model). Alternatively, the system may be configured to use the latitude and longitude information obtained in S302 instead of the location information keyword for the search.

[0033] Here, let's assume we obtained the results of a search for "Monaco May 26th 1:30 PM" using a search engine. In this way, event information related to the image data obtained in S301 can be searched using date and time information and location information keywords.

[0034] In S306, the event information determination unit 242 determines, from the search results obtained by the event information search unit 241, to be event information if the occurrence frequency is 1 or higher than a threshold. The items of event information include, for example, the event name, the name of the competition, and the main subject, and multiple items are linked to the image data. Here, "Event name: Monaco Grand Prix", "Competition name: F1", and "Main subject: Automobile" are determined to be event information and linked to the image data.

[0035] In S307, the metadata setting unit 250 attaches event information to the image data. Here, for example, it adds information in the same format as the information listed in Exif, such as "Date and Time: May 26, 1:30 PM," and "Event Name: Monaco Grand Prix," "Competition Name: F1."

[0036] In S308, the metadata setting unit 250 determines whether event information has been associated with all image data to be processed stored in the second memory 160. If it determines that event information has been associated with all image data, it proceeds to the dataset creation process in S309 to S311. If it determines that there is still unprocessed image data remaining, it returns to S301 and performs the same processing on the remaining image data as described above. Note that S308 may also be configured to determine whether a certain number of images have been associated with event information or whether a certain number of images have been associated with the same event information.

[0037] In steps S309 to S311, the dataset creation unit 260 creates a machine learning dataset 265, including training data, based on image data and event information associated with the image. The formatting of the dataset may include, but is not limited to, adding annotation data or processing the data.

[0038] Here, the dataset creation unit 260 consists of a grouping unit 261, a scraping unit 262, and an annotation data assignment unit 263. We will explain using an example where images are grouped based on user-specified grouping items, annotation data is automatically assigned, and the number of images is expanded to create a dataset 265.

[0039] Figure 4 shows an example of the GUI screen of the dataset creation application. The GUI screen is displayed on the client terminal 20 in relation to the processing in S309 to S311.

[0040] In S309, the grouping unit 261 groups images based on the grouping items specified by the user. In Figure 4, setting windows 401 and 402 are select boxes for setting the grouping items. Setting window 401 displays the type of event information, with "Event Name" selected as the type. Setting window 402 displays a list of event names that can be selected, and here the user has selected "Monaco Grand Prix". From these two settings, "Event Name: Monaco Grand Prix" is set as the grouping item, and multiple image data associated with this grouping item are grouped together.

[0041] Display area 403 is an area for displaying multiple grouped images. It displays thumbnails of images associated with the event name "Monaco Grand Prix" and the information attached to those images. This allows users to efficiently review the dataset by removing images associated with different event information and displaying only the images relevant to their purpose.

[0042] In S310, the scraping unit 262 scrapes (extracts) images associated with the same grouping item via the network through the communication unit 140, thereby expanding the number of images included in the dataset. Here, the image scraping may target the internet, or it may target a given image database (DB) that is externally connected to the client terminal 20. Furthermore, the images obtained through scraping are not limited to captured images, but may also include images generated by image generation AI, etc.

[0043] In Figure 4, the image search tab 404 shows the results of scraping images from the internet using "Monaco Grand Prix" as the keyword. The checkbox 405 is checked by default, and when the download button 406 is pressed, the checked images are downloaded. If there are images that do not meet the user's purpose, the user can choose whether or not to download them by unchecking the corresponding boxes. Image generation AI datasets require a large number of images, but this makes it possible to collect a large number of images that match the user's purpose based on event information.

[0044] In S311, the dataset creation unit 260 adds annotation data to the image data. In training an image generation AI that takes descriptive text as input, it is necessary to learn the relationship between images and descriptive text as training data. Therefore, the user is asked to input information associated with the image, such as date and time information, location information, and competition information, into the settings window 407.

[0045] In Figure 4, the settings window 407 is a text box for inputting the format of the text that will become the annotation data for the image generation AI. By combining the information corresponding to the items entered in the text box into a single text, annotation data is added to the image. For example, "Location information, date and time information, event name, competition name, main subject" As the corresponding text, "Monaco, May 26, 1:30 PM, Monaco Grand Prix, F1, automobile" This is added as annotation data.

[0046] In S312, the learning unit 270 uses the dataset 265 created by the dataset creation unit 260 to train (additional training) the image generation AI. In other words, the user can obtain an image generation AI that suits their purpose (generating images in the style of the "Monaco Grand Prix"). The dataset 265 may be used to train the machine learning model, or it may be used as validation data or evaluation data to measure performance.

[0047] As described above, according to the first embodiment, event information is determined based on the information at the time of image data capture, and multiple image data related to the determined event information are included in the dataset. At this time, the information at the time of capture and the event information are added to the image data as annotation data. By training an image generation AI using the dataset created in this way, an image generation AI that suits the user's purpose will be generated. In other words, it becomes possible to create a dataset for obtaining an image generation AI that suits the user's purpose. Furthermore, by performing scraping, it becomes possible to include a large amount of image data necessary for training in the dataset.

[0048] (Variation 1) Modification 1 describes a method for grouping images based on the similarity between event information.

[0049] Figure 5 is a block diagram showing the functional configuration of the client terminal in Modification Example 1. Unless otherwise specified, client terminal 50 has the same configuration as client terminal 20 (Figure 2), and functional units with the same names perform the same operations.

[0050] The similarity calculation unit 580 calculates the similarity between event information for multiple image data stored in the second memory 160. The similarity may be calculated using commonalities in the event information, such as location information or competition names. Alternatively, keywords such as event names may be converted into word vectors, and the similarity between these word vectors may be calculated. Here, an example is provided where an event name is input as a keyword, and a list of image data associated with event names that have a high similarity between word vectors is displayed.

[0051] Figure 6 shows an example of the GUI screen of the dataset creation application in Modification Example 1. Unless otherwise specified, UI components with the same symbols as in Figure 4 have similar functions.

[0052] The settings window 601 is a select box for setting the items used to calculate similarity. The same options as in the grouping item settings window 401 are set here, and in this case, "Event Name" is selected as the item used to calculate similarity.

[0053] Display area 602 is an area that groups and displays images associated with highly similar event information. Here, images associated with "event names" that are highly similar to "event name: Monaco Grand Prix" in the second memory 160 are displayed, along with the information attached to the images and their thumbnails. In Figure 6, the "event names" associated with the images in the second memory 160 are displayed in descending order of similarity.

[0054] When an image associated with a highly similar "event name" matches the user's purpose, the user checks checkbox 603. This explicitly indicates that the corresponding image should be included in the dataset.

[0055] As explained above, according to Modification 1, it becomes possible to create a dataset that includes image data highly similar to the user's purpose.

[0056] (Modification 2) Modification 2 describes how to retrieve event information by referencing the user's schedule information.

[0057] Figure 7 is a block diagram showing the functional configuration of the client terminal in modified example 2. Unless otherwise specified, client terminal 70 has the same configuration as client terminal 20 (Figure 2), and functional units with the same names perform the same operations.

[0058] The schedule management unit 780 manages the user's schedule information by linking event information to location and date / time information. For example, it can be implemented as a schedule management app or a calendar app.

[0059] Figure 8 is a flowchart of the dataset creation process in Modification Example 2. Note that steps S801-S804 and S807-S811 are the same as steps S301-S304 and S307-S311, respectively, and therefore their explanations are omitted. Figure 9 shows an example of the GUI screen of the schedule management application.

[0060] Here, Figure 9 shows the app screen of the schedule management app displayed on the client terminal 70, illustrating the schedule information for the day the user participated in the "Japan Championships" as an example.

[0061] In Figure 9, event boxes 901 to 908 are boxes associated with individual event information. Each event information may be pre-entered by the user, including the event name, date and time, and location information, or it may be automatically reflected from event information acquired via the network.

[0062] Here, event box 901 is associated with "Event Name: Japan Championships," "Location: National Stadium," and "Date and Time: 10:00 AM to 8:00 PM." Event box 902 is associated with "Event Name: Opening Ceremony" as a detailed schedule for the "Japan Championships." Event boxes 903 through 908 are also associated with corresponding detailed schedules.

[0063] Checkbox 909 is a checkbox that indicates whether a user participated in the event. Here, the user can manually enter whether or not they participated, or the checkbox can be automatically entered based on the information acquired by the location information acquisition unit 220 and the date and time information acquisition unit 230.

[0064] In S805, the event information search unit 241 takes date and time information and location information (or location information keyword) as input and searches for schedule information from the schedule management unit 780. For example, if there is an image associated with "Date and Time: 2 PM" and "Location: National Stadium," schedule information such as "Event: Japan Championships" and "Event: 10,000m Race" is obtained from the event box 905. On the other hand, for an image associated with "Date and Time: 3 PM" and "Location: National Stadium," the event box 906 shows that the image does not participate in "Long Jump," so no schedule information is associated with it.

[0065] In S606, the event information search unit 241 searches for event information based on schedule information via the network. For example, by searching for "Japan National Championships 10,000m race 2pm National Stadium", detailed information such as "race order" and "participants" for the relevant event can be obtained.

[0066] As explained above, according to Modification 2, it becomes possible to determine the event information in which the user participated in in more detail and with greater accuracy.

[0067] (Second Embodiment) In the second embodiment, we will describe a method for obtaining a trained model based on event information and creating a dataset using the trained model.

[0068] Figure 10 is a block diagram showing the functional configuration of a client terminal in the second embodiment. Unless otherwise specified, client terminal 100 has the same configuration as client terminal 20 (Figure 2), and functional units with the same names perform the same operations.

[0069] The annotation data assignment unit 263 adds descriptive text to images using the trained model acquired by the model acquisition unit 1080, which will be described later. Specifically, it adds descriptive text to both the images grouped by the grouping unit 261 and the images scraped by the scraping unit 262.

[0070] For example, based on the input image, a model that analyzes the image content (scene recognition model) is used to add detailed annotation data. The scene recognition model outputs descriptions such as object detection, scene category (indoor, outdoor, urban, etc.), and specific activity (driving, collision, etc.). As a scene recognition model, for example, the technology disclosed in the following document A can be used. (Reference A) Jiahui Yu et al., "CoCa: Contrastive Captioners are Image-Text Foundation Models", arXiv:2205.01917, May 2022

[0071] The model acquisition unit 1080 acquires a pre-trained scene recognition model suitable for annotation data via the network based on the grouping items set in the grouping unit 261. Alternatively, the system may be configured to select (acquire) a model based on the matching rate between the grouping items and the tags associated with the pre-trained scene recognition model.

[0072] The acquired, trained scene recognition model is used to analyze the content of the image and obtain the analysis result (description). For example, by inputting date and time information, location information, and analysis results, the image generation AI can learn the user's purpose in detail. Here, for example, in image 408 of Figure 4, "Location information, date and time information, event name, scene analysis results" As the corresponding text, "Monaco, May 26, 1:30 PM, Monaco Grand Prix, a black F1 car in motion with its driver inside." This is added as annotation data.

[0073] As described above, according to the second embodiment, it is possible to add detailed annotation data to image data by utilizing a scene recognition model. In particular, by selecting a scene recognition model based on the set grouping items, it becomes possible to add more accurate annotation data.

[0074] (Variation 3) Modification 3 describes a method for extracting images that are more suitable for the user's purpose from grouped and scraped images using a trained model.

[0075] When grouping or scraping images using date, time, and location information as input, the content of the images may not match the user's purpose (even though the date, time, and location information is linked). For example, images of landscapes or people completely unrelated to the user's purpose, such as "Event Name: Monaco Grand Prix," may be retrieved.

[0076] Therefore, the image content is analyzed using a pre-trained scene recognition model, and the similarity calculation unit 580 is used to calculate the similarity between the event information and the scene analysis results. Here, the similarity between the event name associated with the image and the word vector of the scene analysis results is calculated, and images with a similarity above a threshold are extracted from the dataset. This makes it possible to extract images that are more suitable for the user's purpose with greater accuracy.

[0077] (Third embodiment) In the third embodiment, a method for extracting an image using information from a machine learning model used during image capture will be described. The following description will use an example where image capture is performed using a vehicle detection model that detects the position of a vehicle in the image. However, the vehicle detection model is merely an example and does not limit the present invention.

[0078] Figure 11 is a block diagram showing the functional configuration of a client terminal in the third embodiment. Unless otherwise specified, client terminal 110 has the same configuration as client terminal 20 (Figure 2), and functional units with the same names perform the same operations.

[0079] The model information acquisition unit 1180 acquires model usage information from the camera unit 210 when an image is captured. For example, if a car detection model was used when an image was captured, the model usage information includes the type of detection model, the name of the detection model, and whether or not a subject was detected. Here, we will explain assuming that an image taken at a car race (Monaco Grand Prix) has "Detection Model: Car Detection Model" and "Detection Possible / Not Possible: Yes" assigned to it.

[0080] Figure 12 shows an example of the GUI screen of the dataset creation application in the third embodiment. Unless otherwise specified, UI components with the same reference numerals as in Figure 4 have similar functions.

[0081] In the display area 403, images associated with "Event Name: Monaco Grand Prix" and information attached to the images are displayed as thumbnails. Here, the information attached to the images includes the model information 1204 at the time of shooting. The model information 1204 at the time of shooting includes information acquired by the information acquisition unit 117, such as "Model Name: Automobile" and information on whether an automobile was detected, "Detection possible: ○".

[0082] The model relevance calculation unit 1164 calculates the degree of relevance between event information and the automobile detection model. For example, if the event information includes "Main subject: automobile" and the model information includes "Detectability: ○", the unit calculates "Relevance: ◎". On the other hand, if the event information includes "Main subject: automobile" and the model information includes "Detectability: ×", the unit calculates "Relevance: ×".

[0083] Note that the model used for shooting is not limited to a car detection model; for example, a car race could be photographed using a person detection model. For example, in image 1205, if a person detection model was used for shooting and the "main subject: car," "other subjects: person," and "detectability: ○" were met, the relevance would be calculated as "○." Here, the relevance is shown in three stages: "◎," "○," and "×," but it is not limited to these three stages.

[0084] The settings window 1202 is a select box for setting the relevance threshold for extracting images as a dataset. In Figure 12, "Threshold: ○" is selected, and images with a relevance of "○" or higher (i.e., "◎" or "○") are ultimately extracted as a dataset. The extracted images are displayed as thumbnails in the display area 1203.

[0085] The user can obtain (create) an object detection model suitable for their purpose by training the object detection model in the learning unit 270 using the dataset 265 created through the above process. The created object detection model may be mounted on a camera and used for shooting (e.g., autofocus). It may also be used in the annotation data assignment unit 263 to assign annotation data.

[0086] As explained above, according to the third embodiment, it is possible to create a dataset that suits the user's purpose by extracting images related to the model used when capturing the image data.

[0087] The disclosures herein include the following information processing devices, control methods, image generating devices, imaging devices, and programs. (Item 1) Image acquisition means for acquiring captured image data, Information acquisition means for acquiring date and time information and location information of the captured image data, An event information acquisition means that acquires event information based on the aforementioned date and time information and location information, A means for acquiring multiple image data associated with the aforementioned event information and creating a training dataset that can be used for machine learning of an AI model, Equipped with, The creation means creates the training dataset by assigning annotation data containing correct answer information to each of the multiple image data. An information processing device characterized by the following: (Item 2) The event information acquisition means acquires the event information by inputting the date and time information and the location information into a given search means. The information processing device described in item 1, characterized by the features described herein. (Item 3) The event information acquisition means determines, as the event information, a string whose frequency of occurrence is higher than a threshold among the strings included in the search results obtained by the given search means. The information processing device described in item 2, characterized in that it is an information processing device. (Item 4) The system further comprises metadata setting means for setting the aforementioned event information as metadata for image data. An information processing device according to any one of items 1 to 3, characterized by the above. (Item 5) The creation means includes a grouping means for dividing the plurality of image data into a plurality of groups based on a specified grouping item. An information processing device according to any one of items 1 to 4, characterized by the above. (Item 6) The creation means includes scraping means for obtaining the plurality of image data from a given image database based on the event information. An information processing device according to any one of items 1 to 5, characterized by the above. (Item 7) The image acquisition means acquires a plurality of captured image data, The event information acquisition means acquires event information for each of the plurality of captured image data, The grouping means calculates the similarity between event information for each of the plurality of captured image data, and divides the plurality of captured image data into a plurality of groups based on the similarity. The information processing device described in item 5, characterized by the features described herein. (Item 8) Each system further includes a scheduling management mechanism for managing multiple event pieces of information, each associated with date / time and location information. The event information acquisition means acquires the event information by inputting the date and time information and the location information into the schedule management means. An information processing device according to any one of items 1 to 7, characterized by the above. (Item 9) The system further includes a model acquisition means for acquiring a trained model that performs scene analysis of image data related to the aforementioned event information, The creation means adds the analysis results by the trained model to the plurality of image data as annotation data. An information processing device according to any one of items 1 to 8, characterized by the above. (Item 10) A model information acquisition means for acquiring model information of a trained model used when capturing the aforementioned image data, A calculation means for calculating the degree of correlation between the event information and the model information, Furthermore, The creation means creates the training dataset using one or more image data from the plurality of image data that have a relevance of a predetermined threshold or higher. An information processing device according to any one of items 1 to 9, characterized by the above. (Item 11) The system further comprises a learning means for performing machine learning on the AI ​​model using the aforementioned training dataset. An information processing device according to any one of items 1 to 10, characterized by the above. (Item 12) The system includes an inference means that performs inference and generates images using an AI model trained with machine learning by the information processing device described in item 11. An image generation device characterized by the following features. (Item 13) The system includes an inference means that performs inference and controls shooting using an AI model trained by machine learning via the information processing device described in item 11. An imaging device characterized by the following features. (Item 14) A method for controlling an information processing device, The image acquisition process involves obtaining captured image data, An information acquisition step is to acquire the date and time information and location information of the captured image data, An event information acquisition step, which acquires event information based on the aforementioned date and time information and location information, A creation process involves acquiring multiple image data associated with the aforementioned event information and creating a training dataset that can be used for machine learning of an AI model. Includes, In the creation process described above, the training dataset is created by adding annotation data containing correct answer information to each of the multiple image data. A control method characterized by the following: (Item 15) A program to cause a computer to execute the control method described in item 14.

[0088] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0089] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0090] 20 Client terminal; 220 Location information acquisition unit; 230 Date and time information acquisition unit; 240 Event information acquisition unit; 250 Metadata setting unit; 260 Dataset creation unit; 270 Learning unit

Claims

1. Image acquisition means for acquiring captured image data, Information acquisition means for acquiring date and time information and location information of the captured image data, An event information acquisition means that acquires event information based on the aforementioned date and time information and location information, A means for acquiring multiple image data associated with the aforementioned event information and creating a training dataset that can be used for machine learning of an AI model, Equipped with, The creation means creates the training dataset by assigning annotation data containing correct answer information to each of the multiple image data. An information processing device characterized by the following:

2. The event information acquisition means acquires the event information by inputting the date and time information and the location information into a given search means. The information processing apparatus according to feature 1.

3. The event information acquisition means determines, as the event information, a string whose frequency of occurrence is higher than a threshold among the strings included in the search results obtained by the given search means. The information processing apparatus according to feature 2.

4. The system further comprises metadata setting means for setting the aforementioned event information as metadata for image data. The information processing apparatus according to feature 1.

5. The creation means includes a grouping means for dividing the plurality of image data into a plurality of groups based on a specified grouping item. The information processing apparatus according to feature 1.

6. The creation means includes scraping means for obtaining the plurality of image data from a given image database based on the event information. The information processing apparatus according to feature 1.

7. The image acquisition means acquires a plurality of captured image data, The event information acquisition means acquires event information for each of the plurality of captured image data, The grouping means calculates the similarity between event information for each of the plurality of captured image data, and divides the plurality of captured image data into a plurality of groups based on the similarity. The information processing apparatus according to feature 5.

8. Each system further includes a scheduling management mechanism for managing multiple event pieces of information, each associated with date / time and location information. The event information acquisition means acquires the event information by inputting the date and time information and the location information into the schedule management means. The information processing apparatus according to feature 1.

9. The system further includes a model acquisition means for acquiring a trained model that performs scene analysis of image data related to the aforementioned event information, The creation means adds the analysis results by the trained model to the plurality of image data as annotation data. The information processing apparatus according to feature 1.

10. A model information acquisition means for acquiring model information of a trained model used when capturing the aforementioned image data, A calculation means for calculating the degree of correlation between the event information and the model information, Furthermore, The creation means creates the training dataset using one or more image data from the plurality of image data that have a relevance of a predetermined threshold or higher. The information processing apparatus according to feature 1.

11. The system further comprises a learning means for performing machine learning on the AI ​​model using the aforementioned training dataset. The information processing apparatus according to feature 1.

12. The information processing device described in claim 11 comprises inference means for performing inference and generating an image using an AI model trained by machine learning. An image generation device characterized by the following features.

13. The information processing device described in claim 11 comprises inference means that perform inference using an AI model trained by machine learning and control the shooting process. An imaging device characterized by the following features.

14. A method for controlling an information processing device, The image acquisition process involves obtaining captured image data, An information acquisition step is to acquire the date and time information and location information of the captured image data, An event information acquisition step, which acquires event information based on the aforementioned date and time information and location information, A creation process involves acquiring multiple image data associated with the aforementioned event information and creating a training dataset that can be used for machine learning of an AI model. Includes, In the creation process described above, the training dataset is created by adding annotation data containing correct answer information to each of the multiple image data. A control method characterized by the following:

15. A program for causing a computer to execute the control method described in claim 14.

Images