Information processing device, information processing system, information processing method, and program

Abstract

This invention provides an information processing device, an information processing system, an information processing method, and a program that reduce the cost of training machine learning models. [Solution] The information processing device acquires conditions for image generation including an image of a predetermined region, and generates prompts based on those image generation conditions. The information processing device then determines if there are any unprocessed prompts, and if there are, generates training images and ground truth data corresponding to the predetermined region based on the generated prompts.

Description

Technical Field

[0001] The present invention relates to information processing technology using images.

Background Art

[0002] Conventionally, there has been a technology for detecting whether a vehicle (for example, an automobile) is parked in a parking area of a parking lot. As a method for detecting the presence or absence of parking in individual parking areas of a parking lot, there is a method of installing a vehicle detection device such as an infrared sensor to detect a vehicle. In addition, in Patent Document 1, since a part of the detection target may be hidden by another object, a technique of using a large number of learning images covering various hiding methods as teacher data to train a machine learning model (object detection model) is disclosed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technology described in Patent Document 1, although a machine learning model can be trained using learning images covering various hiding methods, it cannot cope with various changes such as weather, shadow, and brightness. In order to be able to cope with such various changes such as weather, shadow, and brightness, it is necessary to prepare an even larger amount of learning images, which increases the cost of training the machine learning model.

[0005] Therefore, an object of the present invention is to reduce the cost of training a machine learning model.

Means for Solving the Problems

[0006] The information processing apparatus of the present invention is characterized by comprising: acquisition means for acquiring conditions relating to image generation including an image of a predetermined region; prompt generation means for generating a prompt based on the conditions relating to image generation; and generation means for generating a training image and correct answer data corresponding to the predetermined region based on the prompt. [Effects of the Invention]

[0007] According to the present invention, the cost of training machine learning models can be reduced. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the hardware configuration of an image generation device. [Figure 2] This figure shows an example of the functional configuration of an image generation device. [Figure 3] This figure shows an example of a parking lot image input screen. [Figure 4] This is a diagram showing an example of a parking lot image. [Figure 5] This figure shows an example of an image generation condition input screen. [Figure 6] This figure shows an example of a screen for setting vehicle compartment information, vehicle traffic area, and vehicle restricted area. [Figure 7] This figure shows an example of a generated prompt. [Figure 8] This figure shows an example table of image generation conditions and prompts. [Figure 9] This figure shows training images, ground truth data, and an example of a ground truth image. [Figure 10] This figure shows an example of an image selection screen. [Figure 11] This figure shows an example of a test image, test correct data, and inference results. [Figure 12] This figure shows an example of a correct test image and an inference result image. [Figure 13] This is a flowchart of the information processing involved in image generation. [Figure 14] This is a flowchart of the information processing involved in learning and verification. [Figure 15] It is a diagram showing a configuration example of an information processing system. [Figure 16] It is a diagram showing a hardware configuration example of an imaging device. [Figure 17] It is a diagram showing a functional configuration example of an imaging device. [Figure 18] It is a diagram showing an example of a determination area setting screen. [Figure 19] It is a flowchart of information processing related to an imaging device and an image generation device. [Figure 20] It is a flowchart of parking detection processing.

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the plurality of features described in the present embodiments are essential for the solution means of the present invention, and those plurality of features may be arbitrarily combined. The configuration of the embodiment can be appropriately modified or changed according to the specifications of the device to which the present invention is applied and various conditions (usage conditions, usage environments, etc.). Also, a part of the embodiments described later may be appropriately combined to form a configuration. In the following embodiments, duplicate explanations of the same configuration and processing steps (processing steps) will be omitted.

[0010] In the present embodiment, an application example to information processing for analyzing whether an object exists in a predetermined area within a predetermined location using an image captured at the predetermined location will be described. In the following embodiments, a parking lot will be cited as an example of the predetermined location, and an application example to information processing for detecting the presence or absence of vehicle parking in the parking area within the parking lot will be cited and described as an example of the predetermined area. Here, as a method for detecting the presence or absence of parking in one or more individual parking areas provided in a parking lot, there is a method of installing a vehicle detection device such as an infrared sensor as described above to detect a vehicle. However, generally, such a vehicle detection device has a limited range in which it can detect a vehicle, and in a large-scale parking lot, the number of installed vehicle detection devices may become extremely large. On the other hand, as a method for detecting the presence or absence of parking in individual parking areas, there is a method of imaging the parking area with a camera and analyzing the captured image to detect whether a vehicle is parked. In this method, by imaging a plurality of parking areas with a single camera, there is an advantage that the presence or absence of parking in the parking areas for multiple vehicles can be detected from the images of a small number of cameras. In the present embodiment, a case where the presence or absence of parking in the parking areas for multiple vehicles is detected from the captured images of a small number of cameras by imaging a plurality of parking areas provided in a parking lot with a single camera will be described. That is, in the present embodiment, a case where the presence or absence of parking in individual parking areas within the parking lot is detected using an image obtained by imaging the parking lot with an imaging device is assumed. In the present embodiment, although an application example to a system for determining a parking status is given, the present embodiment is not limited to an example of detecting the presence or absence of parking, and is applicable to any system that analyzes an image and outputs predetermined information.

[0011] Also, in the present embodiment, the parking detection process for detecting the presence or absence of parking using an image obtained by imaging a plurality of parking areas as described above is assumed to be performed, for example, using a vehicle detector based on a machine learning model (vehicle detection model) learned in advance by machine learning. However, even when the parking detection process is performed using a vehicle detector based on a vehicle detection model, for example, the detection accuracy of the presence or absence of parking may decrease due to various environmental changes such as the installation information of the parking lot or parking area, such as weather, shadow, brightness, and the position of the camera.

[0012] Therefore, in the present embodiment, an example that enables learning of a machine learning model (vehicle detection model) capable of coping with various environmental changes and the like will be described. Below, as an example of an information processing device according to this embodiment, an image generation device that generates training images to enable the learning of a vehicle detection model capable of responding to various environmental changes will be described. Figure 1 shows an example of the hardware configuration of the image generation device 100 according to this embodiment. The image generation device 100 is composed of a general-purpose computer such as a PC, and as shown in Figure 1, it includes, for example, a processor 101, RAM 102, ROM 103, HDD 104, and communication I / F 105. The processor 101 consists of one or more CPUs (Central Processing Units) and MPUs (Micro Processing Units) and is responsible for the overall control of the image generation device 100.

[0013] The RAM (Random Access Memory) 102 is used by the processor 101 as a temporary storage area such as main memory or work area. The ROM (Read Only Memory) 103 stores the processor 101's control program and various setting data. The HDD 104 is used by the processor 101 as a storage area for storing data. The information processing program for the image generation device 100 to realize the image generation process according to this embodiment is stored in the ROM 103 or the HDD 104. The communication interface 105 is a circuit for the processor 101 to communicate with external devices.

[0014] Figure 2 shows an example of the functional configuration of the image generation device 100. The image generation device 100 realizes the functions of each functional unit shown in Figure 2 by having the processor 101 execute the information processing program according to this embodiment, which is stored in the ROM 103 or HDD 104.

[0015] As shown in Figure 2, the image generation device 100 includes, for example, a control unit 201, a network communication unit 202, an operation display unit 203, a setting processing unit 204, a prompt generation unit 205, an image generation unit 206, a learning unit 207, and a storage unit 208. The control unit 201 controls the network communication unit 202, operation display unit 203, setting processing unit 204, storage unit 208, image generation unit 206, and learning unit 207 to perform the corresponding processing as described later. The network communication unit 202, under the control of the control unit 201, connects to the internet via the communication interface 105, for example, and communicates with external devices. This is just one example; the network communication unit 202 may also communicate with external devices via an interface such as USB (Universal Serial Bus).

[0016] The operation display unit 203 controls an input device (not shown) that accepts user operations and an output device (not shown) that notifies the user, via the communication interface 105. In this embodiment, the input device is a mouse, keyboard, etc., and acquires user operation instructions for them. However, it is not limited to this, and the input device may be any other device that can detect the user's intent, such as a touch panel or microphone. The output device is, for example, a display, which displays information such as notifications to the user on the screen. However, it is not limited to this, and the output device may present information to the user by methods other than screen display, such as sound or vibration.

[0017] The setting processing unit 204 detects the area of ​​the parking lot and individual parking spaces by performing area detection processing on the captured image of the parking lot, and further acquires vehicle space information indicating the vehicle space based on the parking area. The setting processing unit 204 also has a function to acquire images of parking lots where no vehicles are present from the captured image of the parking lot. Furthermore, the setting processing unit 204 also acquires information such as one or more conditions related to image generation that are input by the user via the operation display unit 203. In this embodiment, the setting processing unit 204 acquires one or more conditions related to image generation, which are used when the prompt generation unit 205 (described later) generates a prompt, as information input by the user. Details of the processing in the setting processing unit 204 will be described later.

[0018] The prompt generation unit 205 generates a prompt based on the vehicle space information and parking lot image where no vehicle is present, acquired by the setting processing unit 204, and one or more conditions regarding image generation received from the user by the setting processing unit 204. In other words, the prompt generation unit 205 generates a prompt based on one or more conditions regarding image generation, including the parking lot image. The prompt generation unit 205 then inputs the generated prompt to the image generation unit 206, which will be described later. As will be described in detail in Figure 7, the prompt generation unit 205 generates a prompt that includes a command such as "Create an image by compositing a vehicle onto the parking lot image," the coordinates of the vehicle space based on the vehicle space information, and the parking lot image. Note that the format of the prompt is not limited to text including natural language or images, but may also include, for example, audio or video.

[0019] The image generation unit 206 generates an image based on the prompt generated by the prompt generation unit 205. In this embodiment, the image generation unit 206 uses known image generation techniques to generate training images and corresponding ground truth data to be used for training in the learning unit 207, which will be described later.

[0020] The learning unit 207 trains a machine learning model using the training images and ground truth data generated by the image generation unit 206. A machine learning model is a model (program) trained using a technique called machine learning, which finds patterns hidden in a large amount of data through iterative calculations. An example of a machine learning model is an object detection model obtained by applying the techniques described in the references below. In this embodiment, the learning unit 207 trains a vehicle detector (vehicle detection model) used for parking detection processing in a parking lot as the object detection model.

[0021] References: J. Redmon, A. Farhadi, "YOLO9000:Bette r Faster Stronger",Computer Vision and P Attern Recognition (CVPR) 2016.

[0022] Furthermore, the learning unit 207 also has a function to verify whether the inference results obtained by inputting test images using the trained machine learning model fit the ground truth data. In this embodiment, since the machine learning model is a vehicle detection model, the learning unit 207 verifies whether the inference results obtained by using, for example, an image of a parking lot as a test input image fit the ground truth data. The storage unit 208 saves the setting data set by the setting processing unit 204, the training images and correct answer data generated by the image generation unit 206, etc., to the HDD 104.

[0023] Figure 3 shows an example of a parking lot image input screen 300 that is generated and displayed by the setting processing unit 204 as a preliminary step before the user sets one or more conditions related to image generation. The parking lot image input screen 300 is a screen that is generated by the setting processing unit 204 and displayed on the screen by the operation display unit 203 under the control of the control unit 201. In this embodiment, when the setting processing unit 204 receives a display instruction for the parking lot image input screen from the user via the operation display unit 203 and is transmitted via the control unit 201, it displays the parking lot image input screen 300 as shown in Figure 3.

[0024] As shown in Figure 3, the parking lot image input screen 300 includes a parking lot image display area 301, a file form 302, a URL form 303, and a next button 304. File form 302 is a form for the user to input the file paths of one or more image or video files stored, for example, on HDD 104. URL form 303 is a form for the user to enter a URL for obtaining a video stream from an imaging device, such as a network camera. The parking lot image display area 301 is an area for displaying images obtained according to the information entered into the file form 302 or the URL form 303. In this embodiment, a parking lot image is displayed in the parking lot image display area 301. The next button 304 is a button that the user will press to transition to the image generation condition input screen 320, which will be described later in Figure 5.

[0025] Figure 4(a) shows an example of a parking lot image according to this embodiment. In the example in Figure 4(a), there are six parking areas 311a to 311f within the parking lot image 310. Hereafter, when describing these six parking areas 311a to 311f without distinction, they will be referred to as "parking area 311". Also, in the example in Figure 4(a), a vehicle 312 is parked in parking area 311e.

[0026] As mentioned above, the setting processing unit 204 also has a function to remove vehicles from parking lot images, and Figure 4(b) shows image 313 after the setting processing unit 204 has removed the vehicles 312 from the parking lot image 310 in Figure 4(a). The setting processing unit 204 removes the vehicles 312 by, for example, selecting an image that does not contain vehicles 312 from among multiple images of the parking lot 310, or by detecting vehicles 312 from the parking lot image 310 and compositing the background over the detected area of ​​the vehicles 312.

[0027] Figure 5 shows an example of the image generation condition input screen 320, which is accessed when the user presses the next button 304 on the parking lot image input screen 300 in Figure 3. The image generation condition input screen 320 is a screen displayed when the user sets one or more conditions related to image generation. It is generated by the setting processing unit 204 and displayed by the operation display unit 203 under the control of the control unit 201.

[0028] As shown in Figure 5, the image generation condition input screen 320 includes all elements from the vehicle interior form 321 to the generate button 330. The parking space form 321 is a form for receiving user input regarding parking space information (parking space area) within a parking lot image, vehicle traffic area (area where vehicles pass), and vehicle prohibited area (area where vehicles cannot pass). For example, if the user selects the parking space form 321, the setting processing unit 204 displays the parking space input screen 340 shown in Figure 6 on the screen using the operation display unit 203 under the control of the control unit 201.

[0029] Figure 6 shows an example of the vehicle interior input screen 340. The setting processing unit 204 detects the white lines of parking areas 311a to 311f from the parking lot image 310 shown in Figure 4, and displays the result of converting the detected parking areas 311a to 311f into vehicle space information 341a to 341f on the vehicle space input screen 340. Hereafter, when these vehicle space information 341a to 341f are not distinguished, they will be referred to as "vehicle space information 341". Furthermore, the setting processing unit 204 also provides a user interface (UI) that allows the user to arbitrarily change the shapes of the vehicle compartment information 341, the vehicle traffic area 342, and the vehicle restricted area 343.

[0030] The vehicle compartment displacement amount form 322 in Figure 5 is a form for the user to input the amount of displacement from the vehicle compartment when the image generation unit 206 places and synthesizes the vehicle image based on the vehicle compartment information entered in the vehicle compartment form 321. The vehicle count form 323 is a form for the user to input the upper and lower limits of the number of vehicles to be composited into the training image. The image generation unit 206 places and composites the vehicle images within the range of the number of vehicles set in the vehicle count form 323. The setting processing unit 204 may also automatically set the upper limit of the number of vehicles according to the number of vehicle compartments.

[0031] The time form 324 is a form for the user to input time information regarding the background to be composited onto the training image. In this embodiment, the time information set by the user is, for example, information specifying a time period such as 9:00 to 17:00, or a time category such as morning or evening. The image generation unit 206 uses a background corresponding to that time period or time category as the background of the training image. The vehicle type form 325 is a form for the user to input the type of vehicle, such as a passenger car, truck, bus, or light vehicle. The image generation unit 206 synthesizes the vehicle image of the type specified in the vehicle type form 325 with the training image. Note that the types of vehicles that can be specified are not limited to four-wheeled vehicles, but may also include two-wheeled vehicles such as bicycles and motorcycles. Furthermore, when generating training images assuming that there are people in the parking lot, it may also be possible to specify people in the vehicle type form 325.

[0032] The weather form 326 is a form for the user to input weather conditions such as sunny, rainy, foggy, snowy, or cloudy. The image generation unit 206 generates an image corresponding to the weather conditions entered in the weather form 326. The user additional input form 327 is a form for the user to input other information that is missing for prompt generation. Examples of other information that is missing for prompt generation include the setting location of the imaging device that captured the parking lot image, whether the parking lot is indoors or outdoors, the characteristics of the land, the characteristics of the language, and the characteristics of the country. The image generation unit 206 generates an image corresponding to the information entered in the user additional input form 327.

[0033] The training image format form 328 is a form for the user to input the format of the training image. Examples of training image formats that the user can input include the format and size of the training image. The image generation unit 206 generates a training image that corresponds to the training image format entered in the training image format form 328. The correct answer data format form 329 is a form for the user to input the correct answer data format, which will be described later. Examples of correct answer data formats that the user can input include formats suitable for machine learning models that the learning unit 207 learns, such as bounding boxes and segmentation. In the case of the vehicle detector of this embodiment, it is assumed that the user has input the bounding box for the vehicle. The generation button 330 is a button used by the user to instruct the prompt generation unit 205 to start generating prompts. When the generation button 330 is pressed by the user, the control unit 201 of the image generation device 100 starts the image generation process, which will be described later.

[0034] Although Figures 4 and 6 show examples of parking lots where each vehicle has its own designated parking space, parking lots may also be of a type where vehicles are parked freely without designated spaces. In this case, the setting processing unit 204 may set the area where parking is possible as the vehicle space information 341 and set it as an area where multiple vehicles can park. Furthermore, the parking area may be a parking space such as the shoulder of a road. In this case, the setting processing unit 204 may use the area of ​​the road shoulder as the vehicle space information 341 and set it as an area where multiple vehicles can park.

[0035] Figure 7 shows an example of a prompt 400 generated by the prompt generation unit 205 based on one or more conditions related to image generation, including vehicle information and parking lot images acquired by the setting processing unit 204. The prompt 400 exemplified in Figure 7 consists of a command 401 in natural language text, an image 402, and coordinates 403.

[0036] Command 401 is a textual instruction for a known image generation technique included in the image generation unit 206, such as "Create an image by compositing a vehicle onto the parking space 001 in the parking lot image 001." Image 402 is an example of the parking lot image 310 shown in Figure 4(a), for example. The "parking lot image 001" in command 401 of prompt 400 refers to this image 402.

[0037] Coordinate 403 is an example of parking space information 341. "Parking space 001" in command 401 of prompt 400 refers to this coordinate 403. Coordinate 403 shows the coordinates of each parking area in image 402 in JSON (JavaScript Object Notation) format. In the example of coordinate 403 shown in Figure 7, the JSON keys are "id" and "position", which correspond to parking space information 341a to 341f in Figure 6, respectively. "id" indicates the index of parking space information 341 in image 402. "position" indicates the (X, Y) coordinate pair of the vertex of parking space information 341. (X, Y) indicates the pixel values ​​of the X and Y axes with the top left of the image as the origin.

[0038] Furthermore, "Garage 001" in command 401 of prompt 400 refers to index "1" of the garage information 341 at coordinate 403. In Figure 7, the "position" value of index "1" in garage information 341 is "[{"x":436, "y":122}, {"x":759, "y":120}, {"x":744, "y":443}, {"x":353, "y":446}]}. Therefore, "Garage 001" refers to a quadrilateral with four vertices at coordinates (436,122), (759,120), (744,443), and (353,446). Note that coordinate 403 is also used when specifying the vehicle traffic area 342, vehicle prohibited area 343, etc. in Figure 4.

[0039] The following are some examples of prompts generated by the prompt generation unit 205. For example, if the input value for the vehicle compartment displacement form 322 is "30cm", the prompt generation unit 205 will generate a prompt such as, "Create an image in which the vehicle is superimposed at a position 30cm shifted relative to the vehicle compartment 001 in the parking lot image 001." The smaller the displacement value entered in the vehicle compartment displacement form 322, the more prompts the prompt generation unit 205 will generate that superimpose the vehicle inside the vehicle compartment. The larger the displacement value entered in the vehicle compartment displacement form 322, the more prompts the prompt generation unit 205 will generate that superimpose the vehicle at a position that extends beyond the vehicle compartment.

[0040] For example, if the input value in the number of vehicles form 323 is "3 vehicles", the prompt generation unit 205 generates a prompt such as, "Create an image by combining vehicles with parking spaces 001, 002, and 003 of the parking lot image 001."

[0041] For example, if the input value of the time form 324 is "morning", the prompt generation unit 205 generates a prompt such as, "Create an image where the time of day for the background of the parking lot image 001 is morning." Also, for example, if the input value of the time form 324 is daytime, the prompt generation unit 205 generates a prompt instructing to composite a bright background. Also, for example, if the input value of the time form 324 is nighttime, the prompt generation unit 205 generates a prompt instructing to composite a dark background.

[0042] For example, if the input value in the vehicle form 325 is "truck", the prompt generation unit 205 will generate a prompt such as, "Create an image by compositing a truck onto the parking space 001 of the parking lot image 001."

[0043] For example, if the input value of the weather form 326 is "rain", the prompt generation unit 205 generates a prompt such as, "Create an image where the background of the parking lot image 001 is rainy."

[0044] For example, if the input value in the user addition input form 327 is "Set the installation position of the imaging device to a height of 10m", the prompt generation unit 205 will generate a prompt such as "Create an image of the parking lot image 001 taken from a position of 10m."

[0045] For example, if the input value in the user addition input form 327 is "Japanese car", the prompt generation unit 205 generates a prompt such as, "Create an image by compositing a vehicle with a Japanese license plate onto parking space 001 in parking lot image 001."

[0046] For example, if the input value of the learning image format form 328 is "JPEG format", the prompt generation unit 205 will generate a prompt such as, "Create a JPEG image by compositing a vehicle onto the parking space 001 of the parking lot image 001." Note that JPEG is an abbreviation for "Joint Photographic Experts Group" and refers to a type of image compression format.

[0047] For example, if the input value for the learning image format form 328 is "Size is 300x300", the prompt generation unit 205 will generate a prompt such as "Create an image (300x300) by combining a vehicle with the parking space 001 of the parking lot image 001." Note that "300x300" indicates that the vertical and horizontal pixel values ​​of the image are each 300 pixels.

[0048] For example, if the input value of the correct answer data format form 329 is "bounding box", the prompt generation unit 205 generates a prompt such as "Create an image with a vehicle superimposed on the parking space 001 of the parking lot image 001, and create the correct answer data (bounding box)."

[0049] For example, if the input value for the vehicle type form 325 is "3 people", the prompt generation unit 205 will generate a prompt such as, "Create an image by combining 3 people with the parking lot image 001."

[0050] For example, if a vehicle traffic area is specified in the parking space form 321, the prompt generation unit 205 generates a prompt such as, "Create an image by compositing a vehicle onto the roadway area of ​​the parking lot image 001." The roadway area refers to the vehicle traffic area 342 shown in Figure 6.

[0051] For example, if a vehicle-restricted area is specified in the parking space form 321, the prompt generation unit 205 generates a prompt such as, "Create an image in which a vehicle is superimposed on an area of ​​the parking lot image 001 other than the restricted area." The restricted area refers to the vehicle-restricted area 343 shown in Figure 6.

[0052] The prompts described above provided examples of image generation conditions received by the setting processing unit 204 from the user. However, the prompt generation unit 205 can generate one or more prompts by combining one or more image generation conditions. Of course, the combination of image generation conditions may be a combination of all conditions, or a combination of some of the conditions.

[0053] Figure 8 shows an example of a table 500 that illustrates the combinations of prompts generated by the prompt generation unit 205. Table 500 consists of No. 501, image ID 502, vehicle compartment position 503, displacement amount 504, number of vehicles 505, time 506, vehicle type 507, weather 508, user input 509, training image format 510, correct answer data format 511, and prompt (command) 512.

[0054] No. 501 indicates the index of the prompt generated by the prompt generation unit 205. Image ID 502 indicates the address of the parking lot image. Parking space position 503 indicates the address of the parking space information. Shift amount 504 indicates one or more values ​​entered in the parking space shift amount form 322 in Figure 5. Number of vehicles 505 indicates one or more values ​​entered in the number of vehicles form 323 in Figure 5. Time 506 indicates one or more values ​​entered in the time form 324 in Figure 5. Vehicle type 507 indicates one or more values ​​entered in the vehicle type form 325 in Figure 5. Weather 508 indicates one or more values ​​entered in the weather form 326 in Figure 5. User input 509 indicates one or more values ​​entered in the user additional input form 327 in Figure 5. Learning image format 510 indicates one or more values ​​entered in the learning image format form 328 in Figure 5. The correct answer data format 511 shows one or more values ​​in the correct answer data format form 329 of Figure 5. The prompt 512 shows the command in the text included in the prompt generated by the prompt generation unit 205.

[0055] In the image generation conditions illustrated in Figure 8, there is one parking lot image, one space information field, two values ​​for the number of spaces form 323 ("0-1"), three values ​​for the time form 324 ("morning, noon, night"), and two values ​​for the weather form 326 ("sunny, rainy"). Additionally, the learning image format form 428 has one value for the image size ("300x300"), the correct answer data format form 329 has one value ("bounding box"), and no other image generation conditions are specified. Furthermore, in the example in Figure 8, the prompt generation unit 205 generates 12 prompts by combining the image generation conditions.

[0056] Furthermore, the prompt generation unit 205 may generate more or fewer specific prompts depending on the input conditions related to image generation. For example, if the value entered in the user additional input form 327 is "indoors," the prompt generation unit 205 may reduce the number of prompts that involve environmental changes such as changes in weather, and if it is "outdoors," it may increase the number of prompts that involve environmental changes. For example, if the value entered in the user additional input form 327 is "heavy snowfall area", the prompt generation unit 205 may generate more prompts that include "snow", or generate prompts that increase the amount of snow. For example, if there are deficiencies or biases in the generated training images, the prompt generation unit 205 may accept information necessary to generate prompts that compensate for the deficiencies in the user additional input form 327. For example, if there are deficiencies in images with dark backgrounds, and the user additional input form 327 receives the information "Increase the number of images with dark backgrounds," the prompt generation unit 205 will increase the generation of prompts related to "dark backgrounds." Furthermore, the prompt generation unit 205 may generate prompts for generating multiple images. For example, the prompt generation unit 205 may generate a prompt such as, "Create 10 images by combining 0 to 6 vehicles for any of the parking spaces 001 through 006 in the parking lot image 001."

[0057] Figure 9 shows an example of training data (training image and ground truth data) generated by the image generation unit 206 in response to a prompt such as, "Create an image and ground truth data (bounding box) by combining a vehicle with the vehicle in the vehicle compartments 003 and 005 of image 001." The training data includes a training image 601 as shown in Figure 9(a) and ground truth data 604 as shown in Figure 9(b). In the example of Figures 9(a) and 9(b), image 001 is image 313 in Figure 4(b), vehicle compartment 003 is vehicle compartment information 341c in Figure 6, and vehicle compartment 005 is vehicle compartment information 341e in Figure 6.

[0058] As shown in Figure 9(a), the training image 601 is an image of a parking lot including vehicles 602 and 603, synthesized by the image generation unit 206. Furthermore, as shown in Figure 9(b), the ground truth data 604 is JSON format data showing the bounding boxes representing vehicles 602 and 603 synthesized by the image generation unit 206. In the ground truth data 604 in Figure 9(b), "type" indicates the vehicle type, and "position" indicates the position and size of the rectangle surrounding the vehicle. Also, "x" and "y" indicate the coordinates of the top-left vertex of the rectangle, and "width" and "height" indicate the width and height of the rectangle. Of course, the ground truth data 604 is just one example and is not limited to this. Image 605, shown in Figure 9(c), is the ground truth image. The ground truth image is an image obtained by visualizing the bounding box of the ground truth data 604 and superimposing it on the training image 601.

[0059] Figure 10 shows an example of an image selection screen 700 in which the image generation unit 206 of the image generation device 100 according to this embodiment displays the training image and correct answer data generated by the image generation unit 206, allowing the user to make a selection. The training image and correct answer data are displayed on the screen by the operation display unit 203 under the control of the control unit 201. The image generation device 100 according to this embodiment allows the user to select the training image and correct answer data on the image selection screen as shown in Figure 10.

[0060] The image selection screen 700 displays the training images generated by the image generation unit 206 in a list format. In the example shown in Figure 10, the image selection screen 700 consists of selected images 701a to 701f, a toggle button 702 for displaying correct answer data, and a training button 703. Hereafter, when selecting images 701a to 701f are not distinguished, they will be referred to as "selected image 701".

[0061] The selected image 701 displays the training images, and the user can select and deselect the selected image 701. For example, by deselecting the selected image 701, the user can exclude training images that are inappropriate for training from the training target. The correct answer data display toggle button 611 is operated by the user to switch between displaying the training image or the correct answer image as the selected image 701. When the correct answer data display toggle button 702 is "OFF", the training image 601 is displayed as the selected image 701, while when it is "ON", the correct answer image is displayed as the selected image 701. The learning button 703 is a button operated by the user to initiate the learning of a machine learning model (vehicle detection model) using the learning image and ground truth data associated with the selected image 701. When the learning button 703 is pressed, the control unit 201 of the image generation device 100 controls the learning unit 207 to execute the learning process described later. The cancel button (back button) 704 is a button that the user operates when they want to transition the screen to the image generation condition input screen 320 (i.e., return to the image generation condition input screen 320).

[0062] Figures 11 and 12 show an example of the verification results of a machine learning model (vehicle detection model) trained by the learning unit 207 of the image generation device 100. The learning unit 207 uses the training images and a portion of the ground truth data generated by the image generation unit 206 as a test dataset to validate the trained machine learning model.

[0063] Figure 11(a) shows test image 800 from the test dataset, and represents the training images used as the test dataset. The test image 800 exemplified in Figure 11(a) is a composite of two vehicles, vehicle 801a and vehicle 801b. Figure 11(b) shows the test ground truth data 802 from the test dataset, which represents the ground truth data for test image 800. The test ground truth data 802 exemplified in Figure 11(b) shows that it has bounding boxes for vehicle 801a and vehicle 801b. Figure 11(c) shows the inference result 803, which is the output when the test image 800 is input to the machine learning model trained by the learning unit 207. As illustrated in Figure 11(c), the inference result 803 is output in the same format as the ground truth data.

[0064] Figure 12(a) shows the correct test image 804. The correct test image 804 is an image in which the bounding box 805 of the correct test data 802 is superimposed on the test image 800. In the example in Figure 12(a), the bounding box for vehicle 801a is defined as bounding box 805a, and the bounding box for vehicle 801b is defined as bounding box 805b.

[0065] Figure 12(b) shows the inference result image 806. The inference result image 806 is an image in which the bounding box 807 of the inference result 803 is superimposed on the test image 800. In the example of Figure 12(b), no bounding box is detected for vehicle 801a, and the bounding box for vehicle 801b is detected as bounding box 807b. Also, in the example of the test correct answer image 804 shown in Figure 12(a), the vehicle type in the test correct answer data 802 for vehicle 801a is "car", whereas the vehicle type in the inference result 803 of Figure 12(b) is "bus". In other words, it can be seen that the machine learning model trained by the learning unit 207 of this embodiment is making false detections for vehicle 801a. Furthermore, in the example shown in Figure 12(b), the bounding box of the inference result 803 is not detected for vehicle 801b, indicating that the machine learning model trained by the learning unit 207 of this embodiment is making false detections for vehicle 801b.

[0066] Figure 13 is a flowchart showing the flow of the image generation process performed in the image generation device 100 according to this embodiment. Each processing step shown in the flowchart of Figure 13 is realized by the processor 101 of the image generation device 100 executing the information processing program according to this embodiment. However, this is merely an example, and some or all of each processing step may be realized not only by the image generation device 100 but also by external devices or dedicated hardware. The control unit 201 of the image generation device 100 according to this embodiment starts the process shown in the flowchart when the user presses the generation button 330 in Figure 5.

[0067] First, in step S901, the setting processing unit 204 of the image generation device 100 obtains conditions related to image generation, including images of parking lots, through the parking lot image input screen 300 shown in Figure 3 and the image generation condition input screen 320 shown in Figure 5. That is, the setting processing unit 204 obtains conditions related to image generation, including the parking lot images and vehicle information obtained through the parking lot image input screen 300, and the values ​​entered from the vehicle form 321 to the correct answer data format form 329 of the image generation condition input screen 320.

[0068] Next, in step 902, the prompt generation unit 205 generates one or more prompts as described above, based on the image generation conditions acquired in step S901. Next, in steps S903 and S904, the image generation unit 206 repeats the process in step S904 for all prompts generated in step 902. That is, in step S903, the image generation unit 206 determines if there are any unprocessed prompts, and if there are, in step S904 it generates training images and ground truth data based on those unprocessed prompts.

[0069] On the other hand, if it is determined in step S903 that there are no unprocessed prompts, the control unit 201 of the image generation device 100 terminates the processing of the flowchart in Figure 13, and then displays the image selection screen 700 shown in Figure 10. For example, if the user determines that sufficient training images 601 have not been generated, the control unit 201 of the image generation device 100 may, triggered by the user pressing the cancel button 704 in Figure 10, re-execute the flowchart process in Figure 13. For example, the prompt generation unit 205 may adjust the prompt generation parameters so that it is difficult to generate an image similar to the training image 601 that was deselected on the image selection screen 700. For example, the prompt generation unit 205 may adjust the generation parameters so that it uses a combination of conditions as image generation conditions that are different from the image generation conditions used when the image was generated, that is, conditions that are far removed from the image generation conditions used when the image was generated.

[0070] Figure 14 is a flowchart showing the flow of the learning process using the image selected by the image selection screen described above, and the verification process of the machine learning model obtained from that learning process, in the image generation device 100 according to this embodiment. Each processing step shown in the flowchart of Figure 14 is realized by the processor 101 of the image generation device 100 executing the information processing program according to this embodiment. However, this is only an example, and some or all of each processing step may be realized not only by the image generation device 100 but also by external devices or dedicated hardware. The control unit 201 of the image generation device 100 according to this embodiment starts the processing in this flowchart when the user presses the learning button 703 in Figure 10 as a trigger.

[0071] First, in step S911, the storage unit 208 of the image generation device 100 saves the training image and correct answer data selected on the image selection screen 700 in Figure 10, along with the associated prompts, to the HDD 104. Next, in step S912, the learning unit 207 of the image generation device 100 trains a machine learning model using the training images and a portion of the ground truth data saved in step S911. Next, in step S913, the learning unit 207 verifies the accuracy of the machine learning model using the training images and some of the ground truth data saved in step S911.

[0072] Next, in step S914, the learning unit 207 determines whether the accuracy verified in step S913 is above a predetermined threshold. If it is above the predetermined threshold, the process shown in the flowchart of Figure 14 is terminated. On the other hand, if the accuracy verified in step S913 is below the predetermined threshold, the learning unit 207 proceeds to step S915.

[0073] When the process proceeds to step S915, the prompt generation unit 205 adjusts the prompt generation parameters to facilitate the generation of a training image 601 that approximates the test image 800 in which the inference result 803 did not match the test ground truth data 802 during the verification in step 913. For example, the prompt generation unit 205 adjusts the generation parameters so that it uses a combination of the image generation conditions and the approximation conditions as the image generation conditions. Alternatively, the prompt generation unit 205 may, for example, include the generated training image or ground truth data in the prompt to generate a prompt such as "Generate an image that approximates the given image." After adjusting the prompt generation parameters in step S915, the image generation device 100 returns to the image generation process flowchart shown in Figure 13. As a result, the image generation device 100 performs learning based on the prompt after the adjustment of the generation parameters.

[0074] As described above, the image generation device 100 of this embodiment generates training images and ground truth data based on image generation conditions that include parking lot images acquired through the parking lot image input screen 300 and various setting values ​​acquired through the image generation condition input screen 320. In other words, the image generation device 100 of this embodiment can generate training images and ground truth data corresponding to image generation conditions with various setting values ​​such as weather, shadows, brightness, and camera position. The image generation device 100 can then train a machine learning model using these training images and ground truth data corresponding to various image generation conditions. Furthermore, since the image generation device 100 of this embodiment can automatically generate a machine learning model that corresponds to various setting values ​​such as weather, shadows, brightness, and camera position, the cost of training the machine learning model can be reduced.

[0075] The following describes an information processing system using the machine learning model trained as described above. In this embodiment, an information processing system that detects whether or not a vehicle is parked using the machine learning model trained in the learning unit 207 of the image generation device 100 described above will be used as an example. However, the information processing system of this embodiment is not limited to a system that detects whether or not a vehicle is parked, and can be applied to any system that analyzes an image and outputs predetermined information.

[0076] Figure 15 is a diagram showing one example configuration of the information processing system 1000 according to this embodiment. The information processing system 1000 according to this embodiment comprises one or more imaging devices (in this example, four imaging devices 1001a to 1001d), a network 1002, an image generation device 100, and an external device 1003. The configuration of the image generation device 100 is the same as described above. Hereafter, when describing the four imaging devices 1001a to 1001d without distinction, they will be referred to as "imaging device 1001".

[0077] The imaging device 1001 is an imaging device such as a network camera. The imaging device 1001 according to this embodiment also has a function to detect whether or not a vehicle is parked using a vehicle detector based on a machine learning model (vehicle detection model) learned by the first image generation device 100. In this embodiment, the imaging device 1001 has a built-in processing unit for image processing, but is not limited to this. For example, an external computer such as a PC (personal computer) connected to the imaging device 1001 may perform calculations such as image processing, and such a combination with a PC may be treated as the imaging device 1001.

[0078] The imaging device 1001, the image generation device 100, and the external device 1003 are connected to each other via the network 1002 so that they can communicate with one another. Network 1002 is comprised of multiple routers, switches, cables, etc., that comply with communication standards such as Ethernet®. Network 1002 may be any network enabling communication between the imaging device 1001, the image generation device 100, and the external device 1003, and may be constructed with any size, configuration, and compliant communication standards. For example, network 1002 could be the Internet, a wired LAN (Local Area Network), a wireless LAN, a WAN (Wide Area Network), etc. Network 1002 may also be configured to enable communication using a communication protocol compliant with, for example, the ONVIF (Open Network Video Interface Forum) standard. However, it is not limited to these. Network 1002 may also be configured to communicate using other communication protocols, such as a proprietary communication protocol.

[0079] The external device 1003 receives the result of the parking detection process performed by the imaging device 1001 and performs specific processing. The external device 1003 may be a PC (personal computer), a patrol light (registered trademark), a recording device, a speaker, an access control device, etc., and the processing content will be determined by the device. For example, if the external device 1003 is a patrol light, the external device 1003 will turn on the lamp when the imaging device 1001 notifies that it has detected a vehicle in a designated parking space. On the other hand, when the imaging device 1001 notifies that it has detected that the designated parking space is empty and no vehicle is parked there, the external device 1003 will turn off the lamp.

[0080] Figure 16 shows an example of the main configuration of the imaging device 1001. As shown in Figure 16, the imaging device 1001 is configured to include, for example, an imaging unit 1201, an image processing unit 1202, an arithmetic processing unit 1203, and a communication unit 1204.

[0081] The imaging unit 1201 includes a lens unit for forming a light image of a subject, etc., and an image sensor for converting the formed light image into an analog signal. The lens unit has a zoom function for adjusting the angle of view and an aperture function for adjusting the amount of light. The image sensor has a gain function for adjusting the sensitivity when converting light into an analog signal. These functions are adjusted based on setting values ​​notified by the image processing unit 1202. The analog signal acquired by the imaging unit 1201 is converted into a digital signal by an analog-to-digital conversion circuit (not shown) and sent to the image processing unit 1202 as an image signal.

[0082] The image processing unit 1202 consists of an image processing engine and its peripheral devices. Peripheral devices include, for example, RAM (Random Access Memory) and drivers for each I / F (interface). The image processing unit 1202 generates image data by applying image processing, such as development, filtering, sensor correction, and noise reduction, to the image signal from the imaging unit 1201. The image processing unit 1202 also sends set values ​​to the lens unit and image sensor and performs exposure adjustment to obtain an appropriately exposed image. The image data generated by the image processing unit 1202 is sent to the arithmetic processing unit 1203.

[0083] The arithmetic processing unit 1203 consists of one or more processors such as a CPU or MPU, memory such as RAM or ROM, and drivers for each I / F. Note that CPU is an acronym for Central Processing Unit, MPU for Micro Processing Unit, RAM for Random Access Memory, and ROM for Read Only Memory.

[0084] The communication unit 1204 consists of a network communication engine and peripheral devices such as RAM and an ETH PHY module. The ETH PHY module is a module that performs processing on the physical (PHY) layer of Ethernet. The communication unit 1204 converts image data and processing result data acquired from the arithmetic processing unit 1203 into a format that can be transmitted over the network 1002, and outputs the converted data to the network 1002.

[0085] Figure 17 is a diagram showing an example of the functional configuration of the imaging device 1001 according to this embodiment. As shown in Figure 17, the imaging device 1001 includes the following functional units: an imaging control unit 1301, a signal processing unit 1302, a storage unit 1303, a control unit 1304, an analysis unit 1305, a network communication unit 1306, and a setting unit 1307. The imaging device 1001 realizes the functions of each functional unit shown in Figure 17 by having a processor such as a CPU execute an information processing program according to this embodiment stored in ROM or the like.

[0086] The imaging control unit 1301 controls the imaging unit 1201 to image the surrounding environment. The signal processing unit 1302 performs predetermined signal processing on the image captured by the imaging unit 1201 to generate captured image data. In this embodiment, this captured image data is simply referred to as the "captured image." The signal processing unit 1302 also encodes the captured image. For example, if the captured image is a still image, the signal processing unit 1302 encodes the still image using an encoding method such as JPEG (Joint Photographic Experts Group). If the captured image is a moving image, the signal processing unit 1302 encodes the moving image using an encoding method such as H.264 / MPEG-4 AVC or HEVC (High Efficiency Video Coding). The signal processing unit 1302 can also encode the image using an encoding method selected by the user via an operation unit (not shown) of the imaging device 1001 from among a plurality of pre-set encoding methods.

[0087] The memory unit 1303 stores temporary data for various processes. The control unit 1304 controls the signal processing unit 1302, storage unit 1303, analysis unit 1305, network communication unit 1306, and setting unit 1307 to perform predetermined processes. Furthermore, as will be described in detail later, the control unit 1304 also performs parking detection processing using a vehicle detector based on a machine learning model learned by the image generation device 100 mentioned above. The analysis unit 1305 performs various image analysis processes on the captured image. The network communication unit 1306 communicates with the image generation device 100 and external device 1003 shown in Figure 15 via the network 1002.

[0088] The setting unit 1307 accepts setting operations from the user. In this embodiment, the setting unit 1307 has the functionality of a web server and accepts setting operations entered by the user using, for example, a browser on a PC that constitutes the image generation device 100. That is, the setting unit 1307 accepts input from input devices such as a mouse or keyboard included in the operation display unit 203 of the image generation device 100. The setting unit 1307 can also control notifications to the user via the display of the operation display unit 203 of the image generation device 100.

[0089] Figure 18 shows an example of a determination area setting screen displayed on the operation display unit 203, which was set by the setting unit 1307 of the imaging device 1001 according to this embodiment and transmitted by the control unit 1304 to the image generation device 100 via the network communication unit 1306. As shown in Figure 18, the judgment area setting screen 1400 consists of an captured image 1401, a judgment area 1402, a prohibited area 1403, a user input form 1404, and a learning button 1405.

[0090] The captured image 1401 is an image captured by the imaging device 1001. The setting unit 1307 of the imaging device 1001 sends the determination area setting screen 1400, which contains the captured image 1401, from the control unit 1304 to the image generation device 100 via the network communication unit 1306, so that it is displayed on the operation display unit 203.

[0091] The determination area 1402 is an area for determining whether a vehicle detected by the vehicle detector is parked in the parking lot. In this embodiment, the setting unit 1307 sets the determination area 1402 based on the vertices specified by the user operating the image generation device 100 while viewing the captured image 1401 on the determination area setting screen 1400. In this embodiment, an example is given in which the user manually sets the determination area 1402, but it is not limited to this. The setting unit 1307 may also be configured to detect the white lines of the parking area on the captured image 1401 using a known Hough transform, and to detect the determination area 1402 with the intersection of the white lines as the vertices.

[0092] The restricted area 1403 indicates an area where vehicles cannot pass. The restricted area 1403 is an area that prevents vehicles from being incorporated into the training images when the image generation device 100 trains the machine learning model of the vehicle detector. When the user inputs the setting of the restricted area through operation of the operation display unit 203 of the image generation device 100, the setting unit 1307 sets the restricted area 1403 on the captured image 1401.

[0093] The user input form 1404 is a form for receiving arbitrary image generation conditions entered by the user through operation of the operation display unit 203 of the image generation device 100. The learning button 1405 is a button operated by the user to save the information of the judgment area 1402 set by the setting unit 1307 and to instruct the image generation device 100 to start the learning process. When the image generation device 100 receives notification that the user has pressed the learning button 1405, the setting unit 1307 saves the setting value set on the judgment area setting screen 1400 to the storage unit 1303.

[0094] Figure 19 is a flowchart showing the flow of information processing performed in the information processing system 1000 according to this embodiment. Each processing step in the flowchart shown on the left side of Figure 19 is realized by the control unit 1304 of the imaging device 1001 executing the information processing program according to this embodiment. On the other hand, each processing step in the flowchart shown on the right side of Figure 19 is realized by the processor 101 of the image generation device 100 executing the information processing program according to this embodiment. However, this is merely an example, and some or all of each processing step may be realized not only by the image generation device 100 and the imaging device 1001, but also by other external devices or dedicated hardware. The control unit 1304 of the imaging device 1001 according to this embodiment starts the processing in the flowchart on the left side of Figure 19 when the user presses the learning button 1405 in Figure 18.

[0095] First, in step S1501, the control unit 1304 of the imaging device 1001 transmits the image generation conditions set by the setting unit 1307 to the judgment area setting screen 1400 in Figure 18 via the network communication unit 1306 to the image generation device 100. The image generation conditions set by the setting unit 1307 of the imaging device 1001 are the captured image 1401, the judgment area 1402, the prohibited area 1403, and the values ​​of the user input form 1404.

[0096] Next, in step S1502, the control unit 201 of the image generation device 100 receives the image generation conditions transmitted from the imaging device 1001 via the network communication unit 202. Next, in step S1503, the control unit 201 of the image generation device 100 uses the image generation conditions received in step S1502 to execute the processes of steps S901 to S904 in Figure 13 described above, and generates one or more training images and ground truth data.

[0097] Next, in step S1503, the prompt generation unit 205 of the image generation device 100 sets the captured image 1401 to the aforementioned parking lot image 310 and the judgment area 1402 to the aforementioned vehicle space information 341. The prompt generation unit 205 also sets the prohibited area 1403 to the aforementioned vehicle prohibited area 343. Furthermore, the prompt generation unit 205 treats the value of the user input form 1404 as one of the values ​​from the vehicle space displacement form 322 to the correct answer data format form 329 in Figure 5, or a combination thereof. The prompt generation unit 205 then uses these as image generation conditions to generate a prompt, and further generates a training image and correct answer data based on the generated prompt.

[0098] Next, in step S1504, the control unit 201 of the image generation device 100 uses the generated training images and ground truth data as training data to execute steps S911 to S915 of the learning process shown in Figure 14, thereby training the machine learning model for the vehicle detector. Next, in step S1505, the control unit 201 of the image generation device 100 transmits the machine learning model of the vehicle detector learned in step S1504 to the imaging device 1001 via the network communication unit 202. Then, in step S1506, the control unit 1304 of the imaging device 1001 receives the machine learning model of the vehicle detector via the network communication unit 1306. This enables the control unit 1304 of the imaging device 1001 to execute the parking detection process flowchart shown in Figure 20.

[0099] Figure 20 is a flowchart showing the flow of parking detection processing performed by the imaging device 1001 according to this embodiment. Each processing step in Figure 20 is realized by the control unit 1304 of the imaging device 1001 executing the information processing program according to this embodiment. However, this is merely an example, and some or all of each processing step may be realized not only by the imaging device 1001 but also by other external devices or dedicated hardware. Note that the control unit 1304 of the imaging device 1001 according to this embodiment can execute the processing shown in the flowchart in Figure 20 if it has a machine learning model for the vehicle detector.

[0100] The control unit 1304 of the imaging device 1001 repeatedly executes the processes from step S1601 to step S1604. First, in step S1601, the control unit 1304 acquires an image of the parking lot via the imaging control unit 1301. Next, in step S1602, the control unit 1304 performs inference using the vehicle detector on the image acquired in step S1601 to detect a vehicle.

[0101] Next, in step S1603, the control unit 1304 determines whether or not a vehicle is parked based on whether or not the vehicle detected in step S1602 is included in each of the vehicle compartments in the determination area 1402. Then, in step S1604, the control unit 1304 notifies the external device 1003 of the result determined in step 1603.

[0102] As described above, the imaging device 1001 in this information processing system performs parking detection processing using a machine learning model of a vehicle detector learned by the image generation device 100 using images of the parking lot, as previously mentioned. As a result, the imaging device 1001 can perform parking detection processing optimized for the parking lot, and can detect the presence or absence of a parked vehicle with high detection accuracy.

[0103] In the embodiments described above, an example was given of determining whether or not a parking space is occupied in a parking lot in order to know the degree of congestion or vacancy of the parking lot, but the example is not limited to parking lots. For example, it can also be applied to situations in which the presence or absence of each table or chair is determined in order to know the degree of congestion or usage of an area such as tables and chairs set up in a predetermined outdoor facility. In this example as well, the image generation device generates training images and ground truth data based on image generation conditions that include images of tables and chairs in the outdoor facility, and uses them to train a machine learning model. The imaging device can then use this machine learning model to determine whether or not each table or chair is occupied.

[0104] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more of the functions. The embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various ways without departing from its technical concept or its main features.

[0105] This embodiment includes the following configurations, methods, and programs. (Composition 1) An acquisition means for acquiring conditions related to image generation including an image of a predetermined region, A prompt generation means that generates a prompt based on the conditions for image generation, A generation means that generates a training image and ground truth data corresponding to the predetermined region based on the prompt, An information processing device characterized by having the following features. (Configuration 2) The system includes a region detection means for detecting a predetermined region from an captured image, The information processing apparatus according to configuration 1, characterized in that the prompt generation means generates a prompt for compositing an image of a predetermined object at the position of the predetermined region. (Composition 3) The information processing apparatus according to configuration 2, characterized in that the region detection means detects the predetermined region from an image after removing the predetermined object captured in the image. (Composition 4) It has region detection means for detecting a region from an captured image in which a predetermined object can be placed, The information processing apparatus according to any one of configurations 1 to 3, characterized in that the prompt generation means generates a prompt for compositing an image of the predetermined object at the location of a region where the predetermined object can be placed. (Composition 5) It has region detection means for detecting regions from captured images where the placement of a predetermined object is prohibited, The information processing apparatus according to any one of configurations 1 to 3, characterized in that the prompt generation means generates a prompt for compositing an image of the predetermined object at a location other than the area where the placement of the predetermined object is prohibited. (Composition 6) The information processing apparatus according to any one of configurations 1 to 5, characterized in that the prompt generation means adjusts the conditions for image generation when generating the prompt so that an image similar to a learning image that was not selected by the user among the learning images generated by the generation means is not generated. (Composition 7) An information processing device according to any one of configurations 1 to 6, characterized in that it has a learning means for training a machine learning model using the training images and ground truth data, and for verifying the accuracy of the machine learning model using the training images and ground truth data. (Composition 8) The information processing apparatus according to configuration 7, characterized in that the prompt generation means adjusts the conditions for image generation when generating the prompt so that, based on the accuracy verification by the learning means, an image is generated that approximates a learning image for which the inference result by the machine learning model was incorrect. (Composition 9) The information processing device according to any one of configurations 1 to 8, characterized in that the predetermined area is a parking space area in a parking lot where vehicles are parked. (Composition 10) The information processing device according to configuration 9, characterized in that the conditions for image generation include one or more of the following: vehicle interior area, vehicle traffic area, vehicle prohibited area, number of vehicles, time, brightness, vehicle type, person, weather, installation information of imaging device, characteristics of parking lot land, language characteristics, characteristics of country, arbitrary input by the user, format of training image, and format of ground truth data. (Composition 11) An acquisition means for acquiring conditions related to image generation, including an image of a predetermined region captured by an imaging device, A prompt generation means that generates a prompt based on the conditions for image generation, A generation means that generates a training image and ground truth data corresponding to the predetermined region based on the prompt, A learning method for training a machine learning model using the aforementioned training images and ground truth data, A communication means for transmitting the learned machine learning model to the imaging device, An information processing device having, An imaging means for capturing an image, A communication means for receiving the machine learning model from the information processing device, A detection means that uses the received machine learning model to detect a predetermined object from an image captured by the imaging means, A determination means for determining the presence or absence of a predetermined object in a predetermined area based on the results of the detection by the detection means, An imaging device having, An information processing system characterized by including the following. (Method 1) An acquisition step to acquire conditions related to image generation including an image of a predetermined region, A prompt generation step that generates a prompt based on the conditions for image generation, A generation step that generates a training image and ground truth data corresponding to the predetermined region based on the prompt, An information processing method characterized by having the following features. (Program 1) A program that causes a computer to function as an information processing device described in any one of configurations 1 through 10. [Explanation of symbols]

[0106] 100: Image generation device, 201: Control unit, 202: Network communication unit, 203: Operation display unit, 204: Setting processing unit, 205: Prompt generation unit, 206: Image generation unit, 207: Learning unit, 208: Storage unit

Claims

1. An acquisition means for acquiring conditions related to image generation including an image of a predetermined region, A prompt generation means that generates a prompt based on the conditions for image generation, A generation means that generates a training image and ground truth data corresponding to the predetermined region based on the prompt, An information processing device characterized by having the following features.

2. The system includes a region detection means for detecting a predetermined region from an captured image, The information processing apparatus according to claim 1, characterized in that the prompt generation means generates a prompt for compositing an image of a predetermined object at the position of the predetermined region.

3. The information processing apparatus according to claim 2, characterized in that the region detection means detects the predetermined region from an image after removing the predetermined object captured in the image.

4. It has region detection means for detecting a region from an captured image in which a predetermined object can be placed, The information processing apparatus according to claim 1, characterized in that the prompt generation means generates a prompt for compositing an image of the predetermined object at the location of a region in which the predetermined object can be placed.

5. It has region detection means for detecting regions from captured images where the placement of a predetermined object is prohibited, The information processing apparatus according to claim 1, characterized in that the prompt generation means generates a prompt for compositing an image of the predetermined object at a location other than the area where the placement of the predetermined object is prohibited.

6. The information processing apparatus according to claim 1, characterized in that the prompt generation means adjusts the conditions for image generation when generating the prompt so that an image similar to a training image that was not selected by the user among the training images generated by the generation means is not generated.

7. The information processing apparatus according to claim 1, characterized in that it has a learning means for training a machine learning model using the training images and ground truth data, and for verifying the accuracy of the machine learning model using the training images and ground truth data.

8. The information processing apparatus according to claim 7, characterized in that the prompt generation means adjusts the conditions for image generation when generating the prompt so that, based on the accuracy verification by the learning means, an image is generated that approximates a learning image for which the inference result by the machine learning model was incorrect.

9. The information processing apparatus according to any one of claims 1 to 8, characterized in that the predetermined area is a parking space area in a parking lot where vehicles are parked.

10. The information processing apparatus according to claim 9, characterized in that the conditions for image generation include one or more of the following: a vehicle compartment area, a vehicle traffic area, a vehicle-restricted area, the number of vehicles, time, brightness, vehicle type, people, weather, information on the installation of the imaging device, the characteristics of the parking lot land, the characteristics of the language, the characteristics of the country, arbitrary input by the user, the format of the training image, and the format of the ground truth data.

11. An acquisition means for acquiring conditions related to image generation, including an image of a predetermined region captured by an imaging device, A prompt generation means that generates a prompt based on the conditions for image generation, A generation means that generates a training image and ground truth data corresponding to the predetermined region based on the prompt, A learning method for training a machine learning model using the aforementioned training images and ground truth data, A communication means for transmitting the learned machine learning model to the imaging device, An information processing device having, An imaging means for capturing an image, A communication means for receiving the machine learning model from the information processing device, A detection means that uses the received machine learning model to detect a predetermined object from an image captured by the imaging means, A determination means for determining the presence or absence of a predetermined object in a predetermined area based on the results of the detection by the detection means, An imaging device having, An information processing system characterized by including the following.

12. An acquisition step to acquire conditions related to image generation including an image of a predetermined region, A prompt generation step that generates a prompt based on the conditions for image generation, A generation step that generates a training image and ground truth data corresponding to the predetermined region based on the prompt, An information processing method characterized by having the following features.

13. Computers, An acquisition means for acquiring conditions related to image generation including an image of a predetermined region, A prompt generation means that generates a prompt based on the conditions for image generation, A generation means that generates a training image and ground truth data corresponding to the predetermined region based on the prompt, A program that makes an information processing device function as having a certain feature.

Images