Information processing apparatus, information processing method, and program

Abstract

To appropriately generate an image to be captured by a camera from a projection image.SOLUTION: There is provided an information processing apparatus including: an acquisition unit that acquires three-dimensional point cloud data and a captured image captured by an image capturing apparatus under a first image capturing condition; a generation unit that generates a projection image under a second image capturing condition, based on the three-dimensional point cloud data, and generates a pseudo-projection image, based on the projection image and the captured image; and a setting unit that sets learning data for performing supervised learning by using the pseudo-projection image as an explanatory variable and using the captured image as ground truth data.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses a technology that enables accurate estimation of the three-dimensional coordinates of an object captured in a satellite image. In Patent Document 1, a projection view is generated, which is a view in which a three-dimensional shape represented by three-dimensional data is projected onto a two-dimensional plane along the line of sight of a sensor. A pseudo-projection view that simulates the projection view is then generated based on the satellite image. Points in the projection view are then associated with points in the pseudo-projection view, and a mapping is derived that associates points in the pseudo-projection view with points in the three-dimensional shape represented by the three-dimensional data. Based on this mapping, a correspondence relationship between points of the object in the satellite image and points in the three-dimensional shape represented by the three-dimensional data is then derived. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-177855 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 does not consider restoring (inferring, restoring) an image captured by a camera from a projection drawing (projected image), which is a drawing in which the three-dimensional shape represented by the stereoscopic data is projected onto a two-dimensional plane along the line of sight of the sensor.

[0005] In view of the above-described problems, an object of the present disclosure is to provide a technology that can appropriately generate an image captured by a camera from a projected image. [Means for solving the problem]

[0006] In a first aspect of the present disclosure, there is provided an information processing device including: an acquisition unit that acquires three-dimensional point cloud data and a captured image captured under first capturing conditions by a capturing device; a generation unit that generates a projection image under second capturing conditions based on the three-dimensional point cloud data and generates a pseudo-projection image based on the projection image and the captured image; and a setting unit that sets learning data for supervised learning using the pseudo-projection image as an explanatory variable and the captured image as correct answer data.

[0007] In addition, a second aspect of the present disclosure provides an information processing method that acquires three-dimensional point cloud data and a captured image captured under first capturing conditions by a capturing device, generates a projection image under second capturing conditions based on the three-dimensional point cloud data, generates a pseudo-projection image based on the projection image and the captured image, and sets learning data for supervised learning using the pseudo-projection image as an explanatory variable and the captured image as correct answer data.

[0008] In addition, a third aspect of the present disclosure provides a program for causing a computer to execute a process of acquiring three-dimensional point cloud data and a captured image captured under first capturing conditions by a capturing device, generating a projection image under second capturing conditions based on the three-dimensional point cloud data, generating a pseudo-projection image based on the projection image and the captured image, and setting learning data for supervised learning using the pseudo-projection image as an explanatory variable and the captured image as correct answer data.

[0009] In addition, a fourth aspect of the present disclosure provides an information processing device including: an acquisition unit that acquires first three-dimensional point cloud data; a generation unit that generates a first projection image based on the first three-dimensional point cloud data; and an inference unit that uses a pseudo projection image generated based on a first captured image captured by an imaging device under first capturing conditions and a second projection image under second capturing conditions generated based on second three-dimensional point cloud data as an explanatory variable, and infers a second captured image based on the first projection image using a trained model generated by supervised learning with the first captured image as correct answer data. [Effects of the Invention]

[0010] According to one aspect, an image captured by a camera can be appropriately generated from a projected image. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information processing device that performs processing in a learning phase according to an embodiment. [Figure 2] FIG. 1 is a diagram illustrating an example of a configuration of an information processing device that performs processing in an inference phase according to an embodiment. [Figure 3] FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing apparatus according to an embodiment. [Figure 4] 10 is a flowchart illustrating an example of a process for generating learning data in the information processing device according to the embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of a process for generating learning data in the information processing device according to the embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of a learning data DB (Data Base) according to the embodiment. [Figure 7] 10 is a flowchart illustrating an example of a process for generating a trained model of the information processing device according to the embodiment. [Figure 8] 10 is a flowchart illustrating an example of processing in an inference phase of the information processing device according to the embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of an application that uses an inference result according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] The principles of the present disclosure will be described with reference to some exemplary embodiments. It should be understood that these embodiments are set forth for illustrative purposes only, to aid those skilled in the art in understanding and practicing the present disclosure, without implying any limitation on the scope of the disclosure. The disclosure described herein may be implemented in various ways other than those described below.

[0013] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0014] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessarily required to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0015] <Configuration> <<Configuration of information processing device 10 that performs processing in the learning phase>> The configuration of an information processing device 10 that performs processing in the learning phase according to the embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of an information processing device 10 that performs processing in the learning phase according to the embodiment. The information processing device 10 has an acquisition unit 11, a generation unit 12, and a setting unit 13. Each of these units may be realized by cooperation between one or more programs installed in the information processing device 10 and hardware such as a processor and memory of the information processing device 10.

[0016] The acquisition unit 11 acquires three-dimensional point cloud data and a captured image captured at a first location by an imaging device. The generation unit 12 generates a projection image in a specific line of sight direction from a second location based on the three-dimensional point cloud data acquired by the acquisition unit 11, and generates a pseudo projection image based on the projection image and the captured image acquired by the acquisition unit 11.

[0017] The setting unit 13 sets learning data for performing supervised learning using the pseudo projection images generated by the generating unit 12 as explanatory variables and the photographed images acquired by the acquiring unit 11 as correct answer data.

[0018] <<Configuration of the information processing device 20 that performs the processing in the inference phase>> The configuration of the information processing device 20 that performs processing of the inference phase according to the embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the information processing device 20 that performs processing of the inference phase according to the embodiment. The information processing device 20 has an acquisition unit 21, a generation unit 22, and an inference unit 23. Each of these units may be realized by cooperation between one or more programs installed in the information processing device 20 and hardware such as a processor and memory of the information processing device 20.

[0019] The acquisition unit 21 acquires first three-dimensional point cloud data. The generation unit 22 generates a first projection image based on the first three-dimensional point cloud data.

[0020] The inference unit 23 infers the second captured image based on the first projection image using a trained model generated by supervised learning using learning data generated by the information processing device 10. The learning data includes, as explanatory variables, pseudo projection images generated based on the first captured image captured by the imaging device under first shooting conditions and the second projection image under second shooting conditions generated based on the second three-dimensional point cloud data, and the first captured image as ground truth data.

[0021] <Hardware configuration> 3 is a diagram showing an example of the hardware configuration of the information processing device 10 and the information processing device 20 according to the embodiment. In the example of FIG. 3, the information processing device 10 and the information processing device 20 (computer 100) include a processor 101, a memory 102, and a communication interface 103. These components may be connected by a bus or the like. The memory 102 stores at least a part of a program 104. The communication interface 103 includes an interface required for communication with other network elements.

[0022] When the program 104 is executed by the processor 101, memory 102, and the like in cooperation with each other, the computer 100 performs at least some of the processing of the embodiments of the present disclosure. The memory 102 may be of any type. As a non-limiting example, the memory 102 may be a non-transitory computer-readable storage medium. The memory 102 may also be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. Although only one memory 102 is shown in the computer 100, several physically different memory modules may exist in the computer 100. The processor 101 may be of any type. The processor 101 may include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture, as a non-limiting example. The computer 100 may have multiple processors, such as application-specific integrated circuit chips that are time-slaved to a clock that synchronizes the main processor.

[0023] Embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device.

[0024] The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, that execute on a target real or virtual processor or device to perform the processes or methods of the present disclosure. Program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or divided among program modules as desired in various embodiments. The machine-executable instructions of the program modules may be executed in local or distributed devices. In a distributed device, the program modules may be located in both local and remote storage media.

[0025] The program code for executing the methods of the present disclosure may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus. When the program code is executed by the processor or controller, the functions / acts in the flowcharts and / or implementing block diagrams are performed. The program code may be executed entirely on the machine, partly on the machine, as a standalone software package, partly on the machine and partly on a remote machine, or entirely on a remote machine or server.

[0026] The program can be stored and provided to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible recording media. Examples of non-transitory computer-readable media include magnetic recording media, magneto-optical recording media, optical disk media, and semiconductor memory. Magnetic recording media include, for example, flexible disks, magnetic tapes, and hard disk drives. Magneto-optical recording media include, for example, magneto-optical disks. Optical disk media include, for example, Blu-ray discs, CD (Compact Disc)-ROMs (Read Only Memory), CD-Rs (Recordable), and CD-RWs (Rewritable). Semiconductor memory includes, for example, solid-state drives, mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory). The program may also be provided to a computer by various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire or an optical fiber, or via a wireless communication path.

[0027] <Processing> <<Learning Phase>> Next, an example of a process for generating learning data of the information processing device 10 according to the embodiment will be described with reference to Fig. 4 to Fig. 6. Fig. 4 is a flowchart showing an example of a process for generating learning data of the information processing device 10 according to the embodiment. Fig. 5 is a diagram showing an example of a process for generating a pseudo-projection image of the information processing device 10 according to the embodiment. Fig. 6 is a diagram showing an example of a learning data DB (Data Base) 501 according to the embodiment. Note that the information processing device 10 may repeatedly execute the process of Fig. 4 a specific number of times (for example, 100,000 times).

[0028] In step S101, the acquisition unit 11 acquires three-dimensional point cloud data. Here, the acquisition unit 11 may acquire one piece of three-dimensional point cloud data randomly selected from a DB in which a plurality of pieces of three-dimensional point cloud data are recorded, for example.

[0029] The three-dimensional point cloud data may be acquired by, for example, LiDAR (Light Detection and Ranging), a stereo camera, a laser scanner, or a 3D scanner. The three-dimensional point cloud data may include, for example, the three-dimensional position (coordinate information) of each point on the surface of an object. The three-dimensional point cloud data may also include color information of each point.

[0030] Next, the acquisition unit 11 acquires a photographed image photographed under a first photographing condition by a photographing device (camera) (step S102). The first photographing condition may include, for example, conditions such as the photographing position and direction. Here, the acquisition unit 11 may acquire, for example, one photographed image randomly selected from a DB in which a plurality of photographed images are recorded. The photographed image may be an image of an arbitrary subject photographed under arbitrary photographing conditions.

[0031] Examples of captured images may include, for example, photographs taken by a camera, panoramic photographs taken by a spherical camera, still images extracted from videos, and images created by simulation using 3DCG technology. Examples of captured images may also include, for example, generated images created by known technology that automatically generates images that look like photographs, images taken by a hyperspectral camera that captures images by dispersing light into wavelengths (hyperspectral images), and images taken by a thermal camera that can visualize temperature based on the amount of infrared light emitted from an object (thermal images).

[0032] Next, the generation unit 12 generates a projection image under a second shooting condition different from the first shooting condition based on the 3D point cloud data acquired by the acquisition unit 11 (step S103). The second shooting condition may include conditions such as a projection position and direction. The combination of positions and directions included in the first shooting condition is different from the combination of positions and directions included in the second shooting condition.

[0033] Here, the generation unit 12 may, for example, set the number of vertical pixels and the number of horizontal pixels of the projected image to be the same as the number of vertical pixels and the number of horizontal pixels of the captured image, respectively.

[0034] The generation unit 12 may generate a projection image by, for example, perspective projection, which projects a three-dimensional object so that the three-dimensional object converges to one point. Alternatively, the generation unit 12 may generate a projection image by, for example, orthogonal projection, which projects a three-dimensional object directly in parallel onto a two-dimensional object.

[0035] The generation unit 12 may generate a projected image as a binary image consisting of only white (pixel value 1) and black (pixel value 0) pixels. In this case, the generation unit 12 may set the pixel value of a specific pixel on the projected image to white only when, for example, 3D point cloud data of the subject exists in a direction from the camera position corresponding to the specific pixel (the line of sight direction collides with a point in the 3D point cloud data).

[0036] Furthermore, as shown in FIG. 5, the generation unit 12 may generate multiple (two or more) projection images 501 and 502, each having different shooting conditions (projection conditions), based on one or more 3D point cloud data acquired by the acquisition unit 11. Then, as shown in FIG. 5, the generation unit 12 may generate one projection image 503 based on the projection images 501 and 502. This allows for the generation of more random projection images. FIG. 5 illustrates an example of a process for generating pseudo projection images by the information processing device 10 according to the embodiment. In the example of FIG. 5, the generation unit 12 generates a projection image 501 in a certain direction from a certain point, and a projection image 502 in a point and direction that is different from the projection image 501 in at least one of the point and the direction. Then, the generation unit 12 generates a projection image 503 by extracting and combining, for example, pixels of a proportion X of the pixels of the projection image 501 and pixels of a proportion 1-X of the pixels of the projection image 502. As a result, a projection image 503 is generated in which pixels in the projection image 502 are replaced with pixels at a ratio X of pixels in the projection image 501.

[0037] Next, the generation unit 12 generates a pseudo projection image based on the projected image and the captured image acquired by the acquisition unit 11 (step S104). Here, the generation unit 12 may, for example, set the value of pixels in the projected image where 3D point cloud data of the subject exists in a direction from the camera position corresponding to a specific pixel on the projected image (e.g., white pixels) to 1. The generation unit 12 may then generate the pseudo projection image by, for example, performing a logical product (AND operation) of each pixel value in the projected image and each pixel value in the captured image. This makes it possible to generate a pseudo projection image that looks like a projection image based on 3D point cloud data at the point and direction at which the captured image was captured. In this case, the generation unit 12 may, for example, set the logical product of the pixel value of each coordinate in the projected image and the pixel value of the corresponding coordinate (e.g., the same coordinate) in the captured image as the pixel value of each corresponding coordinate in the pseudo projection image. This generates a pseudo projection image in which the value of each pixel in the captured image is set to black (pixel value 0) if the value of the corresponding pixel in the projection image is black (pixel value 0).

[0038] In the example of FIG. 5, the generation unit 12 generates a pseudo projection image 521 based on the projection image 503 and the captured image 511 at a specific probability p (p is a value in the range of 0 to 1), and generates a pseudo projection image 522 based on the random noise image 531 and the captured image 511 at the remaining probability (1-p) of the specific probability. In the example of FIG. 5, the generation unit 12 generates the pseudo projection image 521 based on the logical product of each pixel value of the captured image 511 and each pixel value of the projection image 503 at the specific probability p. Also, in the example of FIG. 5, the generation unit 12 generates the pseudo projection image 522 based on the logical product of the captured image 511 and the random noise image 531 at the remaining probability (1-p) of the specific probability. This makes it possible to generate a more random pseudo projection image. Therefore, improvement in the performance of inference based on the learning results of machine learning can be expected.

[0039] The generating unit 12 may set p=1 and generate only the pseudo projection image 521. Alternatively, the generating unit 12 may generate both the pseudo projection image 521 and the pseudo projection image 522. In this case, for example, the number of pieces of training data can be increased.

[0040] The generation unit 12 may change (determine) the value of the identification probability p according to the progress of supervised learning (for example, the number of epochs or the number of learning times). In this case, the generation unit 12 may, for example, increase the value of the identification probability p as the learning progresses. This makes it possible to perform learning in which knowledge acquired in the domain of the normal captured image 511 (for example, a city street) is corrected to the domain of the projected images 501 and 502 (for example, a specific facility). In this case, the generation unit 12 may, for example, change the value of the identification probability p so that it approaches 0 in the early stage of learning and approaches 1 in the final stage of learning.

[0041] Furthermore, the generation unit 12 may change (determine) the value of the ratio X according to the progress of learning (for example, the number of epochs or the number of times of learning). In this case, the generation unit 12 may, for example, increase the value of the ratio X as the learning progresses. This makes it possible to perform learning in which knowledge acquired in a broader domain is corrected to the domain of the projection image 501 (for example, a specific facility). In this case, the generation unit 12 may, for example, change the value of the ratio X so that it approaches 0 in the early stage of learning and approaches 1 in the final stage of learning.

[0042] Furthermore, the generation unit 12 may determine (generate) the random noise image 531 so that the proportion of black pixels (pixel value 0) increases as the learning progresses, for example. As a result, for example, in the first half of the learning, a random noise image 531 with a high proportion of white pixels (pixel value 1) is generated so that a relatively large proportion of information from the original captured image 511 is retained. Also, for example, in the second half of the learning, a random noise image 531 with a high proportion of black pixels (pixel value 0) is generated so that a relatively large proportion of information from the original captured image 511 is lost. Therefore, for example, even when a sparse projection image (with a relatively large proportion of black pixels) is input in the inference phase, learning can be performed so that the captured image can be appropriately inferred.

[0043] Furthermore, when data on a combination of a projection image 501 and a captured image 511 captured (generated) under the same capturing conditions can be acquired, the generation unit 12 may use the data on the combination in the final stage of learning and set at least one of the ratio X and the specific probability p to a relatively large value (for example, a value close to 1). In this case, the calculated pseudo projection image 521 resembles a projection image obtained when a three-dimensional object is captured under the same capturing conditions as the captured image. This allows fine tuning to be performed, in which the machine learning results are finely adjusted to suit a specific task.

[0044] Note that data of a combination of the projection image 501 and the captured image 511 captured (generated) under the same capturing conditions can be acquired using a sensor, such as a stereo camera, that can simultaneously acquire the color and structure of a three-dimensional object. Furthermore, if three-dimensional data of the subject of the captured image exists, the generation unit 12 may calculate the capturing conditions of the captured image based on the three-dimensional data, for example, using a technique such as Visual Localization. Then, the generation unit 12 may generate a projection image with the same capturing information as the captured image, for example, based on the calculated capturing conditions and the three-dimensional data.

[0045] Next, the setting unit 13 sets (records) learning data for performing supervised learning in the learning data DB 601 using the pseudo projection image as an explanatory variable and the captured image acquired by the acquisition unit 11 as correct answer data (step S105). This allows the trained model to be trained so that the representation of the projection image resembles the representation of the captured image, for example.

[0046] 6, a plurality of records of combinations of pseudo projection images and captured images are stored in the learning data DB 601 in association with learning data IDs. The learning data IDs are identification information of the learning data, which is a combination of a pseudo projection image and a captured image.

[0047] Next, an example of a process for generating a trained model of the information processing device 10 according to the embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of a process for generating a trained model of the information processing device 10 according to the embodiment.

[0048] In step S201, the acquiring unit 11 acquires a data set of learning data from the learning data DB 601.

[0049] Next, the generation unit 12 performs supervised learning for each record in the training data DB 601 using the pseudo projection images as explanatory variables and the captured images as correct data to generate a trained model that infers the captured image from the projection image (step S202). Here, the generation unit 12 may, for example, calculate a loss that indicates the low degree of similarity between the captured image, which is correct data, and the captured image generated by inference from the pseudo projection images. The loss may, for example, be the average value of the differences between each pixel value in one image and each pixel value in the other image. The loss may also, for example, be the loss (perceptual loss) between feature maps obtained by inputting each image into a specific deep learning model. Then, the generation unit 12 may, for example, perform machine learning to reduce the loss.

[0050] Furthermore, the generation unit 12 may perform machine learning using, for example, generative adversarial networks (GANs). In this case, the generation unit 12 may use, for example, a deep convolutional GAN ​​(DCGAN) that uses convolutional neural networks (CNNs) for two networks, a generator network and a discriminator network.

[0051] <<Inference Phase>> Next, an example of processing in the inference phase of the information processing device 20 according to the embodiment will be described with reference to Fig. 8 and Fig. 9. Fig. 8 is a flowchart showing an example of processing in the inference phase of the information processing device 20 according to the embodiment. Fig. 9 is a diagram showing an example of an application that uses the inference result according to the embodiment.

[0052] In step S301, the acquisition unit 21 acquires three-dimensional point cloud data. Here, the acquisition unit 11 may acquire three-dimensional point cloud data of a specific facility from, for example, a database in which a plurality of pieces of three-dimensional point cloud data are recorded.

[0053] Next, the generation unit 22 generates a projection image at a randomly determined point and direction based on the three-dimensional point cloud data acquired by the acquisition unit 11 (step S302). Note that the method for generating the projection image may be the same as that used in the process of step S103 in FIG.

[0054] 7, the inference unit 23 infers the captured image based on the projected image generated by the generation unit 22 (step S303). As a result, the captured image at the point and direction determined in the process of step S302 can be generated based on the three-dimensional point cloud data.

[0055] The inference unit 23 may identify the position and orientation of the camera at the time of capturing the image by, for example, performing feature matching between each captured image inferred for each point and each direction and an image actually captured by the camera. The example of Fig. 9 shows an example of 3D point cloud data 901 of a specific facility and an image 911 actually captured by the camera. In the example of Fig. 9, the inference unit 23 identifies the position 321 and orientation 322 of the camera at the time of capturing the image.

[0056] In addition, the inference unit 23 may identify the position of the subject in the captured image by, for example, performing feature matching between each captured image inferred for each point and each direction and an image actually captured by an imaging device.

[0057] <Other> Consider the case where a trained model that generates (infers) captured images (photographs) from 3D point cloud data is generated by supervised learning. In this case, a large amount of training datasets, which are combinations of projected images and captured images that are ground truth data, is required for learning. However, preparing such datasets is not easy. On the other hand, according to the present disclosure, a large amount of datasets for supervised learning can be generated based on any captured image and any 3D point cloud data. This makes it possible to appropriately generate (infer) captured images from projected images.

[0058] <Modification> The information processing device 10 and the information processing device 20 may each be a device contained in a single housing, but the information processing device of the present disclosure is not limited to this. Each unit of the information processing device 10 and each unit of the information processing device 20 may be realized by cloud computing configured with one or more computers, for example. Furthermore, the information processing device 10 and the information processing device 20 may be the same information processing device. Such an information processing device 10 is also included in an example of the "information processing device" of the present disclosure.

[0059] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0060] The present invention is not limited to the above-described embodiment, and can be modified as appropriate within the scope of the invention.

[0061] Some or all of the above embodiments may also be described as, but are not limited to, the following appendices. Note that some or all of the elements (e.g., configurations and functions) described in each appendix dependent on appendix 1 may also be dependent on independent appendices in other categories in a similar dependency relationship. Some or all of the elements described in any appendix may be applied to various hardware, software, recording means for recording software, systems, and methods. (Appendix 1) an acquisition unit that acquires three-dimensional point cloud data and a captured image captured under a first capturing condition by an imaging device; a generation unit that generates a projection image under a second photographing condition different from the first photographing condition based on the three-dimensional point cloud data, and generates a pseudo projection image based on the projection image and the photographed image; a setting unit that sets learning data for supervised learning using the pseudo projection images as explanatory variables and the captured images as correct answer data; An information processing device comprising: (Appendix 2) a combination of positions and directions included in the first photographing condition is different from a combination of positions and directions included in the second photographing condition; 10. The information processing device according to claim 1. (Appendix 3) the projection image includes a plurality of projection images; the generating unit generates the plurality of projection images each having a different photographing condition based on the three-dimensional point cloud data, and generates the pseudo projection image based on the plurality of projection images and the photographed image. 3. The information processing device according to claim 1 or 2. (Appendix 4) The generation unit generating a first projected image and a second projected image, each of which is captured under different imaging conditions, based on the three-dimensional point cloud data; generating a third projection image in which a specific ratio of pixels in the first projection image is replaced with pixels in the second projection image; generating the pseudo projection image based on a logical product of each pixel value of the third projection image and each pixel value of the captured image; 4. The information processing device according to claim 3. (Appendix 5) The generation unit generating the pseudo-projected image based on the projected image and the captured image with a specific probability; generating the pseudo-projection image based on the random noise image and the captured image with the remaining probability of the specific probability; 3. The information processing device according to claim 1 or 2. (Appendix 6) the generation unit increases the value of the specific probability as the supervised learning progresses. 6. The information processing device according to claim 5. (Appendix 7) The generation unit performs supervised learning based on the learning data to generate a trained model that infers a captured image from a projected image. 3. The information processing device according to claim 1 or 2. (Appendix 8) Acquire three-dimensional point cloud data and a photographed image photographed under a first photographing condition by an imaging device; generating a projection image under a second photographing condition based on the three-dimensional point cloud data, and generating a pseudo projection image based on the projection image and the photographed image; setting learning data for performing supervised learning using the pseudo projection images as explanatory variables and the captured images as correct answer data; Information processing methods. (Appendix 9) Acquire three-dimensional point cloud data and a photographed image photographed under a first photographing condition by an imaging device; generating a projection image under a second photographing condition based on the three-dimensional point cloud data, and generating a pseudo projection image based on the projection image and the photographed image; setting learning data for performing supervised learning using the pseudo projection images as explanatory variables and the captured images as correct answer data; A program that causes a computer to perform a process. (Appendix 10) an acquisition unit that acquires first three-dimensional point cloud data; a generation unit that generates a first projection image based on the first three-dimensional point cloud data; an inference unit that uses a pseudo projection image generated based on a first captured image captured under a first capturing condition by an imaging device and a second projection image generated under a second capturing condition based on second three-dimensional point cloud data as explanatory variables, and infers a second captured image based on the first projection image using a trained model generated by supervised learning using the first captured image as correct answer data; An information processing device comprising: [Explanation of symbols]

[0062] 10. Information processing equipment 11 Acquisition Department 12 Generation part 13 Setting section 20 Information processing equipment 21 Acquisition Department 22 Generation part 23 Reasoning part

Claims

1. an acquisition unit that acquires three-dimensional point cloud data and a captured image captured under a first capturing condition by an imaging device; a generation unit that generates a projection image under a second photographing condition different from the first photographing condition based on the three-dimensional point cloud data, and generates a pseudo projection image based on the projection image and the photographed image; a setting unit that sets learning data for supervised learning using the pseudo projection images as explanatory variables and the captured images as correct answer data; An information processing device comprising:

2. a combination of positions and directions included in the first photographing condition is different from a combination of positions and directions included in the second photographing condition; The information processing device according to claim 1 .

3. the projection image includes a plurality of projection images; the generating unit generates the plurality of projection images each having a different photographing condition based on the three-dimensional point cloud data, and generates the pseudo projection image based on the plurality of projection images and the photographed image.

3. The information processing device according to claim 1.

4. The generation unit generating a first projected image and a second projected image, each of which is captured under different imaging conditions, based on the three-dimensional point cloud data; generating a third projection image in which a specific ratio of pixels in the first projection image is replaced with pixels in the second projection image; generating the pseudo projection image based on a logical product of each pixel value of the third projection image and each pixel value of the captured image; The information processing device according to claim 3 .

5. The generation unit generating the pseudo-projected image based on the projected image and the captured image with a specific probability; generating the pseudo-projection image based on the random noise image and the captured image with the remaining probability of the specific probability; 3. The information processing device according to claim 1.

6. the generation unit increases the value of the specific probability as the supervised learning progresses. The information processing device according to claim 5 .

7. The generation unit performs supervised learning based on the learning data to generate a trained model that infers a captured image from a projected image.

3. The information processing device according to claim 1.

8. Acquire three-dimensional point cloud data and a photographed image photographed under a first photographing condition by the photographing device; generating a projection image under a second photographing condition based on the three-dimensional point cloud data, and generating a pseudo projection image based on the projection image and the photographed image; setting learning data for performing supervised learning using the pseudo projection images as explanatory variables and the captured images as correct answer data; Information processing methods.

9. Acquire three-dimensional point cloud data and a photographed image photographed under a first photographing condition by the photographing device; generating a projection image under a second photographing condition based on the three-dimensional point cloud data, and generating a pseudo projection image based on the projection image and the photographed image; setting learning data for performing supervised learning using the pseudo projection images as explanatory variables and the captured images as correct answer data; A program that causes a computer to perform a process.

10. an acquisition unit that acquires first three-dimensional point cloud data; a generation unit that generates a first projection image based on the first three-dimensional point cloud data; an inference unit that uses a pseudo projection image generated based on a first captured image captured by an imaging device under a first capturing condition and a second projection image generated under a second capturing condition based on second three-dimensional point cloud data as explanatory variables, and infers a second captured image based on the first projection image using a trained model generated by supervised learning using the first captured image as correct answer data; An information processing device comprising:

Images