Learning system, learning method, learning program, image generation system, and estimation system

Abstract

To provide a learning system, a learning method, and a learning program that can improve estimation accuracy of a depth image having distance information that can be expressed in a unit of length.SOLUTION: A learning system 10 includes: an absolute depth image estimation model 102 that uses a captured image as input information and an absolute depth image as output information; a parameter estimation model 103 that uses a preceding captured image and a subsequent captured image as input information, and outputs a change amount parameter representing an amount of change in a position and an orientation of an image capturing device when the preceding captured image has been captured and the position and the orientation of the image capturing device when the subsequent captured image has been captured; and an image reconstruction unit 104 that reconstructs the preceding captured image using the absolute depth image, the change amount parameter, and color information of the subsequent captured image. The absolute depth image estimation model 102 and the parameter estimation model 103 are trained to reduce a first loss, a second loss, and a third loss.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a machine learning learning system, a learning method, a learning program, an image generation system, and an estimation system. [Background technology]

[0002] Various techniques for generating a depth image from a captured image have been proposed. Regarding such techniques, Patent Document 1 discloses a training data generation system that generates a sample image using a target image and a depth map of the target image, estimates depth values ​​of pixels included in the sample image to generate a depth map for the sample image, and generates at least a portion of training data using the depth map. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7285986 [Non-patent literature]

[0004] [Non-Patent Document 1] Wei Yin and six others, "Learning to Recover 3D Scene Shape from a Single Image," [online], [February 27, 2024], Internet (https: / / openaccess.thecvf.com / content / CVPR2021 / papers / Yin_Learning_To_Recover_3D_Scene_Shape_From_a_Single_Image_CVPR_2021_paper.pdf) [Non-patent document 2] Clement Godard and three others, "Digging Into Self-Supervised Monocular Depth Estimation," [online], [January 24, 2024], Internet (URL: https: / / openaccess.thecvf.com / content_ICCV_2019 / papers / Godard_Digging_Into_Self-Supervised_Monocular_Depth_Estimation_ICCV_2019_paper.pdf) [Non-patent document 3] Libo Sun, 5 others, “SC-DepthV3: Robust Self-supervised Monocular Depth Estimation for Dynamic Scenes”, [online], [January 24, 2020], Internet (URL: https: / / arxiv.org / pdf / 2211.03660.pdf) [Non-patent document 4] Ke Xian and five others, "Structure-Guided Ranking Loss for Single Image Depth Prediction," [online], [February 27, 2024], Internet (https: / / openaccess.thecvf.com / content_CVPR_2020 / papers / Xian_Structure-Guided_Ranking_Loss_for_Single_Image_Depth_Prediction_CVPR_2020_paper.pdf) [Non-patent document 5] Vitor Guizilini, 4 others, “3D Packing for Self-Supervised Monocular Depth Estimation”, [online], [January 24, 2020], Internet (URL: https: / / openaccess.thecvf.com / content_CVPR_2020 / papers / Guizilini_3D_Packing_for_Self-Supervised_Monocular_Depth_Estimation_CVPR_2020_paper.pdf) Summary of the Invention [Problem to be solved by the invention]

[0005] However, Patent Document 1 does not disclose a method for generating a depth image having distance information that can be expressed in units of length from a captured image, and the learning data generation method disclosed in Patent Document 1 has the problem that it is not possible to improve the estimation accuracy of such a depth image.

[0006] The present disclosure provides a learning system, a learning method, and a learning program that can improve the estimation accuracy of a depth image having distance information that can be expressed in units of length. [Means for solving the problem]

[0007] A learning system for learning an estimation model by machine learning according to the present disclosure includes: an absolute depth image estimation model that takes a captured image as input information and outputs an absolute depth image, which is a depth image having distance information that can be expressed in units of length; a parameter estimation model that receives as input information a preceding captured image and a subsequent captured image that is temporally consecutive to the preceding captured image, and that outputs as output information change amount parameters that represent the position and orientation of the image capture device when the preceding captured image was captured and the amount of change in the position and orientation of the image capture device when the subsequent captured image was captured; an image reconstruction unit that reconstructs the preceding captured image using an absolute depth image corresponding to the preceding captured image output from the absolute depth image estimation model, change amount parameters corresponding to the preceding captured image and the subsequent captured image output from the parameter estimation model, and color information of the subsequent captured image; The absolute depth image estimation model and the parameter estimation model are trained to reduce a first loss, a second loss, and a third loss. Here, the first loss is a photometric error between a previous captured image not reconstructed by the image reconstruction unit and the previous captured image reconstructed by the image reconstruction unit. The second loss is a loss based on the geometric consistency between a relative depth image, which is a depth image of the previous captured image having distance information not expressed in units of length, and an absolute depth image corresponding to the previous captured image output by the absolute depth image estimation model. The third loss is a difference between the distance between the position of the image capture device when the previous captured image was captured and the position of the image capture device when the subsequent captured image was captured, and the distance corresponding to the change in the position of the image capture device indicated by the change amount parameter corresponding to the previous captured image and the subsequent captured image output by the parameter estimation model.

[0008] A learning method for learning an estimation model by machine learning according to the present disclosure includes: outputting an absolute depth image using an absolute depth image estimation model that uses a photographed image as input information and an absolute depth image, which is a depth image having distance information that can be expressed in units of length, as output information; outputting a change amount parameter using a parameter estimation model in which a preceding photographed image and a subsequent photographed image that is temporally consecutive to the preceding photographed image are used as input information, and in which a change amount parameter representing a change amount between the position and orientation of the photographing device when the preceding photographed image was taken and the position and orientation of the photographing device when the subsequent photographed image was taken is used as output information; reconstructing the preceding captured image using an absolute depth image corresponding to the preceding captured image output using the absolute depth image estimation model, change amount parameters corresponding to the preceding captured image and the subsequent captured image output using the parameter estimation model, and color information of the subsequent captured image; An absolute depth image estimation model and a parameter estimation model are trained to reduce the first loss, the second loss, and the third loss.

[0009] A learning program for learning an estimation model by machine learning according to the present disclosure includes: a step of outputting an absolute depth image using an absolute depth image estimation model that uses a captured image as input information and an absolute depth image, which is a depth image having distance information that can be expressed in units of length, as output information; a step of outputting a change amount parameter using a parameter estimation model in which a preceding captured image and a subsequent captured image that is temporally consecutive to the preceding captured image are used as input information, and change amount parameters that represent the amount of change in the position and orientation of the image capturing device when the preceding captured image was captured and the position and orientation of the image capturing device when the subsequent captured image was captured are used as output information; a step of reconstructing the preceding captured image using an absolute depth image corresponding to the preceding captured image output using the absolute depth image estimation model, change amount parameters corresponding to the preceding captured image and the subsequent captured image output using the parameter estimation model, and color information of the subsequent captured image; and training an absolute depth image estimation model and a parameter estimation model so as to reduce the first loss, the second loss, and the third loss.

[0010] The image generation system according to the present disclosure comprises: an absolute depth image estimation model that takes a captured image as input information and outputs an absolute depth image, which is a depth image having distance information that can be expressed in units of length; a three-dimensional point cloud generation unit that generates a three-dimensional point cloud by inversely projecting color information of the captured image into a three-dimensional space using the captured image input to the absolute depth image estimation model and the absolute depth image output by the absolute depth image estimation model; a coordinate conversion unit that converts the coordinates of the 3D point cloud generated by the 3D point cloud generation unit into coordinates as viewed from the position of the virtual second camera device, using a parameter that indicates the distance between the first camera device that generated the captured image and a virtual second camera device that is located at a different shooting position from the first camera device, and a parameter that indicates the amount of change in the attitude of the virtual second camera device relative to the attitude of the first camera device; and a new image generation unit that projects the three-dimensional point cloud whose coordinates have been transformed by the coordinate transformation unit to generate an image as seen from the position of a virtual second imaging device.

[0011] An estimation system for estimating a movement amount of a moving object according to the present disclosure includes: a parameter estimation model trained by the above-described training method; The parameter estimation model outputs change amount parameters corresponding to a preceding captured image and a subsequent captured image generated by an image capturing device mounted on the moving object; The amount of change in the position of the image capture device indicated by the change amount parameter represents the amount of movement of the moving object. [Effects of the Invention]

[0012] The present disclosure provides a learning system, a learning method, and a learning program that can improve the estimation accuracy of a depth image having distance information that can be expressed in units of length. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram illustrating a configuration of an image processing device according to the present disclosure. [Figure 2] FIG. 10 is a diagram illustrating an example of processing executed by an image processing device according to the present disclosure. [Figure 3] FIG. 10 is a diagram illustrating another example of processing executed by the image processing device according to the present disclosure. [Figure 4] 10 is a diagram showing the flow of images and information used when an image reconstruction unit according to the present disclosure reconstructs a previously captured image. FIG. [Figure 5] FIG. 10 is a diagram illustrating an example of an application using a parameter estimation model according to the present disclosure. [Figure 6] FIG. 10 is a diagram illustrating another application example using the parameter estimation model according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014] An embodiment will be described below with reference to the drawings. FIG. 1 is a block diagram showing the configuration of an image processing device 10 according to the present disclosure. The image processing device 10 is a device capable of processing various images. The image processing device 10 corresponds to a learning system for learning an estimation model by machine learning. The image processing device 10 also corresponds to an image generation system.

[0015] The image processing device 10 includes a processor 100, a communication interface (I / F) 120, and a storage device 130. The communication interface 120 is an interface for data communication between the image processing device 10 and other devices. The storage device 130 is a storage device that stores various data such as programs executed by the image processing device 10 and images. Images processed by the image processing device 10 are stored in the storage device 130 via the communication interface 120. The images processed by the image processing device 10 include captured images generated by a camera. The camera may be mounted on a mobile object such as an automobile or a robot. Alternatively, the camera may be fixed to a structure such as a wall or a ceiling.

[0016] The processor 100 is a processor such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit) that controls electronic circuits and devices included in the image processing device 10. The processor 100 executes a method defined by a program by reading the program from the storage device 130 and executing the program. The program executed by the processor 100 may be executed by an integrated circuit such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The processor 100 and these integrated circuits correspond to a computer.

[0017] The processor 100 executes the following programs: a relative depth image generation unit 101, an absolute depth image estimation model 102, a parameter estimation model 103, an image reconstruction unit 104, a first loss calculation unit 105, a second loss calculation unit 106, a third loss calculation unit 107, a learning processing unit 108, a three-dimensional point cloud generation unit 109, a coordinate conversion unit 110, a new image generation unit 111, and an image completion unit 112.

[0018] The relative depth image generation unit 101 is a program that generates a depth image (hereinafter referred to as a "relative depth image") having distance information that is not expressed in units of length such as meters for each of multiple captured images stored in the storage device 130. As a relative depth estimation method, for example, the method described in Non-Patent Document 1 can be adopted. The model described in Non-Patent Document 1 is a monocular depth estimation model that has undergone supervised learning based on a large amount of data. The monocular depth estimation model inputs an image and outputs a depth image. The monocular depth estimation model can be trained so that the estimated depth image and the supervised depth image become closer to each other.

[0019] The absolute depth image estimation model 102 is a program that takes a captured image as input information and outputs a depth image (hereinafter referred to as an "absolute depth image") having distance information that can be expressed in units of length. The absolute depth image estimation model 102 can be trained using a first loss L1, a second loss L2, and a third loss L3, which will be described later.

[0020] The parameter estimation model 103 is a program that takes two temporally consecutive captured images as input information and outputs a change parameter that represents the amount of change between the position and orientation of the camera when the previous image was captured and the position and orientation of the camera when the subsequent image was captured. The amount of change between the position of the camera when the previous image was captured and the position of the camera when the subsequent image was captured corresponds to the distance between these two positions. The orientation of the camera is expressed by the roll angle, pitch angle, and yaw angle of the camera. Therefore, the amount of change in the orientation of the camera corresponds to the amount of change in the roll angle, pitch angle, and yaw angle of the camera. The parameter estimation model 103 can be trained using a first loss L1, a second loss L2, and a third loss L3, which will be described later.

[0021] The image reconstruction unit 104 is a program that reconstructs a preceding captured image using an absolute depth image corresponding to the preceding captured image, variation parameters corresponding to the preceding captured image and the subsequent captured image, and color information of the subsequent captured image. Fig. 4 is a diagram showing the flow of images and information used when the image reconstruction unit 104 reconstructs a preceding captured image.

[0022] We will explain how to reconstruct a previously captured image. When there is a point in space, its world coordinates are p(=[X,Y,Z]^T), the camera matrix (internal parameter) is K, and the camera pose (external parameter) is T(=[R|t]), the homogeneous coordinates u(=[x,y,1]^T) of the corresponding image are expressed as [Du=K[R|t]p=KTp]. Here, D is the depth, and in this case, [p=T -1 K -1 Du]. When there are two images, and the internal camera parameters are K, and the external parameters of image 1 and image 2 are T1 and T2, respectively, the coordinate u2 of image 2 corresponding to the point with coordinate u1 of image 1 is [D2u2=KT2T1 -1 K -1 D1u1]. When restoring image 1 from the color of image 2, all pixels u in image 1 are 1i The pixel u corresponding to 2i In general, pixel u 1iEven if is an integer, pixel u 2i is not necessarily an integer. Therefore, the color of coordinate u2 is calculated from the colors of the pixels around coordinate u2. There are several methods for calculating the color of the corresponding coordinate from the colors of the surrounding pixels, such as nearest neighbor interpolation, bilinear, and bicubic. 1i The corresponding pixel u 2i The reconstructed image is the result of adapting the colors.

[0023] The first loss calculation unit 105 is a program that calculates the first loss L1. The first loss L1 is a photometric error between a preceding captured image that has not been reconstructed by the image reconstruction unit 104 and a preceding captured image that has been reconstructed by the image reconstruction unit 104. Specific examples of this photometric error include, but are not limited to, the mean absolute error, mean square error (MSE), and structural similarity (SSIM) of these images.

[0024] The second loss calculation unit 106 is a program that calculates the second loss L2. The second loss L2 is a loss based on the geometric consistency between the relative depth image of the preceding captured image and the absolute depth image corresponding to the preceding captured image output by the absolute depth image estimation model 102. Specific examples of the second loss L2 include, but are not limited to, a Confident Depth Ranking Loss (to be described later) and the dot product of two normals calculated from depths.

[0025] The third loss calculation unit 107 is a program that calculates the third loss L3. The third loss L3 is the difference between the actual distance between the actual position of the camera when the preceding image was captured and the actual position of the camera when the subsequent image was captured, and the distance corresponding to the change in the position of the camera indicated by the change amount parameter corresponding to the preceding image and the subsequent image output by the parameter estimation model 103. If the camera is mounted on a moving object, the actual distance can be calculated from the time from when the preceding image was captured to when the subsequent image was captured and the speed of the moving object at that time. The speed of the moving object can be obtained from a measuring device installed on the moving object. For example, if the moving object is a self-propelled robot, the speed can be obtained from an inertial measurement unit (IMU). If the moving object is a car, the speed can be obtained from a vehicle speed sensor. Alternatively, the actual distance may be calculated using odometry.

[0026] The learning processing unit 108 is a program that trains the absolute depth image estimation model 102 and the parameter estimation model 103 using the first loss L1, the second loss L2, and the third loss L3. Specifically, the learning processing unit 108 trains the absolute depth image estimation model 102 and the parameter estimation model 103 so as to reduce the first loss L1, the second loss L2, and the third loss L3. For example, the learning processing unit 108 can train the absolute depth image estimation model 102 and the parameter estimation model 103 using a backpropagation algorithm so as to minimize the first loss L1, the second loss L2, and the third loss L3.

[0027] An example in which SSIM is used as the first loss L1 will be described. Based on Formulas 1, 2, and 3 disclosed in Non-Patent Document 2, the absolute depth image estimation model 102 and the parameter estimation model 103 can be trained so that the image reconstructed by the image reconstruction unit 104 approaches the previous captured image, which is the original image.

number

number

number

[0028] An example will be described in which the Confident Depth Ranking Loss disclosed in Non-Patent Document 3 is used as the second loss L2. The Confident Depth Ranking Loss is a loss for making the context of regions in an absolute depth image corresponding to regions with clear context in a relative depth image the same as the context in the relative depth image. The second loss L2 is reduced by imposing constraints on parts in the relative depth image whose depth order is known based on Equation 4, Equation 5, and Equation 6. Note that Equation 4, Equation 5, and Equation 6 correspond to Equation 8, Equation 9, and Equation 10 described in Non-Patent Document 3.

number

number

number

[0029] An example will be described in which the Edge-aware Relative Normal Loss disclosed in Non-Patent Document 3 is used as the second loss L2. Based on Equation 7, constraints are placed on normal vectors near edges in the (RGB) image to clarify the edges. The absolute depth image estimation model 102 and the parameter estimation model 103 can be trained so that the dot product of two normals in an absolute depth image sampled near an edge of the (RGB) image is the same as the dot product of the normals at the corresponding points in the relative depth image. For edge detection, for example, the detection method described in Non-Patent Document 4 can be adopted.

number

[0030] An example will be described in which the velocity supervision loss disclosed in Non-Patent Document 5 is used as the third loss L3. Based on Equation 8, the velocity v of a moving object having a length dimension and the frame rate ΔT t→s Calculate the scaled magnitude of the moving object's self-movement from the estimated self-movement magnitude t^ t→s The absolute depth image estimation model 102 and the parameter estimation model 103 are trained so that approaches the scaled magnitude value of the self-movement. Note that Equation 8 corresponds to Equation 6 described in Non-Patent Document 5.

number

[0031] The 3D point cloud generation unit 109 uses the previous captured image and the absolute depth image of the previous captured image output by the absolute depth image estimation model 102 to reverse-project the color information of the previous captured image into a 3D space to generate a 3D point cloud.

[0032] The coordinate conversion unit 110 converts the coordinates of the 3D point cloud generated by the 3D point cloud generation unit 109 into coordinates as viewed from the position of the virtual second camera device, using a parameter indicating the distance between the first camera device that generated a certain captured image and a virtual second camera device whose shooting position is different from that of the first camera device, and a parameter indicating the amount of change in the attitude of the virtual second camera device based on the attitude of the first camera device. Any position and attitude of the virtual second camera device can be used. In other words, any parameter, such as a randomly determined parameter, can be used as the parameter used to convert the coordinates of the 3D point cloud.

[0033] The new image generation unit 111 projects the three-dimensional point group whose coordinates have been transformed by the coordinate transformation unit 110, and generates an image as seen from the position of a virtual second imaging device.

[0034] The image completion unit 112 completes missing pixels, which are pixels that do not have a projection source pixel, among the pixels of the image generated by the new image generation unit 111. For example, the image completion unit 112 can complete the missing pixel by using pixel values ​​of pixels surrounding the missing pixel.

[0035] 2 is a diagram illustrating an example of processing executed by the image processing device 10. The processing illustrated in FIG. 2 is processing for training the absolute depth image estimation model 102 and the parameter estimation model 103.

[0036] In step S1, the relative depth image generation unit 101 generates a relative depth image of the preceding captured image. In step S2, the absolute depth image estimation model 102 generates an absolute depth image of the preceding captured image. In step S3, the parameter estimation model 103 generates a change amount parameter using the preceding captured image and the subsequent captured image. In step S4, the image reconstruction unit 104 reconstructs the preceding captured image using the absolute depth image generated in step S2, the parameters generated in step S3, and color information of the subsequent captured image.

[0037] In step S5, the first loss calculation unit 105 calculates a first loss L1 using the previous captured image and the previous captured image reconstructed in step S4. In step S6, the second loss calculation unit 106 calculates a second loss L2 using the relative depth image of the previous captured image generated in step S1 and the absolute depth image generated in step S2. In step S7, the third loss calculation unit 107 calculates a third loss L3 using the actual distance between the position of the image capture device when the previous captured image was generated and the position of the image capture device when the subsequent captured image was generated, and the distance of the image capture device indicated by the change amount parameter generated in step S3.

[0038] In step S8, the learning processing unit 108 trains the absolute depth image estimation model 102 and the parameter estimation model 103 to reduce the first loss L1, the second loss L2, and the third loss L3, and the processing of Figure 2 ends.

[0039] Fig. 3 is a diagram showing another example of processing executed by the image processing device 10. In the processing shown in Fig. 3, the absolute depth image estimation model 102 and the parameter estimation model 103 trained by the above-described method are used.

[0040] In step S11, the trained absolute depth image estimation model 102 generates an absolute depth image of the preceding captured image. In step S12, the 3D point cloud generation unit 109 uses the preceding captured image and the absolute depth image of the preceding captured image generated in step S11 to inversely project color information of the preceding captured image into a 3D space, thereby generating a 3D point cloud.

[0041] In step S13, the coordinate conversion unit 110 uses the arbitrary parameters described above to convert the coordinates of the three-dimensional point group generated in step S12 into coordinates as viewed from the position of the virtual second imaging device.

[0042] In step S14, the new image generation unit 111 projects the 3D point cloud whose coordinates have been transformed by the coordinate transformation unit 110, and generates an image as viewed from the position of the virtual second imaging device. In step S15, the image completion unit 112 completes the missing pixels, and the processing in FIG. 3 ends.

[0043] In the above-described embodiment, an absolute depth image estimation model, which uses a captured image as input information and an absolute depth image, which is a depth image having distance information that can be expressed in units of length, as output information, is trained to reduce the first loss, the second loss, and the third loss, thereby improving the estimation accuracy of the depth image having distance information that can be expressed in units of length.

[0044] 5 is a diagram showing an application example using a parameter estimation model trained by the above-described learning method. In this example, the image processing device 10 corresponds to an estimation system that estimates the amount of movement of a moving object. The parameter estimation model 103 outputs change amount parameters corresponding to a preceding captured image and a subsequent captured image generated by an image capture device mounted on the moving object. The amount of change in the position of the image capture device indicated by the change amount parameters represents the amount of movement of the moving object.

[0045] The example in Figure 5 shows a self-propelled robot, which is a moving object. The distance traveled by the self-propelled robot between time t = 0 and time t = 1 is obtained by inputting the image captured at time t = 0 as the preceding image and the image captured at time t = 1 as the succeeding image into parameter estimation model 103. Note that parameter estimation model 103 also outputs a change amount parameter that indicates the amount of change in the posture of the camera provided on the self-propelled robot between time t = 0 and time t = 1.

[0046] Similarly, the distance traveled by the self-propelled robot between time t=1 and time t=2 can be obtained by inputting the image captured at time t=1 as the preceding image and the image captured at time t=2 as the succeeding image into parameter estimation model 103. Parameter estimation model 103 also outputs a change amount parameter that indicates the amount of change in the posture of the camera provided on the self-propelled robot between time t=1 and time t=2.

[0047] Fig. 6 is a diagram showing another application example using a parameter estimation model trained by the above-described training method. In the example shown in Fig. 6, three image capturing devices 61, 62, and 63 are fixed to a structure.

[0048] In this example, the image captured by the camera 61 is input as the preceding image and the image captured by the camera 62 is input as the succeeding image to the parameter estimation model 103, whereby a change parameter indicating the distance between the camera 61 and the camera 62 and a change parameter indicating the amount of change in the attitude of the camera 62 based on the attitude of the camera 61 are output.

[0049] Similarly, by inputting the image captured by the camera 61 as the preceding image and the image captured by the camera 63 as the succeeding image into the parameter estimation model 103, a change parameter indicating the distance between the camera 61 and the camera 63 and a change parameter indicating the amount of change in the attitude of the camera 63 based on the attitude of the camera 61 are output.

[0050] Similarly, by inputting the image captured by the camera 62 as the preceding image and the image captured by the camera 63 as the succeeding image into the parameter estimation model 103, a change parameter indicating the distance between the camera 62 and the camera 63 and a change parameter indicating the amount of change in the attitude of the camera 63 based on the attitude of the camera 62 are output.

[0051] In this way, it is possible to obtain the distances between the three image capturing devices 61, 62, and 63. In addition, it is possible to calculate the amount of change in the attitude of the three image capturing devices 61, 62, and 63, and it is possible to grasp the image capturing directions of these image capturing devices 61, 62, and 63.

[0052] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disk (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0053] The present disclosure is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit of the present disclosure. For example, in the above-described embodiment, a single image processing device 10 executes the above-described program, but in other embodiments, multiple individual image processing devices may execute the above-described program in a distributed manner. [Explanation of symbols]

[0054] 10: Image processing device / learning system, 100: Processor, 101: Relative depth image generation unit, 102: Absolute depth image estimation model, 103: Parameter estimation model, 104: Image reconstruction unit, 105: First loss calculation unit, 106: Second loss calculation unit, 107: Third loss calculation unit, 108: Learning processing unit, 109: 3D point cloud generation unit, 110: Coordinate conversion unit, 111: New image generation unit, 112: Image completion unit, 120: Communication interface, 130: Storage device, 61 to 63: Image capture device

Claims

1. A learning system for learning an estimation model by machine learning, an absolute depth image estimation model that takes a captured image as input information and outputs an absolute depth image, which is a depth image having distance information that can be expressed in units of length; a parameter estimation model that receives as input information a preceding captured image and a subsequent captured image that is temporally consecutive to the preceding captured image, and that receives as output information change amount parameters that represent the position and orientation of the image capture device when the preceding captured image was captured and the amount of change in the position and orientation of the image capture device when the subsequent captured image was captured; an image reconstruction unit that reconstructs the preceding captured image using an absolute depth image corresponding to the preceding captured image output by the absolute depth image estimation model, change amount parameters corresponding to the preceding captured image and the subsequent captured image output by the parameter estimation model, and color information of the subsequent captured image; the absolute depth image estimation model and the parameter estimation model are trained to reduce a first loss, a second loss, and a third loss; the first loss is a photometric error between the previous captured image that has not been reconstructed by the image reconstruction unit and the previous captured image that has been reconstructed by the image reconstruction unit; the second loss is a loss based on geometric consistency between a relative depth image, which is a depth image of the preceding captured image and has distance information not expressed in units of length, and an absolute depth image corresponding to the preceding captured image output by the absolute depth image estimation model; the third loss is a difference between a distance between a position of the image capturing device when the preceding image is captured and a position of the image capturing device when the subsequent image is captured, and a distance corresponding to a change in the position of the image capturing device indicated by a change amount parameter corresponding to the preceding image and the subsequent image output by the parameter estimation model; Learning system.

2. A learning method for learning an estimation model by machine learning, comprising: outputting an absolute depth image using an absolute depth image estimation model that uses a photographed image as input information and an absolute depth image, which is a depth image having distance information that can be expressed in units of length, as output information; a parameter estimation model is used in which a preceding photographed image and a subsequent photographed image that is temporally consecutive to the preceding photographed image are used as input information, and a change amount parameter that represents a change amount between the position and orientation of the photographing device when the preceding photographed image was taken and the position and orientation of the photographing device when the subsequent photographed image was taken is used as output information, and the change amount parameter is output; reconstructing the preceding captured image using an absolute depth image corresponding to the preceding captured image output using the absolute depth image estimation model, change amount parameters corresponding to the preceding captured image and the subsequent captured image output using the parameter estimation model, and color information of the subsequent captured image; training the absolute depth image estimation model and the parameter estimation model so as to reduce the first loss, the second loss, and the third loss; the first loss is a photometric error between the non-reconstructed previous captured image and a reconstructed previous captured image; the second loss is a loss based on geometric consistency between a relative depth image, which is a depth image of the preceding captured image and has distance information not expressed in units of length, and an absolute depth image corresponding to the preceding captured image and output using the absolute depth image estimation model; the third loss is a difference between a distance between a position of the image capturing device when the preceding image is captured and a position of the image capturing device when the subsequent image is captured, and a distance corresponding to a change in the position of the image capturing device indicated by a change amount parameter corresponding to the preceding image and the subsequent image output using the parameter estimation model; How to learn.

3. A learning program for learning an estimation model by machine learning, the learning program comprising: a step of outputting an absolute depth image using an absolute depth image estimation model that uses a captured image as input information and an absolute depth image, which is a depth image having distance information that can be expressed in units of length, as output information; a step of using a parameter estimation model in which a preceding photographed image and a subsequent photographed image that is temporally consecutive to the preceding photographed image are used as input information, and in which output information is a change amount parameter that represents a change amount between the position and orientation of the photographing device when the preceding photographed image was taken and the position and orientation of the photographing device when the subsequent photographed image was taken, and outputting the change amount parameter; reconstructing the preceding captured image using an absolute depth image corresponding to the preceding captured image output using the absolute depth image estimation model, change amount parameters corresponding to the preceding captured image and the subsequent captured image output using the parameter estimation model, and color information of the subsequent captured image; training the absolute depth image estimation model and the parameter estimation model so as to reduce a first loss, a second loss, and a third loss; Execute the first loss is a photometric error between the non-reconstructed previous captured image and a reconstructed previous captured image; the second loss is a loss based on geometric consistency between a relative depth image, which is a depth image of the preceding captured image and has distance information not expressed in units of length, and an absolute depth image corresponding to the preceding captured image and output using the absolute depth image estimation model; the third loss is a difference between a distance between a position of the image capturing device when the preceding image is captured and a position of the image capturing device when the subsequent image is captured, and a distance corresponding to a change in the position of the image capturing device indicated by a change amount parameter corresponding to the preceding image and the subsequent image output using the parameter estimation model; Learning program.

4. an absolute depth image estimation model that takes a captured image as input information and outputs an absolute depth image, which is a depth image having distance information that can be expressed in units of length; a three-dimensional point cloud generation unit that generates a three-dimensional point cloud by inversely projecting color information of the captured image into a three-dimensional space using the captured image input to the absolute depth image estimation model and the absolute depth image output from the absolute depth image estimation model; a coordinate conversion unit that converts the coordinates of the three-dimensional point cloud generated by the three-dimensional point cloud generation unit into coordinates as viewed from the position of the virtual second camera device, using a parameter indicating the distance between a first camera device that generated the captured image and a virtual second camera device that is located at a different shooting position from the first camera device, and a parameter indicating the amount of change in the attitude of the virtual second camera device relative to the attitude of the first camera device; a new image generation unit that projects the three-dimensional point cloud whose coordinates have been transformed by the coordinate transformation unit and generates an image as viewed from the position of the virtual second imaging device; 1. An image generation system comprising:

5. An estimation system for estimating a movement amount of a moving object, a parameter estimation model trained by the training method according to claim 2; the parameter estimation model outputs change amount parameters corresponding to the preceding captured image and the subsequent captured image generated by an image capturing device mounted on a moving object; The amount of change in the position of the image capturing device indicated by the change amount parameter represents the amount of movement of the moving object. Estimation system.

Images