Program, generation method and information processing device

Abstract

To generate an estimation model for estimating a depth from an image with higher accuracy.SOLUTION: A program causes a computer to execute a depth calculation step, a motion calculation step, a correspondence calculation step, and a learning step. The learning step updates parameters of a first estimation model and a second estimation model so as to optimize a first loss function, a second loss function, and a third loss function, and generates the first estimation model and the second estimation model represented by the updated parameters, wherein the first loss function includes a term indicating a difference between correspondence information and corresponding training data which is training data regarding a correspondence between a first pixel and a second pixel, the second loss function includes a term regarding a depth, and the third loss function includes a term indicating a difference in pixel values between the first pixel and the second pixel in which the correspondence is indicated by the correspondence information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to a program, a generation method, and an information processing device. [Background technology]

[0002] A technology has been proposed that uses brightness differences between images based on the geometric relationship between depth and egomotion (camera motion) to train a neural network that estimates depth and a neural network that estimates egomotion.

[0003] In such techniques, for example, egomotion models only the motion of stationary backgrounds captured in an image, and does not take moving objects into account. As a result, for example, a neural network that estimates depth may not be able to accurately estimate the depth (distance) to a moving object. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2021-503134 [Non-patent literature]

[0005] [Non-Patent Document 1] Zhou, Tinghui, et al. "Unsupervised learning of depth and ego-motion from video." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention aims to provide a program, a generation method, and an information processing device that can generate an estimation model for estimating depth from an image with higher accuracy. [Means for solving the problem]

[0007] A program according to an embodiment causes a computer to execute a depth calculation step, a motion calculation step, a correspondence calculation step, and a learning step. The learning step updates parameters of the first estimation model and the second estimation model to optimize a first loss function including a term indicating a difference between the correspondence information and correspondence training data, which is training data regarding the correspondence between a first pixel and a second pixel, a second loss function including a term regarding depth, and a third loss function including a term indicating a difference in pixel value between the first pixel and the second pixel, whose correspondence is indicated by the correspondence information, and generates the first estimation model and the second estimation model represented by the updated parameters. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating a configuration of an information processing apparatus according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of a depth calculation unit. [Figure 3] FIG. 2 is a block diagram showing an example of the configuration of a motion calculation unit. [Figure 4] 10A and 10B are diagrams showing a specific example of correspondence calculation processing by a correspondence calculation unit. [Figure 5] 10 is a flowchart of a learning process according to an embodiment. [Figure 6] 10 is a flowchart of an estimation process according to an embodiment. [Figure 7] FIG. 1 is a hardware configuration diagram of an information processing apparatus according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of a program, a generating method, and an information processing device according to the present invention will be described in detail below with reference to the accompanying drawings.

[0010] The information processing device according to the embodiment models not only static objects such as the background in an image but also the motion of moving objects to learn an estimation model for estimating depth, thereby enabling highly accurate depth estimation using the estimation model.

[0011] In this embodiment, at least the following two models are used. Estimation model MA: A model (first estimation model) that receives an input image and outputs depth information including multiple depths for multiple pixels contained in the input image. Estimation model MB: A model (second estimation model) that inputs two pieces of depth information obtained from two input images and outputs motion information representing the movement in three-dimensional space for each of multiple pixels.

[0012] The estimation models are trained (generated, constructed) by updating the parameters of the two estimation models so as to optimize the loss functions for the two estimation models described above. Since the estimation model MA that estimates depth can be trained together with the estimation model MB that models the motion of the moving object, it is possible to generate the estimation model MA with higher accuracy.

[0013] In this embodiment, the camera parameters of the image capture device are learned as learnable parameters along with the parameters of the estimation model. Because the camera parameters are obtained through learning, prior calibration to obtain the camera parameters is not required. The camera parameters include, for example, at least one of the focal length of the image capture device, the principal point position of the image capture device, and the distortion coefficient of the image capture device.

[0014] 1 is a block diagram showing an example of the configuration of an information processing device 100 according to an embodiment. As shown in FIG. 1, the information processing device 100 includes a receiving unit 101, a depth calculation unit 111, a motion calculation unit 112, a correspondence calculation unit 113, a learning unit 121, an estimation unit 122, an output control unit 102, and a storage unit 130.

[0015] The receiving unit 101 receives input of various information used in the information processing device 100. For example, the receiving unit 101 receives input of an input image used for learning or estimation, and training data used for learning.

[0016] The input images are, for example, a plurality of images captured at a plurality of times by a camera (imaging device). The plurality of images include, for example, an input image IA (first input image) and an input image IB (second input image) captured at a different time from the input image IA. The camera is, for example, a monocular camera.

[0017] The training data includes depth training data, which is training data about depth, and correspondence training data, which is training data about the correspondence between pixels in two images. The depth training data may be obtained by any method, for example, a method using a system that estimates depth from a single image, or a method using a system that estimates depth from multiple images. The correspondence training data may be obtained by any method, for example, a method using a system that estimates correspondence based on optical flow, or a method using a system that estimates correspondence based on corresponding points.

[0018] The depth calculation unit 111 calculates depth information of the input image using the estimation model MA. For example, the depth calculation unit 111 inputs the input image IA and the input image IB to the estimation model MA, and obtains depth information DA (first depth information) for the input image IA and depth information DB (second depth information) for the input image IB.

[0019] The estimation model MA may be a model that outputs the reliability of the depth information together with the depth information. The reliability is used, for example, in the learning process by the learning unit 121. Hereinafter, the reliability of the depth information DA may be referred to as reliability RA (first reliability), and the reliability of the depth information DB may be referred to as reliability RB (second reliability).

[0020] The reliability indicates whether the depth estimated for each pixel of the input image is reliable. For example, if the estimated depth is reliable, a larger value is calculated as the reliability than if it is unreliable. For example, when estimating the depth of an image captured in an urban area, background pixels exist around the outlines of moving objects such as automobiles and pedestrians, so pixels corresponding to the entire moving object are likely to be mistakenly recognized as background. In other words, the depth of the pixel corresponding to the entire moving object is estimated to be a value that is farther away than it actually is. In such cases, a small value (a value indicating unreliability) is calculated as the reliability.

[0021] The motion calculation unit 112 calculates motion information for each of a plurality of pixels included in the input image using two pieces of depth information calculated for the two input images. For example, the motion calculation unit 112 inputs the depth information DA and the depth information DB to the estimation model MB to obtain the motion information.

[0022] The estimation models MA and MB may be models of any structure, for example, a neural network model (hereinafter simply referred to as a neural network) or a machine learning model such as a random forest. Neural networks include, for example, a convolutional neural network, a fully connected neural network, a recurrent neural network, and a transformer.

[0023] In the following, we will mainly explain an example in which the estimation models MA and MB are neural networks. Parameters updated by neural network learning include, for example, weights (weight coefficients) and biases. In the following, the parameters of the estimation model MA are referred to as parameters PA, and the parameters of the estimation model MB are referred to as parameters PB.

[0024] Neural networks can be trained by any method, for example, gradient descent. Gradient descent uses a differentiated value of the loss function. Therefore, each model and loss function is represented by a differentiable function.

[0025] The correspondence calculation unit 113 uses the depth information DA and depth information DB, motion information, and camera parameters of the imaging device to calculate correspondence information indicating the correspondence between pixel PXA (first pixel) included in the input image IA and pixel PXB (second pixel) included in the input image IB from the geometric relationship between these pieces of information.

[0026] Here, the functions of the depth calculation unit 111, the motion calculation unit 112, and the correspondence calculation unit 113 will be described in detail.

[0027] 2 is a block diagram showing an example of the configuration of the depth calculation unit 111. As shown in FIG. 2, the depth calculation unit 111 inputs an input image 201 to an estimation model MA, and calculates a depth 221 and a reliability 222. The input image 201 is, for example, an image captured at time t. In the following, the input image captured at time t is referred to as an input image I t and the input image I t The depth and confidence calculated from t and confidence level σ t This is sometimes the case.

[0028] 2, the estimation model MA used by the depth calculation unit 111 includes an encoder 211 and a decoder 212. The encoder 211 receives an input image 201 and calculates a low-resolution feature map 211a that indicates features of the input image 201. The decoder 212 receives the low-resolution feature map 211a and calculates a high-resolution feature map 212a that has the same resolution as the input image. The estimation model MA calculates a depth 221 and a reliability 222 from the feature map 212a. The depth calculation unit 111 uses this estimation model MA to calculate the depth 221 and the reliability 222 for the input image 201.

[0029] The depth calculation unit 111 calculates and outputs the depth and reliability for each of the two input images received by the reception unit 101, for example, through the above-described processing.

[0030] The encoder 211 and the decoder 212 can also be interpreted as corresponding to the models included in the estimation model MA. The encoder 211 and the decoder 212 are also represented by differentiable functions.

[0031] The depth calculation unit 111 may calculate the depth and reliability for the input image 201 by dividing the input image 201 into multiple divided images and combining the multiple depths and multiple reliability values ​​obtained by applying the estimation model MA to each of the multiple divided images.

[0032] Next, a detailed description will be given of the motion calculation unit 112. Fig. 3 is a block diagram showing an example of the configuration of the motion calculation unit 112.

[0033] The motion calculation unit 112 uses the estimation model MB to calculate three-dimensional motion 312 for each of the plurality of pixels included in the input image 201 and the input image 301. Information including the three-dimensional motion 312 of the plurality of pixels corresponds to motion information.

[0034] The motion calculation unit 112 inputs the input image 201, the input image 301, the depth 221, the depth 321, and the camera parameters 311 to the estimation model MB. The input image 301 is, for example, an image captured at time t-1. The depth 321 is a depth calculated for the input image 301. In the following, the input image captured at time t-1 is referred to as the input image I. t-1 and the input image I t-1 The depth (depth 321) and reliability calculated from t-1 and confidence level σ t-1 This is sometimes the case.

[0035] The estimation model MB calculates and outputs a three-dimensional motion 312 for moving to the viewpoint of the input image 201 for each pixel of the input image 301. For example, the motion calculation unit 112 converts the pixel-by-pixel three-dimensional motion 312 into an optical flow, aligns the viewpoints of the input image 201 and the input image 301, and then inputs the input data (input image 201, input image 301, depth 221, and depth 321) back into the estimation model MB to newly calculate the pixel-by-pixel three-dimensional motion 312.

[0036] The estimation model MB may share some layers (and the parameters of those layers) with the estimation model MA.

[0037] The motion calculation unit 112 is assumed to be capable of repeating the process of estimating pixel-by-pixel three-dimensional motion 312, converting it to optical flow, aligning the viewpoint, and estimating a new pixel-by-pixel three-dimensional motion 312 multiple times.

[0038] In addition, in order to reduce the amount of calculation, the motion calculation unit 112 may, during the calculation process, calculate three-dimensional motion in pixel units at a lower resolution from the input data (input image 201, input image 301, depth 221, and depth 321), and calculate the data obtained by upsampling the low-resolution three-dimensional motion to the same resolution as the input image as the three-dimensional motion 312 to be finally output.

[0039] Next, a description will be given of the details of the correspondence calculation unit 113. Fig. 4 is a diagram showing a specific example of correspondence calculation processing by the correspondence calculation unit 113.

[0040] The correspondence calculation unit 113 calculates the correspondence between the input image 201 and the input image 301 from the geometric relationship between the depth calculated by the depth calculation unit 111, the pixel-by-pixel three-dimensional motion 411 calculated by the motion calculation unit 112, and the set camera parameters.

[0041] Specifically, the correspondence calculation unit 113 first projects a straight line 402 from an imaging position 401 into a three-dimensional space based on the camera parameters, and back-projects a pixel 403 into the three-dimensional space.

[0042] Next, the correspondence calculation unit 113 determines the end position of the line 402 based on the depth, thereby generating a position 404 in three-dimensional space corresponding to a pixel 403 of the input image 301 captured from the viewpoint of the imaging position 401.

[0043] Next, the correspondence calculation unit 113 uses the three-dimensional motion 411 of the pixel 403 to move the position 404 in the three-dimensional space to a position 424 in the three-dimensional space.

[0044] Next, the correspondence calculation unit 113 projects a position 424 in three-dimensional space along a straight line 422 onto a pixel 423 of the input image 201 captured from the viewpoint of the imaging position 421 based on the set camera parameters.

[0045] In this way, correspondence information is calculated that associates the pixel 403 with the pixel 423. Note that the difference in position between the pixel 423 and the pixel 403 is the optical flow. The correspondence information may include the optical flow.

[0046] The correspondence calculation process can be formulated by the following equation (1).

number

[0047] x t-1 are the coordinates of the pixel 403, and the function π -1 () is the back projection based on the camera parameters, D(x t-1 ) is the depth corresponding to pixel 403, T t-1→t represents the three-dimensional motion 411 of the pixel 403, and the function π() represents the projection based on the camera parameters. t-1 )·π -1 (x t-1 ) is a function that calculates the position 404 in three-dimensional space, T t-1→t D(xt-1 )·π -1 (x t-1 ) is a function that calculates the position 424 in three-dimensional space, π(T t-1→t D(x t-1 )·π -1 (x t-1 )) is a function that calculates the coordinates of pixel 423.

[0048] Returning to the explanation of Fig. 1, the learning unit 121 executes a learning process for the estimation models MA and MB. For example, the learning unit 121 updates the parameters of the estimation models MA and MB so as to optimize a loss function as shown below, and generates the estimation models MA and MB represented by the updated parameters. Loss function LA (first loss function) including a term indicating the difference between the correspondence information and the corresponding training data. The correspondence training data is training data regarding the correspondence between pixel PXA and pixel PXB. Loss function LB (second loss function) that includes a term for depth. A loss function LC (third loss function) that includes a term indicating the difference in pixel value between pixel PXA and pixel PXB, whose correspondence is indicated by the correspondence information.

[0049] Through the learning process, the depth calculation unit 111 (estimation model MA), the motion calculation unit 112 (estimation model MB), and the camera parameters can be adapted to the target scene. The loss function will be described in detail later.

[0050] The estimation unit 122 executes estimation processing using the estimation model learned by the learning unit 121. The estimation processing may be used for any purpose, but may be applied to, for example, a technology for calculating the distance from an imaging position to a subject using an image captured by an imaging device, and controlling a moving body (a vehicle such as an automobile, a mobile robot, etc.) using the calculated distance. The imaging device may be mounted on the moving body to be controlled.

[0051] The estimation unit 122 may perform an estimation process using one of the estimation model MA and the estimation model MB. For example, the estimation unit 122 may perform an estimation process to estimate the depth of an input image using the estimation model MA. Since the estimation model MA is trained together with the estimation model MB, the depth can be estimated with higher accuracy.

[0052] The output control unit 102 controls the output of various information used in the information processing device 100. For example, the output control unit 102 stores parameters (parameters PA, parameters PB) of each model (estimation model MA, estimation model MB) obtained by the learning process in the storage unit 130, or outputs the parameters to an external device (such as an estimation device) that executes processing using each model. In addition, the output control unit 102 displays the result of the estimation process by the estimation unit 122 on a display device such as a display, or transmits the result to an external device connected via a network.

[0053] At least a part of each of the above units (reception unit 101, depth calculation unit 111, motion calculation unit 112, correspondence calculation unit 113, learning unit 121, estimation unit 122, and output control unit 102) may be realized by one or more processing units. Each of the above units is realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) execute a program, that is, by software. Each of the above units may be realized by a processor such as a dedicated IC (Integrated Circuit), that is, by hardware. Each of the above units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or may realize two or more of the units.

[0054] The storage unit 130 stores various information used in the information processing device. For example, the storage unit 130 stores an input image 131, depth teacher data 132, corresponding teacher data 133, an estimation model 134, and camera parameters 135. The estimation model 134 includes an estimation model MA and an estimation model MB.

[0055] The storage unit 130 can be configured from any commonly used storage medium, such as a flash memory, a memory card, a RAM (Random Access Memory), an HDD (Hard Disk Drive), or an optical disk.

[0056] At least a portion of each data stored in the memory unit 130 (input image 131, depth teacher data 132, corresponding teacher data 133, estimation model 134, and camera parameters 135) may be stored in physically different storage media, or may be stored in different storage areas of the same physically stored medium.

[0057] The information processing device 100 may be physically configured as one device, or may be physically configured as multiple devices. For example, the information processing device 100 may be constructed in a cloud environment. Furthermore, each unit in the information processing device 100 may be distributed across multiple devices. For example, the information processing device 100 (information processing system) may be configured to include a device (e.g., a learning device) having functions required for learning processing (e.g., the learning unit 121) and a device (e.g., an estimation device) having functions required for estimation processing using the trained estimation model 134 (e.g., the estimation unit 122).

[0058] Next, specific examples of each loss function will be described. First, an example of the loss function LA will be described.

[0059] The learning unit 121 updates the parameters PA (parameters of the estimation model MA), the parameters PB (parameters of the estimation model MB), and the camera parameters, for example, using a loss function LA so as to reduce the difference in luminance values ​​(pixel values) of corresponding pixels (hereinafter referred to as corresponding pixels) in the two input images.

[0060] The magnitude of the increase (decrease) when updating each parameter may be proportional to the absolute value of the differential coefficient.To avoid sudden fluctuations, an upper limit may be set on the range of parameter fluctuations.

[0061] Specifically, the learning unit 121 calculates a loss function L as shown in the following equation (2), for example: pho The parameters PA, PB, and camera parameters are updated so that (an example of a loss function LA) is minimized.

number

[0062] I t-1 and t are the input images IA and IB, respectively. p is l t-1 pixels, F 2d t-1→t is the optical flow from input image IA to input image IB, and the function d() represents the difference in brightness. Also, p and p+F 2d t-1→t is the corresponding pixel.

[0063] The function d() may be any function capable of calculating the difference in luminance values. For example, the function d() may be a function that calculates the difference in luminance values ​​based on a geometric distance concept such as L1 distance and L2 distance, or a function that calculates perceptual similarity such as PSNR and SSIM as the difference in luminance values. The function d() may be a function that combines multiple calculation methods. For example, the function d() may be expressed as a weighted sum of multiple functions corresponding to multiple calculation methods.

[0064] Next, an example of the loss function LB will be described. The learning unit 121 updates the parameter PA so that the lower the reliability calculated by the depth calculation unit 111, the closer the depth calculated by the depth calculation unit 111 is to the depth teacher data (so that the higher the reliability calculated by the depth calculation unit 111, the less the depth calculated by the depth calculation unit 111 is to the depth teacher data).

[0065] Specifically, the learning unit 121 updates the parameter PA using, for example, a loss function such as the following equation (3) (an example of a loss function LB) so that the absolute difference between the depth and the depth teacher data is minimized according to the reliability.

number

[0066] D and D T represent the depth and depth training data, respectively. σ represents the confidence level corresponding to the depth D. Note that D and D T Since the real scale is indefinite, it is normalized by the following equation (4).

number

[0067] The function median() calculates the median value of the depth of all pixels, and the function mean() calculates the average value of the depth of all pixels. T The same calculation applies to Equation (4) only applies to Equation (3).

[0068] The loss function in equation (3) is based on the depth information DA and depth information DB (depth D), and the depth training data (depth D T ) and corresponds to a function that includes a term indicating that the greater the reliability RA and reliability RB(σ), the greater the loss. By using such a loss function, it becomes possible to filter out pixels with low reliability.

[0069] Furthermore, the learning unit 121 may update the parameter PA using the context of pixels obtained from the depth training data, for example, using a loss function such as the following equation (5) (an example of a loss function LB) so that the distance between pixels that are separated by a certain threshold or more is maximized.

number

[0070] p i and p j are pixels with different depths D, which are randomly sampled. δ is a hyperparameter (an example of a specified value) that controls the threshold in the range [0, 1]. Note that l is the pixel context obtained from the depth training data, and is formulated by the following equation (6).

number

[0071] D T is normalized by the following formula (7): Note that formula (7) is only applicable to formula (6).

number

[0072] According to equation (7), D T is normalized to the range [0,1]. This makes it easier to set the hyperparameter δ. For example, if δ is set to 0.1, then pixel p i and pixel p j When the distance is more than 10%, D(p i ) and D(p j ) is maximized.

[0073] The loss function of equation (5) corresponds to a function including a term indicating the depth difference between two pixels whose depth difference in the depth training data is greater than a specified value.

[0074] The learning unit 121 may use either one of Equation (3) and Equation (5), or a combination of both. In the latter case, the learning unit 121 may use, for example, a loss function LB expressed as a weighted sum of Equation (3) and Equation (5).

[0075] Next, an example of the loss function L C will be described. The learning unit 121 updates the parameters PA, PB, and camera parameters so that the correspondence calculated by the correspondence calculation unit 113 becomes closer to the corresponding training data.

[0076] Specifically, the learning unit 121 updates the parameters PA, PB, and camera parameters so that a loss function (an example of a loss function LC) such as the following equation (8) is minimized.

number

[0077] F 2d t-1→t is the optical flow calculated by the correspondence calculation unit 113 for the input images IA and IB. 2d,T t-1→t is the optical flow for the input image IA and the input image IB included in the corresponding training data. 2d t-1→t And, F 2d,T t-1→t The difference is F 2d t-1→t And, F 2d,T t-1→t The distance may be calculated in any manner, but for example, a calculation method based on a geometric distance concept such as L1 distance or L2 distance can be applied.

[0078] As described above, each loss function is represented by a differentiable function. Therefore, the learning unit 121 can differentiate the loss function with the parameters PA, PB, and camera parameters to determine a direction in which the loss function decreases. The learning unit 121 updates the parameters PA, PB, and camera parameters by repeating a process of varying the parameters PA in the determined direction. As a result of such updating, an estimated model MA represented by the parameters PA and an estimated model MB represented by the parameters PB are generated.

[0079] The learning unit 121 updates the three parameters PA, PB, and PC by a learning method such as gradient descent so as to optimize a loss function that includes all of the three loss functions LA, LB, and LC. This allows the learning of an estimation model MA that estimates depth together with an estimation model MB that models the motion of a moving object, making it possible to generate an estimation model MA with higher accuracy.

[0080] The learning unit 121 may further update at least some of the parameters of each generated model (estimation model MA, estimation model MB) using only some of the three loss functions LA, LB, and LC. That is, the learning unit 121 may have a function of updating the parameters of the estimation model MA and the estimation model MB so as to optimize one or two of the loss function LB, loss function LC, and loss function LA.

[0081] For example, the learning unit 121 uses the loss function L pho The learning unit 121 may update the parameters PA, PB, and the camera parameters so as to optimize the loss function of equation (3). The learning unit 121 may update the parameters PA, PB, and the camera parameters so as to optimize the loss function of equation (8).

[0082] Next, a description will be given of the learning process performed by the information processing apparatus 100 according to the embodiment. Fig. 5 is a flowchart showing an example of the learning process according to the embodiment.

[0083] The receiving unit 101 receives two input images IA and IB from an imaging device (step S101).

[0084] Using the input images, depth (depth information), three-dimensional motion (motion information), and correspondence (correspondence information) are calculated (step S102). For example, the depth calculation unit 111 calculates the depth and reliability for each of the two input images. The motion calculation unit 112 calculates the three-dimensional motion for each pixel for the two input images. The correspondence calculation unit 113 calculates the correspondence between the two input images from a geometric relationship using the calculated depth, the calculated three-dimensional motion for each pixel, and set camera parameters.

[0085] The learning unit 121 uses the calculation results (depth information, motion information, correspondence information) in step S102 and the training data to update the parameters PA, PB, and camera parameters so as to optimize the loss function (step S103).

[0086] The learning unit 121 determines whether to end the learning (step S104). For example, the learning unit 121 determines to end the learning when the value of the loss function becomes smaller than a threshold (threshold of the function value) or when the number of iterations exceeds a threshold (threshold of the number of iterations).

[0087] If it is determined that the learning should not be ended (step S104: No), the process returns to step S101 and is repeated. If it is determined that the learning should be ended (step S104: Yes), the learning process ends.

[0088] Next, a description will be given of the estimation process performed by the information processing apparatus 100 according to the embodiment. Fig. 6 is a flowchart showing an example of the estimation process according to the embodiment.

[0089] The receiving unit 101 receives an input image to be estimated from an imaging device (step S201). The estimation unit 122 estimates (calculates) depth (depth information), three-dimensional motion (motion information), and correspondence (correspondence information) for the received input image using trained models (estimation model MA, estimation model MB) (step S202). Note that the estimation unit 122 may estimate part of this information.

[0090] The output control unit 102 outputs the estimation result by the estimation unit 122 (step S203), and ends the estimation process.

[0091] In this way, the information processing device of the embodiment models not only the motion (three-dimensional motion) of stationary objects such as the background in an image but also the motion of moving objects, and performs learning by applying geometric constraints to the estimation model that estimates depth. This makes it possible to bring the distance (depth) to a subject, including a moving object, closer to a desired value. As a result, for example, the operation of the information processing device 100 can be adapted according to the subject captured in the image, thereby improving the accuracy of estimating the distance (depth) to the subject.

[0092] For example, a technology using egomotion that models only a static background cannot adapt the depth estimation operation to a moving object or other subject. For example, if a moving object or other subject is not modeled in a captured image, the moving object or other subject may be erroneously recognized as a static background. As a result, the estimated distance to the moving object or other subject may differ from the actual distance.

[0093] Next, the hardware configuration of the information processing apparatus according to the embodiment will be described with reference to Fig. 7. Fig. 7 is an explanatory diagram illustrating an example of the hardware configuration of the information processing apparatus according to the embodiment.

[0094] The information processing device of the embodiment includes a control device such as a CPU (Central Processing Unit) 51, a storage device such as a ROM (Read Only Memory) 52 and a RAM (Random Access Memory) 53, a communication I / F 54 that connects to a network and communicates, and a bus 61 that connects each part.

[0095] The programs executed by the information processing apparatus according to the embodiment are provided in advance in the ROM 52 or the like.

[0096] The program executed by the information processing device of the embodiment may be configured to be provided as a computer program product by being recorded in an installable or executable format on a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), or a DVD (Digital Versatile Disk).

[0097] Furthermore, the program executed by the information processing apparatus of the embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the program executed by the information processing apparatus of the embodiment may be provided or distributed via a network such as the Internet.

[0098] The programs executed by the information processing device of the embodiment can cause a computer to function as each of the above-mentioned parts of the information processing device. In this computer, the CPU 51 can read the programs from a computer-readable storage medium onto a main storage device and execute them.

[0099] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0100] 100 Information processing device 101 Reception 102 Output control section 111 Depth calculation section 112 Motion calculation unit 113 Correspondence calculation unit 121 Learning Department 122 Estimation Department 130 Storage section

Claims

1. On the computer, a depth calculation step of inputting a first input image and a second input image captured by an imaging device into a first estimation model that receives an input image and outputs depth information including a plurality of depths for a plurality of pixels included in the input image, and obtaining first depth information for the first input image and second depth information for the second input image; a motion calculation step of inputting the first depth information and the second depth information into a second estimation model that receives the two pieces of depth information and outputs motion information representing a movement in three-dimensional space for each of a plurality of pixels, thereby obtaining the motion information; a correspondence calculation step of calculating correspondence information indicating a correspondence between a first pixel included in the first input image and a second pixel included in the second input image using the first depth information, the second depth information, the motion information, and camera parameters of the imaging device; a learning step of updating parameters of the first estimation model and the second estimation model so as to optimize a first loss function including a term indicating a difference between the correspondence information and corresponding training data that is training data regarding the correspondence between the first pixel and the second pixel, a second loss function including a term regarding depth, and a third loss function including a term indicating a difference in pixel value between the first pixel and the second pixel whose correspondence is indicated by the correspondence information, and generating the first estimation model and the second estimation model represented by the updated parameters; A program to execute.

2. the first estimation model is a model that outputs the depth information and a reliability of the depth information; The depth calculation step includes inputting the first input image and the second input image to the first estimation model, obtaining the first depth information and first reliability for the first input image, and the second depth information and second reliability for the second input image, the second loss function indicates a difference between the first depth information and the second depth information and depth teacher data, which is teacher data about depth, and includes a term indicating that the loss increases as the first reliability and the second reliability increase. The program according to claim 1.

3. The second loss function includes a term indicating a depth difference between two pixels whose depth difference in depth training data, which is training data about depth, is greater than a specified value. The program according to claim 1.

4. the first estimation model is a model that outputs the depth information and a reliability of the depth information; The depth calculation step includes inputting the first input image and the second input image to the first estimation model, obtaining the first depth information and first reliability for the first input image, and the second depth information and second reliability for the second input image, The second loss function is a function including a term indicating a difference between the first depth information and the second depth information and depth teacher data, which is teacher data about depth, and indicating that the loss is greater as the first reliability and the second reliability are greater; a function including a term indicating a depth difference between two pixels whose depth difference in the depth teacher data is greater than a specified value; is a weighted sum of The program according to claim 1.

5. The learning step includes: updating the camera parameters to optimize the first loss function, the second loss function, and the third loss function; The program according to claim 1.

6. The computer, further executing an updating step of updating parameters of the first estimation model and the second estimation model so as to optimize one or two of the first loss function, the second loss function, and the third loss function; The program according to claim 1.

7. A generation method executed by an information processing device, a depth calculation step of inputting a first input image and a second input image captured by an imaging device into a first estimation model that receives an input image and outputs depth information including a plurality of depths for a plurality of pixels included in the input image, and obtaining first depth information for the first input image and second depth information for the second input image; a motion calculation step of inputting the first depth information and the second depth information into a second estimation model that receives the two pieces of depth information and outputs motion information representing a movement in three-dimensional space for each of a plurality of pixels, thereby obtaining the motion information; a correspondence calculation step of calculating correspondence information indicating a correspondence between a first pixel included in the first input image and a second pixel included in the second input image using the first depth information, the second depth information, the motion information, and camera parameters of the imaging device; a learning step of updating parameters of the first estimation model and the second estimation model so as to optimize a first loss function including a term indicating a difference between the correspondence information and corresponding training data that is training data regarding the correspondence between the first pixel and the second pixel, a second loss function including a term regarding depth, and a third loss function including a term indicating a difference in pixel value between the first pixel and the second pixel whose correspondence is indicated by the correspondence information, and generating the first estimation model and the second estimation model represented by the updated parameters; A generation method including:

8. a depth calculation unit that inputs a first input image and a second input image captured by an imaging device into a first estimation model that inputs an input image and outputs depth information including a plurality of depths for a plurality of pixels included in the input image, and obtains first depth information for the first input image and second depth information for the second input image; a motion calculation unit that inputs the first depth information and the second depth information into a second estimation model that receives the two pieces of depth information and outputs motion information representing a movement in three-dimensional space for each of a plurality of pixels, thereby obtaining the motion information; a correspondence calculation unit that calculates correspondence information indicating a correspondence between a first pixel included in the first input image and a second pixel included in the second input image using the first depth information, the second depth information, the motion information, and camera parameters of the imaging device; a learning unit that updates parameters of the first estimation model and the second estimation model so as to optimize a first loss function including a term indicating a difference between the correspondence information and corresponding training data that is training data regarding the correspondence between the first pixel and the second pixel, a second loss function including a term regarding depth, and a third loss function including a term indicating a difference in pixel value between the first pixel and the second pixel whose correspondence is indicated by the correspondence information, and generates the first estimation model and the second estimation model represented by the updated parameters; An information processing device comprising:

Images