Program, generation method, and information processing device.

Abstract

To generate more accurate depth estimation models from images. [Solution] This is a program that causes a computer to perform a depth calculation step, a motion calculation step, a correspondence calculation step, and a learning step. The learning step updates the parameters of the first and second estimated models to optimize a first loss function that includes a term showing the difference between correspondence information and correspondence training data, which is training data for the correspondence between the first and second pixels; a second loss function that includes a term for depth; and a third loss function that includes a term showing the difference in pixel values ​​between the first and second pixels whose correspondence is indicated by the correspondence information. The program then generates the first and second estimated models represented by the updated parameters.

Description

Technical Field

[0001] Embodiments of the present invention relate to a program, a generation method, and an information processing apparatus.

Background Art

[0002] Techniques have been proposed for training a neural network that estimates depth using the luminance difference between images based on the geometric relationship between depth (depth) and ego-motion (camera motion), and a neural network that estimates ego-motion.

[0003] In such techniques, for example, ego-motion models only the motion of the stationary background imaged in the image, and moving objects are not considered. For this reason, for example, the neural network that estimates depth may not be able to accurately estimate the depth (distance) to a moving object.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] An object of the present invention is to provide a program, a generation method, and an information processing apparatus that can generate an estimation model for estimating depth from an image with higher accuracy. **Means for Solving the Problems**

[0007] The program according to the embodiment is a program for causing a computer to execute a depth calculation step, a motion calculation step, a correspondence calculation step, and a learning step. The learning step includes a first loss function including a term indicating a difference between correspondence information and correspondence teacher data which is teacher data about the correspondence between a first pixel and a second pixel, a second loss function including a term about depth, and a third loss function including a term indicating a difference in pixel values between the first pixel and the second pixel whose correspondence is indicated by the correspondence information, and updates parameters of the first estimation model and the second estimation model so as to optimize them, and generates the first estimation model and the second estimation model represented by the updated parameters. **Brief Description of the Drawings**

[0008] [Figure 1] Block diagram of the configuration of the information processing apparatus according to the embodiment. [Figure 2] Block diagram showing a configuration example of the depth calculation unit. [Figure 3] Block diagram showing a configuration example of the motion calculation unit. [Figure 4] Diagram showing a specific example of the correspondence calculation process by the correspondence calculation unit. [Figure 5] Flowchart of the learning process in the embodiment. [Figure 6] Flowchart of the estimation process in the embodiment. [Figure 7] Hardware configuration diagram of the information processing apparatus according to the embodiment. **Modes for Carrying Out the Invention**

[0009] Hereinafter, preferred embodiments of the program, the generation method, and the information processing apparatus according to the present invention will be described in detail with reference to the accompanying drawings.

[0010] The information processing device in this embodiment learns an estimation model that estimates depth by modeling not only stationary objects such as backgrounds in an image, but also the motion of moving objects. This makes it possible to perform depth estimation using the estimation model with high accuracy.

[0011] In this embodiment, at least the following two models are used. • Estimation Model MA: A model that takes an input image as input and outputs depth information containing multiple depths for multiple pixels in the input image (first estimation model). • Estimation Model MB: A model (second estimation model) that takes two depth information values ​​obtained from two input images as input and outputs motion information representing the movement in three-dimensional space for each of the multiple pixels.

[0012] The estimation models are learned (generated, constructed) by updating the parameters of the two estimation models to optimize the loss function for each model. Since the estimation model MA, which estimates depth, can be learned together with the estimation model MB, which models the motion of a moving object, it becomes possible to generate the estimation model MA with higher accuracy.

[0013] Furthermore, in this embodiment, the camera parameters of the imaging device are learned as learnable parameters along with the parameters of the estimation model. Since the camera parameters are obtained through learning, prior calibration to determine the camera parameters is unnecessary. The camera parameters include, for example, at least one of the following: the focal length of the imaging device, the principal point position of the imaging device, and the distortion coefficient of the imaging device.

[0014] Figure 1 is a block diagram showing an example of the configuration of an information processing device 100 according to an embodiment. As shown in Figure 1, the information processing device 100 includes a reception unit 101, a depth calculation unit 111, a motion calculation unit 112, a correspondence calculation unit 113, a learning unit 121, an estimation unit 122, an output control unit 102, and a storage unit 130.

[0015] The reception unit 101 receives input of various types of information used by the information processing device 100. For example, the reception unit 101 receives input images used for learning or estimation, and training data used for learning.

[0016] The input images are, for example, multiple images captured at multiple times by a camera (imaging device). These multiple images include, for example, input image IA (first input image) and input image IB (second input image) captured at a different time than input image IA. The camera is, for example, a monocular camera.

[0017] The training data includes depth training data, which is training data about depth, and correspondence training data, which is training data about the correspondence between pixels in two images. Depth training data can be obtained by any method, for example, by using a system that estimates depth from a single image, or by using a system that estimates depth from multiple images. Correspondence training data can be obtained by any method, for example, by using a system that estimates correspondence based on optical flow, or by using a system that estimates correspondence based on corresponding points.

[0018] The depth calculation unit 111 calculates depth information of the input image using the estimation model MA. For example, the depth calculation unit 111 inputs input image IA and input image IB to the estimation model MA and obtains depth information DA (first depth information) for input image IA and depth information DB (second depth information) for input image IB.

[0019] The estimation model MA may also be a model that outputs depth information along with the confidence level of the depth information. The confidence level is used, for example, in the learning process by the learning unit 121. Hereinafter, the confidence level of the depth information DA may be referred to as confidence level RA (first confidence level), and the confidence level of the depth information DB may be referred to as confidence level RB (second confidence level).

[0020] The confidence score indicates whether the estimated depth for each pixel in the input image is reliable or not. For example, if the estimated depth is reliable, a higher confidence score will be calculated than if it is unreliable. For instance, when estimating the depth of an image taken in an urban area, background pixels exist around the outlines of moving objects such as cars and pedestrians, making it easy for pixels corresponding to the entire moving object to be misrecognized as background. In other words, the depth of pixels corresponding to the entire moving object will be estimated to be at a greater distance than it actually is. In such cases, a low confidence score (indicating unreliability) will be calculated.

[0021] The motion calculation unit 112 uses two depth information values ​​calculated for the two input images to calculate motion information for each of the multiple pixels contained in the input images. For example, the motion calculation unit 112 inputs depth information DA and depth information DB into the estimation model MB to obtain motion information.

[0022] The estimated model MA and estimated model MB may be models of any structure, but for example, they may be neural network models (hereinafter also simply referred to as neural networks) or machine learning models such as random forests. Neural networks include, for example, convolutional neural networks, fully connected neural networks, recurrent neural networks, and Transformers.

[0023] The following primarily describes examples where the estimated model MA and estimated model MB are neural networks. Parameters updated during neural network training include, for example, weights (weight coefficients) and biases. Below, the parameters of estimated model MA are referred to as parameter PA, and the parameters of estimated model MB are referred to as parameter PB.

[0024] Neural networks can be trained using various methods, but they are often trained using methods such as gradient descent. Gradient descent uses the derivative of the loss function. Therefore, each model and loss function is expressed as a differentiable function.

[0025] The correspondence calculation unit 113 uses depth information DA and depth information DB, motion information, and camera parameters of the imaging device to calculate correspondence information that shows the correspondence between pixel PXA (first pixel) included in input image IA and pixel PXB (second pixel) included in input image IB, based on the geometric relationships of each of these pieces of information.

[0026] Here, we will explain in detail the functions of the depth calculation unit 111, the motion calculation unit 112, and the corresponding calculation unit 113.

[0027] Figure 2 is a block diagram showing an example configuration of the depth calculation unit 111. As shown in Figure 2, the depth calculation unit 111 inputs the input image 201 to the estimation model MA and calculates the depth 221 and confidence level 222. The input image 201 is, for example, an image taken at time t. In the following, the input image taken at time t is referred to as input image I t And so, input image I t The depth and confidence calculated from this are, Depth D t and confidence σ t There are cases like that.

[0028] In the example shown in Figure 2, the estimation model MA used by the depth calculation unit 111 includes an encoder 211 and a decoder 212. The encoder 211 receives the input image 201 and calculates a low-resolution feature map 211a that shows the features of the input image 201. The decoder 212 receives the low-resolution feature map 211a and calculates a high-resolution feature map 212a that has the same resolution as the input image. The estimation model MA calculates the depth 221 and confidence 222 from the feature map 212a. The depth calculation unit 111 uses this estimation model MA to calculate the depth 221 and confidence 222 for the input image 201.

[0029] The depth calculation unit 111 calculates and outputs the depth and confidence level for each of the two input images received by the reception unit 101, for example, by performing the processing described above.

[0030] The encoder 211 and decoder 212 can also be interpreted as corresponding to the models included in the estimated model MA. The encoder 211 and decoder 212 are also represented by differentiable functions.

[0031] The depth calculation unit 111 may calculate the depth and confidence score for the input image 201 by dividing the input image 201 into multiple segmented images and combining the multiple depths and multiple confidence scores obtained by applying the estimation model MA to each of the multiple segmented images.

[0032] Next, we will describe the details of the motion calculation unit 112. Figure 3 is a block diagram showing an example of the configuration of the motion calculation unit 112.

[0033] The motion calculation unit 112 uses the estimated model MB to calculate the pixel-level three-dimensional motion 312 for each of the multiple pixels contained in the input image 201 and the input image 301. The information including the three-dimensional motion 312 of the multiple pixels corresponds to motion information.

[0034] The motion calculation unit 112 inputs the input image 201, input image 301, depth 221, depth 321, and camera parameters 311 to the estimation model MB. Input image 301 is, for example, an image taken at time t-1. Depth 321 is the depth calculated for input image 301. In the following, the input image taken at time t-1 is input image I t-1 And so, input image I t-1 The depth (depth 321) and confidence level calculated from this are given to depth D t-1 and confidence σ t-1 There are cases like that.

[0035] The estimation model MB calculates and outputs a three-dimensional motion 312 for each pixel of the input image 301 to move to the viewpoint of the input image 201. For example, the motion calculation unit 112 converts the pixel-level three-dimensional motion 312 into optical flow, aligns the viewpoints of the input image 201 and the input image 301, and then inputs the input data (input image 201, input image 301, depth 221, and depth 321) back into the estimation model MB to calculate a new pixel-level three-dimensional motion 312.

[0036] Furthermore, the estimated model MB may share some layers (and their parameters) with the estimated model MA.

[0037] The motion calculation unit 112 is capable of repeatedly estimating pixel-level three-dimensional motion 312, converting it to optical flow, aligning the viewpoint, and estimating new pixel-level three-dimensional motion 312.

[0038] Furthermore, in order to reduce the amount of computation, the motion calculation unit 112 may, during the calculation process, obtain pixel-level three-dimensional motion at a lower resolution from the input data (input image 201, input image 301, depth 221, and depth 321), and then calculate the final output three-dimensional motion 312 by upsampling the lower-resolution three-dimensional motion to the same resolution as the input image.

[0039] Next, the details of the correspondence calculation unit 113 will be explained. Figure 4 is a diagram showing a specific example of the correspondence calculation process performed by the correspondence calculation unit 113.

[0040] The correspondence calculation unit 113 calculates the correspondence between input image 201 and input image 301 based on the geometric relationship between the depth calculated by the depth calculation unit 111, the pixel-level three-dimensional motion 411 calculated by the motion calculation unit 112, and the set camera parameters.

[0041] Specifically, the correspondence calculation unit 113 first emits a straight line 402 from the imaging position 401 into the three-dimensional space based on the camera parameters, and back-projects the pixel 403 into the three-dimensional space.

[0042] Next, the correspondence calculation unit 113 determines the end point position of the straight line 402 based on the depth, thereby generating a position 404 in the three-dimensional space corresponding to the pixel 403 of the input image 301 captured from the viewpoint of the imaging position 401.

[0043] Next, the correspondence calculation unit 113 uses the three-dimensional motion 411 of the pixel 403 to move the position 404 in the three-dimensional space to a position 424 in the three-dimensional space.

[0044] Next, the correspondence calculation unit 113 projects the position 424 in the three-dimensional space along the straight line 422 onto the pixel 423 of the input image 201 captured from the viewpoint of the imaging position 421 based on the set camera parameters.

[0045] In this way, correspondence information for associating the pixel 403 and the pixel 423 is calculated. Note that the difference in the positions between the pixel 423 and the pixel 403 is the optical flow. The correspondence information may include the optical flow.

[0046] The correspondence calculation process can be formulated by the following formula (1).

Equation

[0047] x t-1 represents the coordinates of the pixel 403, the function π -1 () represents back-projection based on the camera parameters, D(x t-1 ) represents the depth corresponding to the pixel 403, T t-1→t represents the three-dimensional motion 411 of the pixel 403, and the function π() represents projection based on the camera parameters. Note that D(x t-1 )·π -1 (x t-1 ) is a function for calculating the position 404 in the three-dimensional space, T t-1→t ·D(xt-1 )·π -1 (x t-1 ) is a function that calculates the position 424 in three-dimensional space, π(T t-1→t ·D(x t-1 )·π -1 (x t-1 )) is a function that calculates the coordinates of pixel 423.

[0048] Returning to the explanation of Figure 1, the learning unit 121 performs the learning process for the estimated model MA and estimated model MB. For example, the learning unit 121 updates the parameters of the estimated model MA and estimated model MB to optimize the loss function shown below, and generates the estimated model MA and estimated model MB represented by the updated parameters. • A loss function LA (first loss function) that includes a term showing the difference between the correspondence information and the corresponding training data. The corresponding training data is the training data for the correspondence between pixel PXA and pixel PXB. • Loss function LB (second loss function) that includes a term related to depth. The loss function LC (third loss function) includes a term that shows the difference in pixel values ​​between pixel PXA and pixel PXB, whose correspondence is indicated by the correspondence information.

[0049] Through the learning process, the depth calculation unit 111 (estimated model MA), motion calculation unit 112 (estimated model MB), and camera parameters can be adapted to the target scene. Details of the loss function will be described later.

[0050] The estimation unit 122 performs estimation processing using the estimation model learned by the learning unit 121. The estimation processing can be used for any purpose, but for example, it can be applied to a technology that uses an image captured by an imaging device to determine the distance from the imaging position to the subject, and then uses the determined distance to control a moving object (such as a car or mobile robot). The imaging device may be mounted on the moving object to be controlled.

[0051] The estimation unit 122 may perform estimation processing using either estimation model MA or estimation model MB. For example, the estimation unit 122 may perform estimation processing to estimate the depth of the input image using estimation model MA. Since estimation model MA is trained together with estimation model MB, it can estimate the depth with higher accuracy.

[0052] The output control unit 102 controls the output of various types of information used by the information processing device 100. For example, the output control unit 102 stores the parameters (parameter PA, parameter PB) of each model (estimated model MA, estimated model MB) obtained through the learning process in the storage unit 130, or outputs them to an external device (such as an estimation device) that performs processing using each model. The output control unit 102 also displays the results of the estimation process performed by the estimation unit 122 on a display device such as a display, or transmits them to an external device connected via a network.

[0053] At least a portion of each of the above components (reception unit 101, depth calculation unit 111, motion calculation unit 112, correspondence calculation unit 113, learning unit 121, estimation unit 122, and output control unit 102) may be implemented by one or more processing units. Each of the above components may be implemented by, for example, one or more processors. For example, each of the above components may be implemented by having a processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) execute a program, i.e., by software. Each of the above components may be implemented by a processor such as a dedicated IC (Integrated Circuit), i.e., by hardware. Each of the above components may be implemented by using both software and hardware. When multiple processors are used, each processor may implement one of the above components, or two or more of the above components.

[0054] The memory unit 130 stores various types of information used in the information processing device. For example, the memory unit 130 stores the input image 131, depth training data 132, corresponding training data 133, estimation model 134, and camera parameters 135. The estimation model 134 includes estimation model MA and estimation model MB.

[0055] The storage unit 130 can be composed of any commonly used storage medium, such as flash memory, memory card, RAM (Random Access Memory), HDD (Hard Disk Drive), and optical disc.

[0056] At least a portion of each piece of data stored in the memory unit 130 (input image 131, depth training data 132, corresponding training data 133, estimation model 134, and camera parameters 135) may be stored in physically different storage media, or in different storage areas of the physically same storage media.

[0057] Furthermore, the information processing device 100 may be composed of one physical device or multiple physical devices. For example, the information processing device 100 may be built on a cloud environment. Also, each part of the information processing device 100 may be distributed and provided on multiple devices. For example, the information processing device 100 (information processing system) may be configured to include a device (e.g., a learning device) equipped with functions necessary for learning processing (e.g., a learning unit 121) and a device (e.g., an estimation device) equipped with functions necessary for estimation processing using the learned estimation model 134 (e.g., an estimation unit 122).

[0058] Next, we will explain specific examples of each loss function. First, we will explain an example of the loss function LA.

[0059] The learning unit 121 updates the parameters PA (parameters of the estimated model MA), PB (parameters of the estimated model MB), and camera parameters, for example, using the loss function LA, so that the difference in brightness values ​​(pixel values) between corresponding pixels (hereinafter referred to as corresponding pixels) of two input images becomes smaller.

[0060] Furthermore, the magnitude of the increase (decrease) when updating each parameter may be proportional to the absolute value of the derivative. To avoid abrupt fluctuations, an upper limit may be set on the range of parameter variation.

[0061] Specifically, the learning unit 121 uses a loss function L such as, for example, equation (2) below. pho Update the parameters PA, PB, and camera parameters so that (an example of the loss function LA) is minimized.

number

[0062] I t-1 and l t These are input image IA and input image IB, respectively. p is l t-1 Pixels, F 2d t-1→t is the optical flow from input image IA to input image IB, and the function d() represents the difference in luminance values. Also, p and p+F 2d t-1→t This is the corresponding pixel.

[0063] The function d() can be any function that can calculate the difference in luminance values. For example, the function d() may be a function that calculates the difference in luminance values ​​based on geometric distance concepts such as L1 distance and L2 distance, or a function that calculates perceptual similarity such as PSNR and SSIM as the difference in luminance values. The function d() may also be a function that combines multiple calculation methods. For example, the function d() may be expressed as a weighted sum of multiple functions corresponding to multiple calculation methods.

[0064] Next, we will explain an example of the loss function LB. The learning unit 121 updates the parameter PA such that the lower the confidence level calculated by the depth calculation unit 111, the closer the depth calculated by the depth calculation unit 111 will be to the depth training data (and the higher the confidence level calculated by the depth calculation unit 111, the less close the depth calculated by the depth calculation unit 111 will be to the depth training data).

[0065] Specifically, the learning unit 121 updates the parameter PA in accordance with the confidence level, using a loss function such as equation (3) below (example of loss function LB), so that the absolute difference between the depth and the depth training data is minimized.

number

[0066] D and D T These represent depth and depth training data, respectively. σ represents the confidence level corresponding to depth D. Note that D and D T Since the actual scale is indeterminate, it is normalized by the following equation (4).

number

[0067] The `median()` function calculates the median depth of all pixels, and the `mean()` function calculates the average depth of all pixels. Note that the depth training data D T The same calculation applies to . Equation (4) applies only to equation (3).

[0068] The loss function in equation (3) is calculated using depth information DA and depth information DB (depth D), and depth training data (depth D), which is training data for depth. T This function represents the difference between ) and , and includes a term that indicates that the larger the confidence levels RA and RB(σ), the greater the loss. By using such a loss function, it becomes possible to filter out pixels with low confidence levels.

[0069] Furthermore, the learning unit 121 may update the parameter PA using the front-to-back relationship of pixels obtained from the depth training data, for example, with a loss function like the following equation (5) (example of loss function LB), so that the distance between pixels that are more than a certain threshold apart is maximized.

number

[0070] p i and p j Each of these is a pixel with a different depth D, and they are sampled randomly. δ represents a hyperparameter (an example of a specified value) that controls the threshold whose value range is [0,1]. l is the front-to-back relationship of pixels obtained from depth training data, and is formulated by the following equation (6).

number

[0071] D T The expression is normalized by the following equation (7). Note that equation (7) applies only to equation (6).

number

[0072] According to equation (7), D T The range of values ​​is normalized to [0,1]. This makes it easier to set the hyperparameter δ. For example, if δ is set to 0.1, pixel p i and pixel p j When they are separated by more than 10%, D(p i ) and D(p j ) is maximized.

[0073] The loss function in equation (5) corresponds to a function that includes a term representing the depth difference for two pixels in the depth training data whose depth difference is greater than a specified value.

[0074] The learning unit 121 may use either equation (3) or equation (5), or a combination of both. In the latter case, the learning unit 121 may use a loss function LB that can be expressed as a weighted sum of equations (3) and (5), for example.

[0075] Next, we will explain an example of the loss function LC. The learning unit 121 updates the parameters PA, PB, and camera parameters so that the correspondence calculated by the correspondence calculation unit 113 becomes closer to the corresponding training data.

[0076] Specifically, the learning unit 121 updates the parameters PA, PB, and camera parameters so that the loss function (example of loss function LC) shown in equation (8) below is minimized.

number

[0077] F 2d t-1→t This is the optical flow obtained by the correspondence calculation unit 113 for input image IA and input image IB. 2d,T t-1→t This is the optical flow for input image IA and input image IB included in the corresponding training data. F in equation (8) 2d t-1→t And, F 2d,T t-1→t The difference is F 2d t-1→t And, F 2d,T t-1→t This corresponds to the distance between two points. The distance can be calculated in any way, but methods based on geometric distance concepts such as L1 distance and L2 distance can be applied.

[0078] As described above, each loss function is represented by a differentiable function. Therefore, the learning unit 121 can determine the direction in which to decrease the loss function by differentiating the loss function with respect to the parameters PA, PB, and camera parameters. The learning unit 121 updates the parameters PA, PB, and camera parameters by repeatedly varying the parameter PA in the determined direction. As a result of such updates, an estimated model MA represented by parameter PA and an estimated model MB represented by parameter PB are generated.

[0079] The learning unit 121 updates the three parameters PA, PB, and PC using a learning method such as gradient descent to optimize a loss function that includes all three loss functions LA, LB, and LC mentioned above. This allows the estimation model MA, which estimates depth, to be learned along with the estimation model MB, which models the motion of the moving object, making it possible to generate the estimation model MA with higher accuracy.

[0080] Furthermore, the learning unit 121 may further update at least some of the parameters of each generated model (estimated model MA, estimated model MB) using only a portion of the three loss functions LA, LB, and LC. In other words, the learning unit 121 may have a function to update the parameters of estimated model MA and estimated model MB in order to optimize one or two of the loss functions LB, LC, and LA.

[0081] For example, the learning unit 121 uses the loss function L in equation (2). pho The parameters PA, PB, and camera parameters may be updated to optimize the loss function in equation (3). The learning unit 121 may also update parameter PA to optimize the loss function in equation (3). The learning unit 121 may also update parameters PA, PB, and camera parameters to optimize the loss function in equation (8).

[0082] Next, the learning process by the information processing device 100 of this embodiment will be described. Figure 5 is a flowchart showing an example of the learning process in this embodiment.

[0083] The reception unit 101 receives two input images IA and IB from the imaging device (step S101).

[0084] Using the input images, depth (depth information), three-dimensional motion (motion information), and correspondence (correspondence information) are calculated (step S102). For example, the depth calculation unit 111 calculates depth and confidence for each of the two input images. The motion calculation unit 112 calculates pixel-level three-dimensional motion for the two input images. The correspondence calculation unit 113 uses the calculated depth, the calculated pixel-level three-dimensional motion, and the set camera parameters to calculate the correspondence between the two input images from a geometrical relationship.

[0085] The learning unit 121 updates the parameters PA, PB, and camera parameters to optimize the loss function using the calculation results from step S102 (depth information, motion information, correspondence information) and the training data (step S103).

[0086] The learning unit 121 determines whether or not to terminate the learning process (step S104). For example, the learning unit 121 determines to terminate the learning process if the value of the loss function becomes smaller than a threshold (a threshold for the function value), or if the number of iterations exceeds a threshold (a threshold for the number of iterations).

[0087] If it is determined that learning should not be terminated (Step S104: No), the process returns to Step S101 and is repeated. If it is determined that learning should be terminated (Step S104: Yes), the learning process is terminated.

[0088] Next, the estimation process by the information processing device 100 of the embodiment will be described. Figure 6 is a flowchart showing an example of the estimation process in the embodiment.

[0089] The reception unit 101 receives the input image to be estimated from the imaging device (step S201). The estimation unit 122 uses trained models (estimation model MA, estimation model MB) to estimate (calculate) depth (depth information), three-dimensional motion (motion information), and correspondence (correspondence information) for the received input image (step S202). The estimation unit 122 may estimate only a portion of this information.

[0090] The output control unit 102 outputs the estimation result from the estimation unit 122 (step S203), and terminates the estimation process.

[0091] In this embodiment, the information processing device models not only stationary objects such as the background in an image but also the motion (three-dimensional motion) of moving objects, and performs learning by applying geometric constraints to the estimation model that estimates depth. This makes it possible to bring the distance (depth) to the subject, including the moving object, closer to a desired value. As a result, for example, the operation of the information processing device 100 can be adapted according to the subject captured in the image, and the accuracy of estimating the distance (depth) to the subject can be improved.

[0092] Furthermore, techniques using EgoMotion, which only models static backgrounds, cannot adapt depth estimation to moving objects or other subjects. For example, if moving objects or other subjects are not modeled in the captured image, they may be mistakenly recognized as static backgrounds. As a result, the estimated distance to the moving object or other subject may differ from the actual distance.

[0093] Next, the hardware configuration of the information processing device of the embodiment will be described using Figure 7. Figure 7 is an explanatory diagram showing an example of the hardware configuration of the information processing device of the embodiment.

[0094] The information processing device of this embodiment includes a control device such as a CPU (Central Processing Unit) 51, a storage device such as a ROM (Read Only Memory) 52 and a RAM (Random Access Memory) 53, a communication interface 54 that connects to a network for communication, and a bus 61 that connects each part.

[0095] The program to be executed in the information processing device of this embodiment is provided pre-installed in a ROM 52 or the like.

[0096] The program executed by the information processing device of this embodiment may be configured to be provided as a computer program product by recording it in an installable or executable file format onto a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), or a DVD (Digital Versatile Disk).

[0097] Furthermore, the program executed by the information processing device of the embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. Alternatively, the program executed by the information processing device of the embodiment may be provided or distributed via a network such as the Internet.

[0098] The program executed in the information processing device of this embodiment can cause the computer to function as a component of the information processing device described above. This computer can read the program from a computer-readable storage medium onto the main memory and execute it using the CPU 51.

[0099] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of symbols]

[0100] 100 Information Processing Devices 101 Reception Department 102 Output control unit 111 Depth calculation section 112 Motion Calculation Unit 113 Correspondence Calculation Unit 121 Learning Department 122 Estimation Department 130 Storage section

Claims

1. On the computer, A depth calculation step involves inputting a first input image and a second input image captured by an imaging device into a first estimation model that outputs depth information including multiple depths for multiple pixels contained in an input image, and obtaining first depth information for the first input image and second depth information for the second input image. A motion calculation step involves inputting the first depth information and the second depth information into a second estimation model that outputs motion information representing the movement of each pixel in three-dimensional space, in order to obtain the motion information. A correspondence calculation step that calculates correspondence information indicating the correspondence between a first pixel included in the first input image and a second pixel included in the second input image, using the first depth information and the second depth information, the motion information and the camera parameters of the imaging device, A learning step that updates the parameters of the first and second estimation models based on a loss function that includes a term showing the difference in pixel values ​​between the first and second pixels whose correspondence is indicated by the correspondence information, and generates the first and second estimation models represented by the updated parameters, A program to execute.

2. The first estimation model is a model that outputs the depth information and the confidence level of the depth information, The depth calculation step involves inputting the first input image and the second input image into the first estimation model and obtaining the first depth information and first confidence level for the first input image, and the second depth information and second confidence level for the second input image. The program according to claim 1.

3. The aforementioned learning steps are: The Camera parameters are updated to optimize the loss function. The program according to claim 1.

4. To the aforementioned computer, Further update steps are performed to update the parameters of the first and second estimation models in order to optimize the loss function. The program according to claim 1.

5. A generation method performed by an information processing device, A depth calculation step involves inputting a first input image and a second input image captured by an imaging device into a first estimation model that outputs depth information including multiple depths for multiple pixels contained in an input image, and obtaining first depth information for the first input image and second depth information for the second input image. A motion calculation step involves inputting the first depth information and the second depth information into a second estimation model that outputs motion information representing the movement of each pixel in three-dimensional space, in order to obtain the motion information. A correspondence calculation step that calculates correspondence information indicating the correspondence between a first pixel included in the first input image and a second pixel included in the second input image, using the first depth information and the second depth information, the motion information and the camera parameters of the imaging device, A learning step that updates the parameters of the first and second estimation models based on a loss function that includes a term showing the difference in pixel values ​​between the first and second pixels whose correspondence is indicated by the correspondence information, and generates the first and second estimation models represented by the updated parameters, A method of production that includes this.

6. A depth calculation unit receives a first input image and a second input image captured by an imaging device as input to a first estimation model that outputs depth information including multiple depths for multiple pixels contained in an input image, and obtains first depth information for the first input image and second depth information for the second input image. A motion calculation unit that inputs the first depth information and the second depth information to a second estimation model that outputs motion information representing the movement of each pixel in three-dimensional space, and obtains the motion information, A correspondence calculation unit calculates correspondence information indicating the correspondence between a first pixel included in the first input image and a second pixel included in the second input image, using the first depth information, the second depth information, the motion information, and the camera parameters of the imaging device. A learning unit updates the parameters of the first and second estimation models based on a loss function that includes a term indicating the difference in pixel values ​​between the first and second pixels whose correspondence is shown by the correspondence information, and generates the first and second estimation models represented by the updated parameters. An information processing device equipped with the following features.

Images