Information processing method and recording medium

Abstract

The present invention provides an information processing method and a recording medium. The information processing method includes: inputting a first image or a second image into a first machine learning model to predict a depth map; inputting the first and second images into a second machine learning model to predict the relative movement and rotation of a camera after capturing the first and second images; generating a first transformed image obtained by applying the following image transformation to the first image, which is based on the predicted relative movement and rotation from the first image to the second image and the depth map predicted for the first image; calculating a loss including an error between the first transformed image and the second image; and inputting the gradient of the loss related to the relative movement and rotation, the first transformed image, and the first and second images into a second machine learning model, repeatedly predicting the relative movement and rotation, and converging the loss including the error between a new first transformed image based on the newly predicted relative movement and rotation and the second image.

Description

Technical Field

[0001] The present invention relates to an information processing method and a recording medium. Background Art

[0002] In recent years, the following technology has become known: images captured by a camera are input into a deep neural network (DNN), and the depth and camera pose (the relative movement and rotation between the camera devices that captured the two images) within the image are estimated through inference processing by the DNN (Non-Patent Documents 1 and 2). In order to estimate the relative camera pose using previous (manual) estimation methods that do not use DNNs, it is necessary to solve the camera pose and depth map that are suitable for the correspondence between the two images as an optimization problem. The above-mentioned method using DNNs can obtain more accurate estimation results for the correspondence between the images and the depth estimation.

[0003] On the other hand, the technologies proposed in Non-Patent Documents 1 and 2 have a problem in that higher performance cannot be achieved in estimating the relative camera posture compared to the case where the relative camera posture is estimated using a conventional (manual) estimation method that does not use a DNN.

[0004] Prior art literature

[0005] Non-patent literature

[0006] Non-Patent Literature 1: Vitor Guizilini and three others, “3D Packing for Self-Supervised Monocular Depth Estimation,” arXiv:1905.02693v4[cs.CV]28 Mar 2020 (https: / / arxiv.org / pdf / 1905.02693.pdf)

[0007] Non-Patent Literature 2: Clement Godard and three others, “Digging Into Self-Supervised Monocular Depth Estimation,” arXiv:1806.01260v4[cs.CV]17Aug 2019 (https: / / arxiv.org / pdf / 1806.01260.pdf)

[0008] Non-Patent Literature 3: John Flynn and seven others, “DeepView: View Synthesis with Learned Gradient Descent,” arXiv:1906.07316v1[cs.CV], 18 Jun 2019 (https: / / arxiv.org / pdf / 1906.07316.pdf) Summary of the Invention

[0009] Problems to be solved by the invention

[0010] Incidentally, the following technology is known: in the inference process using DNN, the gradient of the loss based on the output of the DNN is input to the DNN for repeated calculation, thereby replacing the design involved in the parameter change of the gradient descent method (such as learning coefficients, normalization, etc.) with the DNN. Non-patent document 3 proposes the following technology: performing a calculation on the gradient of the loss input to a DNN (CNN), generating multiple images (multi-plane images) of each depth based on images captured from multiple viewpoints, and generating a composite image of a new viewpoint that is not in the captured image. The technologies involved in the above-mentioned non-patent documents 1 and 2 are merely technologies for estimating the camera posture, etc. through a single DNN calculation. In addition, the technology involved in non-patent document 3 does not consider the structure for estimating the camera posture.

[0011] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an estimation technology that can improve the estimation accuracy of a camera attitude when estimating the camera attitude using a machine learning model.

[0012] Means used to solve problems

[0013] According to the present invention, there is provided an information processing method, which is executed in an information processing device and is characterized in that:

[0014] The information processing method comprises:

[0015] Inputting the first image or a second image paired with the first image into a first machine learning model to predict a depth map;

[0016] Inputting the first image and the second image into a second machine learning model to predict relative movement and rotation of the camera after the first image and the second image are captured;

[0017] generating a first transformed image obtained by applying an image transformation to the first image, the image transformation being based on the predicted relative movement and rotation from the first image to the second image and the depth map predicted for the first image;

[0018] calculating a loss including an error between the first transformed image and the second image; and

[0019] The gradient of the loss related to the relative movement and rotation, the first conversion image, the first image and the second image are input into the second machine learning model, and the relative movement and rotation are repeatedly predicted to converge the loss, including the error between the new first conversion image based on the newly predicted relative movement and rotation and the second image.

[0020] Furthermore, according to the present invention, there is provided a recording medium storing a program for causing a computer to execute each step of the information processing method executed in an information processing apparatus, wherein:

[0021] The information processing method comprises:

[0022] Inputting the first image or a second image paired with the first image into a first machine learning model to predict a depth map;

[0023] Inputting the first image and the second image into a second machine learning model to predict relative movement and rotation of the camera after the first image and the second image are captured;

[0024] generating a first transformed image obtained by applying an image transformation to the first image, the image transformation being based on the predicted relative movement and rotation from the first image to the second image and the depth map predicted for the first image;

[0025] calculating a loss including an error between the first transformed image and the second image; and

[0026] The gradient of the loss related to the relative movement and rotation, the first conversion image, the first image and the second image are input into the second machine learning model, and the relative movement and rotation are repeatedly predicted to converge the loss, including the error between the new first conversion image based on the newly predicted relative movement and rotation and the second image.

[0027] Effects of the Invention

[0028] According to the present invention, when estimating a camera attitude using a machine learning model, the estimation accuracy of the camera attitude can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a block diagram showing an example of the functional configuration of the information processing server according to the first embodiment.

[0030] Figure 2 This is a diagram for explaining depth map prediction and camera pose prediction in the model processing unit 114 according to the first embodiment.

[0031] Figure 3A This is a diagram for explaining image conversion (from time t to time t+1) according to the first embodiment.

[0032] Figure 3B This is a diagram for explaining the reverse image conversion (from time t+1 to time t) according to the first embodiment.

[0033] Figure 4A This is a diagram explaining the loss of the predicted camera posture according to the first embodiment.

[0034] Figure 4B This is a diagram explaining the loss of the predicted depth map according to the first embodiment.

[0035] Figure 5 This is a diagram explaining depth map prediction involving iteration according to the first embodiment.

[0036] Figure 6 This is a diagram explaining camera posture prediction involving iteration according to the first embodiment.

[0037] Figure 7 This is a flowchart showing a series of operations of the prediction process (of the camera posture and the depth map) in the model processing unit according to the first embodiment.

[0038] Figure 8 It is a block diagram showing an example of the functional configuration of a vehicle according to the second embodiment.

[0039] Figure 9 It is a diagram showing a main configuration for driving control of a vehicle according to the second embodiment.

[0040] Description of Reference Numerals

[0041] 100: Information processing server; 113: Image data acquisition unit; 114: Model processing unit; 201: Depth map prediction unit; 202: Camera posture prediction unit; 203: Image conversion unit; 204: Loss calculation unit; 205: Iteration control unit. DETAILED DESCRIPTION

[0042] The following embodiments are described in detail with reference to the accompanying drawings. The following embodiments do not limit the inventions described in the claims, and not all combinations of features described in the embodiments are essential to the invention. Any combination of two or more of the multiple features described in the embodiments may be used. Identical or similar components are denoted by the same reference numerals, and duplicate descriptions are omitted.

[0043] <Configuration of Information Processing Server>

[0044] Next, refer to Figure 1 This section describes an example of the functional configuration of an information processing server. Furthermore, the various functional modules described with reference to the following figures may be combined or separated, and the functions described may be implemented by other modules. Furthermore, components described as hardware may be implemented by software, and vice versa. The information processing server is located outside a mobile object such as a vehicle.

[0045] The control unit 104 includes, for example, a CPU 110, a RAM 111, and a ROM 112, and controls the operation of each component of the information processing server 100. The control unit 104 performs the functions of each component by having the CPU 110 load a computer program stored in the ROM 112 into the RAM 111 and execute the program. In addition to the CPU 110, the control unit 104 may also include a GPU or a dedicated circuit suitable for executing machine learning or neural network processing. The GPU or such circuit may also execute the processing of the model processing unit 114.

[0046] The image data acquisition unit 113 acquires image data transmitted from an external device, such as a vehicle operated by a user. The image data acquisition unit 113 stores the acquired image data in the storage unit 103. To obtain inference results based on the acquired image data, the acquired image data is input into a machine learning model (also referred to as a learning model) in the inference stage. Furthermore, the image data acquired by the image data acquisition unit 113 can also be used to train the model of the model processing unit 114.

[0047] The model processing unit 114 includes the machine learning model of this embodiment and performs the learning and inference phases of the learning model. For example, the learning model receives first and second image data as input and predicts the relative camera pose between camera 1, which captured the first image data, and camera 2, which captured the second image data. This relative camera pose includes, for example, relative movement of one camera relative to the other and relative rotation around three axes. Furthermore, the learning model predicts depth maps corresponding to each of the first and second image data.

[0048] The first image data and the second image data may be, for example, image data captured at different times (e.g., time t and a subsequent time t+1) by a camera mounted in a manner facing the front of the vehicle. In this case, since the camera moves between time t and time t+1 due to the movement of the vehicle, the camera posture of the camera when capturing each image data is different. In addition, the first image data and the second image data may be, for example, two image data captured by two cameras mounted in a manner facing the front of the vehicle. In this case, the camera posture of the camera when capturing each image data is also different. When the prediction processing of this embodiment is used on the image data captured by two fixed cameras, camera calibration and camera anomaly detection can be performed based on the estimated camera parameters.

[0049] The DNN becomes a learned state by performing the processing of the learning phase. By inputting new image data into the learned DNN, it is possible to perform camera pose prediction and depth map prediction (inference phase processing) for the new image data. When the inference processing using the learned model is performed in the information processing server 100, the inference phase processing is performed. The information processing server 100 executes the learned learned model on the information processing server 100 side and sends the inference result to an external device such as a vehicle or an information processing device. Generally speaking, the information processing server 100 can use more abundant computing resources than vehicles, etc., and is therefore suitable for situations where repeated calculations are performed as in the prediction processing described later. In addition, by receiving and accumulating image data captured by various vehicles, image data under various conditions can be collected, and learning corresponding to more conditions can be performed.

[0050] The prediction result providing unit 115 provides the depth map and camera posture information predicted by the model processing unit 114 to an external device such as a vehicle that has transmitted the image data.

[0051] The communication unit 101 is a communication device including, for example, communication circuitry, and communicates with external devices such as vehicles and information processing devices via a network such as the Internet. In addition to receiving actual images from external devices such as vehicles and information processing devices, the communication unit 101 also transmits camera parameters and depth maps predicted by the learning model to the vehicle. The power supply unit 102 supplies power to various components within the information processing server 100. The storage unit 103 is a nonvolatile memory such as a hard disk or semiconductor memory. The storage unit 103 stores the learned parameters and learning data of the DNN, which will be described later.

[0052] <Example of Learning Model of Model Processing Unit>

[0053] Next, an example of a learning model in the model processing unit 114 according to this embodiment will be described. The following description uses as an example a case where two sets of image data captured by a camera mounted on a vehicle at time t and a subsequent time t+1 are input to the model processing unit 114.

[0054] Reference Figure 2 , the configuration of the model processing unit 114 and an example of processing of the learning model included in the model processing unit 114 are described. The model processing unit 114 includes a DNN_D 201, a DNN_P 202, an image conversion unit 203, a loss calculation unit 204, and an iteration control unit 205. Furthermore, the model processing unit 114 includes a learning model. The learning model involved in this embodiment includes, for example, a DNN_D 201, a DNN_P 202, an image conversion unit 203, a loss calculation unit 204, and an iteration control unit 205.

[0055] The depth map prediction unit 201 (hereinafter also referred to as DNN_D201) is composed of one or more deep neural networks (DNNs), which are examples of machine learning models, and outputs a depth map 211 of the input image data as a prediction result. For example, when image data at time t is input, DNN_D201 outputs a depth map corresponding to the image data at time t, and when image data at time t+1 is input, it outputs a depth map corresponding to the image data at time t+1. A depth map is, for example, information representing the predicted depth (the distance from the camera to the object) for each pixel value in the input image, with pixels closer to the camera represented in white and pixels farther away in black. Furthermore, as described below, DNN_D201 iterates (repeatedly calculates) to output the depth map, which can improve the accuracy of the predicted depth map. In this case, DNN_D201 can receive image data, a depth map predicted in the immediately preceding iteration, and the gradient of the loss associated with the depth map (described below), and output the depth map for the current iteration. DNN_D201 can be composed of, for example, CNN (Convolutional Neural Network), autoencoder, etc.

[0056] The camera pose prediction unit 202 (also referred to as DNN_P202) is composed of one or more DNNs as an example of a machine learning model, and outputs the relative camera pose 212 between the two input image data as a prediction result. For example, when the DNN_P202 is input with two image data at time t and time t+1, it outputs the relative camera pose (for example, the amount of movement of the camera and the amount of rotation around three axes) between the image data at time t and time t+1. In addition, as described later, DNN_P202 iterates in order to output the camera pose as a prediction result, which can improve the accuracy of the predicted camera pose. At this time, DNN_P202 can be input with two image data, the camera pose predicted in the immediately previous iteration, and the gradient of the loss related to the camera pose described later, and output the camera pose in the current iteration. DNN_P202 can output the relative camera pose P (I t , I t+1 ). In addition, DNN_P202 can output the relative camera posture P(I t+1 , I t ). DNN_P202 can be composed of, for example, CNN (Convolutional Neural Network) and the like.

[0057] The image conversion unit 203 converts the image data captured at one moment using the predicted (relative) camera pose and depth map to generate a converted image 213 corresponding to another moment. Figure 3A As shown, the image conversion unit 203 receives as input image data 301 captured at time t, a depth map 302 estimated for the image data 301 at time t, and a relative camera pose 303 from time t to time t+1. Based on the input data, the image conversion unit 203 generates a converted image 304 corresponding to time t+1.

[0058] Here, the image conversion performed by the image conversion unit 203 is equivalent to applying an image conversion based on the predicted relative camera pose and depth map to the image data captured by one camera, thereby reproducing virtual image data captured by the other camera. Therefore, the more accurate the (relative) camera pose (and depth map) predicted from the image data at time t and the image data at time t+1 is, the smaller the error between the image data at time t+1 and the converted image 304 corresponding to time t+1. Conversely, if the error between the actual image data at time t+1 and the converted image 304 corresponding to time t+1 is large, the predicted camera pose or depth map is inaccurate. Therefore, by repeatedly predicting the predicted camera pose or depth map to reduce the difference between the converted image corresponding to time t+1 and the actual image data at time t+1, a more accurate (relative) camera pose and depth map can be obtained.

[0059] In addition, the image conversion unit 203 uses the image data I t , relative camera pose P(I t , I t+1 ) and the depth map D t The image conversion unit 203 can also perform image conversion using the image data I t+1 , relative camera pose P(I t+1 , I t ) and depth map D t+1 For example, Figure 3B As shown in FIG. 1 , the image conversion unit 203 can be input with the image data 311 captured at time t+1, the depth map 312 predicted based on the image data at time t+1, and the relative camera posture 313 (P(I t+1 , I t)) to generate a transformed image 314 corresponding to time t. By constraining optimization to minimize errors in both the conversion from time t to the image at time t+1 and the conversion from time t+1 to the image at time t, high prediction accuracy can be achieved. Furthermore, when training a DNN, the presence of a pair of images allows for double learning, enabling efficient learning with less training data.

[0060] The loss calculation unit 204 calculates the loss 214 based on the error between the captured image and the converted image. t+1 The error (for example, the sum of squares of differences in pixel values) between the converted image corresponding to time t+1 output by the image conversion unit 203 is included in the loss. t The error (for example, the sum of squares of differences in pixel values) between the converted image corresponding to the time t output by the image conversion unit 203 is further included in the loss. For example, the loss calculation unit 204 calculates the error using the image data I t and the converted image corresponding to time t output by the image conversion unit 203, as shown in FIG. Figure 4A As shown, using as image data I t The image data 401 and the converted image 402 corresponding to the time t are used to calculate the loss 403 related to the camera posture.

[0061] The loss calculation unit 204 may calculate the loss based on the error between the image data and the converted image, or instead of the error, using an error based on image features, an SSIM (structural similarity) index, or the like. For example, the loss calculation unit 204 may use another DNN that has been previously trained to extract a predetermined feature quantity, and perform the loss calculation on the image data I. t+1 The feature quantities of the converted image corresponding to time t+1 are extracted. In this case, the error based on the image feature can also be calculated based on the difference between the extracted feature quantities. Of course, the loss calculation unit 204 can also calculate the error based on the image data I t The error based on the image characteristics of the converted image corresponding to time t is calculated.

[0062] The loss calculation unit 204 may further calculate the loss by considering the error between the depth map predicted from the captured image and the depth map after image conversion by the image conversion unit 203. For example, the loss calculation unit 204 uses the depth map 411 at time t and the converted image 412 of the depth map corresponding to time t to calculate the loss 413 related to the depth map. At this time, the loss calculation unit 204 considers, for example, the difference between the pixel values ​​(depth values) of the two depth maps of the comparison object. The loss calculation unit 204 may further consider the difference between the gradients (depth smoothness) of the pixel values ​​of the two depth maps of the comparison object on this basis or instead.

[0063] Furthermore, the loss calculation unit 204 may mask hidden areas during image conversion and not use them for error calculation. For example, when obtaining a converted image corresponding to time t+1 from image data at time t, it is essentially impossible to obtain prediction accuracy for areas that appear only in the image at time t+1. Therefore, the loss calculation unit 204 uses the depth map at time t and the depth map at time t+1 to determine the hidden areas and stores mask information representing the hidden areas in the storage unit 103. The mask information representing the hidden areas may be, for example, binary data having a value of 0 or 1, and may also set areas in the converted image corresponding to time t+1 that cannot be accurately predicted to 0, and areas that cannot be accurately predicted to 1.

[0064] For example, as an example, the loss calculation unit 204 can calculate the loss L as follows.

[0065] L=(I t -I t ') 2 +(I t+1 -I t+1 ') 2 +(D t -D t ') 2 +(D t+1 -D t+1 ') 2

[0066] Among them, I t ' represents the transformed image corresponding to t, I t+1 ' represents the transformed image corresponding to t+1, D t represents the depth map predicted from the image data at time t, D t ' represents the converted image of the depth map corresponding to time t. In addition, D t+1 represents the depth map predicted based on the image data at time t+1, D t+1' represents the converted image of the depth map corresponding to time t+1.

[0067] The iteration control unit 205 controls the repeated operation (i.e., iteration) to achieve the optimization based on DNN_D201 and DNN_P202. DNN_D201 improves the accuracy of the predicted depth map by repeated operation as described above. Specifically, Figure 5 As shown, DNN_D201 receives inputs such as image data at time t, a depth map 502 predicted in the immediately preceding (e.g., N) iteration for the image data at time t, and a gradient 503 of the loss associated with the depth map. DNN_D201 then outputs a depth map 504 at the N+1th iteration. In the iterative calculation, the depth map 502 of the Nth iteration and the gradient 503 of the loss associated with the depth map are updated at each iteration. The gradient 503 of the loss associated with the depth map at time t is the depth map (D t ) (e.g., the gradient of the cost function of the loss L above based on this D t In addition, the gradient of the loss associated with the depth map at time t+1 is the depth map at time t+1 in the loss (D t+1 ) (e.g., the gradient of the cost function of the loss L above based on this D t+1 of the partial differential).

[0068] DNN_P202 improves the accuracy of the predicted camera posture by repeated calculations as described above. Figure 6 As shown, DNN_P 202 receives inputs such as image data 601 at time t and time t+1, a camera pose 602 predicted for image data 601 in the immediately preceding (e.g., N) iteration, and a gradient 603 of the loss associated with the camera pose. DNN_P 202 then outputs a camera pose 604 for iteration N+1. In the iterative computation, the camera pose 602 for iteration N and the gradient 603 of the loss associated with the camera pose are updated in each iteration. The gradient 603 of the loss associated with the camera pose (from time t to time t+1) is the gradient of the loss from time t to time t+1 (e.g., the partial derivative of the loss L described above with respect to the camera pose). Furthermore, the gradient of the loss associated with the camera pose (from time t+1 to time t) is the gradient of the loss from time t+1 to time t (e.g., the partial derivative of the loss L described above with respect to the camera pose).

[0069] The iteration control unit 205 repeats iterations until the loss 214 converges. For example, when the calculated loss 214 becomes equal to or less than a threshold after a predetermined number of times, the iteration control unit 205 determines that the loss 214 has converged and ends the iteration.

[0070]

[0071] Next, refer to Figure 7 , a series of operations for the prediction process (of camera parameters and depth maps) in the model processing unit 114 will be described. Furthermore, this process is implemented, for example, by the CPU 110 (or GPU) of the control unit 104 loading a program stored in the ROM 112 into the RAM 111 and executing it. Furthermore, the learning model of the model processing unit 114 of the control unit 104 is executed in a state where learning has been completed (i.e., the aforementioned DNN_D201 and DNN_P202 are in a state where learning has been completed).

[0072] In S701, the image data acquisition unit 113 of the control unit 104 acquires two image data from an external device such as a vehicle. The two image data are, for example, images captured by a camera mounted on the vehicle at time t and at a subsequent time t+1.

[0073] In S702, the model processing unit 114 of the control unit 104 processes the image data (I t ) and the image data at time t+1 (I t+1 )’s depth map (D t and D t+1 Specifically, in the initial iteration, DNN_D201 is input with image data (I t ), predicts the depth map (D1) at time t. In addition, DNN_D201 is input with image data (I t+1 ), the depth map (D t+1 ) is used for prediction. In addition, since the most recent iteration has not yet been executed at the time point of this step, the depth map 502 of the most recent iteration and the gradient 503 of the loss associated with the depth map are not input to DNN_D201.

[0074] In S703, the model processing unit 114 processes the image data (I t ) to the image data (I t+1 ), the relative camera pose P(I t , I t+1 ) is predicted. In addition, the model processing unit 114 performs a prediction on the image data (I t+1 ) to the image data (It ), the relative camera pose P(I t+1 , I t Specifically, in the initial iteration, DNN_P202 is fed with image data (I t ) and image data (I t+1 ), for the relative camera pose P(I t , I t+1 ) is used for prediction. In addition, DNN_P202 is input with image data (I t ) and image data (I t+1 ), for the relative camera pose P(I t+1 , I t ) is used for prediction. In addition, since the most recent iteration has not yet been performed at the time point of this step, the camera pose 602 of the most recent iteration and the gradient 603 of the loss related to the camera pose are not input to DNN_P202.

[0075] In S704, the image conversion unit 203 converts the image data (I t ) and the image data at time t+1 (I t+1 ) to perform image conversion. Specifically, the image conversion unit 203 uses the image data (I t ), the depth map at time t (D t ) and the relative camera pose P(I t , I t+1 ), generates a conversion image corresponding to time t+1. In addition, the image conversion unit 203 uses the image data (I t+1 ), the depth map at time t+1 (D t+1 ) and the relative camera pose P(I t+1 , I t ), generate a conversion image corresponding to time t.

[0076] In addition, the image conversion unit 203 uses the relative camera posture P(I t , I t+1 ) and the depth map at time t (D t ), generate the depth conversion image (D t+1 '). Further, the image conversion unit 203 uses the relative camera posture P(I t+1 , I t ) and the depth map at time t+1 (D t+1 ), generate a depth conversion image (D t ').

[0077] In S705, the loss calculation unit 204 calculates a loss, including the difference between the pre-conversion data and the converted image. The loss can be, for example, the aforementioned loss L. Specifically, the loss calculation unit 204 includes the squared error between each image at time t and time t+1, and each converted image corresponding to time t and time t+1, in the loss. Furthermore, the loss calculation unit 204 includes the squared error between each depth map at time t and time t+1, and each depth map corresponding to time t and time t+1, in the loss. In other words, the camera poses in the two directions of time t and time t+1, as well as the depth maps at time t and time t+1, are optimized to minimize the loss.

[0078] In S706, the iterative control unit 205 monitors the loss calculated by the loss calculation unit 204 and determines whether the loss satisfies the convergence condition. For example, if the calculated loss falls below the loss value threshold after a predetermined number of times, the iterative control unit 205 determines that the convergence condition is satisfied and terminates this series of processing. On the other hand, if the iterative control unit 205 determines that the calculated loss has not fallen below the loss value threshold after a predetermined number of times, the processing proceeds to S707.

[0079] In S707, the iteration control unit 205 calculates the gradient of the loss related to the depth map based on the loss calculated in S705. The gradient of the loss related to the depth map can be calculated for each depth map at time t and time t+1. The depth map at time t in the loss (D t ) is the gradient of the loss L based on D t In addition, the depth map (D t+1 ) is the gradient of the loss L based on D t+1 The partial differential of .

[0080] The iteration control unit 205 uses the gradient of the loss related to the depth map, the current iteration (ie, the prediction result of S702) and each image data (I t , I t+1 ), the depth map in the new iteration (D t 、D t+1 ) to make predictions.

[0081] In S708, the iterative control unit 205 calculates the gradient of the loss related to the camera posture based on the loss calculated in S705. The gradient of the loss related to the camera posture can be calculated for the relative camera posture P(I t , I t+1 ) and P(I t+1 , It ) is used for calculation. The camera pose P(I t , I t+1 ) is, for example, the gradient of the loss L based on the camera pose P(I t , I t+1 ). In addition, the camera posture P(I t+1 , I t ) is, for example, the gradient of the loss L based on the camera pose P(I t+1 , I t ) is the partial derivative of .

[0082] The iteration control unit 205 uses the gradient of the loss related to the camera posture, the current iteration (ie, the prediction result of S703), each image data (I t , I t+1 ), the relative camera pose in the new iteration (P(I t , I t+1 ) and P(I t+1 , I t )) to make predictions.

[0083] After the process of S708 is completed, the iterative control unit 205 proceeds to S704 and repeats the processes of S704 to S708 to continue the iterative process.

[0084] In this way, during the prediction process in the model processing unit 114, the loss is calculated by comparing the original image data with the converted image obtained based on the estimated camera parameters and depth map. The gradient of the loss associated with the depth map and the gradient of the loss associated with the camera parameters are used to optimize the depth map and camera pose through repeated calculations. This allows the camera parameters and depth map predicted by the DNN to be corrected based on the gradient, thereby improving prediction accuracy compared to predictions made through a single DNN process.

[0085] Although not explicitly mentioned in the above prediction process, the model processing unit 114 can transmit the finalized camera pose and depth map to an external device, such as a vehicle. Specifically, the model processing unit 114 can receive two images of the processing target from an external device, such as a vehicle, and return the depth map and camera parameters predicted based on these images to the external device. This allows the external device, such as a vehicle, to perform driving control, such as autonomous driving, while appropriately obtaining inference results based on the captured images from the information processing server 100.

[0086] In addition, the series of actions in the prediction process described above are actions performed by the learning model of the model processing unit 114 during the inference phase (i.e., after learning has been completed). When learning the learning model, the model processing unit 114 can simply add a process for updating the weight parameters of DNN_D201 and DNN_P202 in addition to the processes of S701 to S708 described above. Specifically, after calculating the loss in S705, backpropagation using the gradient of the loss can be performed to update the weight parameters of each DNN. Alternatively, the final camera pose and depth map, which are determined to have met the convergence conditions in S706 (i.e., after the iterative calculations have been completed), can be used to calculate the loss, including the difference between the pre-conversion data and the converted image, in the same manner as S705. In this case, all calculated losses can be transferred to the learning data and accumulated in the learning data, and backpropagation of the DNN can be performed based on the accumulated losses.

[0087] In the above example, the model processing unit 114 generates a depth map based on the acquired image. However, the model processing unit 114 may also use a depth map generated by a vehicle's lidar (light detection and ranging) or radar to estimate camera parameters.

[0088] (Second embodiment)

[0089] Next, a second embodiment of the present invention will be described. In the above embodiment, the case where prediction processing based on a learning model is performed in the information processing server 100 is described as an example. However, this embodiment is not limited to the case where the prediction processing is performed in the information processing server, and can also be applied to the case where the prediction processing is performed in a vehicle as an example of a mobile body. That is, the learning model of the model processing unit in the vehicle is operated, and the camera parameters and depth map can be predicted within the vehicle. The following describes an example of the functional configuration of a vehicle in such an embodiment. In addition, in the following example, the case where the control unit 808 is a control unit assembled in the vehicle 800 is described as an example. However, an information processing device having the configuration of the control unit 808 can also be installed in the vehicle 800. That is, the vehicle 800 can also be a vehicle equipped with an information processing device having a configuration such as a CPU 810 and a model processing unit 814 included in the control unit 808.

[0090] When the inference phase of the learning model is performed in the vehicle, for example, information processing server 100 provides information about the learned model (e.g., weight parameters of the learned neural network) to vehicle 800. This allows vehicle 800 to use the latest learned model learned using various learning data collected by information processing server 100.

[0091] <Vehicle Configuration>

[0092] Reference Figure 8 , an example of the functional configuration of vehicle 800 according to this embodiment is described. Furthermore, the various functional modules described with reference to the following figures may be combined or separated, and the functions described may be implemented by other modules. Furthermore, components described as hardware may be implemented by software, and vice versa.

[0093] The sensor unit 801 includes a camera (imaging device) that outputs images captured in front of (or, further, in the rear direction or the surrounding area of) the vehicle. The sensor unit 801 may also include a LiDAR (Light Detection and Ranging) device that outputs distance images obtained by measuring the distance in front of (or, further, in the rear direction or the surrounding area of) the vehicle. The captured images are used, for example, in prediction processing in the model processing unit 814. Furthermore, the sensor unit 801 may also include various sensors that output information such as acceleration, position information, and steering angle of the vehicle 800.

[0094] Communication unit 802 is, for example, a communication device including communication circuitry, and communicates with information processing server 100, surrounding transportation systems, and the like via mobile communications such as LTE, LTE-Advanced, or so-called 5G. Communication unit 802 obtains learning model information from information processing server 100. Furthermore, communication unit 802 receives partial or full map data, traffic information, and the like from other information processing servers and surrounding transportation systems.

[0095] The operating unit 803 includes operating components such as buttons and a touch panel installed in the vehicle 800, as well as components such as a steering wheel and a brake pedal that receive input for driving the vehicle 800. The power supply unit 804 includes a battery such as a lithium-ion battery, and supplies power to various components in the vehicle 800. The power unit 805 includes, for example, an engine or a motor that generates power for driving the vehicle.

[0096] The driving control unit 806 uses the prediction processing results (e.g., camera pose, depth map) output from the model processing unit 814 to estimate the movement of the vehicle 800 or recognize the outside world, thereby controlling the driving of the vehicle 800. Furthermore, the movement estimation, the recognition of the outside world, and the driving control can be performed using known methods. Furthermore, in the description of this embodiment, the driving control unit 806 is illustrated as a component separate from the control unit 808, but it can also be included in the control unit 808.

[0097] The storage unit 807 includes a nonvolatile, high-capacity storage device such as a semiconductor memory. It temporarily stores actual images output from the sensor unit 801 and various other sensor data output from the sensor unit 801. It also stores, for example, parameters of a learned model received from the external information processing server 100 via the communication unit 802.

[0098] The control unit 808 includes, for example, a CPU 810, a RAM 811, and a ROM 812, and controls the operation of various components of the vehicle 800. Furthermore, the control unit 808 acquires image data from the sensor unit 801 and performs the aforementioned prediction processing. The control unit 808 utilizes the functions of various components of the control unit 808, such as the model processing unit 814, by having the CPU 810 load a computer program stored in the ROM 812 into the RAM 811 and execute the program.

[0099] The CPU 810 includes one or more processors. The RAM 811 is composed of a volatile storage medium such as a DRAM and functions as a working memory for the CPU 810. The ROM 812 is composed of a non-volatile storage medium and stores computer programs executed by the CPU 810, setting values ​​for causing the control unit 808 to operate, and the like. In the following embodiments, the CPU 810 is used as an example to illustrate the processing of the model processing unit 814, but the processing of the model processing unit 814 may also be performed by one or more other processors (e.g., a GPU) not shown.

[0100] The learning parameter acquisition unit 813 acquires information (learning parameters) of the learned model from the information processing server 100 and stores the information in the storage unit 807 .

[0101] The model processing unit 814 has the same Figure 2 The model processing unit 814 executes the inference stage processing of the learning model having the same configuration as that shown in the figure. The inference stage processing executed by the model processing unit 814 can be performed in the same manner as the prediction processing shown in the first embodiment.

[0102] <Main Components for Vehicle Driving Control>

[0103] Next, refer to Figure 9 , the main components used for driving control of the vehicle 800 are explained. The sensor unit 801 captures the front of the vehicle 800, for example, and outputs the captured image data at a predetermined number of frames per second. The image data output from the sensor unit 801 is input to the model processing unit 814 of the control unit 808. The image data input to the model processing unit 814 is used for prediction processing of the camera posture and depth map (processing in the estimation stage). The model processing unit 814 receives the image data output from the sensor unit 801, performs the prediction processing, and outputs the camera posture and depth map to the driving control unit 806.

[0104] The driving control unit 806 performs motion estimation, external environment recognition, and other operations based on the camera attitude, depth map prediction results, and various sensor information such as vehicle acceleration and steering angle obtained from the sensor unit 801. Furthermore, the driving control unit 806 outputs control signals to the power unit 805, for example, to control the vehicle 800. Motion estimation, external environment recognition, and vehicle control using the camera attitude and depth map can be performed using known methods. The power unit 805 controls power generation based on the control signals from the driving control unit 806.

[0105] The learning parameter acquisition unit 813 acquires the information of the learning model transmitted from the information processing server 100. The acquired data is used to enable the DNN of the model processing unit 814 to perform learning.

[0106] Thus, in this embodiment, a learning model that predicts camera pose and depth map is executed in the model processing unit 814 of the vehicle 800. Specifically, the vehicle calculates the loss obtained by comparing the original image data with the transformed image obtained based on the estimated camera parameters and depth map. Using the gradient of the loss associated with the depth map and the gradient of the loss associated with the camera parameters, the depth map and camera pose are optimized through repeated calculations. This allows the learning model executed in the vehicle to achieve higher prediction accuracy than predictions made through a single DNN-based process.

[0107] <Summary of Implementation Methods>

[0108] (1) In the above-mentioned embodiment, an information processing method is provided. The information processing method is executed in an information processing device and is characterized in that:

[0109] The information processing method comprises:

[0110] Inputting the first image or a second image paired with the first image into a first machine learning model to predict a depth map;

[0111] Inputting the first image and the second image into a second machine learning model to predict relative movement and rotation of the camera after the first image and the second image are captured;

[0112] generating a first transformed image obtained by applying an image transformation to the first image, the image transformation being based on the predicted relative movement and rotation from the first image to the second image and the depth map predicted for the first image;

[0113] calculating a loss including an error between the first transformed image and the second image; and

[0114] The gradient of the loss related to the relative movement and rotation, the first conversion image, the first image and the second image are input into the second machine learning model, and the relative movement and rotation are repeatedly predicted to converge the loss, including the error between the new first conversion image based on the newly predicted relative movement and rotation and the second image.

[0115] According to such an embodiment, by providing the gradient of the loss related to relative movement and rotation to the machine learning model and repeatedly calculating it, the estimation accuracy of the camera posture can be improved when the machine learning model is used to estimate the camera posture.

[0116] (2) Furthermore, in the above embodiment,

[0117] The information processing method further comprises:

[0118] generating a second transformed image resulting from applying an image transformation based on the predicted relative movement and rotation from the second image to the first image and the depth map predicted for the second image to the second image, and applying an image transformation to the second image; and

[0119] calculating the loss including the error between the second transformed image and the first image,

[0120] Making the loss converge includes: inputting the gradient of the loss related to the relative movement and rotation, the second conversion image, the first image and the second image into the second machine learning model, and repeatedly predicting the relative movement and rotation.

[0121] According to this embodiment, it is possible to compare data that are correct in both the conversion from the first image to the second image and the conversion from the second image to the first image, thereby further improving estimation accuracy. In addition, it is possible to reduce the amount of learning data required when training the machine learning model.

[0122] (3) Furthermore, in the above embodiment,

[0123] The loss includes both an error between the first converted image and the second image, and an error between the second converted image and the first image.

[0124] According to such an embodiment, since constraints are imposed so as to reduce the loss of both the first converted image and the second converted image, the estimation accuracy of each can be improved.

[0125] (4) Furthermore, in the above embodiment,

[0126] The information processing method further comprises:

[0127] generating a depth map for the first converted image; and

[0128] calculating the loss including an error between a depth map for the first converted image and a depth map for the first image,

[0129] Converging the loss includes inputting the gradient of the loss associated with the depth map, the depth map for the first converted image, and the first image into the first machine learning model, and repeatedly predicting the depth map.

[0130] According to such an embodiment, by applying iterative calculations using machine learning to the estimation of the depth map, it is possible to estimate the depth map with higher accuracy.

[0131] (5) Furthermore, in the above embodiment,

[0132] The information processing device is an information processing server disposed outside the mobile body.

[0133] According to such an embodiment, it is possible to perform processing of the machine learning model in an information processing server with more abundant computing resources.

[0134] (6) Furthermore, in the above embodiment,

[0135] The information processing device is a mobile object including a vehicle.

[0136] By executing the above-described information processing method on a moving object including a vehicle, it is possible to perform calibration of an imaging device and movement prediction on the moving object with high accuracy.

[0137] The present invention is not limited to the above-described embodiment, and various modifications and changes can be made within the scope of the gist of the present invention.

Claims

1. An information processing method, which is executed in an information processing device, characterized in that: The information processing method comprises: Inputting the first image or a second image paired with the first image into a first machine learning model to predict a depth map; Inputting the first image and the second image into a second machine learning model to predict relative movement and rotation of the camera after the first image and the second image are captured; generating a first transformed image obtained by applying an image transformation to the first image, the image transformation being based on the predicted relative movement and rotation from the first image to the second image and the depth map predicted for the first image; calculating a loss including an error between the first transformed image and the second image; and The gradient of the loss related to the relative movement and rotation, the first conversion image, the first image and the second image are input into the second machine learning model, and the relative movement and rotation are repeatedly predicted to converge the loss, including the error between the new first conversion image based on the newly predicted relative movement and rotation and the second image.

2. The information processing method according to claim 1, wherein: The information processing method further comprises: generating a second transformed image resulting from applying an image transformation based on the predicted relative movement and rotation from the second image to the first image and the depth map predicted for the second image to the second image, and applying an image transformation to the second image; and calculating the loss including the error between the second transformed image and the first image, Making the loss converge includes: inputting the gradient of the loss related to the relative movement and rotation, the second conversion image, the first image and the second image into the second machine learning model, and repeatedly predicting the relative movement and rotation.

3. The information processing method according to claim 2, wherein: The loss includes both an error between the first converted image and the second image, and an error between the second converted image and the first image.

4. The information processing method according to claim 1, wherein: The information processing method further comprises: generating a depth map for the first converted image; and calculating the loss including an error between a depth map for the first converted image and a depth map for the first image, Converging the loss includes inputting the gradient of the loss associated with the depth map, the depth map for the first converted image, and the first image into the first machine learning model, and repeatedly predicting the depth map.

5. The information processing method according to any one of claims 1 to 4, characterized in that: The information processing device is an information processing server disposed outside the mobile body.

6. The information processing method according to any one of claims 1 to 4, characterized in that: The information processing device is a mobile object including a vehicle.

7. The information processing method according to claim 1, wherein: The first image and the second image are images captured by the same imaging device, but captured at different times.

8. A recording medium storing a program for causing a computer to execute each step of an information processing method executed in an information processing device, characterized in that: The information processing method comprises: Inputting the first image or a second image paired with the first image into a first machine learning model to predict a depth map; Inputting the first image and the second image into a second machine learning model to predict relative movement and rotation of the camera after the first image and the second image are captured; generating a first transformed image obtained by applying an image transformation to the first image, the image transformation being based on the predicted relative movement and rotation from the first image to the second image and the depth map predicted for the first image; calculating a loss including an error between the first transformed image and the second image; and The gradient of the loss related to the relative movement and rotation, the first conversion image, the first image and the second image are input into the second machine learning model, and the relative movement and rotation are repeatedly predicted to converge the loss, including the error between the new first conversion image based on the newly predicted relative movement and rotation and the second image.

Images