IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD, PROGRAM, METHOD FOR PRODUCING TRAINED MODEL, AND IMAGE PROCESSING SYSTEM

Abstract

To provide an image processing apparatus that can accurately estimate the other viewpoint image from a two-viewpoint image.SOLUTION: An image processing apparatus (103) has: an acquisition unit (103b) that acquires two first images obtained by photographing a single subject from two different viewpoints; and an image processing unit (103c) that inputs the two first images in a machine learning model and estimates second images from one or more viewpoints different from the two viewpoints.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing device that estimates a different viewpoint image from two viewpoint images by using machine learning DL (Deep Learning). [Background technology]

[0002] Non-Patent Document 1 discloses a method for estimating other-viewpoint images from four-viewpoint images using DL. Here, the four-viewpoint images are images of the same subject captured from four different viewpoints, and the other-viewpoint images are equivalent images of the same subject captured from one or more viewpoints other than the four viewpoints. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] NKKalantari, TC.Wang, R.Ramamoorthi, “Learning-Based View Synthesis for Light Field Camera”, SIGGRAPH Asia (2016). Summary of the Invention [Problem to be solved by the invention]

[0004] When estimating a different-viewpoint image from two-viewpoint images with horizontal parallax using the method disclosed in Non-Patent Document 1, the estimation accuracy of a rectangular-shaped object present in the two-viewpoint images decreases. This is because it is difficult to find corresponding points of the rectangular object present in the two-viewpoint images, and the distance (depth) from the camera to the rectangular object is not determined.

[0005] Therefore, an object of the present invention is to provide an image processing device capable of estimating an other-viewpoint image from two-viewpoint images with high accuracy. [Means for solving the problem]

[0006] According to one aspect of the present invention, there is provided an image processing device including: an acquisition unit that acquires two first images obtained by photographing an identical subject from two different viewpoints; and an image processing unit that estimates a second image corresponding to a viewpoint different from the two viewpoints by inputting the two first images to a machine learning model. The image processing unit generates two first feature amounts corresponding to the two first images, respectively, by a first neural network in the machine learning model, and estimates the second image based on the two first feature amounts by a second neural network in the machine learning model. do.

[0007] Other objects and features of the present invention are illustrated in the following examples. Effect of the Invention

[0008] According to the present invention, it is possible to provide an image processing device capable of estimating an other-viewpoint image from two-viewpoint images with high accuracy. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing the flow of learning of a machine learning model in each embodiment. [Diagram 2] 1 is a block diagram of an image processing system according to a first embodiment. [Diagram 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4] 11 is a flowchart relating to weight learning in each embodiment. [Diagram 5] 11 is a flowchart relating to estimation of another viewpoint image in each embodiment. [Figure 6] FIG. 11 is a block diagram of an image processing system according to a second embodiment. [Figure 7] FIG. 11 is an external view of an image processing system according to a second embodiment. [Figure 8] FIG. 11 is a block diagram of an image processing system according to a third embodiment. [Figure 9] 13 is a flowchart related to estimation of another viewpoint image in the third embodiment. [Figure 10] FIG. 2 is an explanatory diagram of an overview of each embodiment. [Figure 11]FIG. 11 is a diagram showing a learning flow of a machine learning model as a modified example of each embodiment. [Figure 12] FIG. 11 is a diagram showing the flow of learning a machine learning model as another modified example of each embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0011] First, before providing a specific description, the gist of each embodiment will be described. In each embodiment, when estimating a different-viewpoint image from two-viewpoint images with horizontal parallax using DL, the different-viewpoint image is estimated robustly for a horizontally elongated subject in the two-viewpoint images.

[0012] In image processing by DL, a neural network is used. In the neural network, a filter that is convolved with an image, a bias that is added, and an activation function that performs nonlinear transformation are used. The filter and bias are called weights, and are generated by learning from training images. In the present invention, two-viewpoint images and corresponding other-viewpoint images are used as training images. Also, a parametric activation function may be used, and its parameters may be generated simultaneously by learning.

[0013] The multidimensional array obtained in the intermediate layer of a neural network by repeatedly convolving a filter with an image, adding a bias, and performing a nonlinear transformation is called a feature map. Note that by convolving multiple filters with an image many times, each point on the feature map contains information about a corresponding large area (receptive field) in the image.

[0014] Next, an overview of each embodiment will be described with reference to FIG. 10. In each embodiment, first, two feature maps (first feature amounts) of the two-viewpoint images (first images) are generated from the two-viewpoint images by a first neural network. Next, the two generated feature maps are compared, and information (second feature amounts) on corresponding points of the two-viewpoint images is extracted. Here, the information on corresponding points of the two-viewpoint images is information for determining the amount of deviation according to the depth (distance) of the same subject in the two-viewpoint images. Finally, a different-viewpoint image (second image) is estimated from the information on corresponding points of the two-viewpoint images (second feature amounts) by a second neural network.

[0015] Each embodiment is characterized by finding corresponding points by comparing two feature maps obtained from two-viewpoint images. This makes it easier to find corresponding points for a horizontally long object than with conventional techniques that compare two-viewpoint images pixel by pixel. This is because even for a horizontally long object, clues to corresponding points (edges of the horizontally long object or texture on the surface of the horizontally long object) can be found when viewing a wide area. As a result, each embodiment makes it possible to estimate other-viewpoint images robustly for horizontally long objects in two-viewpoint images.

[0016] Here, a case where corresponding points of two-viewpoint images are found by template matching will be described. In this case, if the template is made larger, it may be possible to find corresponding points of a horizontally long object in the two-viewpoint image. However, the larger the template, the greater the computation load, so this is not realistic. On the other hand, in each embodiment, a feature map is used, so that the receptive field is expanded simply by deepening the network, and corresponding points can be found in a wider area of ​​the two-viewpoint image. As a result, it becomes easier to find corresponding points of a horizontally long object in the two-viewpoint image.

[0017] In addition, two-viewpoint images with horizontal parallax generally do not have vertical parallax information. For this reason, it is difficult to estimate an image from a two-viewpoint image at a viewpoint that is shifted from the horizontal direction (predetermined direction) determined by the two viewpoints to the vertical direction (direction perpendicular to the predetermined direction, or a direction different from the predetermined direction). However, in each embodiment, the distance from the camera to the subject can be estimated with high accuracy by comparing two feature maps obtained from each of the two-viewpoint images to find corresponding points. By estimating vertical parallax information from the map, each embodiment makes it possible to estimate an image from a two-viewpoint image at a viewpoint that is shifted from the horizontal direction determined by the two viewpoints to the vertical direction.

[0018] The image processing method described above is an example, and each embodiment is not limited to this. Details of other image processing methods will be described in the following embodiments. Each embodiment is not limited to the case where the two viewpoints of the two-viewpoint image are shifted in the horizontal direction, and can be applied to an image where the viewpoints are shifted in any direction (predetermined direction). EXAMPLES

[0019] First, an image processing system according to a first embodiment of the present invention will be described. In this embodiment, a neural network is used to learn and execute image processing for estimating an other-viewpoint image from two-viewpoint images.

[0020] Fig. 2 is a block diagram of an image processing system 100 in this embodiment. Fig. 3 is an external view of the image processing system 100. The image processing system 100 includes a learning device 101, an imaging device 102, an image estimation device 103, a display device 104, a recording medium 105, an output device 106b, and a network 107.

[0021] The learning device (image processing device) 101 includes a storage unit 101a, an image generating unit 101b, a feature generating unit 101c, a feature comparing unit 101d, an image reconstructing unit 101e, and a learning unit 101f.

[0022] The imaging device 102 has an optical system (imaging optical system) 102a and an imaging element 102b. The optical system 102a collects light incident on the imaging device 102 from the subject space. The optical image of the subject formed through the optical system 102a is received to obtain an imaging image (two-viewpoint image). The imaging element 102b is an imaging surface phase difference sensor. Here, the imaging surface phase difference sensor is an image sensor in which one pixel is composed of two independent photodiodes on the left and right. This makes it possible to simultaneously obtain two-viewpoint images in which the same subject is viewed by the right eye and the left eye. The imaging device 102 equipped with the imaging surface phase difference sensor is mainly used for the purpose of calculating the distance to the subject and automatically focusing on the subject (autofocus). The imaging device 102 of this embodiment is not limited to such a configuration, and may be, for example, a stereo camera as long as it can obtain a two-viewpoint image. In this case, the imaging element 102b is a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor. Furthermore, a memory unit that stores the acquired two-viewpoint images, a display unit that displays them, a transmission unit that transmits them to the outside, an output unit that stores them in an external storage medium, and a control unit that controls each part of the imaging device 102 are not shown in Figures 2 and 3.

[0023] The image estimation device (image processing device) 103 has a storage unit 103a, an acquisition unit 103b, and an image processing unit (estimation unit) 103c. The image estimation device 103 performs image processing to estimate a different viewpoint image from the acquired two viewpoint images. A neural network is used for the image processing, and weight information of the neural network is read from the storage unit 103a. The weights are learned by the learning device 101, and the image estimation device 103 reads the weight information from the storage unit 101a via the network 107 in advance and stores it in the storage unit 103a. The stored weight information may be the weight numerical value itself or may be in an encoded format. Details regarding weight learning and image processing using the weights will be described later.

[0024] The estimated other-viewpoint image is output to at least one of the display device 104, the recording medium 105, and the output device 106b. The display device 104 is, for example, a liquid crystal display or a projector. A user can perform editing work while checking an image being processed through the display device 104. The recording medium 105 is, for example, a semiconductor memory, a hard disk, a server on a network, or the like. The output device 106b is, for example, a printer. The image estimation device 103 has a function of performing image processing such as development processing and refocusing as necessary. Refocusing is an image processing that makes the focal position (focus position) variable by superimposing corresponding points of the multi-viewpoint image. That is, the image processing unit 103c can generate a third image (refocus image) in which the focal position is changed from the estimated other-viewpoint image.

[0025] Next, a weight learning method (a method for manufacturing a trained model) executed by the learning device 101 in this embodiment will be described with reference to Fig. 1 and Fig. 4. Fig. 1 is a diagram showing a flow of learning weights of a neural network (machine learning model). Fig. 4 is a flowchart related to weight learning. Each step in Fig. 4 is mainly executed by the image generating unit 101b, the feature generating unit 101c, the feature comparing unit 101d, the image reconstructing unit 101e, or the learning unit 101f.

[0026] First, in step S101 of FIG. 4, the image generating unit 101b generates a two-viewpoint patch (two first images) and a corresponding other-viewpoint patch (at least one second image). In this embodiment, the two-viewpoint patch is an image (first image) obtained by photographing the same subject from two different viewpoints, and the other-viewpoint patch is an image (second image) obtained by photographing the same subject from one or more viewpoints different from the two viewpoints. Note that the patch refers to an image having a predetermined number of pixels (for example, 120×120 pixels, etc.). In this embodiment, the one or more viewpoints different from the two viewpoints include a viewpoint shifted in a direction perpendicular to the direction determined by the two viewpoints, but are not limited thereto, and may include a viewpoint shifted from the direction determined by the two viewpoints.

[0027] In this embodiment, a multi-viewpoint image is acquired by a light field camera, and two-viewpoint patches and other-viewpoint patches are generated therefrom. For example, a four-viewpoint image may be acquired by a light field camera, and a two-viewpoint image equivalent to that acquired by a camera equipped with an image plane phase difference sensor may be generated from the acquired four-viewpoint images. Note that the two-viewpoint image may be generated from the four-viewpoint image by interpolating the viewpoint images based on the relationship between the viewpoints. In addition, the relationship between the viewpoints is determined by the configuration of the light field camera sensor and the image plane phase difference sensor.

[0028] However, this embodiment is not limited to this, and two-viewpoint patches and corresponding other-viewpoint patches may be generated by numerical calculation. For example, two-viewpoint patches and corresponding other-viewpoint patches may be generated by arranging a subject and a camera in a three-dimensional space of 3DCG and rendering an image equivalent to that captured by the camera.

[0029] Next, in step S102, the feature generating unit 101c generates feature maps (two first feature amounts) for each of the two-viewpoint patches from the two-viewpoint patches (two first images) using a first neural network. Note that one first neural network is prepared for each of the two-viewpoint patches (one exists for each of the two-viewpoint patches). In this embodiment, the first neural networks to which the two-viewpoint patches are input have the same network structure and weights. However, this embodiment is not limited to this, and the first neural networks to which the two-viewpoint patches are input may have different network structures or weights.

[0030] In this embodiment, each of the two viewpoint patches is input to the first neural network, but the present invention is not limited to this. For example, information on the shooting conditions and the viewpoint of the other viewpoint patch to be estimated may be input to the first neural network together with the two viewpoint patches. Here, the shooting conditions include the aperture value (F value) of the lens that shot the two viewpoint patches, the focal length, the distance from the camera to the focused subject, the ISO sensitivity and shutter speed of the sensor on the camera side, the pixel pitch, various filter characteristics, etc.

[0031] As a method of inputting information on the shooting conditions and the viewpoint of the other viewpoint image to be estimated into the neural network, for example, a method of linking an aperture image having a value assigned to the lens aperture value as a pixel value with the two-viewpoint patch and inputting it into the first neural network may be used. Alternatively, a method of inputting a value assigned to the lens aperture value into a third neural network, linking an aperture image having the obtained value as a pixel value with the two-viewpoint patch and inputting it into the first neural network may be used. That is, instead of inputting the value assigned to the lens aperture value directly into the first neural network, it may be input after being nonlinearly converted by a third neural network separately prepared. Also, instead of inputting the two-viewpoint patch itself into the first neural network, for example, an image obtained by shifting the two-viewpoint patch horizontally by sub-pixels by interpolation may be input.

[0032] Next, in step S103, the feature comparison unit 101d extracts information (second feature) about corresponding points of the two-viewpoint patches by comparing two feature maps (two first feature) generated from the two-viewpoint patches (two first images). In this embodiment, the feature comparison unit 101d calculates corresponding points of the two feature maps (similarity between the two feature maps) and compares them. More specifically, the feature comparison unit 101d calculates corresponding points of the two feature maps by performing processing based on the matrix product of the two feature maps. However, this embodiment is not limited to a comparison method using processing based on the matrix product of the feature maps. For example, classic feature amounts used for alignment such as SIFT, SURF, and HoG may be used. Alternatively, the feature maps may be shifted horizontally by sub-pixels and then subtracted for each pixel before being compared. Alternatively, the feature maps may be juxtaposed and processed by a fourth neural network for comparison.

[0033] Next, in step S104, the image reconstruction unit 101e generates an estimated patch (output image) 202 by a second neural network from information (second feature amount) on the corresponding points of the two-viewpoint patch. , two training images ) 201 obtained from other viewpoint patches (second image , Correct image ) 200 estimates (estimated images).

[0034] In this embodiment, information on the corresponding points of the two-viewpoint patch (second feature amount) is input to the second neural network, but the present invention is not limited to this. For example, information on the shooting conditions and the viewpoint of the other-viewpoint image to be estimated may be input to the second neural network together with information on the corresponding points of the two-viewpoint patch. Here, the shooting conditions are the same as in step S102. Also, the method of inputting the shooting conditions and the information on the viewpoint of the other-viewpoint image to be estimated to the second neural network is also the same as in step S102. Also, the two-viewpoint patch may be warped using information on the corresponding points of the two-viewpoint image, and then input to the second neural network.

[0035] In this embodiment, as shown in FIG. 1, the first branched network represents the first neural network, and the second network represents the second neural network. CN in FIG. 1 represents a convolutional layer. In the convolutional layer CN, the sum of the input, the convolution of the filter, and the bias is calculated, and the result is nonlinearly transformed by the activation function. The initial values ​​of each component of the filter and the bias are arbitrary, and are determined by random numbers in this embodiment. The activation function can be, for example, ReLU (Rectified Linear Unit) or a sigmoid function. The multidimensional array output in each layer except the final layer is called a feature map. The skip connection 203 is a shortcut route that combines feature maps output from discontinuous layers. The combination of feature maps may be performed by taking the sum of each element, or by concatenation in the channel direction. In this embodiment, the sum of each element is adopted.

[0036] Reference numeral 204 in Fig. 1 denotes a comparison unit that compares two feature maps (two first feature amounts) generated from two-viewpoint patches (two first images) 201, and corresponds to the function of the feature amount comparison unit 101d in Fig. 2. As described above, in this embodiment, the comparison method of the feature maps is matrix multiplication, but is not limited to this.

[0037] The elements within the dotted frame 205 in FIG. 1 represent residual blocks. Although not enclosed by dotted lines, the elements before and after the residual block in the first neural network are also residual blocks. Similarly, the second neural network is also composed of residual blocks. In this way, a neural network in which residual blocks are multi-layered is called a residual network, and is widely used in DL image processing. However, this embodiment is not limited to this, and other elements may be multi-layered to configure a neural network. For example, an inception module can be used in which convolution layers having different convolution filter sizes are arranged side by side, and the resulting feature maps are integrated to form a final feature map. A dense block having dense skip connections may also be used.

[0038] In addition, the processing load (number of convolutions) can be reduced by downsampling the feature map in a layer close to the input and upsampling the feature map in a layer close to the output, thereby reducing the size of the feature map in the intermediate layer. Here, pooling, stride, de-pixel shuffle, etc. can be used for downsampling the feature map. In addition, deconvolution (or transposed convolution), pixel shuffle, interpolation, etc. can be used for upsampling the feature map.

[0039] Next, in step S105 of FIG. 4, the learning unit 101f updates the weights of the neural network shown in FIG. 1 based on the error between the other-viewpoint patch (second image) 200 and the estimated patch (output image or estimated image) 202. Here, the weights include the components and biases of the filters of each layer. Although the backpropagation method is used to update the weights, the present embodiment is not limited to this method. In the mini-batch learning, the errors between the multiple other-viewpoint patches 200 and the corresponding estimated patches 202 are obtained, and the weights are updated. For example, the L2 norm or the L1 norm may be used as the loss function. The weight update method (learning method) is not limited to the mini-batch learning, and may be batch learning or online learning. Note that only the second neural network may be trained using the weights of the first neural network previously trained. In other words, the weights of the first neural network may be fixed without being trained, and only the second neural network may be trained. Alternatively, only the first neural network may be trained. Although not shown in Fig. 1, when a third neural network is used in step S102, its weights are updated at the same time. The same applies when a fourth neural network is used in step S103.

[0040] Next, in step S106, the learning unit 101f determines whether or not learning of the weights is completed. Completion can be determined by, for example, whether the number of iterations of learning (updating the weights) reaches a specified value, or whether the amount of change in the weights at the time of updating is smaller than a specified value. If it is determined that learning is not completed, the process returns to step S101, and multiple new two-viewpoint patches (first images) 201 and other-viewpoint patches (second images) 200 are obtained. On the other hand, if it is determined that learning is completed, the learning device 101 (learning unit 101f) ends learning and stores weight information in the storage unit 101a.

[0041] In this embodiment, the configuration shown in FIG. 1 is used as the machine learning model, but is not limited thereto. For example, the configuration shown in FIG. 11 or FIG. 12 may be used. FIG. 11 is a diagram showing the learning flow of the machine learning model as a modified example of this embodiment. As shown in FIG. 11, a machine learning model in which a two-view patch (two first images) 201 is input to a common neural network to generate an estimated patch (output image) 202 may be used. That is, the machine learning model of this embodiment does not need to be separated into two neural networks such as a first neural network and a second neural network. FIG. 12 is a diagram showing the learning flow of the machine learning model as another modified example of this embodiment. As shown in FIG. 12, a machine learning model in which two outputs from the first neural network of each of the two-view patches (two first images) 201 are directly input to a second neural network to generate an estimated patch (output image) 202 may be used. That is, the machine learning model of this embodiment does not need to use a comparison unit 204 that compares the outputs of the first neural networks.

[0042] Next, the estimation of other-viewpoint images performed by the image estimation device 103 in this embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart related to the estimation of other-viewpoint images. Each step in Fig. 5 is mainly performed by the acquisition unit 103b and the image processing unit 103c of the image estimation device 103.

[0043] First, in step S201, the acquisition unit 103b acquires information on a captured image and weights. The captured image is a two-viewpoint image as in the case of learning, and is an image transmitted from the imaging device 102 in this embodiment. The weight information is transmitted from the learning device 101 and stored in the storage unit 103a. In addition, when information on the shooting conditions and the viewpoint of the other-viewpoint image to be estimated are input to the first neural network together with the first image in step S102 of FIG. 4, the information is also acquired. Note that the imaging conditions are transmitted from the imaging device 102 together with the captured image and stored in the storage unit 103a. In addition, information on the viewpoint of the other-viewpoint image to be estimated is input by the user from, for example, the input device 106a, and stored in the storage unit 103a.

[0044] Next, in step S202, the image processing unit 103c inputs the captured image to the neural network to which the acquired weight information is applied, and estimates the other-view image. Here, the other-view image is an estimate of an image captured from one or more viewpoints different from the captured image. A neural network with the same configuration as that shown in FIG. 1 can be used to estimate the other-view image. When inputting the captured image to the neural network, it is not necessary to cut it out to the same size as the two-view patch used during learning, but in order to speed up processing, the captured image may be decomposed into multiple overlapping patches and then processed. In this case, the patches obtained after processing may be merged to form the other-view image.

[0045] As described above, the image estimation device 103 of this embodiment has an acquisition unit 103b and an image processing unit 103c. The acquisition unit 103b acquires two first images obtained by photographing the same object from two different viewpoints. The image processing unit 103c inputs the two first images into a machine learning model and estimates a second image from one or more viewpoints different from the two viewpoints. The learning device 101 of this embodiment also has an image generation unit 101b and a learning unit 101f. The image generation unit 101b generates two first images obtained by photographing the same object from two different viewpoints and a second image from one or more viewpoints different from the two viewpoints. The learning unit 101f inputs the two first images into the machine learning model and compares the output image with the second image to learn the machine learning model.

[0046] In this embodiment, the learning device 101 and the image estimation device 103 are separate devices, but the present invention is not limited to this, and the learning device 101 and the image estimation device 103 may be integrated. In other words, learning (the process shown in FIG. 4) and estimation (the process shown in FIG. 5) may be performed within an integrated device.

[0047] With the above configuration, according to this embodiment, it is possible to provide an image processing device that estimates another viewpoint image robustly for a horizontally elongated subject in two viewpoint images with parallax in the horizontal direction. EXAMPLES

[0048] Next, an image processing system according to a second embodiment of the present invention will be described. In this embodiment, similar to the first embodiment, a neural network is used to learn and execute DL image processing for estimating another viewpoint image from two viewpoint images. The image processing system of this embodiment differs from the first embodiment in that an imaging device acquires captured images (two viewpoint images) and performs image processing.

[0049] Fig. 6 is a block diagram of an image processing system 300 in this embodiment. Fig. 7 is an external view of the image processing system 300. The image processing system 300 includes a learning device (image processing device) 301 and an imaging device 302 connected via a network 303. Note that the learning device 301 and the imaging device 302 do not need to be constantly connected via the network 303.

[0050] The learning device 301 has a similar configuration to the learning device 101 of Example 1. That is, the learning device 301 has a storage unit 311, an image generation unit 312, a feature generation unit 313, a feature comparison unit 314, an image reconstruction unit 315, and a learning unit 316. In this configuration, the learning device 301 learns weights for performing image processing for estimating another viewpoint image from two viewpoint images using a neural network.

[0051] The imaging device 302 captures an image of the subject space to obtain a captured image (two-viewpoint image), and estimates a different-viewpoint image from the two-viewpoint image using the read-out weight information. Details of the image processing performed by the imaging device 302 will be described later. The imaging device 302 has an optical system (imaging optical system) 321 and an image sensor 322. The image estimation unit 323 has an acquisition unit 323a and an image processing unit 323b, and estimates a different-viewpoint image from the two-viewpoint image using weight information stored in a storage unit 324. Note that the learning of the weights of the neural network performed by the learning device 301 is the same as in the first embodiment, so the description thereof will be omitted, and only the details of the image processing performed by the imaging device 302 will be described later.

[0052] The weight information is learned in advance by the learning device 301 and stored in the storage unit 311. The imaging device 302 reads out the weight information from the storage unit 311 via the network 303 and stores it in the storage unit 324. The estimated other-view image is stored in the recording medium 325a. When an instruction on displaying the estimated other-view image is given by the user, the stored image (estimated other-view image) is read out and displayed on the display unit 325b. Note that the captured image (two-view image) already stored in the recording medium 325a may be read out, and the other-view image may be estimated by the image estimation unit 323. Also, when an instruction is given by the user, a refocus image may be generated from the estimated other-view image. The above series of controls are performed by the system controller 327.

[0053] Next, the estimation of other-viewpoint images performed by the image estimation unit 323 in this embodiment will be described. The procedure of image processing is similar to that of FIG. 5 in the embodiment 1. Each step of the image processing is mainly performed by the acquisition unit 323a or the image processing unit 323b of the image estimation unit 323.

[0054] First, in step S201, the acquisition unit 323a acquires captured images (two-viewpoint images) and weight information. The two-viewpoint images are captured by the imaging device 302 and stored in the storage unit 324. The weight information is transmitted from the learning device 301 and stored in the storage unit 324.

[0055] Next, in step S202, the image processing unit 323b inputs the captured images (two-viewpoint images) to a neural network to which the acquired weights are applied, and estimates other-viewpoint images. A neural network similar to the configuration shown in FIG. 1 can be used to estimate other-viewpoint images.

[0056] With the above configuration, according to this embodiment, it is possible to provide an image processing system that estimates another viewpoint image robustly for a horizontally elongated subject in two viewpoint images with parallax in the horizontal direction. EXAMPLES

[0057] Next, an image processing system according to a third embodiment of the present invention will be described. The image processing system of this embodiment differs from the first and second embodiments in that it includes a processing device (computer) that transmits captured images (two-viewpoint images) to be processed to an image estimation device and receives other-viewpoint images estimated from the two-viewpoint images from the image estimation device.

[0058] 8 is a block diagram of an image processing system 400 in this embodiment. The image processing system 400 includes a learning device 401, an imaging device 402, an image estimation device (second device) 403, and a computer (processing device, first device) 404. The learning device 401 and the image estimation device 403 are, for example, servers. The computer 404 is, for example, a user terminal (personal computer or smartphone). The computer 404 is connected to the image estimation device 403 via a network 405. The image estimation device 403 is connected to the learning device 401 via a network 406.

[0059] That is, the computer 404 and the image estimation device 403 are configured to be able to communicate with each other, and the image estimation device 403 and the learning device 401 are configured to be able to communicate with each other. Note that the configurations of the learning device 401 and the imaging device 402 are similar to those of the learning device 101 and the imaging device 102 in the first embodiment, respectively, and therefore their description will be omitted.

[0060] The image estimation device 403 includes a storage unit 403a, an acquisition unit 403b, a communication unit (reception unit) 403c, and an image processing unit 403d. The storage unit 403a, the acquisition unit 403b, and the image processing unit 403d are similar to the storage unit 103a, the acquisition unit 103b, and the image processing unit 103c of the image estimation device 103 of the embodiment 1, respectively. The communication unit 403c has a function of receiving a request transmitted from the computer 404, and a function of transmitting a different viewpoint image estimated by the image estimation device 403 to the computer 404.

[0061] The computer 404 has a communication unit (transmission unit) 404a, a display unit 404b, an input unit 404c, an image processing unit 404d, and a recording unit 404e. The communication unit 404a has a function of transmitting a request to the image estimation device 403 to cause the image estimation device 403 to execute processing on the captured images (two first images, two-viewpoint image), and a function of receiving an output image (other-viewpoint image) estimated by the image estimation device 403. The display unit 404b has a function of displaying various information. The information displayed by the display unit 404b includes, for example, the captured images (two-viewpoint images) to be transmitted to the image estimation device 403, the other-viewpoint image received from the image estimation device 403, and a refocused image (third image) generated from the other-viewpoint image. The input unit 404c inputs instructions from a user. The image processing unit 404d has a function of receiving the other-viewpoint image estimated by the image estimation device 403, and further performing image processing on it. Here, image processing applied to the estimated other viewpoint image includes refocus processing, etc. The recording unit 404e records the captured image acquired from the imaging device 402, the output image received from the image estimation device 403, and the like.

[0062] Next, image processing in this embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart related to estimation of other viewpoint images in this embodiment. Note that the image processing in this embodiment is equivalent to the image processing (Fig. 5) described in the first embodiment. The image processing shown in Fig. 9 is started when an instruction to start image processing is given by a user via computer 404.

[0063] First, the operation of the computer 404 will be described. First, in step S401, the computer 404 transmits a request for processing the captured image (two-viewpoint image) to the image estimation device 403. Note that the method of transmitting the two-viewpoint image to be processed to the image estimation device 403 does not matter. For example, the two-viewpoint image may be uploaded to the image estimation device 403 at the same time as step S401, or may be uploaded to the image estimation device 403 before step S401. Also, the two-viewpoint image may be an image stored on a server different from the image estimation device 403. Note that in step S401, the computer 404 may transmit an ID for authenticating a user, imaging conditions, information on the viewpoint of the other-viewpoint image to be estimated, and the like together with the request for processing the two-viewpoint image. Next, in step S402, the computer 404 receives the other-viewpoint image estimated by the image estimation device 403.

[0064] Next, a description will be given of the operation of the image estimation device 403. First, in step S501, the image estimation device 403 receives a request for processing a captured image (two-viewpoint image) transmitted from the computer 404. The image estimation device 403 determines that processing for the two-viewpoint image has been instructed, and executes the processes from step S502 onward.

[0065] Next, in step S502, the image estimation device 403 acquires weight information. The weight information is information (trained model) learned in the same manner as in the first embodiment (FIG. 4). The image estimation device 403 may acquire the weight information from the learning device 401, or may acquire weight information previously acquired from the learning device 401 and stored in the storage unit 403a. The next step S503 is the same as step S202 in the first embodiment. Next, in step S504, the image estimation device 403 transmits the estimated other-viewpoint image to the computer 404.

[0066] As described above, as in this embodiment, the image estimation device 403 may be configured to be controlled by using the computer 404 communicably connected to the image estimation device 403 .

[0067] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0068] According to each embodiment, when estimating a different-viewpoint image from two-viewpoint images having parallax in the horizontal direction using machine learning DL, the different-viewpoint image can be robustly estimated for a horizontally elongated subject in the two-viewpoint images. Therefore, according to each embodiment, it is possible to provide an image processing device, an image processing method, a program, a method for manufacturing a trained model, and an image processing system capable of estimating a different-viewpoint image from two-viewpoint images with high accuracy.

[0069] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0070] 103 Image Estimation Device (Image Processing Device) 103b Acquisition Department 103c Image processing unit

Claims

1. an acquisition unit that acquires two first images obtained by photographing the same subject from two different viewpoints; and an image processing unit that estimates a second image corresponding to a viewpoint different from the two viewpoints by inputting the two first images into a machine learning model; The image processing unit includes: generating two first feature amounts corresponding to the two first images, respectively, by a first neural network in the machine learning model; The image processing device is characterized in that the second image is estimated based on the two first feature amounts by a second neural network in the machine learning model.

2. 2. The image processing device according to claim 1, wherein the image processing unit generates a second feature based on the two first feature values, and estimates the second image based on the second feature values ​​using the second neural network.

3. the two first feature amounts are two feature maps, The image processing apparatus according to claim 2 , wherein the second feature amount is generated by calculating corresponding points in the two feature maps.

4. 4. The image processing device according to claim 1, wherein the image processing unit processes each of the two first images using the first neural network having the same network structure and weights.

5. 5 . The image processing apparatus according to claim 1 , wherein the viewpoint different from the two viewpoints includes a viewpoint that is shifted from a direction of a line segment connecting the two viewpoints.

6. 6. The image processing apparatus according to claim 5, wherein the viewpoint different from the two viewpoints includes a viewpoint that is shifted in a direction perpendicular to a direction of a line segment connecting the two viewpoints.

7. The image processing device according to claim 1 , wherein the image processing unit generates a third image by changing a focus position in the second image.

8. an acquisition step of acquiring two first images obtained by photographing the same subject from two different viewpoints; and estimating a second image corresponding to a viewpoint different from the two viewpoints by inputting the two first images into a machine learning model; The estimation step includes: generating two first feature amounts corresponding to the two first images, respectively, by a first neural network in the machine learning model; An image processing method comprising: estimating the second image based on the two first features by a second neural network in the machine learning model.

9. A program for causing a computer to execute the image processing method according to claim 8.

10. an image generating unit that generates two training images obtained by photographing the same subject from two different viewpoints and a correct answer image corresponding to a viewpoint different from the two training images; a learning unit that estimates an output image by inputting the two training images into a machine learning model, and learns the machine learning model based on the output image and the correct answer image; The learning unit is generating two first features corresponding to the two training images, respectively, by a first neural network in the machine learning model; A learning device characterized in that the output image is estimated based on the two first features by a second neural network in the machine learning model.

11. a feature generator that generates two features corresponding to the two training images, respectively, by a first neural network in the machine learning model; an image reconstruction unit that estimates the output image based on the two feature amounts by a second neural network in the machine learning model; 11. The learning device according to claim 10, wherein the learning unit performs learning of at least one of the first neural network and the second neural network.

12. an image generating step of generating two training images obtained by photographing the same subject from two different viewpoints and a correct answer image corresponding to a viewpoint different from the two training images; a learning step of estimating an output image by inputting the two training images into a machine learning model, and learning the machine learning model based on the output image and the correct answer image; The learning step includes: generating two first features corresponding to the two training images, respectively, by a first neural network in the machine learning model; A learning method comprising: estimating the output image based on the two first features by a second neural network in the machine learning model.

13. A program for causing a computer to execute the image processing method according to claim 12.

14. an image generating step of generating two training images obtained by photographing the same subject from two different viewpoints and a correct answer image corresponding to a viewpoint different from the two training images; A learning process of estimating an output image by inputting the two training images into a machine learning model, and learning the machine learning model based on the output image and the ground truth image, The learning step includes: generating two first features corresponding to the two training images, respectively, by a first neural network in the machine learning model; A method for generating a trained model, characterized in that the output image is estimated based on the two first features by a second neural network in the machine learning model.

15. 8. An image processing system comprising the image processing device according to claim 1 and a control device capable of communicating with the image processing device, The image processing device according to claim 1, wherein the control device has a transmission unit that transmits a request for executing processing on the two first images.

Images