Image processing device, three-dimensional model generating device, and program

Abstract

To accurately compensate for focal blur included in an image.SOLUTION: An image processing device includes: a function estimation unit that estimates a local point spread function of a first image on the basis of a three-dimensional model based on a first image obtained by imaging a subject and imaging conditions when the first image is imaged; and a blur compensation unit that generates a second image in which focal blur is compensated for each local area of the first image on the basis of the point spread function.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, a three-dimensional model generating device, and a program. [Background technology]

[0002] There are known techniques for compensating for out-of-focus blurring in an image. For example, Non-Patent Document 1 discloses a technique for calculating a local point spread function (PSF) of an image based on a partial auto-correlation (PARCOR) analysis method. Non-Patent Document 1 also discloses a technique for estimating three-dimensional spatial information based on a point spread function obtained by the PARCOR analysis method. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Akihiko Sugiura and Mitsunobu Kamata, "A method for extracting regions in 3D space from a single image using focus blur estimation," Transactions on Electrical Engineers of Japan, Vol. 121, No. 9, 2001 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, there is room for improving the accuracy of compensation for defocus blur. For example, in the PARCOR analysis method, the correct point spread function may not be obtained depending on the texture of the object. If the correct point spread function is not used, it is difficult to compensate for defocus blur with high accuracy.

[0005] In view of the above technical problems, one aspect of the present disclosure has an object to accurately compensate for focus blur. [Means for solving the problem]

[0006] An image processing device according to one aspect of the present disclosure includes a function estimation unit that estimates a local point spread function of a first image based on a three-dimensional model based on a first image of a subject and on imaging conditions when the first image was captured, and a blur compensation unit that generates a second image in which focus blur is compensated for for each local area of ​​the first image based on the point spread function. Effect of the Invention

[0007] According to one aspect of the present disclosure, it is possible to compensate for defocus with high accuracy. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of the overall configuration of an image processing system according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of a hardware configuration of a computer. [Diagram 3] FIG. 2 is a block diagram showing an example of a functional configuration of the three-dimensional model generating device. [Figure 4] 1 is a block diagram showing an example of a functional configuration of a blur compensation device. [Diagram 5] 1 is a flowchart illustrating an example of a three-dimensional model generating method. [Figure 6] 13 is a flowchart illustrating an example of a blur compensation process. [Figure 7] FIG. 11 is a block diagram showing an example of the overall configuration of an image processing system according to a second embodiment. [Figure 8] FIG. 2 is a block diagram showing an example of a functional configuration of the image processing device. [Figure 9] 1 is a flowchart illustrating an example of an image correction method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0010] [First embodiment] The first embodiment of the present disclosure is an image processing system that generates a three-dimensional model of a subject based on a plurality of real-life images of the subject. The image processing system in this embodiment has a function of compensating for out-of-focus blurring that occurs in the real-life images and generating a detailed three-dimensional model.

[0011] Conventionally, various techniques for compensating for out-of-focus blurring in an image have been known. For example, there is the unsharp masking technique. The unsharp masking technique is a technique for obtaining an image with emphasized high-frequency components by superimposing a film on which an image to be compensated for has been further blurred and toned-inverted, and a film on which the image to be compensated for has been captured, and then printing the resulting image onto photographic paper or the like (or viewing the resulting image directly as a transparent original). There is also image processing software that digitally executes a process similar to that of the unsharp masking technique.

[0012] When the point spread function of the focus blur is known or can be estimated, there is a technique for compensating the focus blur based on the point spread function. In this type of technique, a convolution of the inverse function of the point spread function or a filter based on the point spread function (e.g., a Wiener filter) is applied to an image in which focus blur occurs (hereinafter, also referred to as a "blurred image"). As the point spread function, for example, a pillbox function, which is an ideal lens blur, or a Gaussian function, which approximates a realistic lens blur, is used.

[0013] There is a technique for modeling the process of blurring an image using a functional F. This type of technique models the process of blurring an image X as F[X], and determines the result of blurring compensation as X such that the error ||YF[X]|| between the result of applying the functional F to the image X, F[X], and the blurred image Y is minimized. Here, ||·|| represents an arbitrary norm for the function, for example the H2 norm. Therefore, the result of blurring compensation is expressed by equation (1).

[0014]

number

[0015] In this case, a regularization term intended to suppress high frequency noise may be added to make Equation (2) the blur compensation result.

[0016]

number

[0017] where λ is a positive constant and ||·|| A and ||·|| B are arbitrary norms for functions. The norm ||| A and ||·|| B may be different norms or may be the same norm. A typical example is ||·|| A is the H2 norm, and ||·|| B is the TV (Total Variation) norm.

[0018] There is a technology that uses artificial intelligence (AI) to obtain blur compensation results from a blurred image. This type of technology estimates a focused image from a blurred image based on a trained model that is machine-learned from teacher data consisting of a pair of an image without focus blur (hereinafter also referred to as a "focused image") and a blurred image. For example, a convolutional neural network (CNN) can be used for the artificial intelligence.

[0019] Non-Patent Document 1 discloses a technique for calculating a local point spread function of an image based on the PARCOR analysis method, and also discloses a technique for estimating three-dimensional spatial information based on the point spread function obtained by the PARCOR analysis method.

[0020] In the conventional techniques described above, there is room for improving the accuracy of blur compensation. For example, when applying techniques such as an unsharp mask or a Wiener filter, the point spread function needs to be known. Non-Patent Document 1 discloses that the point spread function can be estimated by the PARCOR analysis method or the like, but the PARCOR analysis method may not be able to obtain a correct point spread function depending on the texture of the object. For example, it is difficult to distinguish between a case where an object with a sharp texture is blurred by the optical system and a case where an object with an originally blurred texture is in focus.

[0021] An embodiment of the present disclosure aims to accurately compensate for out-of-focus blur. In an embodiment of the present disclosure, blur compensation is applied to each of a plurality of real-life images obtained by capturing an object from a plurality of viewpoints, and a three-dimensional model is generated based on the real-life images in which the out-of-focus blur has been compensated. In one aspect, according to the present embodiment, a fine three-dimensional model in which the out-of-focus blur has been compensated can be generated based on the images in which the out-of-focus blur has occurred.

[0022] <Overall composition> The overall configuration of an image processing system according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the overall configuration of an image processing system according to this embodiment.

[0023] 1, an image processing system 1000 in this embodiment includes M imaging devices 10 and a three-dimensional model generating device 20. Each imaging device 10 and the three-dimensional model generating device 20 are electrically connected. Here, M is an integer of 2 or more. M may be, for example, about 10 to 100. In this embodiment, M=26.

[0024] Hereinafter, when distinguishing between the M blur compensation devices 22, they will be denoted by sub-numbers such as "blur compensation device 22-1" and "blur compensation device 22-2."

[0025] The imaging device 10 is an optical device that captures an image of a subject 1 from a specific viewpoint. The M imaging devices 10 included in the image processing system 1000 are arranged so as to capture images of the same subject 1 from different viewpoints, and the angles of view, shooting conditions, etc. are adjusted. The M imaging devices 10 may be multiple cameras mounted on one camera array system.

[0026] The three-dimensional model generating device 20 is an information processing device such as a personal computer, a workstation, or a server that generates a three-dimensional model (hereinafter also referred to as an "output model") 2 of the subject 1. Electronic data representing M real-life images captured by the imaging device 10 is input to the three-dimensional model generating device 20. The three-dimensional model generating device 20 generates the output model 2 based on the M real-life images, and outputs electronic data representing the output model 2.

[0027] The overall configuration of the image processing system 1000 shown in Fig. 1 is an example, and various system configuration examples are possible depending on the application and purpose. For example, the image processing system 1000 may include a plurality of three-dimensional model generating devices 20. For example, the three-dimensional model generating device 20 may be realized by a plurality of computers, or may be realized as a cloud computing service. The division of devices such as the imaging device 10 and the three-dimensional model generating device 20 shown in Fig. 1 is an example.

[0028] <Hardware configuration> The hardware configuration of the image processing system 1000 in this embodiment will be described with reference to Fig. 2. The three-dimensional model generating device 20 is realized by, for example, a computer. Fig. 2 is a block diagram showing an example of the hardware configuration of the computer.

[0029] 2, the computer 500 includes a central processing unit (CPU) 501, a read only memory (ROM) 502, a random access memory (RAM) 503, a hard disk drive (HDD) 504, an input device 505, a display device 506, a communication interface (I / F) 507, and an external I / F 508. The CPU 501, the ROM 502, and the RAM 503 form a so-called computer. Each piece of hardware of the computer 500 is connected to each other via a bus line 509. The input device 505 and the display device 506 may be connected to the external I / F 508 for use.

[0030] The CPU 501 is a calculation device that reads out programs and data from a storage device such as the ROM 502 or the HDD 504 onto the RAM 503 and executes processing to realize the overall control and functions of the computer 500. The computer 500 may have a GPU (Graphics Processing Unit) in addition to or instead of the CPU 501.

[0031] The ROM 502 is an example of a non-volatile semiconductor memory (storage device) that can retain programs and data even when the power is turned off. The ROM 502 functions as a main storage device that stores various programs and data required for the CPU 501 to execute various programs installed in the HDD 504. Specifically, the ROM 502 stores boot programs such as a Basic Input / Output System (BIOS) and an Extensible Firmware Interface (EFI) that are executed when the computer 500 is started, and data such as an Operating System (OS) setting and a network setting.

[0032] The RAM 503 is an example of a volatile semiconductor memory (storage device) in which programs and data are erased when the power is turned off. The RAM 503 is, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM). The RAM 503 provides a working area in which various programs installed in the HDD 504 are expanded when the CPU 501 executes them.

[0033] The HDD 504 is an example of a non-volatile storage device that stores programs and data. The programs and data stored in the HDD 504 include an OS, which is basic software that controls the entire computer 500, and applications that provide various functions on the OS. Note that the computer 500 may use a storage device that uses a flash memory as a storage medium (e.g., an SSD (Solid State Drive) or the like) instead of the HDD 504.

[0034] The input device 505 includes a touch panel, operation keys or buttons, a keyboard or mouse, a microphone for inputting sound data such as voice, etc., which are used by the user to input various signals.

[0035] The display device 506 is composed of a display such as a liquid crystal display or an organic EL (Electro-Luminescence) display for displaying a screen, a speaker for outputting sound data such as voice, and the like.

[0036] The communication I / F 507 is an interface that connects to a communication network and enables the computer 500 to perform data communication.

[0037] The external I / F 508 is an interface with external devices, such as a drive device 510.

[0038] The drive device 510 is a device for setting the recording medium 511. The recording medium 511 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, or a magneto-optical disk. The recording medium 511 may also include semiconductor memories that record information electrically, such as a ROM or a flash memory. This allows the computer 500 to read and / or write data from and to the recording medium 511 via the external I / F 508.

[0039] The various programs to be installed in the HDD 504 are installed, for example, by setting the distributed recording medium 511 in a drive device 510 connected to the external I / F 508 and reading out the various programs recorded in the recording medium 511 by the drive device 510. Alternatively, the various programs to be installed in the HDD 504 may be installed by being downloaded from a network different from the communication network via the communication I / F 507.

[0040] <Functional configuration> The functional configuration of the image processing system in this embodiment will be described with reference to FIGS.

[0041] <3D model generation device> Fig. 3 is a block diagram showing an example of a functional configuration of a three-dimensional model generating device. As shown in Fig. 3, the three-dimensional model generating device 20 includes a modeling unit 21 (an example of a first modeling unit), M blur compensation devices 22 (22-1, 22-2), and a modeling unit 23 (an example of a second modeling unit). In Fig. 3, only two blur compensation devices 22-1, 22-2 of the M blur compensation devices 22 are shown.

[0042] The modeling unit 21, the blur compensation device 22, and the modeling unit 23 are realized by processing that is executed by a CPU 501 in accordance with a program loaded onto a RAM 503 from an HDD 504 shown in Fig. 2. The blur compensation device 22 may be realized by a computer 500 shown in Fig. 2, or may be an independent device having a housing different from that of the three-dimensional model generating device 20.

[0043] A real-life image 11 of a subject 1 captured by an imaging device 10 and a camera parameter θ of the imaging device 10 are input to the three-dimensional model generating device 20. The real-life images 11 and the camera parameters θ are input in the same number as the imaging devices 10 (i.e., M of each). The real-life image 11 is an example of a first image.

[0044] In the real-life images 11, out-of-focus blur may occur when the subject 1 is captured. The out-of-focus blur may occur over the entire real-life image 11, or may occur locally in a part of the real-life image 11. The M real-life images 11 may include an in-focus image without out-of-focus blur.

[0045] The camera parameter θ is information indicating the imaging conditions when the real image 11 is captured by the imaging device 10. In this embodiment, the camera parameter θ may include, for example, information regarding the relative position and relative attitude between the imaging device 10 and the subject 1, the angle of view of the imaging device 10, the focal distance, and the lens aperture.

[0046] Specifically, the camera parameter θ is expressed as the camera position T=[T X ,T Y ,T Z ] T , the rotation matrix R from the world coordinate system to the camera coordinate system, and the image coordinate c of the point where the optical axis intersects with the image plane = [c x ,c y ] T , the horizontal and vertical grid spacing r of the pixel array of the image x and r y(In the case of a rectangular grid with each side parallel to the x-axis and y-axis), the rotation matrix R may include a focal length f, a focus position d, and an aperture ratio F. Here, the rotation matrix R is an example of orientation information. The focal length f is an example of information related to the angle of view. The aperture ratio F is an example of information related to the lens aperture. T (superscript T) is the transpose of a vector.

[0047] The camera coordinate system is a coordinate system fixed to the imaging device 10. The ξ and η axes of the camera coordinate system are parallel to the horizontal and vertical axes of the image plane, and the ζ axis is perpendicular to the image plane, with the direction from the imaging device 10 toward the subject 1 defined as positive. The image coordinate system is a coordinate system of an actual image captured by the imaging device 10. The x and y axes of the image coordinate system are parallel to and oriented in the same direction as the ξ and η axes of the camera coordinate system, respectively. In the image coordinate system, coordinates are expressed in units of the number of pixels, and the origin can be set at any location. The origin may be, for example, the top left corner of the image.

[0048] The modeling unit 21 generates a three-dimensional model (hereinafter also referred to as a "temporary model") 12 of the subject 1 based on M real-life images 11-1,...,11-M and M camera parameters θ-1,...,θ-M. The temporary model 12 is an example of a first three-dimensional model.

[0049] The temporary model 12 may be a model that indicates only the shape of the subject 1. The modeling unit 21 may generate the temporary model 12 based on a part of the M real-life images 11-1, . . . , 11-M. The number of real-life images 11 used to generate the temporary model 12 may be, for example, five.

[0050] The position information of each part included in the temporary model 12 is recorded in the world coordinate system. In this embodiment, the temporary model 12 is described by a point cloud. Therefore, in the temporary model 12, the shape of the subject 1 is represented by N vertex coordinates P (n) =[P X (n) ,P Y (n) ,P Z (n) ] Twhere N is a natural number, n∈{1, 2, , N}.

[0051] Any method can be used as a method for generating the temporary model 12. For example, the method for generating the temporary model 12 may be a method for constructing the temporary model 12 using a three-dimensional shape estimation method such as a stereo method, or a method for constructing the temporary model based on measurement data obtained by a distance sensor or the like.

[0052] For example, the modeling unit 21 may use a photogrammetry technique to generate the temporary model 12. Photogrammetry is a technique for constructing a three-dimensional model of a subject based on images from multiple viewpoints and their camera parameters.

[0053] Also, for example, the modeling unit 21 may generate the temporary model 12 by a shape from shading (SFS) technique. The SFS technique is a technique for estimating the shape of a subject based on a single-viewpoint image and information about known illumination. Note that the information about illumination includes information indicating the position and brightness of the illumination.

[0054] Furthermore, for example, the modeling unit 21 may generate the temporary model 12 by photometric stereo, which is a method for estimating the shape of a subject based on single-viewpoint images captured under two or more different lighting conditions and information about the lighting under each lighting condition.

[0055] Furthermore, for example, the modeling unit 21 may use artificial intelligence to generate the temporary model 12. The method using artificial intelligence is, for example, a method of estimating depth information from an image of a single viewpoint by artificial intelligence, and estimating the shape of the subject based on the estimated depth information and camera parameters.

[0056] The blur compensation device 22-m (m∈{1,2,...,M}) generates an image 13-m (hereinafter also referred to as a "compensated image") by applying blur compensation to the real-life image 11-m captured by the imaging device 10-m, based on the temporary model 12 generated by the modeling unit 21 and the camera parameter θ-m. In this embodiment, M blur compensation devices 22 each process one real-life image 11 in parallel, thereby generating M compensated images 13. The compensated image 13 is an example of a second image.

[0057] The 3D model generating device 20 may be configured to include less than M blur compensation devices 22, with each blur compensation device 22 processing a plurality of real-life images 11. The 3D model generating device 20 may be configured to include one blur compensation device 22, with each blur compensation device 22 processing M real-life images 11.

[0058] The modeling unit 23 generates an output model 2 based on the M compensated images 13-1, . . . , 13-M generated by the blur compensation devices 22-1, . . . , 22-M and the M camera parameters θ-1, . . . , θ-M. The output model 2 is an example of a second three-dimensional model.

[0059] The output model 2 may be a three-dimensional model that indicates at least one of the texture and the material quality in addition to the shape of the subject 1. The output model 2 may be a three-dimensional model that indicates only the shape of the subject 1, similar to the temporary model 12. The modeling unit 23 may generate the output model 2 based on all M real-life images 11.

[0060] The method of generating the output model 2 may be the same as or different from the method of generating the temporary model 12. When the method of generating the output model 2 is the same as the method of generating the temporary model 12, the three-dimensional model generation device 20 may include only one modeling unit, and the temporary model 12 and the output model 2 may be generated by the modeling unit.

[0061] <Blur compensation device> 4 is a block diagram showing an example of the functional configuration of the blur compensation device 22. As shown in FIG 4, the blur compensation device 22 includes a depth estimation unit 101, a coordinate estimation unit 102, a function estimation unit 103, a map generation unit 104, a hidden surface processing unit 105, and a blur compensation unit 106.

[0062] The depth estimation unit 101 estimates the depth value of each vertex included in the temporary model 12 based on the temporary model 12 and the camera parameter θ-m. The depth value is the distance from the image capture device 10 to each part of the subject 1. Specifically, the depth estimation unit 101 estimates the camera coordinate [ξ (n) ,η (n) ,ζ (n) ] T The value of the ζ-axis of the camera coordinates is estimated. (n) However, the vertex coordinate P (n) The depth value is up to

[0063]

number

[0064] The coordinate estimation unit 102 calculates image coordinates onto which each vertex included in the temporary model 12 is projected, based on the temporary model 12 and the camera parameter θ-m. Specifically, the coordinate estimation unit 102 calculates the image coordinates onto which each vertex included in the temporary model 12 is projected, based on the lattice interval r x ,r y Using the focal length f and the image coordinate c, the image coordinate p is calculated by equation (4). (n) =[p x (n) ,p y (n) ] T Calculate.

[0065]

number

[0066] The function estimation unit 103 estimates the depth value ζ (n)and the camera parameter θ-m, the function estimation unit 103 estimates a point spread function for each local area. In this embodiment, the function estimation unit 103 estimates a point spread function for each pixel, but the function estimation unit 103 may estimate a point spread function for each region consisting of multiple pixels.

[0067] When the camera lens is ideal, the point spread function is a cylindrical pillbox function. When the lens is adjusted to focus on a plane of depth d, the diameter of the point spread function (hereinafter also referred to as the "diameter of the circle of confusion") B is a function of the absolute value (depth value) ζ of the optical axis (ζ-axis) direction component of the vector from the first principal point of the camera lens to the subject's part. Specifically, the diameter B (ζ) of the circle of confusion is expressed by Equation (5) using the focal length f and the focal distance F included in the camera parameter θ-m.

[0068]

number

[0069] Vertex P included in temporary model 12 (n) The depth value of is ζ (n) and vertex P (n) The image on the image plane is [p x (n) ,p y (n) ] T Therefore, the image coordinates [p x (n) ,p y (n) ] is the diameter of the circle of confusion at B(ζ (n) )

[0070] The point spread function φ due to the pillbox function of diameter d is d (x, y) is expressed by equation (6). Therefore, the diameter of the circle of confusion B(ζ (n) ) is the image coordinate [p x (n) ,p y (n) ] is information that specifies the point spread function at

[0071]

number

[0072] The map generating unit 104 generates a map (hereinafter also referred to as a "circle of confusion diameter map") indicating the arrangement of the point spread function for each local area estimated by the function estimating unit 103. Specifically, the map generating unit 104 generates a circle of confusion diameter map M expressed by equation (7).

[0073]

number

[0074] The hidden surface processor 105 performs hidden surface processing on the circle of confusion diameter map generated by the map generator 104. The hidden surface processor 105 omits parts of the vertices of the temporary model 12 that are hidden and cannot be seen from the viewpoint.

[0075] When a three-dimensional model is composed of multiple vertices, multiple different vertices may be projected onto the same or nearby image coordinates. Here, nearby means that the image coordinates are within a predetermined range of each other. For example, when the density of the point cloud that composes the three-dimensional model of the subject is high, the range of the neighborhood may be determined by, for example, the Chebyshev distance (chessboard distance or L∞ - The distance between adjacent pixels (also called the distance between adjacent pixels) can be within one pixel. On the other hand, when the density of the point cloud is low, the neighborhood range can be set according to the spacing between successive surfaces of the subject when they are projected onto the image. Here, a high density point cloud means that when the point cloud is projected onto the image plane, the successive surfaces of the subject are projected at a density equal to or denser than the grid spacing of the pixel arrangement on the image.

[0076] If multiple different vertices are projected onto the same or nearby image coordinates, then the vertex with the deepest depth value ζ (n) The diameter of the circle of confusion B(ζ (n) ) into image coordinates [px (n) ,p y (n) ] is the diameter of the circle of confusion at the point 1 in the image. This allows hidden surface removal to be achieved.

[0077] Specifically, the hidden surface processor 105 first sets D(x, y) = ∞ and M(x, y) = 0 for all pixel positions ∀(x, y) on the image. Next, the hidden surface processor 105 executes equation (9) only when equation (8) is satisfied for all integers n between 1 and N.

[0078]

number

[0079] However, w x and w y is an integer greater than or equal to 0, and is a parameter that defines the range of the neighborhood. In particular, w x =w y If =0, equation (9) should be executed instead of equation (8) only when equation (10) is satisfied.

[0080]

number

[0081] The hidden surface processor 105 outputs the confusion circle diameter map M obtained by equation (9) as the confusion circle diameter map M after hidden surface processing.

[0082] The blur compensation unit 106 generates a compensated image 13-m by applying blur compensation to each local area of ​​the real-life image 11-m based on the point spread function for each local area estimated by the function estimation unit 103. Specifically, the blur compensation unit 106 performs image restoration on the real-life image 11-m, based on the circle of confusion diameter map M output from the hidden surface processing unit 105, to cancel the point spread function for each local area.

[0083] As an example, the blur compensation unit 106 may derive a compensated image by optimization. As another example, the blur compensation unit 106 may derive a compensated image by deconvolution. The method of applying blur compensation is not limited to these, and any method that compensates for focus blur based on a point spread function may be used.

[0084] <Deriving a compensated image through optimization> When an ideal image without focus blur is I and a blurred image with focus blur is J, the process of blur generation by the circle of confusion diameter map M can be modeled by Equation (11).

[0085]

number

[0086] Where Φ M [I] is a functional from image to image, which means that image I is transformed into a blurred image by the circle of confusion diameter map M.

[0087] Given a blurred image J and a circle of confusion diameter map M, the defocus-compensated image ^I can be derived by optimizing equation (12) using a functional E[J,K] that quantifies the error between the images with a scalar value.

[0088]

number

[0089] However, the functional E[J,K] may be defined by the square error of pixel values. For example, the functional [J,K] may be defined by Equation (13).

[0090]

number

[0091] The optimization of equation (12) is done by perturbing the pixel values ​​of image I to obtain the E[J,Φ M This can be achieved by repeatedly searching for the smallest possible [I].

[0092] More efficiently, we can say that E[J,Φ M Based on the increment ∂E / ∂I of [I], the compensated image ̂I can be calculated by a nonlinear least squares method (e.g., the Levenberg-Marquardt method), where the increment ∂E / ∂I can be calculated analytically or approximated numerically.

[0093] The integral operations of equations (11) and (13) may be implemented by summation operations by discretizing the image coordinates.

[0094] <<Deriving the compensated image by deconvolution>> First, the point spread function φ d The transfer function of (x, y) is calculated by the point spread function φ d It is obtained by performing a two-dimensional Fourier transform on (x,y).

[0095]

number

[0096] where F[·] is the Fourier transform, and u and v are the horizontal and vertical spatial frequencies.

[0097] Next, the transfer function G d (u, v) is determined by equation (15). As an example of the filter, a Wiener filter or the like can be used.

[0098]

number

[0099] however,·* (Superscript *) denotes complex conjugate. The function Γ(u,v) is a non-negative real-valued function that models noise at spatial frequency (u,v). The function Γ(u,v) takes a larger value when larger noise is expected. More simply, it can be expressed as Γ(u,v)=γ (γ is a non-negative real constant) regardless of spatial frequency (u,v).

[0100] Next, the impulse response of the filter G d Calculate (x,y). Impulse response G d (x,y) is the transfer function G of the filter according to equation (16). d It is obtained by performing the inverse Fourier transform of (u,v).

[0101]

number

[0102] However, F -1 [·] is the inverse Fourier transform.

[0103] Impulse Response G d (x, y) may be found by applying a window function w as in equation 17. As an example, the window function w may be a rectangular window, a Hamming window, or a Hanning window.

[0104]

number

[0105] Then, equation (18) is calculated based on the blurred image J and the confusion circle diameter map M. This results in a compensated image ^I.

[0106]

number

[0107] The integral range of equation (18) may be truncated within a finite range (for example, the tail portion where the absolute value of the window function w in equation (17) is less than a predetermined value is truncated). Moreover, equation (18) may be implemented by a summation operation by discretizing pixel positions.

[0108] <Processing Procedure> A three-dimensional model generating method executed by the image processing system 1000 in this embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing an example of the three-dimensional model generating method.

[0109] In step S1, M imaging devices 10-m capture an image of the subject 1. The imaging devices 10-m output real-life images 11-m of the subject 1 and camera parameters θ-m used when capturing the real-life images 11-m. The real-life images 11-m and the camera parameters θ-m are input to the three-dimensional model generating device 20.

[0110] In step S2, the modeling unit 21 of the three-dimensional model generating device 20 acquires the M real-life images 11-1,...,11-M and the M camera parameters θ-1,...,θ-M input to the three-dimensional model generating device 20. Next, the modeling unit 21 generates a temporary model 12 based on the real-life images 11-1,...,11-M and the camera parameters θ-1,...,θ-M. Then, the modeling unit 21 sends the generated temporary model 12 to each of the blur compensation devices 22-m.

[0111] In step S3, the blur compensation device 22-m of the three-dimensional model generating device 20 receives the temporary model 12 from the modeling unit 21. The blur compensation device 22-m also acquires the real-life image 11-m and the camera parameters θ-m input to the three-dimensional model generating device 20. Next, the blur compensation device 22-m generates a compensated image 13-m by applying blur compensation to the real-life image 11-m based on the temporary model 12 and the camera parameters θ-m. Then, the blur compensation device 22-m sends the compensated image 13-m to the modeling unit 23.

[0112] In step S4, the modeling unit 23 of the three-dimensional model generating device 20 receives the compensated images 13-m from each of the blur compensation devices 22-m. The modeling unit 23 also acquires the M camera parameters θ-1,...,θ-M input to the three-dimensional model generating device 20. Next, the modeling unit 23 generates an output model 2 based on the compensated images 13-1,...,13-M and the camera parameters θ-1,...,θ-M. Then, the modeling unit 23 outputs electronic data indicating the generated output model 2.

[0113] <Blur compensation processing> The blur compensation process (step S3 in FIG. 5) in this embodiment will be described in detail with reference to FIG 6. FIG 6 is a flowchart showing an example of the blur compensation process.

[0114] In step S101, the depth estimation unit 101 of the blur compensation device 22-m calculates the camera coordinates [ξ (n) ,η (n) ,ζ (n) ] T The depth estimation unit 101 estimates the camera coordinates [ξ (n) ,η (n) ,ζ (n) ] T to the coordinate estimation unit 102. The depth estimation unit 101 also transmits the depth value ζ (n) is sent to the function estimation unit 103.

[0115] In step S102, the coordinate estimation unit 102 of the blur compensation device 22-m calculates the camera coordinates [ξ (n) ,η (n) ,ζ (n) ] T Next, the coordinate estimation unit 102 receives the image coordinates p (n) =[p x (n) ,p y (n) ] TThe coordinate estimation unit 102 calculates the image coordinate p (n) is sent to the map generation unit 104.

[0116] In step S103, the function estimation unit 103 of the blur compensation device 22-m receives the depth value ζ (n) Next, the function estimation unit 103 receives the depth value ζ (n) Based on the camera parameters θ-m and θ-m, the point spread function for each region (the diameter of the circle of confusion B(ζ (n) The function estimation unit 103 estimates the diameter B(ζ (n) ) is sent to the map generating unit 104.

[0117] In step S104, the map generating unit 104 of the blur compensation device 22-m receives the image coordinate p (n) The map generating unit 104 also receives the diameter B(ζ (n) ) is received. Next, the map generation unit 104 generates a confusion circle diameter map M. Then, the map generation unit 104 sends the confusion circle diameter map M to the hidden surface removal unit 105.

[0118] In step S105, the hidden surface processor 105 of the blur compensation device 22-m receives the confusion circle diameter map M from the map generator 104. Next, the hidden surface processor 105 performs hidden surface processing on the confusion circle diameter map M. The hidden surface processor 105 sends the confusion circle diameter map M after the hidden surface processing to the blur compensation unit 106.

[0119] In step S106, the blur compensation unit 106 of the blur compensation device 22-m receives the confusion circle diameter map M from the hidden surface processor 105. Next, the blur compensation unit 106 applies blur compensation to each local area of ​​the real-life image 11-m based on the confusion circle diameter map M. The blur compensation unit 106 outputs a compensated image 13-m in which the out-of-focus blur occurring in the real-life image 11-m is locally compensated for.

[0120] <Advantages of the First Embodiment> The blur compensation device 22 in this embodiment estimates a local point spread function of the real-life image 11 based on a temporary model 12 based on a real-life image 11 captured of a subject 1 and a camera parameter θ when the real-life image 11 was captured, and generates a compensated image 13 in which focus blur is compensated for locally in the real-life image 11 based on the point spread function. In one aspect, this embodiment makes it possible to generate a fine image in which focus blur is appropriately compensated for locally.

[0121] The blur compensation device 22 in this embodiment may estimate a point spread function based on the depth value ζ of a vertex included in the temporary model 12. The blur compensation device 22 may select a point spread function estimated based on a vertex having a smallest depth value ζ among a plurality of vertices within a predetermined range. In one aspect, according to this embodiment, the point spread function can be estimated based on an appropriate vertex among vertices arranged in the vicinity.

[0122] A three-dimensional model generating device 20 in this embodiment estimates a local point spread function of the real-life image 11 based on a temporary model 12 based on multiple real-life images 11 captured from multiple viewpoints of a subject 1 and on a camera parameter θ when the real-life image 11 was captured, and generates an output model 2 based on a compensated image 13 in which focus blur is compensated for locally on the real-life image 11 based on the point spread function. In one aspect, this embodiment makes it possible to generate a detailed three-dimensional model in which focus blur is appropriately compensated for locally.

[0123] The three-dimensional model generating device 20 in this embodiment may generate the temporary model 12 using some of the multiple real-life images 11, and generate the output model 2 using all of the multiple real-life images 11. According to this embodiment, the amount of calculation required to generate the temporary model 12 is reduced, and the output model 2 can be generated at high speed.

[0124] The temporary model 12 in this embodiment may be a three-dimensional model showing the shape of the subject 1, and the output model 2 may be a three-dimensional model showing at least one of the shape and the texture or material quality of the subject 1. According to this embodiment, a three-dimensional model with improved shape accuracy and texture or material quality can be generated.

[0125] [Second embodiment] In the first embodiment, an image processing system is described that compensates for out-of-focus blur occurring in a plurality of real-life images obtained by capturing an object, and generates a detailed three-dimensional model that does not include out-of-focus blur. In the second embodiment, an image processing system is described that compensates for out-of-focus blur occurring in a plurality of real-life images obtained by capturing an object, and outputs a detailed real-life image that does not include out-of-focus blur. In one aspect, according to this embodiment, it is possible to accurately compensate for out-of-focus blur in a plurality of images obtained by capturing the same object.

[0126] The image processing system of this embodiment will be described below, focusing on the differences from the first embodiment.

[0127] <Overall composition> The overall configuration of the image processing system in this embodiment will be described with reference to Fig. 7. Fig. 7 is a block diagram showing an example of the overall configuration of the image processing system in this embodiment.

[0128] 7, the image processing system 1000 in this embodiment includes M image capturing devices 10 and an image processing device 30. Each image capturing device 10 and image processing device 30 are electrically connected to each other.

[0129] The image processing device 30 is an information processing device such as a personal computer, a workstation, or a server that applies blur compensation to real-life images captured of the subject 1. Electronic data representing M real-life images captured by the imaging device 10 is input to the image processing device 30. The image processing device 30 outputs electronic data representing M compensated images obtained by applying blur compensation to each of the M real-life images.

[0130] <Functional configuration> The functional configuration of the image processing system in this embodiment will be described with reference to FIG.

[0131] Image Processing Device Fig. 8 is a block diagram showing an example of a functional configuration of an image processing device in this embodiment. As shown in Fig. 8, an image processing device 30 in this embodiment includes a modeling unit 21, M blur compensation devices 22 (22-1, 22-2), and an image output unit 24. Therefore, the image processing device 30 differs from the three-dimensional model generating device 20 in that it includes the image output unit 24 instead of the modeling unit 23.

[0132] The image output unit 24 outputs image data 3 including M compensated images 13-1,...,13-M generated by the blur compensation devices 22-1,...,22-M. The image data 3 may include all of the M compensated images 13-1,...,13-M, or may include one or more compensated images 13 among the M compensated images 13-1,...,13-M. The compensated images 13 to be included in the image data 3 may be specified by a user of the image processing system 1000.

[0133] <Processing Procedure> The image correction method executed by the image processing system 1000 in this embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of the image correction method in this embodiment.

[0134] In step S11, M imaging devices 10-m capture an image of the subject 1. The imaging devices 10-m output real-life images 11-m of the subject 1 and camera parameters θ-m used when capturing the real-life images 11-m. The real-life images 11-m and the camera parameters θ-m are input to the image processing device 30.

[0135] In step S12, the modeling unit 21 of the image processing device 30 acquires the M real-life images 11-1,...,11-M and the M camera parameters θ-1,...,θ-M input to the image processing device 30. Next, the modeling unit 21 generates a temporary model 12 based on the real-life images 11-1,...,11-M and the camera parameters θ-1,...,θ-M. Then, the modeling unit 21 sends the generated temporary model 12 to each of the blur compensation devices 22-m.

[0136] In step S13, the blur compensation device 22-m of the image processing device 30 receives the temporary model 12 from the modeling unit 21. The blur compensation device 22-m also acquires the real-life image 11-m and the camera parameters θ-m input to the image processing device 30. Next, the blur compensation device 22-m generates a compensated image 13-m by applying blur compensation to the real-life image 11-m based on the temporary model 12 and the camera parameters θ-m. Then, the blur compensation device 22-m sends the compensated image 13-m to the image output unit 24.

[0137] In step S14, the image output unit 24 of the image processing device 30 receives the compensated images 13-m from each of the blur compensation devices 22-m. Then, the image output unit 24 outputs the image data 3 including the compensated images 13-1, . . . , 13-M.

[0138] <Effects of the second embodiment> The image processing device 30 in this embodiment estimates a local point spread function of the real-life image 11 based on a temporary model 12 based on multiple real-life images 11 captured from multiple viewpoints of the subject 1 and on the camera parameter θ when the real-life image 11 was captured, and outputs a compensated image 13 in which the focus blur is compensated for locally in the real-life image 11 based on the point spread function. In one aspect, according to this embodiment, it is possible to generate a fine image in which the focus blur is appropriately compensated for locally.

[0139] [supplement] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) implemented by an electronic circuit, and an ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), conventional circuit module, and other devices designed to execute each function described above.

[0140] Although the embodiments of the present disclosure have been described in detail above, the embodiments disclosed herein are illustrative and not restrictive in all respects. The embodiments can be modified and improved in various forms without departing from the scope and spirit of the appended claims. The matters described in the above embodiments can be configured in other ways as long as they are not inconsistent, and can be combined as long as they are not inconsistent. [Explanation of symbols]

[0141] 1. Subject 10. Imaging device 20. Three-dimensional model generation device 21 Modeling Department 22 Blur compensation device 23 Modeling Department 24 Image output section 30 Image processing device 101 Depth Estimation Unit 102 Coordinate Estimation Unit 103 Function Estimation Unit 104 Map Generation Unit 105 Shadow Surface Processing Unit 106 Blur compensation unit 1000 Image Processing System

Claims

1. a function estimator configured to estimate a local point spread function of a first image based on a three-dimensional model based on a first image of an object and on an imaging condition when the first image was captured; a blur compensation unit configured to generate a second image in which focus blur is compensated for locally in the first image based on the point spread function; An image processing device comprising:

2. 2. The image processing device according to claim 1, the function estimation unit is configured to estimate the point spread function based on depth values ​​of vertices included in the three-dimensional model; Image processing device.

3. 3. The image processing device according to claim 2, a hidden surface processor configured to select the estimated point spread function based on the vertex having the smallest depth value among the vertices within a predetermined range, Image processing device.

4. 4. The image processing device according to claim 1, a modeling unit configured to generate the three-dimensional model based on a plurality of the first images obtained by capturing the subject from a plurality of viewpoints; an image output unit configured to output a plurality of the second images corresponding to the plurality of the first images, The image processing device further comprises:

5. a first modeling unit configured to generate a first three-dimensional model based on a plurality of first images obtained by capturing an object from a plurality of viewpoints; a function estimator configured to estimate a local point spread function of the first image based on the first three-dimensional model and an imaging condition when the first image was captured; a blur compensation unit configured to generate a second image in which focus blur is compensated for locally in the first image based on the point spread function; a second modeling unit configured to generate a second three-dimensional model based on a plurality of the second images corresponding to the plurality of the first images, respectively; A three-dimensional model generating device comprising:

6. The three-dimensional model generating device according to claim 5, the first modeling unit is configured to generate the first three-dimensional model using a portion of the plurality of first images; The second modeling unit is configured to generate the second three-dimensional model using all of the plurality of first images. 3D model generation device.

7. 7. The three-dimensional model generating device according to claim 5, the first three-dimensional model is a three-dimensional model that represents a shape of the subject; the second three-dimensional model is a three-dimensional model that represents at least one of a shape and a texture or a material quality of the subject; 3D model generation device.

8. A program for causing a computer to function as the image processing device according to claim 1 or the three-dimensional model generating device according to claim 5.

Images