Information processing apparatus, image processing device, information processing method, and program

Abstract

To propose a method for estimating a plurality of homography transformations between a plurality of images.SOLUTION: A feature point is detected from a reference image among a plurality of images, a corresponding point corresponding to the feature point is detected from an image other than the reference image, a non-homogeneous equation including a plurality of homography transformations is calculated based on a correspondence between the feature point and the corresponding point, and the homography transformations are estimated by estimating an approximate solution of the non-homogeneous equation.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and the like.

Background Art

[0002] As an image synthesis technology for capturing an image with a wide dynamic range, HDR (High Dynamic Range) synthesis technology is known. For example, Patent Document 1 discloses an image synthesis apparatus that can generate an appropriate composite image even when the subject moves during imaging.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the image synthesis apparatus described in Patent Document 1, for example, the movement between input images is corrected using the movement information of pixels between images, and alignment between images is performed. At this time, due to reasons such as a plurality of subjects being included in the image, the alignment accuracy between images has become loose, and artifacts have occurred in the HDR-synthesized image in some cases.

Means for Solving the Problems

[0005] The present invention has been made in view of the above problems, and an object thereof is to propose a method for improving the alignment accuracy between a plurality of images. Another object is to propose a method for estimating a plurality of homography transformations between a plurality of images, which is required to achieve the above object.

[0006] According to one aspect of the present invention, an information processing apparatus capable of estimating a plurality of homography transformations between a plurality of images detects feature points from a reference image among the images, detects corresponding points corresponding to the feature points from images other than the reference image, calculates a non-homogeneous equation including a plurality of homography transformations based on the correspondence relationship between the feature points and the corresponding points, and estimates a plurality of homography transformations by estimating an approximate solution of the non-homogeneous equation.

[0007] Further, according to one aspect of the present invention, a correction unit of an image processing apparatus including a correction unit that performs alignment between a plurality of input images includes a preprocessing unit that converts an input image into a luminance image, the information processing apparatus, and a difference image generation unit that calculates a difference image between a conversion result image by a plurality of homography transformations estimated by the information processing apparatus and the luminance image. The image is generated based on the luminance image and a mask image for designating a region for estimating the homography transformation among the luminance images. The mask image is updated based on the difference image. The correction unit estimates a plurality of homography transformations, calculates a difference image, and updates the mask image when a predetermined condition is satisfied.

[0008] Further, according to one aspect of the present invention, the image processing apparatus further includes an image generation unit for generating an image in which a luminance image other than the reference luminance image is aligned with the reference luminance image among the luminance images based on the difference image for each mask image update and the plurality of estimated homography transformations for each mask image update.

Advantages of the Invention

[0009] According to the information processing apparatus of the present invention, an effect is obtained that a plurality of homography transformations between a plurality of images can be appropriately estimated. Further, according to the image processing apparatus of the present invention, an effect is obtained that high-precision alignment between images can be realized even when a complex movement occurs between the images.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Mode for Carrying Out the Invention

[0011] Hereinafter, an example of an embodiment for carrying out the present invention will be described with reference to the drawings. In the description of the drawings, the same reference numerals may be assigned to the same elements, and redundant descriptions may be omitted. Also, the constituent elements described in this embodiment are merely examples, and are not intended to limit the scope of the present invention thereto.

[0012] [Embodiment] Hereinafter, an example of an embodiment for realizing the information processing technology, image processing technology, and image alignment technology of the present invention will be described.

[0013] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing apparatus 1 according to an aspect of the present embodiment. The image processing apparatus 1 may also be referred to as an image alignment apparatus or a homography conversion output apparatus. The image processing apparatus 1 includes, for example, a hybrid homography estimation unit 120 and a rendering unit 140. These are, for example, functional units (functional blocks) of a processing unit (processing apparatus) or a control unit (control apparatus) (not shown) of the image processing apparatus 1, and are configured to include a processor such as a CPU or a DSP and an integrated circuit such as an ASIC.

[0014] The hybrid homography estimation unit 120 receives, for example, "N" images (where "N">2) captured by an imaging unit 310 outside the image processing apparatus 1 as input.

[0015] The hybrid homography estimation unit 120 has a function of calculating (estimating) information for performing alignment between, for example, one reference image (hereinafter referred to as the "base image") among the input "N" images and other images (hereinafter referred to as the "query images").

[0016] The rendering unit 140 has a function of drawing an image (hereinafter referred to as the "aligned image") in which the query image is aligned (transformed) with the base image based on the information for performing alignment calculated by the hybrid homography estimation unit 120. Then, the rendering unit 140 has a function of outputting, for example, the aligned image to the display unit 340.

[0017] The display unit 340 has a function of receiving the aligned image generated by the rendering unit 140 as input and displaying the same.

[0018] Here, the "display" of an image is a type of "output" of the image. The "output" of an image can include, in addition to the display (display output) of the image on the own apparatus, for example, the output (internal output) of the image to other functional units on the own apparatus, the output (external output) or transmission (external transmission) of the image to an apparatus (external apparatus) other than the own apparatus.

[0019] Note that, as a component of the image processing apparatus 1, the rendering unit 140 may not be included. Further, for example, when the image processing apparatus 1 receives a plurality of images as inputs, it may have a function of outputting information for aligning the base image and the query image without outputting the aligned image.

[0020] The "output" of information can include, for example, output of information to other functional units in the apparatus (internal output), output of information to an apparatus other than the own apparatus (external apparatus) (external output), transmission (external transmission), etc.

[0021] FIG. 2 is a diagram showing the concept of processing in the image processing apparatus 1. In the conventional method, for example, a single homography transformation is used as information for aligning the base image and the query image. By using the homography transformation, for example, alignment between images captured in two camera coordinates obtained by projecting the same real plane or between two camera coordinates when projecting a point in the real space onto a rotating camera can be realized.

[0022] However, in FIG. 2, when comparing the base image and the query image, for example, the person moves away and the leaves of the tree are shaking. When the subject has a plurality of movements like this, if a single homography transformation is applied to the entire image, regions where alignment cannot be accurately realized will occur.

[0023] Therefore, in the present invention, for example, in the hybrid homography estimation unit 120, respective homography transformations corresponding to a plurality of regions are calculated. In FIG. 2, for example, a homography transformation H1 including the tree region and a homography transformation H2 including the person region are calculated. For example, based on the base image and "N - 1" query images, the calculated "N - 1" homography transformations H1, homography transformation H2, ···, homography transformation H N-1The set with [it] is referred to as the "homography set". Also, the mixed homography estimation unit 120 calculates, for example, a difference image corresponding to the weights in each image region of each homography transformation. Further, the mixed homography estimation unit 120 estimates "mixed homography", which is a set of homography sets obtained by repeatedly estimating the homography sets corresponding to a plurality of regions.

[0024] For example, the information for aligning the base image and the query image may be the mixed homography and the difference image.

[0025] Then, the rendering unit 140 calculates, for example, a mixing ratio serving as a weight in each image region of each homography transformation based on the difference image. Then, based on the mixed homography and the mixing ratio, a registered image obtained by aligning the query image with the base image is generated.

[0026] For example, the information for aligning the base image and the query image may be the mixed homography and the mixing ratio.

[0027] [Image processing procedure] FIG. 3 is a flowchart showing an example of the image processing procedure in the present embodiment. The processing in the flowchart of FIG. 3 is realized, for example, by the processing unit of the image processing apparatus 1 reading the code of the alignment application program stored in a storage unit (not shown) into a RAM (not shown) and executing it.

[0028] Each symbol S in the flowchart of FIG. 3 means a step. Also, the flowchart described below is merely an example of the image processing procedure in the present embodiment, and it goes without saying that other steps may be added or some steps may be deleted.

[0029] First, a control unit (not shown) of the image processing apparatus 1 realizes an image acquisition process (S11). In the image acquisition process, the control unit of the information processing apparatus 1 receives, for example, "N" image sensor images captured by the imaging unit 310. Each image sensor image may be an image with different exposures. Then, the control unit of the information processing apparatus 1 designates, for example, the image at the head of the time-series data as the base image and the remaining images as query images.

[0030] Note that the control unit of the information processing apparatus 1 may, for example, designate an image with intermediate exposure (e.g., exposure correction "EV = 0") as the base image, and a high-brightness image (e.g., "EV = 1") and a low-brightness image (e.g., "EV = -1") with different exposures as query images. For example, HDR synthesis can be realized by aligning and synthesizing images with different exposures.

[0031] Further, the control unit of the information processing apparatus 1 may, for example, acquire the base image and the query images from an external device through a communication unit (not shown).

[0032] Next, the hybrid homography estimation unit 120 realizes, for example, a hybrid homography estimation process (S13).

[0033] FIG. 4 is a block diagram showing an example of the functional configuration of the hybrid homography estimation unit 120. The hybrid homography estimation unit 120 includes, for example, a preprocessing unit 121, a multi-input homography estimation unit 123, a warped image generation unit 125, a difference image generation unit 127, and a binary image generation unit 129. Note that in FIG. 4, for example, the number of acquired images "N = 3" is shown, and the case where the base image, the first query image, and the second query image are input to the hybrid homography estimation unit 120 is illustrated. For the case where the number of acquired images "N>3", it can be similarly configured by increasing the query images and the corresponding respective images.

[0034] The preprocessing unit 121 has a function of generating, for example, a luminance image (referred to as a "reduced luminance image") with the sizes of the base image, the first query image, and the second query image reduced. Hereinafter, for example, the reduced luminance image of the base image is referred to as the "base reduced image", the reduced luminance image of the first query image is referred to as the "first reduced image", and the reduced luminance image of the second query image is referred to as the "second reduced image", respectively.

[0035] Note that the preprocessing unit 121 may, for example, not reduce the sizes of the base image, the first query image, and the second query image.

[0036] The multi-input homography estimation unit 123 has a function of calculating a plurality of homography transformations (a set of homographies) based on, for example, the base image, a plurality of query images, and a mask image. Here, the mask image is information for specifying an area to which each homography transformation is applied. The mask image is composed of, for example, the same number of images as the query images, and may be, for example, a binary image that takes "1" in the entire area as an initial value. Also, the size of the mask image may be, for example, the same as the size of the reduced luminance image. Hereinafter, for example, the mask image associated with the first reduced image is referred to as the "first mask image", and the mask image associated with the second reduced image is referred to as the "second mask image", respectively.

[0037] The warped image generation unit 125 has a function of applying the homography transformation H1 estimated by the multi-input homography estimation unit 123 to the first reduced image to generate a first warped image. Similarly, it has a function of generating a second warped image and the like.

[0038] The difference image generation unit 127 has a function of generating a first difference image, which is a difference image between the first warped image and the base reduced image. Similarly, it has a function of generating a second difference image and the like.

[0039] When the binary image generation unit 129 receives, for example, the first difference image as input, it has a function of generating a first binary image obtained by binarizing the first difference image. Similarly, it has a function of generating a second binary image and the like.

[0040] Fig. 5 shows a flowchart illustrating an example of the procedure of the mixed homography estimation process. In the mixed homography estimation process, first, the preprocessing unit 121 acquires, for example, a base image and a query image (S131).

[0041] Then, the preprocessing unit 121 executes, for example, a reduced luminance image generation process (S132). In the reduced luminance image generation process, the preprocessing unit 121 applies a conversion to a YUV color space, a YCbCr color space, etc. to the base image having RGB pixel values, for example, to generate a luminance image composed of luminance values Y. Then, the preprocessing unit 121 reduces the luminance image to a predetermined size (for example, one-fourth of the size of the base image), for example, to generate a base reduced image. Also, the preprocessing unit 121 executes the same process for each query image to generate an n-th reduced image ("n = 1, ···, N - 1"). For example, the sizes of the base reduced image and the n-th reduced image may be the same. In the subsequent processes, by using an image with a smaller size than the current image, the processing speed can be increased.

[0042] Note that in the reduced luminance image generation process, the preprocessing unit 121 may not reduce the luminance image. That is, the sizes of the base reduced image and the n-th reduced image may be equal to the size of the base image, for example.

[0043] Then, the multi-input homography estimation unit 123 executes a multi-input homography estimation process based on, for example, the base reduced image, the n-th reduced image, and the n-th mask image (S133). In this example, "n = 1, 2".

[0044] Fig. 6 shows an overview of the multi-input homography estimation process. The multi-input homography estimation process may be executed, for example, according to the following algorithm.

[0045] (1). In the input frame image I N extract feature points Φ N (k) . The feature points may be detected, for example, based on the Harris feature amount. Here, "k" is a variable indicating each feature point. For example, when "K" feature points are extracted, "k = 1, ···, K". For example, in Fig. 6, it is assumed that one feature point Φ3 (1) is detected from the frame image I3.

[0046] (2). In the past frame images I1, ···, frame image I N-1 detect corresponding points Φ N (k) corresponding to Φ n,1 (k) , ···, Φ n,N (k) ("n = 1, ···, N - 1", "k = 1, ···, K"). The corresponding points may be detected, for example, based on the block matching algorithm. For example, in Fig. 6, as the corresponding points of the feature point Φ3 (1) in the past frame images, corresponding points Φ 3,2 (1) and corresponding points Φ 3,1 (1) are detected.

[0047] (3). Obtain the correspondence between the feature points and the corresponding points, including the past frame inputs. For example, in Fig. 6, if the correspondence between the frame image I1 and the frame image I3 is the homography transformation H1, and the correspondence between the frame image I2 and the frame image I3 is the homography transformation H2, the following simultaneous equations hold. ·Φ3 (1) ~H2Φ 3,2 (1) ·Φ3 (1)~H1Φ 3,1 (1) ·H2Φ2 (1) ~H1Φ 2,1 (1) Here, "~" indicates equality allowing for a constant multiple.

[0048] Generalizing from here, considering the properties of homography, for points Φ m ,I n (m,n≠N,m≠n) on the image I n,m (k) = vector "p" and the homogeneous coordinates of Φ n (k) = vector "q", with the degrees of freedom of the homography transformation being "8", are written as in the following equation (1).

Equation

Equation

Equation

Equation

[0049] Note that the homography set L that minimizes the sum of residuals may be estimated, for example, by the steepest descent method.

[0050] In the multi-input homography estimation process in this apparatus, for example, the base-reduced image is used as the frame image I N and the product image of the n-th reduced image of each frame and the n-th mask image may be used as the frame image I n respectively. And the multi-input homography estimation unit 123 may output the estimated homography set L.

[0051] Note that the multi-input homography estimation unit 123 is not limited to estimating the homography set between images along the time series. More generally, between a plurality of images, the homography set between a base image defined as a representative and other query images may be estimated.

[0052] When the multi-input homography estimation process is executed, the warped image generation unit 125 executes, for example, a warped image generation process (S134). In the warped image generation process, the warped image generation unit 125 applies, for example, the estimated homography transformation H n to the n-th reduced image to generate the n-th warped image. The warped image may be, for example, an image in which the n-th reduced image is aligned with the base-reduced image by homography transformation.

[0053] And the hybrid homography estimation unit 120 executes a mask image update process (S135). In the mask image update process, first, the difference image generation unit 127 calculates, for example, the n-th difference image between the n-th warped image and the base-reduced image. The difference image generation unit 127 may also output the calculated n-th difference image.

[0054] Next, the binary image generation unit 129 binarizes, for example, the n-th difference image to calculate the n-th difference binary image. Then, the mixed homography estimation unit 120 updates the logical product image of the n-th mask image and the n-th difference binary image as a new n-th mask image.

[0055] Then, the mixed homography estimation unit 120 determines whether to end the mixed homography estimation process (S136). The mixed homography estimation unit 120 may determine to end the mixed homography estimation process, for example, in the following cases. (a). When the total area of the true value regions (regions where the image becomes white) of the difference binary image is equal to or less than a threshold value, or smaller than the threshold value (when the difference image has converged, etc.). (b). When the estimated homography H deviates from the identity mapping (when artifacts occur when performing conversion according to the homography or when it becomes a non-linear transformation, etc.).

[0056] Note that, for example, when the area of the true value region of the n-th mask image is equal to or less than the threshold value, the mixed homography estimation unit 120 may exclude the n-th reduced image and the n-th mask image from the objects of the multi-input homography estimation process. And, for example, when the number of reduced images and mask images to be processed is less than 2, it may be determined to end the mixed homography estimation process.

[0057] When it is determined not to end the mixed homography estimation process (S136: NO), the mixed homography estimation unit 120 repeats the processes after the multi-input homography estimation process, for example, based on the updated mask image.

[0058] When it is determined to end the mixed homography estimation process (S136: YES), the mixed homography estimation unit 120 may output the mixed homography {H (1) , ···, H (C)} and the "N i -1" difference images as the mixed homography estimation process result. Here, the homography set L in loop "i" where the "i"-th multi-input homography estimation process is performed i =H (i) ={H1 (i) , ···, H Ni-1 (i)}, and the number of query images (which may also be referred to as the number of reduced images · mask images) used when the "i"-th multi-input homography estimation process is performed is set to "N i -1". Also, for example, it is assumed that the mixed homography estimation process is completed by "C" loops. Note that the mixed homography may also be referred to as a mixed homography set or a multi-input mixed homography.

[0059] Returning to FIG. 3, the rendering unit 140 executes a rendering process (S15) based on, for example, the mixed homography estimation process result. In the rendering process, the rendering processing unit 140 designates, for example, the query image I ρ and generates an aligned image (rendering image) that is aligned with the base image I β based on the homography set. Here, each pixel value of the image I ρ transformed by the mixed homography {H (1) , ···, H (C)} estimated by the multi-input homography estimation process in "C" loops is represented by, for example, the following formula. ρ Here, the pixel value of the image I

Equation

Equation

[0060] For example, the size of the image I ρ ´ becomes the size of the reduced image used in the mixed homography estimation process. Therefore, the rendering processing unit 140 may, for example, perform an enlargement process on the image I ρ ´ and generate an aligned image.

[0061] Then, the rendering processing unit 140 executes rendering image output processing (S17). In the rendering image output processing, the rendering processing unit 140 causes, for example, the aligned image to be displayed and output to the display unit 340.

[0062] Note that in the rendering image output processing, the rendering processing unit 140 may output (transmit) the aligned image to an external device by, for example, a communication unit (not shown).

[0063] Also, the rendering processing unit 140 may output (transmit) information (for example, a set of homographies and mixing ratios) based on the mixed homography estimation processing result necessary for generating the aligned image to an external device by, for example, a communication unit (not shown).

[0064] Then, the control unit of the information processing apparatus 1 determines whether to end the acquisition of the image (S19). If it is determined that the acquisition of the image continues (S19: YES), the control unit of the information processing apparatus 1 returns the process to, for example, the image acquisition processing (S11).

[0065] [Image Processing Example] FIG. 7 shows a specific example of alignment performed by the image processing apparatus 1 according to the present invention when, for example, three images are input ("N = 3"). In this example, a high-luminance image with high brightness with respect to the base image is used as the first query image, and a low-luminance image is used as the second query image, respectively. In the upper part of FIG. 7, a base image and query images are shown.

[0066] In the middle part of FIG. 7, an example of a calculated mask image is shown when the mixed homography estimation process is performed based on the images in the upper part. In these mask images, it can be seen that the background area moving with camera shake and the human contour area moving with fine shaking of the person are represented in white and are the true value areas of the mask image. In the human contour area, the true value area spreads particularly around the face and near the contour of the raised arm, indicating that it is an area where homographies are mixed.

[0067] In the lower part of FIG. 7, an example of the rendering image output process result is shown. The aligned image 1 is a rendering image to which the mixed homography is applied to the first query image, and the aligned image 2 is a rendering image to which the mixed homography is applied to the second query image. It can be seen from these images that the background / human area of the base image and the positions of the background / human areas in each aligned image match with high accuracy.

[0068] FIG. 8 is an example in which the number of processing times (loop times) of the multi-input homography estimation process and the area to which the estimated mixed homography is applied are compared in the mixed homography estimation process for the images used in the example of FIG. 7. Each image in FIG. 8 may be referred to as an image in which the mixing ratio is visualized in grayscale (hereinafter referred to as a "seed map").

[0069] In this example, for instance, in the hybrid homography estimation process, in the first multi-input homography estimation process, in order to align the entire image, the entire seed map floats upward in white centered on the background and is imaged. Therefore, it can be seen that a set of homographies for aligning the background region in particular has been calculated.

[0070] In the second multi-input homography estimation process, the seed map floats upward in white centered on the person region. Therefore, it can be seen that a set of homographies for aligning the person region has been calculated. Also, in the second multi-input homography estimation process, in the left seed map, the face and shoulder regions are shown in white, and in the right seed map, the head and leg regions are shown in white. Therefore, it is presumed that in the second multi-input homography estimation process, the set of homographies is composed of the homography for aligning the face and shoulder regions and the homography for aligning the head and leg regions.

[0071] In the third multi-input homography estimation process, the seed map floats upward in white centered on the region where fine texture of the background or fine blurring of the person region occurs. Therefore, it can be seen that a set of homographies for aligning the details has been calculated.

[0072] [Embodiment] Next, embodiments of a terminal, an electronic device (electronic apparatus), and an information processing device to which the above-described image processing apparatus 1 is applied or which includes the above-described image processing apparatus 1 will be described. Here, as an example, an embodiment of a smartphone, which is a type of mobile phone with a camera function (imaging function), will be described. However, it goes without saying that the embodiments to which the present invention is applicable are not limited to this embodiment.

[0073] FIG. 9 is a diagram showing an example of the functional configuration of the smartphone 10. The smartphone 10 includes, for example, a processing unit 100, a storage unit 200, an imaging unit 310, an inertial measurement unit 320, an operation unit 330, a display unit 340, a sound input unit 350, a sound output unit 360, and a communication unit 370.

[0074] The processing unit 100 is a processing device that comprehensively controls each part of the video editing PC 10 according to various programs such as system programs stored in the storage unit 200, and performs various processes related to video editing processing. It is configured with processors such as a CPU, GPU, DSP, and integrated circuits such as an ASIC.

[0075] The processing unit 100 mainly has a hybrid homography estimation unit 120, a rendering unit 140, and a display control unit 180. As its functional units, the hybrid homography estimation unit 120 has, for example, a preprocessing unit 121, a multi-input homography estimation unit 123, a warped image generation unit 125, a difference image generation unit 127, and a binary image generation unit 129. These functional units respectively correspond to the functional units included in the image processing device 1 of FIG. 1.

[0076] Further, the display control unit 180 has a function of, for example, causing the display unit 340 to display and output the processing result in the processing unit 100.

[0077] The storage unit 200 is a storage device configured with volatile or non-volatile memories such as ROM, EEPROM, flash memory, RAM, and a hard disk device.

[0078] The storage unit 200 stores, for example, a registration application program 210 and a camera image temporary storage unit 220.

[0079] The registration application program 210 is a program that is read by the processing unit 100 and executed as registration application processing.

[0080] The camera image temporary storage unit 220 is, for example, a buffer (frame buffer) that stores captured images (image sensor images) captured by the imaging unit 310 and output images of the rendering unit 140.

[0081] The imaging unit 310 is an imaging device configured to be able to capture images of any scene, and is configured to include imaging elements (semiconductor elements) such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor. The imaging unit 310 forms an image of light emitted from an imaging object on the light receiving plane of the imaging element by a lens (not shown), and converts the brightness and darkness of the light of the image into an electrical signal by photoelectric conversion. The converted electrical signal is converted into a digital signal by an A / D (Analog Digital) converter (not shown) and output to the processing unit 100.

[0082] The inertial measurement unit 320 is configured to include, for example, a gyro sensor that detects angular velocities around the axes of three axes (pitch, roll, yaw) and an acceleration sensor that detects inertial forces in the axial directions of the three axes (pitch, roll, yaw). The detection results of the inertial measurement unit 320 are output to the processing unit 100 at any time.

[0083] The operation unit 330 is configured to include an input device such as operation buttons and operation switches for the user to perform various operation inputs on the smartphone 10. The operation unit 330 also includes a touch panel (not shown) configured integrally with the display unit 340, and this touch panel functions as an input interface between the user and the smartphone 10. An operation signal according to the user operation is output from the operation unit 330 to the processing unit 100.

[0084] The display unit 340 is a display device configured to include an LCD (Liquid Crystal Display), an OELD (Organic Electro-luminescence Display), etc., and performs various displays based on a display signal output from the display control unit 180.

[0085] The sound input unit 350 is a sound input device configured with a microphone, an A / D converter, etc., and performs various sound inputs based on the sound input signal input to the processing unit 100.

[0086] The sound output unit 360 is a sound output device configured with a D / A converter, a speaker, etc., and performs various sound outputs based on the sound output signal output from the processing unit 100.

[0087] The communication unit 370 is a communication device for transmitting and receiving information used inside the device to and from an external information processing device. As the communication method of the communication unit 370, there are various methods applicable, such as a wired connection via a cable conforming to a predetermined communication standard such as Ethernet or USB (Universal Serial Bus), a wireless connection using a wireless communication technology conforming to a predetermined communication standard such as Wi-Fi (registered trademark) or 5G (fifth-generation mobile communication system), and a connection using short-range wireless communication such as Bluetooth (registered trademark).

[0088] The processing unit 100 of the smartphone 10 performs image capture processing and alignment processing according to the alignment application program 210 stored in the storage unit 200. Each process may be executed, for example, according to the flowchart of FIG. 3.

[0089] [Actions and Effects of Embodiments and Examples] The information processing device (for example, the multi-input homography estimation unit 123) in the present embodiment is a device capable of estimating a plurality of homography transformations between a plurality of images. Among the images, it detects feature points from a reference image (for example, a base reduced image), detects corresponding points corresponding to the feature points from an image other than the reference image (for example, a reduced luminance image of a query image), calculates a non-homogeneous equation (for example, Equation (2)) including a plurality of homography transformations based on the correspondence relationship between the feature points and the corresponding points, and estimates an approximate solution of the non-homogeneous equation to show an example of a configuration for estimating a plurality of homography transformations. According to this, from the relationship between feature points and corresponding points among a plurality of images, a non - homogeneous equation is derived, and by estimating an approximate solution thereof, it becomes possible to estimate a plurality of homography transformations among the plurality of images.

[0090] Further, the image processing apparatus (for example, the image processing apparatus 1) in the present embodiment includes correction means (for example, the hybrid homography estimation unit 120) for performing alignment among a plurality of input images. The correction means includes a pre - processing unit (for example, the pre - processing unit 121) that converts the input image into a luminance image (for example, a reduced luminance image), an information processing apparatus (for example, the multi - input homography estimation unit 123), and a difference image generation unit (for example, the warp image generation unit 125, the difference image generation unit 127, and the binary image generation unit 129) that calculates a difference image between the conversion result image (for example, the warp image) by a plurality of homography transformations estimated by the information processing apparatus and the luminance image. The information processing apparatus is generated based on the luminance image and a mask image for designating a region for estimating the homography transformation among the luminance images. The mask image is updated based on the difference image. The correction means shows an example of a configuration in which when a predetermined condition is satisfied, a plurality of homography transformations are estimated, a difference image is calculated, and the mask image is updated. According to this, the correction means of the image processing apparatus can efficiently obtain the homography transformation and the difference image necessary for performing alignment among a plurality of input images by updating the mask image based on the difference image while estimating a plurality of homography transformations indicated by the mask image. Also, when a predetermined condition is satisfied, by recalculating the homography transformation and the difference image, it is possible to obtain information necessary and sufficient for performing alignment.

[0091] Further, the image processing apparatus in the present embodiment further includes image generation means (for example, the rendering unit 140) for generating an image (for example, an aligned image) in which a luminance image other than the reference luminance image is aligned with the reference luminance image among the luminance images based on the difference image for each mask image update and the plurality of estimated homography transformations for each mask image update. An example of the configuration is shown. According to this, the image generation means can image and output an image in which alignment between a plurality of input images is performed based on the processing result in the correction means.

[0092] [Modification Example] The embodiments to which the present invention is applicable are not limited to the above embodiments. Hereinafter, modification examples will be described.

[0093] <Modification Example of Multi-Input Homography Estimation Processing> In the above embodiment, the multi-input homography estimation unit 123 obtains the correspondence relationship between the feature points and the corresponding points by, for example, extracting the feature points in the input latest frame image and detecting the corresponding points in the previous frame image, but it is not limited to this. For example, the multi-input homography estimation unit 123 may use the correspondence relationship between the feature points and their corresponding points in both directions between the images by extracting the feature points in each image and obtaining the corresponding points of the feature points detected in other images.

[0094] In this case, the multi-input homography estimation unit 123 may estimate the homography set as follows, for example. (A1). In the images I1, ···, image I N , detect the feature points {Φ n (1) , ···, Φ n (Kn)} (where "n = 1, ···, N"). (A2). In the images other than the image itself where the feature points are detected, detect the corresponding points {Φ m,n (1) , ···, Φ m,n (Kn)} (where "m ≠ n" and "n = 1, ···, N"). (A3). Estimate the homography set L{H1, ···, H N-1} that minimizes the sum of the residuals in each correspondence relationship. However, the sum of the residuals is obtained, for example, by the following formula (7). [Equation]

[0095] By increasing the relationship between the feature points and the corresponding points in this way, the homography set can be estimated without depending on the input relationship of the image. Further, as the relationship between the feature points and the corresponding points increases, the estimation accuracy of the homography set can be improved.

[0096] In addition, when the input image is input along a time series such as a moving image, the homography set between the latest "N" frames may be estimated for each frame.

[0097] In this case, the multi-input homography estimation unit 123 may estimate the homography set as follows, for example. (B1). After the "(N + 1)"-th frame, discard the image I1 and set the image I n as the image I n+1 . (B2). Set the latest frame as the image I N . (B3). In the latest frame as the image I N , detect the feature points {Φ N (1) , ···, Φ N (Kn)}. (B4). In the past frame images I1 ···, the image I N―1 , detect the corresponding points {Φ n,N (1) , ···, Φ n,N (Kn)} (where "n = 1, ···, N - 1"). (B5). In the latest frame as the image I N , detect the corresponding points {Φ N,n (1) , ···, Φ N,n (Kn)} of the past frames (where "n = 1, ···, N - 1"). (B6). Obtain the correspondence between the feature points and the corresponding points, and minimize the sum of the residuals in the correspondence calculated in the past "(N - 1)" frames to obtain the homography set L{H1, ···, H N-1Estimate {}. However, the sum of the residuals is obtained, for example, according to Equation (7). (B7). Return the process to (B1) and repeat until a new frame input is completed.

[0098] By paying attention to the relationship of past frames in this way, the computational complexity that was O(N 2 ) when simply using the relationship between feature points and corresponding points bidirectionally can be reduced to O(N), and the estimation speed can be increased while maintaining the accuracy.

[0099] <Modification example of the mask image> In the above embodiment, the mask image is, for example, a binary image that takes "1" in the entire area as an initial value, but is not limited to this. For example, the initial value of the mask image may be input from an image area segmentation unit (not shown).

[0100] When the image area segmentation unit acquires a base image, for example, it executes image area segmentation processing using, for example, a depth sensor on the base image. Thereby, the image area segmentation unit may extract, for example, a person area and a background area from the base image. Then, the image area segmentation unit may set, for example, the person area and the background area extracted from the base image as the initial value of the mask image.

[0101] Thereby, for example, in portrait photography composed of a person area and a background area that are likely to be moving subjects, it is possible to improve the processing convergence speed and the image quality.

[0102] <Various devices> In the above example, the case where the present invention is applied to a smartphone, which is an example of an image processing apparatus, a terminal, an electronic apparatus (electronic device), and an information processing apparatus, is illustrated, but it is not limited to this. The present invention is applicable to various devices such as digital telescopes, video cameras, still cameras, tablet terminals, and wearable terminals such as smart glasses.

[0103] <Recording medium> In the above embodiments, various programs and data related to image processing are stored in the storage unit 200, and the processing unit reads and executes these programs, thereby realizing the image processing in each of the above embodiments. In this case, the storage unit of each device may have, in addition to internal storage devices such as ROM, EEPROM, flash memory, hard disk, and RAM, tangible recording media (recording media, external storage devices, storage media) such as memory cards (SD cards), CompactFlash (registered trademark) cards, memory sticks, USB memories, CD-RWs (optical disks), and MOs (magneto-optical disks), which are not temporary, and it is also possible to store the above various programs and data in these recording media. These storage media are an example of computer-readable non-temporary recording media (storage media).

[0104] FIG. 10 is a diagram showing an example of a recording medium in this case. In this example, the image processing apparatus 1 is provided with a card slot 410 for inserting a memory card 430, and a card reader / writer (R / W) 420 for reading information stored in the memory card 430 inserted into the card slot 410 or writing information to the memory card 430.

[0105] The card reader / writer 420 performs an operation of writing programs and data recorded in a storage unit (not shown) to the memory card 430 according to the control of the processing unit. The programs and data recorded in the memory card 430 can be read by an external device other than the image processing apparatus 1, and are configured to be able to realize the image processing in the above embodiments in the external device.

[0106] Note that the above recording media can also be applied to various devices such as terminals (smartphones) equipped with the image processing apparatus 1 described in the above embodiments, image processing apparatuses, electronic devices (electronic equipment), and information processing apparatuses.

Explanation of reference numerals

[0107] 1 Image processing apparatus 10 Smartphone 120 Hybrid homography estimation unit 121 Preprocessing unit 123 Multi-input homography estimation unit 125 Warped image generation unit 127 Difference image generation unit 129 Binary image generation unit 140 Rendering unit

Claims

1. An information processing apparatus capable of estimating a plurality of homography transformations between a plurality of images, wherein a control unit of the information processing apparatus detects feature points from a reference image among the images, detects corresponding points corresponding to the feature points from the images other than the reference image, calculates a non-homogeneous equation including the plurality of homography transformations based on the correspondence relationship between the feature points and the corresponding points, and estimates the plurality of homography transformations by estimating an approximate solution of the non-homogeneous equation.

2. The information processing apparatus according to claim 1, wherein the control unit of the information processing apparatus estimates the approximate solution by the Gauss-Newton method.

3. The information processing apparatus according to claim 1, wherein the control unit of the information processing apparatus detects the feature points while changing the reference image to each of the images, detects the corresponding points corresponding to the feature points in each of the images, and calculates a non-homogeneous equation including the homography transformation based on the correspondence relationship between the feature points in each of the images and the corresponding points corresponding to the feature points in each of the images.

4. An image processing apparatus including correction means for performing alignment between a plurality of input images, wherein the correction means includes a preprocessing unit that converts the input image into a luminance image, the information processing apparatus according to claim 1, and a difference image generation unit that calculates a difference image between a conversion result image by the plurality of homography transformations estimated by the information processing apparatus and the luminance image, and has wherein the image is generated based on the luminance image and a mask image for designating a region for estimating the homography transformation in the luminance image, the mask image is updated based on the difference image, and when a predetermined condition is satisfied, the correction means estimates the plurality of homography transformations, calculates the difference image, and updates the mask image.

5. The image processing apparatus according to claim 4, wherein the predetermined condition includes a case where a binary image true value region of the difference image is equal to or greater than a threshold value and greater than the threshold value.

6. The image processing apparatus according to claim 4, wherein the predetermined condition includes a case where the homography transformation estimated by the information processing apparatus does not deviate from the identity mapping.

7. The image processing apparatus according to claim 4, Image generation means for generating an image in which, among the luminance images, an image obtained by aligning a luminance image other than the reference luminance image with the reference luminance image is generated based on the difference image for each update of the mask image and the plurality of estimated homography transformations for each update of the mask image.

8. The image processing apparatus according to claim 7, wherein the preprocessing unit converts the input image into the reduced luminance image, and the image generation means enlarges and outputs the aligned image according to the input image.

9. The image processing apparatus according to claim 4, wherein the input image corresponding to the reference luminance image is an image that has not been subjected to exposure adjustment, and the input image corresponding to the luminance image other than the reference luminance image is an image that has been subjected to exposure adjustment.

10. An information processing method capable of estimating a plurality of homography transformations between a plurality of images, comprising detecting feature points from a reference image among the images, detecting corresponding points corresponding to the feature points from the images other than the reference image, calculating a non-homogeneous equation including the plurality of homography transformations based on the correspondence between the feature points and the corresponding points, and estimating the plurality of homography transformations by estimating an approximate solution of the non-homogeneous equation.

11. A program for causing an information processing apparatus capable of estimating a plurality of homography transformations between a plurality of images to detect feature points from a reference image among the images, detect corresponding points corresponding to the feature points from the images other than the reference image, calculate a non-homogeneous equation including the plurality of homography transformations based on the correspondence between the feature points and the corresponding points, and estimate the plurality of homography transformations by estimating an approximate solution of the non-homogeneous equation. ​ ​

Images