Image processing device, imaging device, image processing method, program, and storage medium

Abstract

The present invention provides an image processing device capable of reducing areas in 3D models that cannot be viewed. [Solution] The image processing device (100) includes an acquisition means (104) for acquiring an image and distance information of a subject in the image, and a control means (105) for generating an evaluation value of the image. The control means calculates area information relating to the area of ​​the subject in the image in three-dimensional space based on the distance information, and generates an evaluation value using the area information.

Description

Technical Field

[0006] , , ,

[0004] , , , , , , , ,

[0005]

[0001] The present invention relates to an image processing apparatus, an imaging apparatus, an image processing method, a program, and a storage medium.

Background Art

[0002] Non-Patent Document 1 discloses that a 3D model created based on information obtained by photographing an object with an RGBD camera such as a stereo camera has 3D information of the object as viewed from the photographing direction. Generally, when a user views a 3D model, the user opens the 3D model using a viewer application, operates a virtual camera, and views images sequentially rendered on the screen. When the virtual camera is operated, self-occlusion may occur where a part of the 3D model on the back side as viewed from the viewpoint of the virtual camera is hidden by the one on the front side.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When self-occlusion occurs in a subject, an area that cannot be viewed by the user occurs in the 3D model. One of the factors is that the area where the subject is hidden due to self-occlusion is wide.

[0006] Therefore, the present invention aims to provide an image processing apparatus that can reduce areas in a 3D model that cannot be viewed. [Means for solving the problem]

[0007] An image processing apparatus as one aspect of the present invention comprises an acquisition means for acquiring an image and distance information of a subject in the image, and a control means for generating an evaluation value of the image, wherein the control means calculates region information relating to the area of ​​the subject in the three-dimensional space in the image based on the distance information, and generates the evaluation value using the region information.

[0008] Other objects and features of the present invention are described in the following examples. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an image processing device that can reduce areas in a 3D model that cannot be viewed. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram of the image processing device in Example 1. [Figure 2] This is an explanatory diagram of the imaging unit in Example 1. [Figure 3] This is a flowchart showing the image processing method in Example 1. [Figure 4] This is an example of an image from Example 1. [Figure 5] This is an example of an image from Example 1. [Figure 6] This is an example of an image from Example 1. [Figure 7] This is an example of an image from Example 1. [Figure 8] This is an example of an image from Example 1. [Figure 9] This is an example of an image from Example 1. [Figure 10] This is an example of an image from Example 1. [Figure 11] It is a flowchart showing an image processing method in Example 2. [Figure 12] It is an example of an image in Example 2. [Figure 13] It is an example of an image in Example 2. [Figure 14] It is an explanatory diagram of the effect in Example 2. [Figure 15] It is an example of an image in Example 2. [Figure 16] It is a block diagram of an image processing apparatus in Example 3. [Figure 17] It is an example of a plurality of images in Example 3. [Figure 18] It is a flowchart showing an image processing method in Example 3.

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Although a plurality of features are described in each embodiment, not all of these plurality of features are essential, and the plurality of features may be arbitrarily combined. Also, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant explanations are omitted.

Examples

[0012] First, Example 1 of the present invention will be described. In this example, a case where a user adjusts the camera angle while looking at a rear liquid crystal monitor or an EVF using an imaging device such as a digital camera and captures an image suitable for creating a 3D model will be described. In this example, the subject is a plant in a flowerpot, but it is not limited thereto. Hereinafter, the imaging device will be described as an example of the image processing apparatus, but the image processing apparatus of this example is not limited to the imaging device, and can be executed in an electronic device such as a desktop computer, a laptop computer, or a portable computer.

[0013] Figure 1 is a block diagram showing the configuration of the imaging device (image processing device) 100. In Figure 1, the optical system 101 includes a lens group (lens unit, imaging optical system) consisting of a zoom lens and a focus lens, an aperture adjustment device, and a shutter device. The optical system 101 adjusts the magnification, focus position, or light intensity of the subject image reaching the imaging unit 102. In this embodiment, the optical system 101 is integrally configured with the camera body, but it is not limited to this. This embodiment can also be applied to imaging systems that consist of a camera body (imaging device) and an optical system (lens device) that can be attached to or detached from the camera body.

[0014] The imaging unit 102 has a photoelectric conversion element (image sensor) such as a CCD sensor or CMOS sensor that converts the light beam of the subject that has passed through the optical system 101 into an electrical signal. The A / D conversion unit 103 converts the input video signal into a digital image.

[0015] In addition to normal signal processing, the image processing unit 104 performs calculation of the distance image in this embodiment, information regarding the area of ​​the subject in three-dimensional space, generation of evaluation values, and processing of the display image for notification based on the evaluation values. In other words, the image processing unit 104 has the functions of an image acquisition means, a distance map acquisition means (acquisition means), and an evaluation value generation means (generation means). Here, normal processing signals refer to processes such as noise reduction processing, development processing, and gradation compression processing by gamma conversion to compress the gradation to a predetermined output range.

[0016] The control unit (control means) 105 is, for example, a CPU, and controls the operation of each block of the imaging device 100. For example, it calculates the exposure amount during shooting to obtain an input image with appropriate brightness, and controls the optical system 101 and the imaging unit 102 to achieve this, controlling the aperture, shutter speed, and analog gain of the sensor.

[0017] The display unit (display means) 106 sequentially displays images output from the image processing unit 104 on a display component such as an LCD (liquid crystal display) that constitutes a rear LCD monitor or EVF. The storage unit (storage means) 107 is a non-volatile memory such as ROM and has the function of storing images. The storage unit 107 may include an information storage medium such as a memory card equipped with semiconductor memory or a package containing a rotating storage element such as a magneto-optical disk. The RAM 108 is a volatile memory that temporarily stores data. The RAM 108 may also be the internal memory of the control unit 105.

[0018] Figures 2(A) to 2(D) are explanatory diagrams of the imaging unit 102. Figure 2(A) shows the positional relationship between the optical system 101 and the imaging unit 102 in the imaging device 100 of Figure 1. The dashed line in Figure 2(A) is the optical axis 101a of the optical system 101, and the imaging unit 102 is positioned approximately perpendicular to the optical axis 101a.

[0019] Figure 2(B) shows the pixel arrangement diagram of the imaging unit 102 in Figure 1. The direction of the optical axis 101a coincides with the z-axis direction. Each pixel 200 is composed of a microlens 201, a color filter 202, and photoelectric conversion units 203A and 203B, as shown in the cross-sectional view in Figure 2(C). The imaging unit 102 provides each pixel with Red, Green, and Blue spectral characteristics corresponding to the wavelength band detected by the color filter 202, and the color filters are arranged according to known color patterns. Photoelectric conversion units 203A and 203B with sensitivity corresponding to the wavelength band are formed on the substrate 204.

[0020] Figure 2(D) is a view of the exit pupil from the intersection of the optical axis 101a of the optical system 101 and the imaging unit 102. Light beams that have passed through the first pupil region 210 and the second pupil region 220, respectively, are incident on the photoelectric conversion units 203A and 203B. By photoelectrically converting the light beams incident on the photoelectric conversion units 203A and 203B, images A and B are generated. The detected images A and B are transmitted to the image processing unit 104, where distance information is calculated through distance measurement calculation processing and stored in the storage unit 107. The image obtained by adding images A and B can also be used as image information.

[0021] In Figure 2(C), reference numeral 211 indicates the centroid position of the pupil region 210, and reference numeral 221 indicates the centroid position of the pupil region 220. In this embodiment, the centroid position 211 of the pupil region 210 is eccentric (moves) in the x-axis direction from the center of the exit pupil, and the centroid position 221 of the pupil region 220 is eccentric (moves) in the opposite direction. The distance between these centroids becomes the baseline length 222, and the direction of this line segment is called the pupil division direction. Images A and B change position in the same direction as the pupil division direction. The relative position change between these images, i.e., the parallax between images A and B, is a value corresponding to the amount of defocus. This parallax can be detected and converted using known methods and converted into the amount of defocus, or distance information from the imaging device 100 to the subject.

[0022] (Image processing method) Referring to Figure 3, the image processing method performed in the imaging device 100 of this embodiment will be described. Figure 3 is a flowchart of the image processing method. The processing corresponding to each step in Figure 3 is realized by the operation of each unit, where the control unit 105 reads the corresponding processing program stored in the non-volatile memory of the storage unit 107, for example, and then loads it into the volatile memory of the control unit 105 and executes it.

[0023] The following describes the process of adjusting the camera angle while viewing evaluation values ​​displayed on the rear LCD monitor or EVF during still image capture, and storing images suitable for 3D model creation. The processing within the flowchart loops represents the processing of one captured image frame. Figure 4 shows image 401, which is an example of an input image (captured image) in this embodiment. The user is attempting to photograph a plant 402 in a flowerpot from the side.

[0024] First, in step S301, the control unit 105 uses the image processing unit 104 to acquire the image 401 captured by the imaging unit 102. Next, in step S302, the control unit 105 uses the image processing unit 104 to generate a distance map (distance information) corresponding to the image 401 acquired in step S301, and acquires the distance map generated by the image processing unit 104.

[0025] Specifically, the control unit 105 calculates the parallax between image A and image B, converts the parallax into a defocus amount, and then converts the defocus amount into the distance to the object surface (the surface of the subject) using the lens formula. The calculation of parallax and the conversion of the defocus amount may be performed by generating a distribution of defocus amounts for each pixel using a known method, for example, as disclosed in Patent Document 1. The distance from the imaging device 100 to the subject is calculated from the lens formula in geometrical optics as shown in equation (1) below.

[0026]

number

[0027] A: Distance from the object surface to the principal point of optical system 101 B: Distance from the principal point of optical system 101 to the image formation position F: Focal length of optical system 101 In equation (1), the value of B is determined from the distance from the principal point of the optical system 101 to the image sensor (imaging unit 102) and the amount of defocus, and F is either design data or a value obtained by measurement. This allows the distance A to the object surface to be calculated for each pixel and a distance map to be generated.

[0028] Next, in step S303, the control unit 105 calculates region information relating to the area of ​​the subject in the three-dimensional space of the image based on the distance map acquired in step S302. In this embodiment, the region information is calculated to be related to the surface area of ​​the subject in the image. First, the control unit 105 cuts out the subject region 502 in the image, as shown in image 501 in Figure 5. The cutting method may be, for example, an existing method such as DL, or a method of separating the foreground and background based on the distance map acquired in step S302.

[0029] Next, the control unit 105 calculates the surface area based on the distance map of the extracted region. The calculation method involves, for example, calculating the surface area of ​​the object side for each pixel based on the pixel size and focal length of the imaging device 100 and the distance map of the subject region, and then finding the sum of the surface areas of each pixel in the subject region. The surface area s per pixel can be calculated, for example, as shown in equation (2) below.

[0030]

number

[0031] dx: Pixel width dy: vertical width of the pixel L: Distance measurement value corresponding to the pixel f :focal length Nz: Component of the unit normal vector in the same direction as the optical axis. Here, the unit normal vector is obtained by regularizing the cross product vector obtained from the object-side coordinate positions of adjacent pixels. When adjacent pixels are not connected, s=0 or s<(dxL / f)(dyL / f). Connection is determined when Nz is close to zero or when the distance difference between adjacent distance values ​​is large.

[0032] Next, in step S304, the control unit 105 uses the image processing unit 104 to generate an evaluation value for the image 401 based on the surface area of ​​the subject. In this embodiment, the surface area value is used directly as the evaluation value. For example, the evaluation value is higher the larger the surface area of ​​the subject.

[0033] Next, in step S305, the control unit 105 generates notification content (notification information regarding the evaluation value) using the image processing unit 104. In this embodiment, as shown in image 601 in Figure 6, the evaluation value is represented as a bar graph 602 and numbers 603 (the notification information is generated in image format), and the image with the evaluation value superimposed on image 401 is used as the notification content. Next, in step S306, the control unit (notification means) 105 notifies the user by displaying image 601 on the display unit 106.

[0034] Next, in step S307, the control unit 105 determines whether or not to store the image 401. That is, the control unit 105 obtains the user's determination regarding whether or not to store the image 401 in the storage unit 107 via the input unit (not shown) of the imaging device 100. The input unit is, for example, a button. If the button is pressed, the control unit 105 determines to store the image 401. On the other hand, if a certain period of time has elapsed without the button being pressed, the control unit 105 determines not to store the image 401. If the image 401 is to be stored, the process proceeds to step S308. On the other hand, if the image is not to be stored, the process proceeds to step S301 and the same process is repeated. The evaluation value and image reported are updated as the user holding the imaging device 100 moves the imaging device 100 as appropriate. In step S308, the control unit 105 stores the image 401 in the storage unit 107.

[0035] The above describes the process for capturing images suitable for creating 3D models when the user adjusts the camera angle while viewing the information displayed on the rear LCD monitor or EVF (in this embodiment, an evaluation value indicating surface area) during still image capture. Since the user can select the image to store while viewing the evaluation value, it is possible to adjust the camera angle while storing the image with the highest evaluation value.

[0036] Unlike regular images, 3D models have the advantage of being viewable from different viewpoints than those used during capture, thanks to a 3D viewer. When the surface area of ​​the subject visible during capture is large, a wider area of ​​the subject can be captured for the 3D model, reducing the areas that become unviewable due to the lack of a 3D model when the viewpoint is changed.

[0037] In this embodiment, the flow from image acquisition to notification is explained in the flowchart of Figure 3. However, it is also possible to generate a notification image at high speed and low quality, and a storage image at low speed and high quality. That is, the image includes a first image for generating evaluation values ​​and a second image for storage in the storage unit 107. The control unit 105 may use the image processing unit 104 to acquire the first image at a higher speed than the second image and acquire the second image at a higher quality than the first image.

[0038] Alternatively, to enable high-speed display, a different image from the processed image may be acquired again from a similar angle using a high-quality method and stored. This increases the display frame rate, allowing for more comfortable verification of images and evaluation values, and enables the acquisition of images from almost the same angle as the verification at a higher quality. Methods for acquiring images and evaluation values ​​at high speed include, for example, limiting the maximum image quality to the resolution of the display unit of the imaging device 100, or limiting the parallax search range for distance measurement calculation to only the subject area.

[0039] In this embodiment, the system determines whether or not to store an image based on the user's intent, but it may also determine whether or not to store an image based on an evaluation value. Specifically, the evaluation values ​​of multiple images containing the same subject may be stored, and an image may be stored if it exceeds the maximum value of past evaluation values. This prevents the storage of images with high evaluation values ​​from being missed.

[0040] Furthermore, the upper and lower limits 703 and 704 of the display range of the bar graph 702 may be adjusted and notified to the user, as shown in image 701A of Figure 7(A), based on the evaluation values ​​of multiple images containing the same subject. Since the display range of the evaluation value is optimized for the subject, the user will be able to more easily recognize the magnitude of the evaluation value as it is updated in real time.

[0041] Alternatively, the evaluation range for the same subject may be divided into three categories, A, B, and C, in descending order of evaluation, and each category may be reported. This allows the user to easily understand the results of the image evaluation at a glance, rather than simply reporting the numerical evaluation values. Alternatively, the evaluation range may not be based on the evaluation values ​​of the same subject, but rather a predetermined range of evaluation values ​​that is appropriately divided and reported. Even for images of a subject being photographed for the first time, the evaluation category can be obtained and reported.

[0042] In this embodiment, the surface area of ​​the subject in the image is used as the evaluation value, but the evaluation value may also be calculated considering the proportion of the image occupied by the subject. For example, the larger the proportion of the image occupied by the subject, the higher the evaluation value can be. Even at the same angle, if the subject is captured larger, the resolution of the 3D model's texture can be increased. The method for calculating the evaluation value may be, for example, the product of the surface area and the proportion. Alternatively, the evaluation value may be represented as a two-dimensional vector, as in image 701B in Figure 7(B), and visually indicated as rectangle 705. The horizontal axis of the rectangle is the proportion, and the vertical axis is the surface area. The number 706 directly indicates the proportion as text.

[0043] In this embodiment, the surface area is calculated using equation (2), but it is also possible to construct a mesh by appropriately connecting adjacent points in the point cloud on the object side of each pixel, and then use the sum of the areas of the mesh as the surface area. This makes it possible to obtain a more accurate surface area.

[0044] Furthermore, since this embodiment aims to acquire images for 3D models, the calculation and notification of evaluation values ​​may be performed when the 2.5D shooting mode (second mode), which is different from the normal image shooting mode (first mode), is set to ON. Since the visibility of the displayed image is not reduced by the notification, the user can perform normal shooting comfortably. In addition, at this time, at least one of the upper or lower limits of the parameters of the imaging unit 102 (camera parameters), such as the F-number and ISO sensitivity, may be set for 3D models. This makes it easier to acquire images that are more suitable for creating 3D models.

[0045] Furthermore, when adjusting the camera angle of the same subject, a smaller version of image 802, representing the highest previously rated image, may be superimposed, as shown in image 801 of Figure 8. Users can then save the image while referring to how the image with the highest rating appeared.

[0046] The control unit 105 also acquires information regarding the position and orientation of the imaging device 100 during imaging, and may, for example, notify the user of the position and orientation of the imaging device 100 when the image with the highest evaluation value in the past was captured, as shown in image 901 in Figure 9. The position and orientation at the highest evaluation value is represented as a thick arrow 903 on a figure 902 that shows the position and orientation relative to the subject, while the current position and orientation is represented as a thin arrow 904, and this is notified to the user. The user can store the image while referring to the position and orientation at the highest evaluation value.

[0047] The control unit 105 may also generate and report an evaluation value only when the subject is contained within the image. If the subject is not contained within the image, an appropriate evaluation value cannot be obtained, thus preventing an inappropriate evaluation value from affecting the report content. To determine whether the subject is contained within the image, for example, when detecting the subject area, if the subject is present in pixels 1002 along the four sides of the image, as in image 1001 shown in Figure 10, it can be determined that the subject is not contained within the image.

[0048] Furthermore, the control unit 105 may choose not to generate and notify evaluation values ​​when a specific subject is detected. For example, a person's head generally has the largest surface area when viewed from the side, so the evaluation value is highest when photographed from the side. If you want to image a face from the front, this prevents notification from interfering with the shooting. For example, if the subject is not a face, the control unit 105 generates an evaluation value.

[0049] In this embodiment, the user is notified using visual information from images, but the evaluation value may also be notified to the user using auditory information from sound. Since the notification content is not superimposed on the image, the user can see the image clearly and concentrate their visual attention on the current camera angle. Possible methods for conveying the evaluation value by sound include, for example, associating the evaluation value with a musical scale and having a higher pitch sound when the evaluation is higher, or increasing the frequency of sound when the evaluation value is high. [Examples]

[0050] Next, Embodiment 2 of the present invention will be described. In this embodiment, a user adjusts the camera angle while viewing the rear LCD monitor or EVF using an imaging device such as a digital camera, and captures an image suitable for creating a 3D model. In this embodiment, the subject is a cube with a pattern, but it is not limited to this. In this embodiment, if the configuration and processing are the same as in Embodiment 1, the explanation will be omitted. The image processing device of this embodiment is not limited to an imaging device, but can be implemented in electronic devices such as desktop computers, laptop computers, and portable computers.

[0051] The following describes the process of adjusting the camera angle while viewing evaluation values ​​displayed on the rear LCD monitor or EVF during still image capture, and storing images suitable for creating 3D models. Image 1201 shown in Figure 12 is an example of an input image (captured image) in this embodiment. The user is attempting to photograph a cube 1202 with a dot pattern from an oblique angle above.

[0052] (Image processing method) Referring to Figure 11, the image processing method performed in the computer of this embodiment will be described. Figure 11 is a flowchart of the image processing method. Steps S1101, S1102, and S1105-S1108 in Figure 11 are the same as steps S301, S302, and S305-S308 in Figure 3 described in the embodiment, respectively, so their explanations will be omitted.

[0053] In step S1103, the control unit 105 calculates region information relating to the area of ​​the subject on the image based on the distance map acquired in step S1102. In this embodiment, the projected area is calculated as region information. Here, the projected area is the surface area of ​​the subject in the image projected onto a plane substantially perpendicular to the optical axis 101a. The control unit 105 calculates the projected area based on the distance map of the extracted region. The calculation method involves, for example, calculating the projected area of ​​the object on each pixel based on the pixel size and focal length of the imaging device 100 and the distance map of the subject region, and then finding the sum of the projected areas of each pixel in the subject region. The projected area s per pixel can be calculated, for example, as shown in equation (3) below.

[0054]

number

[0055] dx: Pixel width dy: vertical width of the pixel L: Distance measurement value corresponding to the pixel f :focal length Next, in step S1104, the control unit 105 generates an evaluation value for the image 1201 based on the projection area. In this embodiment, the value of the projection area is used directly as the evaluation value. Thus, in this embodiment, the region information is information about the projection area of ​​the subject on a plane perpendicular to the optical axis. For example, the evaluation value is higher the larger the projection area. The processing from step S1105 onward is the same as in Embodiment 1.

[0056] The above describes the process for capturing images suitable for creating 3D models, where the user adjusts the camera angle while viewing the notification content (in this embodiment, an evaluation value indicating the projected area) displayed on the rear LCD monitor or EVF during still image capture.

[0057] When three faces of a cube are visible, if the surface area of ​​the subject on the image is used as the evaluation value, similar evaluation values ​​can be obtained at any angle, as in Example 1. However, when the number of pixels constituting face 1302 of the cube on the image is extremely small, as in image 1301 shown in Figure 13(A), problems arise when operating the virtual camera to view that face in the 3D model. Image 1303 in Figure 13(B) is an image rendered in this way, and the area of ​​face 1302 on the image is wider compared to Figure 13(A), resulting in a decrease in perceived resolution.

[0058] On the other hand, the effect of using the projected area as the evaluation value will be explained using the schematic diagram in Figure 14. Figure 14 is a schematic diagram of a 3D model generated based on an image of a cube taken from the side, viewed from directly above. L-shapes 1401 and 1402 represent the sides of the 3D model, and the sides of the cube that are not visible in the image are not shown. Each L-shape is photographed from the direction of arrow 1403. Therefore, L-shape 1402 includes a region 1404 with extremely low resolution. The lengths of arrows 1405 and 1406 indicate the size of the projected area of ​​L-shapes 1401 and 1402, respectively, and the projected area of ​​L-shape 1402, which includes the region 1404 with extremely low resolution, is smaller. Consequently, when using the projected area as the region information, the evaluation value of the image that forms the basis of the 3D model containing a region with extremely low resolution will be small and less likely to be remembered, thus reducing the possibility of a decrease in perceived resolution when viewing the 3D model.

[0059] In this embodiment, the projection area, which requires less computation, is calculated as region information and used directly as the evaluation value. However, an evaluation value that takes into account the inclination of the surface may be generated by other methods. Specifically, similar to Embodiment 1, the surface area may be calculated as region information, and the inclination of the subject's surface with respect to the optical axis of the imaging device 100 may be directly determined, generating a higher evaluation value for areas where the inclination is closer to perpendicular. In other words, the evaluation value may be higher the smaller the angle between the optical axis of the optical system 101 and the surface normal of the subject. This will allow the same effects as in this embodiment to be obtained.

[0060] Furthermore, the image processing unit 104 may acquire a contrast map (contrast information) in which the contrast value C of each pixel of the subject in the image is in map format. In this case, the control unit 105 may determine the evaluation value by using the area information of high-contrast regions as the projected area and the area information of low-contrast regions as the surface area. Alternatively, the evaluation value may be calculated based on the contrast value C from the values ​​of the projected area and the surface area, respectively. For example, in regions where the value of the contrast map is higher than a predetermined value, the control unit 105 may increase the influence of the projected area on the evaluation value compared to the surface area.

[0061] Specifically, the evaluation value r of a pixel may be calculated as shown in equation (4) below. Here, the contrast value C of each pixel can be calculated as C = (I1 - I2) / (I1 + I2) based on the maximum pixel value I1 and minimum pixel value I2 in a small grayscale region surrounding a pixel. The contrast value C approaches 0 as the contrast is small and approaches 1 as the contrast is large. In other words, the larger the contrast value, the larger the ratio of the projected area s1 to the evaluation value r, and the smaller the contrast value, the larger the ratio of the surface area s2.

[0062]

number

[0063] C: Pixel contrast value s1: Projection area of ​​the pixel s2: Surface area of ​​a pixel Image 1501, shown in Figure 15(A), is an image of cube 1502 taken from an oblique angle above. Here, face 1503 of cube 1502 has an extremely small number of pixels in the image, but there is no dot-like pattern and the contrast is zero. Image 1504, shown in Figure 15(B), is an image rendered by manipulating a virtual camera to view face 1503 of a 3D model generated from image 1501. Although the area of ​​face 1503 in image 1504 is larger than that of face 1503 in image 1501, the resolution does not decrease due to the zero contrast, and this is not a problem.

[0064] Thus, depending on the contrast of the subject, low-resolution areas may or may not be problematic when viewing 3D models. By calculating evaluation values ​​while considering contrast, it becomes easier to remember images that are more suitable for generating 3D models. [Examples]

[0065] Next, Embodiment 3 of the present invention will be described. This embodiment describes a case in which an image processing device such as a desktop computer is used to automatically select an image suitable for creating a 3D model from a plurality of captured images. In this embodiment, the subject is a plant in a flowerpot, but it is not limited to this. Note that if the configuration and processing are the same as in Embodiment 1, the explanation will be omitted. The image processing device in this embodiment is not limited to a desktop computer, but can be run on electronic devices such as laptop computers and portable computers, for example.

[0066] Referring to Figure 16, the configuration of the computer (image processing device) 1600 that performs image processing will be described. Figure 16 is a block diagram showing an example of the functional configuration of the computer 1600 in this embodiment.

[0067] The input unit 1610 is an interface that acquires user actions such as those from a mouse, keyboard, or joystick, as well as information necessary for generating multiple captured images and evaluation values. The information necessary for generating evaluation values ​​includes the distance map itself, camera parameters necessary for calculating the distance map, image A, and image B. In other words, the input unit 1610 has image acquisition means and distance map acquisition means (acquisition means).

[0068] The control unit (control means) 1611 controls the data from the input unit 1610, the calculation unit 1612, and the storage unit 1613, as well as the control of each part of the computer 1600 as a whole. The control unit 1611 has the function of calculating a distance map from the information necessary for generating a distance map acquired by the input unit 1610, and the function of generating an evaluation value. In other words, the control unit 1611 has distance map acquisition means (acquisition means) and evaluation value generation means (generation means).

[0069] The arithmetic unit 1612 has an arithmetic processing unit and executes arithmetic processing programs stored in the storage unit 1613. The storage unit 1613 consists of a primary storage device such as RAM or memory and a secondary storage device such as an SSD or hard disk. The storage unit 1613 stores information about 3D models and their position and orientation, as well as information about the position and orientation of the viewpoint, which are input via the input unit 1610. The storage unit 1613 also stores information about image processing performed by the arithmetic unit 1612.

[0070] The following describes the process by which images suitable for creating a 3D model are automatically selected from multiple images. Images 1701 to 1704 shown in Figure 17 are examples of multiple images (captured images) in this embodiment. The user is trying to select an image suitable for generating a 3D model from multiple images of plants.

[0071] (Image processing method) Referring to Figure 18, the image processing method performed in the computer of this embodiment will be described. Figure 18 is a flowchart of the image processing method.

[0072] First, in step S1801, the control unit 1611 acquires images 1701 to 1704. Next, in step S1802, the control unit 1611 acquires distance maps for each of the images 1701 to 1704. The computer 1600 can calculate the distance maps using, for example, the same method as in Embodiment 1. The following steps S1803 and S1804 are the same as in Embodiment 1, so their explanation is omitted. The control unit 1611 determines an evaluation value for each of the multiple images. Next, in step S1805, the control unit 1611 selects the image with the highest evaluation value from among the multiple images. In this embodiment, since image 1702 has the highest evaluation value, image 1702 is selected.

[0073] The above describes the process of automatically selecting images suitable for creating a 3D model from multiple already captured images. Subsequent processing can be performed as appropriate depending on the purpose.

[0074] For example, if a 3D model is needed quickly, the system immediately converts the selected image into a 3D model. If the user wants to check the 3D models of all images, the system generates them in order from highest to lowest evaluation score. The user can then check the outputted 3D models and interrupt the process once they are satisfied, efficiently obtaining appropriate 3D models from multiple images. Alternatively, the images may be sorted in descending order of evaluation score and communicated to the user along with the notification content, similar to Example 1. This helps the user select images for 3D model generation.

[0075] Additionally, during the process of generating a 3D model from the selected image, inpainting (modifying a portion of the image) may be performed to fill in the shape and texture of any holes in the subject. Because automatic selection allows for the selection of an image from among multiple images that provides a wide area for the subject to be used as the 3D model, the area to be filled in by inpainting is reduced, leading to lower computational load and improved quality, making it easier to obtain a better 3D model.

[0076] (Other examples) The present invention can also be realized by supplying a program that implements the functions of the above embodiment to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0077] According to each embodiment, it is possible to provide an image processing device, an imaging device, an image processing method, a program, and a storage medium that can reduce areas in a 3D model that cannot be viewed.

[0078] Each embodiment's disclosure includes the following configuration and method. (Composition 1) An acquisition means for acquiring an image and distance information of a subject in the image, It includes a control means for generating an evaluation value of the aforementioned image, The control means is characterized by calculating region information relating to the area of ​​the subject in the three-dimensional space in the image based on the distance information, and generating the evaluation value using the region information. (Configuration 2) The image processing apparatus according to configuration 1, characterized in that the region information is information relating to the surface area of ​​the subject. (Composition 3) The image processing apparatus according to configuration 2, characterized in that the larger the surface area of ​​the subject, the higher the evaluation value. (Composition 4) The image processing apparatus according to any one of configurations 1 to 3, characterized in that the smaller the angle between the optical axis of the imaging optical system and the surface normal of the subject, the higher the evaluation value. (Composition 5) The region information is information relating to the projected area of ​​the subject on a plane perpendicular to the optical axis, The image processing apparatus according to configuration 4, characterized in that the larger the projection area, the higher the evaluation value. (Composition 6) The region information includes information relating to the surface area of ​​the subject and information relating to the projected area of ​​the subject on a plane perpendicular to the optical axis of the imaging optical system. The acquisition means acquires the contrast map of the subject in the image, The image processing apparatus according to any one of configurations 1 to 5, characterized in that the control means makes the influence of the projection area on the evaluation value greater than the influence of the surface area in a region where the value of the contrast map is higher than a predetermined value. (Composition 7) The image processing apparatus according to any one of configurations 1 to 6, characterized in that the control means generates the evaluation value when the subject is contained in the image. (Composition 8) The image processing apparatus according to any one of configurations 1 to 7, characterized in that the larger the proportion of the subject in the image, the higher the evaluation value. (Composition 9) The image processing apparatus according to any one of configurations 1 to 8, characterized in that the control means broadcasts notification information relating to the evaluation value. (Composition 10) The image processing apparatus according to configuration 9, characterized in that the control means broadcasts the broadcast information when at least one of the upper limit or lower limit of the parameters relating to the image sensor is set. (Composition 11) The system further includes a display means for displaying the aforementioned image, The image processing apparatus according to configuration 9 or 10, characterized in that the control means displays the notification information together with the image on the display means. (Composition 12) The image processing apparatus according to configuration 11, characterized in that the control means determines the image with the highest evaluation value among a plurality of images in which the same subject exists, and displays it on the display means. (Composition 13) The image processing apparatus according to any one of configurations 9 to 12, characterized in that the control means provides information regarding the position and orientation of the imaging device in which the image with the maximum evaluation value is captured among a plurality of images in which the same subject exists. (Composition 14) The aforementioned image includes a first image for generating the evaluation value and a second image for storage in the storage means. The acquisition means is, The first image is acquired faster than the second image. An image processing apparatus according to any one of configurations 1 to 13, characterized in that the second image is acquired with higher image quality than the first image. (Composition 15) The image processing apparatus according to configuration 9 or 10, characterized in that the control means notifies the notification information by sound. (Composition 16) An image processing apparatus according to any one of configurations 1 to 15, further comprising a storage means for storing the aforementioned image. (Composition 17) The image processing apparatus according to configuration 16, characterized in that the storage means stores the image if the evaluation value of each of the multiple images in which the same subject exists exceeds the maximum value of the respective images. (Composition 18) The image processing apparatus according to any one of configurations 1 to 17, characterized in that, if the subject is not a face, the control means generates the evaluation value. (Composition 19) The acquisition means acquires a plurality of images and a plurality of distance information in each of the plurality of images, The control means is A plurality of evaluation values ​​are generated for each of the plurality of images, An image processing apparatus according to any one of configurations 1 to 18, characterized in that it selects the image with the highest evaluation value from among the plurality of images and creates a 3D model. (Composition 20) The image processing apparatus according to configuration 19, characterized in that the control means performs inpainting on the 3D model. (Composition 21) The image processing apparatus according to any one of configurations 1 to 20, characterized in that the evaluation value is a value used to evaluate the image as an image for a 3D model. (Composition 22) An imaging device characterized by having an image processing device according to any one of configurations 1 to 21 and an image sensor. (Method 1) An acquisition step to acquire an image and distance information of a subject in the image, The process includes a generation step for generating an evaluation value for the aforementioned image, An image processing method characterized in that, in the generation step, region information relating to the area of ​​the subject in the image is calculated based on the distance information, and the evaluation value is generated using the region information. (Composition 23) A program characterized by causing a computer to execute the image processing method described in Method 1. (Composition 24) A computer-readable storage medium characterized by storing the program described in configuration 23.

[0079] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence. [Explanation of Symbols]

[0080] 100 Imaging device (image processing device) 104 Image processing unit (acquisition means) 105 Control Unit (Control Means)

Claims

1. An acquisition means for acquiring an image and distance information of a subject in the image, It includes a control means for generating an evaluation value of the aforementioned image, The control means is characterized by calculating region information relating to the area of ​​the subject in the three-dimensional space in the image based on the distance information, and generating the evaluation value using the region information.

2. The image processing apparatus according to claim 1, characterized in that the region information is information relating to the surface area of ​​the subject.

3. The image processing apparatus according to claim 2, characterized in that the larger the surface area of ​​the subject, the higher the evaluation value.

4. The image processing apparatus according to claim 1, characterized in that the smaller the angle between the optical axis of the imaging optical system and the surface normal of the subject, the higher the evaluation value.

5. The region information is information relating to the projected area of ​​the subject on a plane perpendicular to the optical axis, The image processing apparatus according to claim 4, characterized in that the larger the projection area, the higher the evaluation value.

6. The region information includes information relating to the surface area of ​​the subject and information relating to the projected area of ​​the subject on a plane perpendicular to the optical axis of the imaging optical system. The acquisition means acquires the contrast map of the subject in the image, The image processing apparatus according to claim 1, characterized in that the control means increases the influence of the projection area on the evaluation value more than the surface area in a region where the value of the contrast map is higher than a predetermined value.

7. The image processing apparatus according to claim 1, characterized in that the control means generates the evaluation value when the subject is contained in the image.

8. The image processing apparatus according to claim 1, characterized in that the evaluation value is higher the larger the proportion of the subject in the image.

9. The image processing apparatus according to claim 1, characterized in that the control means broadcasts notification information relating to the evaluation value.

10. The image processing apparatus according to claim 9, characterized in that the control means broadcasts the broadcast information when at least one of the upper limit or lower limit of the parameters relating to the image sensor is set.

11. The system further includes a display means for displaying the aforementioned image, The image processing apparatus according to claim 9, characterized in that the control means displays the notification information together with the image on the display means.

12. The image processing apparatus according to claim 11, characterized in that the control means determines the image with the highest evaluation value among a plurality of images in which the same subject exists, and displays it on the display means.

13. The image processing apparatus according to claim 9, characterized in that the control means provides information regarding the position and orientation of the imaging apparatus in which the image with the maximum evaluation value is captured among a plurality of images in which the same subject exists.

14. The aforementioned image includes a first image for generating the evaluation value and a second image for storage in the storage means. The acquisition means is, The first image is acquired faster than the second image. The image processing apparatus according to claim 1, characterized in that the second image is acquired with higher quality than the first image.

15. The image processing apparatus according to claim 9, characterized in that the control means notifies the notification information by sound.

16. The image processing apparatus according to claim 1, further comprising a storage means for storing the aforementioned image.

17. The image processing apparatus according to claim 16, wherein the storage means stores the image if the evaluation value of each of the multiple images in which the same subject exists exceeds the maximum value.

18. The image processing apparatus according to claim 1, characterized in that, if the subject is not a face, the control means generates the evaluation value.

19. The acquisition means acquires a plurality of images and a plurality of distance information in each of the plurality of images, The control means is A plurality of evaluation values ​​are generated for each of the plurality of images, The image processing apparatus according to claim 1, characterized in that it selects the image with the highest evaluation value from among the plurality of images and creates a 3D model.

20. The image processing apparatus according to claim 19, characterized in that the control means performs inpainting on the 3D model.

21. The image processing apparatus according to claim 1, characterized in that the evaluation value is a value used to evaluate the image as an image for a 3D model.

22. An imaging device characterized by having an image processing device according to any one of claims 1 to 21 and an image sensor.

23. An acquisition step to acquire an image and distance information of a subject in the image, The process includes a generation step for generating an evaluation value for the aforementioned image, An image processing method characterized in that, in the generation step, region information relating to the area of ​​the subject in the three-dimensional space in the image is calculated based on the distance information, and the evaluation value is generated using the region information.

24. A program characterized by causing a computer to execute the image processing method described in claim 23.

25. A computer-readable storage medium characterized by storing the program described in claim 24.

Images