Information processing system, image processing device, optical device, information processing method, and program

Abstract

Provided are an information processing system, an image processing device, an optical device, an information processing method, and a program that can acquire good images from a plurality of polarized images obtained by imaging an object. The information processing system comprises one or more processors. The one or more processors: acquire a plurality of polarized images acquired by imaging the object in a state in which polarization directions are different with respect to a plurality of object imaging directions; and acquire a first image from two or more polarized images.

Description

Information processing system, image processing device, optical device, information processing method and program

[0001] The present invention relates to an information processing system, an image processing device, an optical device, an information processing method, and a program.

[0002] Patent document 1 discloses a camera that includes an optical lens, a rotatable polarizing filter, and a control means that detects the rotation angle of the polarizing filter, captures multiple images, and records each image by associating it with rotation angle information.

[0003] JP 2014-182328 A

[0004] One embodiment of the technique of the present disclosure provides an information processing system, an image processing device, an optical device, an information processing method, and a program that are capable of acquiring a good image from a plurality of polarized images of an object.

[0005] The information processing system of the first aspect is an information processing system having one or more processors, which acquire multiple polarized images obtained by photographing an object in multiple shooting directions with different polarization directions, and acquire a first image from the two or more polarized images.

[0006] In the information processing system of the second aspect, in the first aspect, the one or more processors acquire supplementary information including information on the polarization direction of the polarized image.

[0007] In the information processing system of the third aspect, in the first or second aspect, the one or more processors acquire a first image generated based on analysis of two or more images with different polarization directions.

[0008] In the information processing system of the fourth aspect, in the third aspect, the analysis is an analysis in units of pixels.

[0009] In a fifth aspect of the information processing system, in the fourth aspect, the pixel-by-pixel analysis of corresponding portions corresponding to the same position in two or more polarization images with different polarization directions is comparative dark synthesis.

[0010] An information processing system of a sixth aspect is any one of the first to fifth aspects, in which, when generating a first image, one or more processors convert at least one of the two or more images with different polarization directions into the coordinate system of at least one other image by geometric transformation based on information between the two or more images with different polarization directions.

[0011] The information processing system of the seventh aspect is any one of the first to sixth aspects, wherein one or more processors determine whether or not it is appropriate to apply a first image to the generation of a plurality of polarization images.

[0012] In an information processing system of an eighth aspect, in the second aspect, the supplementary information includes position information of the imaging device at the time of imaging.

[0013] The information processing system of a ninth aspect is the information processing system of the second or eighth aspect, in which the supplementary information is associated with the first image.

[0014] An information processing system of a tenth aspect is any one of the first to ninth aspects, wherein one or more processors generate a 3D model of the object based on first images generated from a plurality of shooting directions.

[0015] An information processing system of an eleventh aspect is any one of the first to tenth aspects, in which one or more processors analyze two or more images with different polarization directions acquired in one of a plurality of shooting directions, and determine the polarization direction in the shooting direction different from the one of the plurality of shooting directions.

[0016] An information processing system of a twelfth aspect is any one of the first to eleventh aspects, in which one or more processors determine conditions for capturing two or more images with different polarization directions based on the environmental conditions under which the object is photographed and the photographing position for photographing the object.

[0017] An information processing system of a thirteenth aspect is any one of the first to eleventh aspects, in which one or more processors accept a shooting mode and determine conditions for capturing two or more images with different polarization directions based on the shooting mode.

[0018] The information processing system of the 14th aspect is any one of the 1st to 13th aspects, in which one or more processors calculate specular reflectance or diffuse reflectance at coordinate positions in two or more images with different polarization directions based on information on the shooting direction and polarization direction.

[0019] The information processing system of the 15th aspect is any one of the first to 14th aspects, in which one or more processors determine the conditions for capturing multiple polarized images based on a pre-image of the object before acquiring the multiple polarized images.

[0020] An information processing system of a sixteenth aspect is any one of the first to fifteenth aspects, wherein the two or more polarized images are time frame images included in a moving image.

[0021] The image processing device of the 17th aspect is an image processing device that includes an optical device equipped with a polarizing filter that can capture an image of an object, and an information processing device, and is equipped with one or more processors included in either the optical device or the information processing device, and acquires multiple polarized images obtained by capturing images of the object in multiple shooting directions with different polarization directions, and acquires a first image from the two or more polarized images.

[0022] An image processing device of an eighteenth aspect is the image processing device of the seventeenth aspect, wherein the one or more processors acquire supplementary information including information on the polarization direction of the polarized image.

[0023] The image processing device of a 19th aspect is the image processing device of the 17th or 18th aspect, wherein the one or more processors generate a 3D model of the object based on first images generated in a plurality of shooting directions.

[0024] The image processing device of the 20th aspect is any one of the 17th to 19th aspects, in which one or more processors determine the conditions for capturing multiple polarized images based on a pre-image of the object before acquiring the multiple polarized images.

[0025] The image processing device of a 21st aspect is any one of the 17th to 20th aspects, wherein the one or more processors perform control to move the optical device or the object.

[0026] An image processing device of a 22nd aspect is any one of the 17th to 21st aspects, wherein the one or more processors perform control to change the polarization direction of the polarizing filter.

[0027] An optical device of the 23rd aspect is an optical device capable of capturing images of an object, which is equipped with one or more processors and a polarizing filter, wherein the one or more processors acquire multiple polarized images obtained by capturing images of the object in multiple shooting directions with different polarization directions, and acquire a first image from the two or more polarized images.

[0028] The optical device of a 24th aspect is the 23rd aspect, wherein the one or more processors acquire supplementary information including information on the polarization direction of the polarized image.

[0029] In the optical device of the 25th aspect, in the 23rd or 24th aspect, one or more processors determine the conditions for capturing multiple polarization images based on a pre-image of the object before acquiring the multiple polarization images.

[0030] The optical device of aspect 26 is any one of aspects 23 to 25, wherein one or more processors generate a 3D model of the object based on first images generated in multiple shooting directions.

[0031] The information processing method of the 27th aspect is an information processing method executed by an information processing system having one or more processors, in which the one or more processors acquire multiple polarized images obtained by photographing an object in multiple shooting directions with different polarization directions, and acquire a first image from the two or more polarized images.

[0032] The program of the 28th aspect is a program that causes an information processing method to be executed by an information processing system having one or more processors, and causes the one or more processors to acquire multiple polarized images obtained by photographing an object in multiple shooting directions with different polarization directions, and to acquire a first image from the two or more polarized images.

[0033] FIG. 1 is a diagram illustrating an overview of an image processing device. FIG. 2 is a diagram illustrating an image processing device different from FIG. 1. FIG. 3 is a block diagram illustrating a schematic configuration of a camera. FIG. 4 is a block diagram illustrating a schematic configuration of a computer. FIG. 5 is a flowchart for explaining an information processing method. FIG. 6 is a diagram for explaining an example of polarized image shooting. FIG. 7 is a diagram for explaining an example of polarized image shooting. FIG. 8 is a block diagram illustrating functions related to acquisition of a first image. FIG. 9 is a block diagram illustrating functions related to 3D model generation. FIG. 10 is a flowchart illustrating a method for identifying polarized images to be excluded. FIG. 11 is a flowchart for explaining an information processing method including pre-shooting. FIG. 12 is a diagram for explaining an example of condition setting for polarized image shooting.

[0034] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings.

[0035] [Summary] In 3D modeling using photogrammetry, if the image used to create the model contains reflective areas, the accuracy of the generated 3D model may be degraded. In situations where it is not possible to create an environment such as external lighting, it is necessary to obtain images to be used for 3D modeling while minimizing the effects of reflections.

[0036] Therefore, in the embodiment, a polarizing member is used to obtain an image with reduced reflection so that the quality of the 3D model can be improved.

[0037] 1 is a diagram showing a system 1 according to an embodiment. The system 1 includes a camera 10 and a computer 20. An object 30 to be photographed is placed on a photographing stand 40.

[0038] The camera 10 can move around the object 30. The camera 10 is configured to capture images of the object 30 from multiple directions as it moves around the object 30. The camera 10 is equipped with a rotatable polarizing filter 50. When capturing an image of the entire object 30, the camera 10 moves around the object 30 (e.g., 360° or more). When capturing an image of only a portion of the object 30, the camera 10 moves within a specific range of the object 30 (a range less than 360°). The camera 10 can be moved around the object 30 by a user holding the camera 10 or by a mobile object supporting the camera 10. The mobile object may be, for example, an arm attached to the imaging stand 40. The arm is configured to be movable around the imaging stand 40 by a motor or the like. If the object 30 is large, the mobile object may be a vehicle or a drone. The polarizing filter 50 is an example of a polarizing member of the present invention. The polarizing member is not limited to the polarizing filter 50, as long as it can control the polarization of light oscillating in a specific direction and remove the light component (reflected light component) from a specific direction.

[0039] The camera 10 is configured to capture images of the object 30 in a shooting direction at each position of the camera 10 while moving around the object 30. The camera 10 can also acquire multiple polarized images of the object 30 captured in different polarization directions in the shooting direction through a polarizing filter 50, the polarization direction of which can be set to any angle. As will be described later, the polarization direction is the rotation angle of the polarizing filter 50 relative to a reference angle when the object 30 is captured by the camera 10. The shooting direction is an angle that represents the relative positional relationship between the camera 10 and the object 30 with respect to the reference positions of the camera 10 and the object 30 when the object 30 is captured by the camera 10. The polarization direction of the polarizing filter 50 can be changed manually by the user or automatically according to the user's instructions. A polarized image is an image captured through the polarizing filter 50.

[0040] The camera 10 is configured to store polarized images of the object 30 captured through the polarizing filter 50. While the camera 10 is being moved around the object 30, the camera 10 can acquire multiple polarized images captured in different polarization directions (polarized image capture) with respect to the shooting direction of the object 30, either manually by a user or automatically.

[0041] The computer 20 has a display and a keyboard. The display is an example of a display device that displays various information. The keyboard is an example of an input device that allows a user to input instructions. The computer 20 is configured to process various data including images, input and output various data, and store various data. The computer 20 is configured to store programs and the like for executing its functions and to be able to execute the programs and the like.

[0042] By executing the program, the computer 20 can acquire multiple polarization images for the shooting direction captured by the camera 10. Furthermore, the computer 20 can acquire a first image from multiple (two or more) polarization images for the shooting direction. As a result, the computer 20 can acquire multiple first images.

[0043] The computer 20 is configured to execute a program to generate a 3D model of the object 30 by photogrammetry from a plurality of first images. Photogrammetry is a technology that analyzes a plurality of images of the object 30 taken from different angles and synthesizes the plurality of images to generate (restore) a three-dimensional shape or structure (3D model). In the embodiment, the plurality of images are a plurality of first images.

[0044] The object 30 is not particularly limited in shape or size as long as it is an object for which a 3D model is to be generated. The object 30 may be any object having a physical shape.

[0045] The imaging table 40 has a plurality of marks 41 on its placement surface (top surface). The marks 41 serve as indicators for the imaging direction. The distance between two marks 41 serves as a reference for the size when creating a 3D model. The imaging table 40 has a cylindrical shape with a flat top surface. The shape of the imaging table 40 is not limited as long as it can accommodate the object 30. The imaging table 40 may be a rectangular parallelepiped. However, the imaging table 40 is not essential when considering conditions such as the size of the object 30 and the location where the object 30 is placed.

[0046] Fig. 2 is a diagram showing an overview of system 2 that generates a 3D model different from system 1. In Fig. 2, parts common to the above-described system 1 are given the same reference numerals, and their description will be omitted. System 2 in Fig. 2 includes a camera 10 and a computer 20. The camera 10 is fixed to a tripod 44 and is in a stationary state. The tripod 44 is an example of a support member that keeps the camera 10 in a stationary state.

[0047] The object 30 is placed on an imaging table 42 having a mark 43. The imaging table 42 is rotatable, unlike the imaging table 40 of the system 1. The imaging table 42 can rotate the object 30 at any angle.

[0048] When the entire object 30 is to be imaged, the imaging table 42 rotates the object 30 (for example, 360° or more). When a part of the object 30 is to be imaged, the imaging table 42 rotates the object 30 within a specific range (a range smaller than 360°). The imaging table 42 includes a motor or the like and can rotate at any speed.

[0049] Similar to system 1, camera 10 is configured to acquire multiple polarized images taken with different polarization directions in the shooting direction of object 30 while object 30 is rotated by imaging stand 42. Camera 10 can manually or automatically capture images of object 30 from multiple shooting directions. While object 30 is being rotated, camera 10 can automatically or manually acquire multiple polarized images taken with different polarization directions in the shooting direction of object 30, either manually by a user's operation.

[0050] The camera 10 and computer 20 of system 2 are essentially the same as the camera 10 and computer 20 of system 1 .

[0051] [Camera] Fig. 3 is a block diagram showing a schematic configuration of the camera 10. As shown in Fig. 3, the camera 10 includes a lens device 100. The lens device 100 includes an imaging optical system 102 including a lens group 104 and an aperture 106, a lens driver 110, an aperture driver 112, a polarizing filter 50, a filter driver 160, a polarization direction detector 162, and the like.

[0052] The lens device 100 may be detachable from the camera 10 or may be integrated with the camera 10 .

[0053] The lens group 104 includes at least a focus lens that is movable in the optical axis direction. This focus lens is a lens for focus adjustment. The focus of the imaging optical system 102 is adjusted by moving the focus lens back and forth along the optical axis. The focus lens is driven and operated by the lens driver 110. The focus lens can be moved to a focus position where it focuses on the target object 30.

[0054] The diaphragm 106 is configured by, for example, an iris diaphragm. The amount of light passing through the imaging optical system 102 is adjusted by the diaphragm 106. The diaphragm 106 is driven by a diaphragm driver 112 to operate.

[0055] The polarizing filter 50 is an optical filter that controls the polarization of light oscillating in a specific direction and removes light components (reflected light components) from a specific direction. The polarizing filter 50 can change its polarization direction by rotating it. The polarizing filter 50 is driven and operated by the filter driver 160. The filter driver 160 is controlled by commands from the system controller 146. A user can change the polarization direction by rotating the polarizing filter 50 without using the filter driver 160. The polarizing filter 50 can be detachably attached to the lens device 100. The camera 10 can capture an image of the object 30 without using the polarizing filter 50. The filter driver 160 can include a motor that rotates the polarizing filter 50.

[0056] The polarization direction detector 162 detects the rotation angle of the polarization filter 50 at the time of shooting, relative to the reference angle of the polarization filter 50. The polarization direction detector 162 can include, for example, an encoder.

[0057] As shown in FIG. 3, the camera 10 includes an image sensor 130, a shutter 132, a shutter driver 134, a memory 136, a digital signal processor 138, an input / output interface 140, a display 142, an operation unit 144, and a system controller 146.

[0058] The image sensor 130 is, for example, a complementary metal-oxide semiconductor (CMOS) image sensor having a predetermined color filter array (e.g., a Bayer array). In the camera 10 of this embodiment, the image sensor 130 includes a drive unit, an analog-to-digital converter (ADC), a signal processor, and the like. In this case, the image sensor 130 is driven by the built-in drive unit. The signals from each pixel are converted into digital signals by the built-in ADC. Furthermore, the signals from each pixel are subjected to processes such as correlated double sampling, gain processing, and correction processing as needed by the built-in signal processor. The signal processing can be performed on the analog signals of each pixel, or on the digital signals of each pixel.

[0059] The imaging element 130 may be configured as an organic thin-film imaging element, an XY address type, a CCD (Charged Coupled Device) type, or other image sensor, in addition to a CMOS type image sensor.

[0060] The shutter 132 is disposed between the aperture 106 and the image sensor 130. The shutter 132 is driven to operate by a shutter driver 134. The shutter driver 134 controls the opening and closing of the shutter 132, and controls the exposure time (shutter speed) at the image sensor 130.

[0061] The memory 136 includes a flash memory, a read-only memory (ROM), a random access memory (RAM), an auxiliary storage device, etc. The flash memory and ROM store a camera control program, a focus bracket shooting program for executing shooting in focus bracket shooting mode, an image processing program, various data necessary for camera control, etc. The RAM temporarily stores shooting data and functions as a work area for processing by the system control unit 146. It also temporarily stores the camera control program, image processing program, etc. stored in the flash memory, etc. Note that part of the memory 136 (RAM) may be built into the system control unit 146.

[0062] The digital signal processing unit 138 performs signal processing such as offset processing, gamma correction processing, demosaic processing, and RGB / YCrCb conversion processing on the image obtained by shooting, thereby generating image data.

[0063] The input / output interface 140 includes a connection unit connectable to an external display device, a connection unit connectable to an external recording device, a card connection unit for inserting and removing a memory card, a communication unit connectable to a network, etc. For example, the input / output interface 140 may be a Universal Serial Bus (USB), a High-Definition Multimedia Interface (HDMI) (HDMI is a registered trademark), or the like.

[0064] The display unit 142 is used as a playback monitor when playing back captured images, a live view monitor that displays live view images during capture, and a setting monitor when performing various settings. The display unit 142 is configured with a display such as an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diode), for example.

[0065] The operation unit 144 is configured to include various operation members for operating the camera 10. The operation members include a power button, a shutter button, and various other operation buttons. The various operation buttons include a button for turning the function of the image stabilization mechanism 150 on and off. Furthermore, if the display unit 142 is configured as a display unit with a touch panel, the operation members that make up the operation unit 144 include the touch panel. The operation unit 144 outputs signals corresponding to the operation of each operation member to the system control unit 146. For example, the user can set the conditions for shooting using the polarizing filter 50 (polarized image shooting) from the operation unit 144.

[0066] The system control unit 146 performs overall control of the camera 10. The system control unit 146 also calculates various physical quantities required for control. The system control unit 146 is configured, for example, by a microcomputer equipped with a processor, memory, etc. The processor is configured, for example, by a CPU (Central Processing Unit), etc. The system control unit 146 can execute a program for capturing polarized images. The system control unit 146 also controls the shake correction mechanism 150.

[0067] The camera 10 may include a shake correction mechanism 150. The shake correction mechanism 150 is a BIS (Body Image Stabilizer) control type shake correction mechanism that corrects shake by shifting (including rotating) the image sensor 130 in a plane perpendicular to the optical axis in the direction opposite to the shake direction. The shake correction mechanism 150 includes a shake control unit 152, an image sensor drive unit 154, a shake detection unit 156, and a position detection unit 158.

[0068] The image sensor driver 154 may include an actuator that moves the image sensor 130. The shake detector 156 may include an acceleration sensor and / or a gyro sensor. The position detector 158 may include a Hall element that generates a voltage signal corresponding to the position of the image sensor 130. The shake controller 152 controls the image sensor driver 154 based on the signal from the shake detector 156 and the signal from the position detector 158 to shift the image sensor 130 in a plane perpendicular to the optical axis so as to offset shake of the camera 10. The shake correction mechanism may be provided in the lens device 100. The camera 10 is an example of an optical device of the present invention.

[0069] [Computer] FIG. 4 is a block diagram showing an example of the hardware configuration of the computer 20. As shown in FIG. 4, the computer 20 is configured to include a CPU 200, a RAM 202, a ROM 204, an auxiliary storage device 206, an input / output interface (IF) 207, an input device 208, and a display device 209. The ROM 204 and / or the auxiliary storage device 206 store programs executed by the CPU 200 and various data. The auxiliary storage device 206 is configured, for example, by a hard disk drive (HDD) or a solid state drive (SSD). The input device 208 is configured, for example, by a keyboard, a mouse, a touch panel, etc. The display device 209 is configured, for example, by an LCD or an OLED. Images captured by the camera 10 are imported into the computer 200 via the input / output interface 207 and stored, for example, in the auxiliary storage device 206.

[0070] The computer 20 includes a program for creating a first image from two or more polarization images, and further includes a program for generating a three-dimensional (3D) model from the plurality of first images. The CPU 200 executes the program to generate the first image and further generate the 3D model. The computer 20 is an example of an information processing device of the present invention.

[0071] [Information Processing Method] Next, an information processing method using the camera 10, the computer 20, and the photographing stand 40 will be described. Fig. 5 is a flowchart showing the information processing method. Figs. 6 and 7 are diagrams for explaining polarized image photographing.

[0072] 5, a user prepares a camera 10 equipped with a polarizing filter 50 and capable of capturing polarized images, a computer 20 capable of acquiring a first image and creating a 3D model, and an object 30. The user places the object 30 on an imaging stand 40 (step S1).

[0073] The user holds the camera 10 and moves to a shooting position to start shooting (step S2). The user fixes the object 30 to the shooting stand 40, and while holding the camera 10, moves around the object 30, taking polarized images of the object 30 from multiple shooting directions to acquire polarized images. Therefore, the user moves to an arbitrary reference shooting position as a shooting position to start shooting. At the reference shooting position, a shooting direction that serves as a reference for the relative positions of the camera 10 and the object 30 is determined.

[0074] At the photographing position, the user photographs a polarized image of the object 30 in a certain photographing direction using the camera 10 (step S3).

[0075] As shown in 6-1 in Figure 6, the user holds the camera 10 and moves to a shooting position (here, the reference shooting position), and determines settings (shooting position, camera focal length, etc.) that will capture the entire object 30 in the shooting range in a certain shooting direction of the object 30. When starting polarized image shooting, a shooting direction θ that represents the relative positional relationship between the camera 10 and the object 30 is determined. At 6-1, this is the position where polarized image shooting will start, so the shooting direction θ is determined to be 0 degrees.

[0076] Next, the user switches the camera 10 to the polarized image capturing mode. As shown in 6-2 in Figure 6, the system control unit 146 controls the filter drive unit 160, which moves the polarizing filter 50 to the polarization direction φ (= 0 degrees), which is the reference angle.

[0077] Next, the user points the camera 10 at the object 30 and presses the shutter button. As shown in 6-2 in Figure 6, the camera 10 is controlled by the system control unit 146 and uses its continuous shooting function to capture multiple polarized images while changing the polarization direction of the polarizing filter 50. This makes it possible to obtain multiple polarized images of the same object 30 at the same shooting position (θ = 0 degrees) and with the same angle of view, but with different polarization directions only.

[0078] As shown in 6-3 in Figure 6, each polarization image can be expressed as Img(θ1, φ1), Img(θ1, φ2), Img(θ1, φ3), Img(θ1, φ4) ... Img(θn, φm) by associating the shooting direction θ with the polarization direction φ. As mentioned above, θ is the shooting direction, and φ is the polarization direction. m is a parameter that identifies the shooting direction, and n is a parameter that identifies the polarization direction.

[0079] Each time a polarization image is captured, the polarization image is stored in the memory 136 as Img(θm, φn). The camera 10 can acquire multiple polarization images.

[0080] Specifically, the polarization images can be stored in memory 136 as, for example, Img(0,0), Img(0,30), Img(0,90), ..., Img(0,175). Furthermore, position information of camera 10, for example, (X,Y,Z,Pitch,Yaw,Roll)=(1,1,1.2,3,3,5), may be stored as additional information in association with the polarization images. Furthermore, the output value of shake detection unit 156 may also be stored in association with the polarization images.

[0081] The polarizing filter 50 may move smoothly (continuously). Alternatively, the polarizing filter 50 may move in discrete increments, rotating a set angle, stopping, and then rotating again. While the polarizing filter 50 is rotating, the camera 10 captures polarized images in each of the set polarization directions.

[0082] For example, under the control of the system control unit 146, the camera 10 can capture 12 polarized images (180 degrees) with the polarization direction φ changed by 15 degrees each time. After the system control unit 146 finishes capturing a polarized image, it can rotate the polarizing filter 50 to a reference angle to capture the next polarized image. The captured polarized image can include data on the capturing direction θ and the polarization direction φ in a tag area within the same file, such as an EXIF ​​file. The polarized image and the data on the capturing direction θ and the polarization direction φ may be stored in separate files.

[0083] Furthermore, when the user half-presses the shutter button, the camera 10 analyzes the live view image and determines the angle at which the polarizing filter 50 should be adjusted to capture a polarized image. The conditions for capturing a polarized image may be determined based on this determination. For example, if the camera 10 determines that the object 30 has a large number of specular areas, it may adjust the polarization direction finely (in 5-degree increments). Therefore, the direction in which polarized images are captured may differ depending on the shooting direction. For example, the polarization direction of the polarizing filter 50 may be changed in 15-degree increments from one shooting direction, and the polarization direction of the polarizing filter 50 may be changed in 5-degree increments from another shooting direction. This configuration reduces the user's effort and optimizes the number of polarized images captured.

[0084] Furthermore, lighting conditions (e.g., lighting position, direction, intensity, etc.) can be input in advance to the camera 10 or the computer 20. Using this information and information on the current position of the camera 10, the camera 10 or the computer 20 can determine the conditions for capturing a polarized image, such as the polarization direction of the polarizing filter 50.

[0085] The user can manually set the reflection suppression rate (low, medium, high) as a mode, or the camera 10 can automatically set it. Depending on the setting, the number of polarized images taken in a certain shooting direction θi can be increased or decreased.

[0086] Next, after step S3, it is determined whether or not the image capture of the object 30 has been completed (step S4). If the result of this determination is that the image capture has not been completed (step S4: No), the camera 10 is moved to the next image capture position (step S5). Then, the process returns to step S3, and the user uses the camera 10 to capture a polarized image of the object 30 in the next image capture direction at the next image capture position (step S3).

[0087] As shown in 7-1 of Figure 7, the user takes the camera 10 and moves to the next shooting position, and the user determines the settings so that the entire object 30 can be captured in the shooting range at the next shooting direction (θ = 15 degrees).

[0088] Next, the user points the camera 10 at the object 30 and presses the shutter button. As shown in 7-2 of FIG. 7, the camera 10 is controlled by the system control unit 146 and captures multiple polarized images while changing the polarization direction of the polarizing filter 50 using the continuous shooting function. This allows multiple polarized images of the same object 30 to be acquired at the same shooting position (θ = 15 degrees). Specifically, as shown in 7-3 of FIG. 7, polarized images captured at a shooting direction θ (= 15 degrees) are stored in the memory 136 as, for example, Img(15,0), Img(15,30), Img(15,90), ..., Img(15,175). As in FIG. 6, additional information may be stored in association with the polarized images. Furthermore, the output value of the shake detection unit 156 may also be stored in association with the polarized images.

[0089] The camera 10 or computer 20 may also analyze information such as the polarization direction φi and the polarized image for another shooting direction θi for which polarized image shooting has already been completed, to determine the polarization direction for a certain shooting direction θk. If the shooting direction θi and the shooting direction θk are close to each other, the polarization direction φi can be applied to the polarization direction for the shooting direction θk as a condition for polarized image shooting in the shooting direction θk. Polarized image shooting in the shooting direction θk is performed without rotating the polarizing filter 50. This reduces the user's effort and optimizes the number of images shot.

[0090] The user moves around the object 30 (for example, θ=360 degrees) and repeats the processing from step S3 to step S5 until polarized image capture from each shooting direction is completed. As the user moves around the object 30, the current position may be identified using an acceleration sensor and / or gyro sensor of the shake detection unit 156 of the camera 10, and the shooting direction may be presented to the user as auxiliary information.

[0091] In the above embodiment, the polarized image specified by Img(θ1, φ1) is acquired as a still image, but it may also be acquired from a moving image. Specifically, the camera 10 captures a moving image of the object 30 while rotating the polarizing filter 50 from a reference shooting direction. A plurality of time frame images included in the captured moving image can be acquired, and the time frame images can be used as polarized images. When acquiring the time frame images, the shooting direction θ and the polarization direction φ can be associated and stored as supplementary information.

[0092] In step S4, the system control unit 146 may determine whether the image capturing operation in the set image capturing direction has been completed. The system control unit 146 may also display images that have already been captured on the display unit 142 so that the user can easily determine whether the image capturing operation has been completed.

[0093] On the other hand, if the image capturing of the object 30 has been completed (step S4: Yes), the camera 10 ends the polarized image capturing.

[0094] Next, the computer 20 acquires a plurality of polarization images of the object 30 in the shooting direction captured by the camera 10 (step S6). The computer 20 acquires the plurality of polarization images from the memory 136 of the camera 10 via the input / output interface 207, either wired or wirelessly, or from a recording medium such as a memory card.

[0095] All of the polarization images are stored in the auxiliary storage device 206 of the computer 20, with the shooting direction θm and polarization direction φn associated with each other, as shown in Table 1 below. Table 1 includes polarization images shot in shooting directions θ1 to θm and polarization directions φ1 to φn. The conditions (number of images, polarization direction, etc.) of the included polarization images do not all need to be the same for shooting directions θ1 to θm.

[0096]

[0097] Next, the computer 20 acquires a first image from two or more polarized images in the shooting direction (step S7). That is, the first image is acquired from two or more polarized images that have the same shooting direction θ but differ only in polarization direction φ. Specifically, the first image is acquired by the CPU 200 of the computer 20 performing comparative dark compositing of two or more polarized images. Note that the comparative dark compositing technique itself is a well-known technique. The outline of this process is as follows.

[0098] Fig. 8 shows functional blocks related to generation of the first image by the CPU 200. As shown in Fig. 8, the image acquisition unit 210 acquires all polarization images for the shooting direction from the auxiliary storage device 206 (1: Img(θ1, φ1), Img(θ1, φ2), ... Img(θ1, φn), 2: Img(θ2, φ1), Img(θ2, φ2), ... Img(θ2, φn), ..., m: Img(θm, φ1), Img(θm, φ2), ... Img(θm, φn)).

[0099] The alignment unit 211 aligns the two or more polarization images used to acquire the first image. For example, if camera shake or the like occurs, the multiple polarization images may be aligned to the same coordinate system using a geometric transformation such as an affine transformation. Aligning to the same coordinate system may involve using one of the two or more polarization images as a reference polarization image and aligning the other polarization images to the reference polarization image, or the two or more polarization images may be aligned to a different coordinate system. The presence or absence of camera shake can be determined, for example, from the output value of the shake detection unit 156, which is one of the additional information. However, camera shake detection is not limited to the output value of the shake detection unit 156, and other methods may also be used. Accurately aligning the positions of the multiple polarization images can prevent unnatural edges and blurring during synthesis.

[0100] The image analysis unit 212 of the CPU 200 selects the pixel value with the lowest brightness for pixels at the same coordinates from the polarization images in the shooting direction (for example, Img(θ1, φ1), Img(θ1, φ2), ... Img(θ1, φn)) as the pixel value for that coordinate position. In other words, the image analysis unit 212 performs image analysis on pixels (pixel by pixel) at the same coordinates in each polarization image with the same shooting direction θ but different polarization directions φ.

[0101] Next, the synthesis unit 213 of the CPU 200 replaces the pixel value at the same pixel position in the two or more polarization images with the smaller pixel value selected for each pixel, and obtains a new first image.

[0102] This is because if the polarization angle is set inappropriately for reflected light, the effect of reflected light cannot be eliminated, and as a result, the intensity of light reaching each pixel becomes high (i.e., it becomes bright and white), resulting in a large pixel value.On the other hand, if reflection is suppressed, the other side of the reflecting object (water surface or glass) is photographed, so objects and scenery that are (basically) darker than the reflected light are photographed.

[0103] Even if the object 30 has a highly reflective surface, the computer 20 can obtain a first image in which the influence of reflection is suppressed. As will be described later, a high-quality three-dimensional (3D) model can be generated even for an object 30 with a highly reflective surface.

[0104] Furthermore, specular reflectance and diffuse reflectance at each position (in this case, each coordinate in the polarized image) of the object 30 may be calculated from multiple polarized images and polarization directions. The specular reflectance and diffuse reflectance can be calculated using the well-known technique of BRDF (Bidirectional Reflectance Distribution Function).

[0105] By calculating the specular reflectance and diffuse reflectance, it is possible to express the gloss, diffusion, and reflected color, which differ depending on the material of the object 30, when a 3D model is generated.

[0106] The composition unit 213 obtains a first image (Img_syn(θ1), Img_syn(θ2) ... Img_syn(θm)) from the polarized images (1: Img(θ1, φ1), Img(θ1, φ2) ... Img(θ1, φn), 2: Img(θ2, φ1), Img(θ2, φ2) ... Img(θ2, φn), ..., m: Img(θm, φ1), Img(θm, φ2) ... Img(θm, φn)). The image analysis and composition methods used in comparative dark composition are not particularly limited, and known techniques can be applied.

[0107] When the first image is acquired in the shooting direction, the CPU 200 stores the first image (Img_syn(θ1), Img_syn(θ2), ... Img_syn(θm)) in the auxiliary storage device 206. Furthermore, the first image may be stored in association with additional information about the polarized image, such as specular reflectance or diffuse reflectance.

[0108] In the above description, an example has been described in which the first image is acquired from two or more polarized images in the imaging directions θ1 to θm. However, this is not limiting, and for example, the polarization direction φ may be determined for a certain imaging direction θ. In other words, there is a case in which multiple polarized images are not acquired in a certain imaging direction θ. Therefore, in the imaging direction θi, one polarized image is acquired, and that polarized image is acquired as the first image.

[0109] Next, it is determined whether to generate a 3D model (step S8). If it is determined in step S8 that a 3D model will not be generated (step S8: No), the information processing ends. The stored multiple first images can be provided to another information system, for example, as images for generating a 3D model.

[0110] As an example of acquiring the first image, a case has been described in which all polarization images are acquired and then the first image is acquired for each shooting direction. However, this is not limited to this. It is also possible to acquire a polarization image for a certain shooting direction, acquire the first image, move to the next shooting position, acquire a polarization image for the next shooting direction, and then acquire the first image. In other words, acquisition of polarization images and acquisition of the first image may be performed sequentially.

[0111] In step S8, if it is determined that a 3D model is to be generated (step S8: Yes), the process proceeds to step S9.

[0112] Next, the computer 20 generates a 3D model from the plurality of first images (step S9). Note that the technology for generating a 3D model using photogrammetry is a well-known technology. The outline of this process is as follows.

[0113] 9 shows functional blocks related to 3D model generation by the CPU 200. As shown in Fig. 9, the image acquisition unit 220 acquires first images (Img_syn(θ1), Img_syn(θ2), ... Img_syn(θm)) in the shooting direction from the auxiliary storage device 206, thereby acquiring a plurality of first images.

[0114] The point cloud data generation unit 221 analyzes multiple first images and generates three-dimensional point cloud data of feature points. The point cloud data generation unit 221 extracts feature points from each first image. Next, the point cloud data generation unit 221 sets corresponding feature points between different first images as corresponding points and matches the corresponding points. The point cloud data generation unit 221 estimates camera parameters (e.g., fundamental matrix, fundamental matrix, internal parameters, etc.) of the camera and estimates the shooting position and orientation based on the estimated camera parameters. Then, it obtains the three-dimensional positions of the feature points of the object 30. Bundle adjustment is performed as necessary. Point cloud data (point cloud) is generated by combining the three-dimensional coordinates of the estimated feature points.

[0115] The 3D patch model generation unit 222 performs processing to generate a 3D patch model of the subject based on the three-dimensional point cloud data of the object 30 generated by the point cloud data generation unit 221. Specifically, patches (meshes) are generated from the generated three-dimensional point cloud to generate a 3D patch model. This makes it possible to represent the undulations of the surface with a small number of points.

[0116] The 3D model generation unit 223 generates a textured 3D model (three-dimensional model) by performing texture mapping on the 3D patch model generated by the 3D patch model generation unit 222. The 3D model generation unit 223 maps the texture onto the mesh to give the 3D patch model a realistic appearance of the object 30. By adding specular reflectance and diffuse reflectance, the material, etc. of the object 30 can be expressed when the 3D model is generated.

[0117] The data of the generated 3D model is stored in the auxiliary storage device 206, etc. Furthermore, the data of the 3D model is displayed on the display device 209 as necessary.

[0118] In the embodiment, the case of the system 1 has been described in which the object 30 is placed on the imaging stand 40 and the camera 10 moves around the object 30. However, the present invention is not limited to this, and a 3D model can also be generated in the system 2.

[0119] In the embodiment, a 3D model is generated by photogrammetry from a plurality of first images of the object 30 in which reflections are suppressed, so that a highly accurate 3D model can be generated.

[0120] When creating a 3D model, the size of the 3D model can be determined by providing information about the distance between the two marks 41 on the imaging stand 40. The positional relationship between the camera 10 and the object 30 can be determined based on the multiple marks 41 on the imaging stand 40, and a 3D model can be generated.

[0121] [Preferred Embodiment] Next, a preferred embodiment will be described. In a preferred embodiment, the computer 20 identifies polarization images to be excluded when acquiring the first image. The computer 20 also identifies first images to be excluded when generating the 3D model.

[0122] 10 shows a flowchart for identifying polarized images to be excluded when acquiring a first image. As shown in FIG. 10, a plurality of polarized images are acquired for the shooting direction (step S11). Specifically, the image acquisition unit 210 acquires a plurality of polarized images.

[0123] Next, the multiple polarized images are aligned in the shooting direction (step S12). Specifically, the alignment unit 211 performs the alignment. For example, a reference image is selected from the multiple polarized images, and feature points are extracted from this reference image. The positions to which corresponding points in the reference image have moved in the remaining polarized images are tracked. Based on the results, the remaining polarized images are subjected to translation, rotation, enlargement, and reduction processing using affine transformation, etc. The multiple polarized images are aligned. This makes it possible to align the polarized images even if there is some camera shake, etc.

[0124] Next, the inappropriate polarized image is excluded from the targets when acquiring the first image (step S13). Specifically, the alignment unit 211 excludes the polarized image from the targets when acquiring the first image. The alignment unit 211 can determine not to apply the polarized image that could not be aligned with the reference image in step S12 to acquiring the first image.

[0125] Furthermore, it can be determined that a polarized image with significant camera shake is not to be applied to the first image. The presence or absence of camera shake can be determined from the high frequency components of the polarized image or from the output value of the camera shake detection unit 156. The alignment unit 211 associates information indicating that the polarized image should not be applied to obtaining the first image with the polarized image, and stores the information in the auxiliary storage device 206.

[0126] When excluding a polarized image from the target, the polarized image itself may be excluded, or only a specific region within the polarized image may be excluded.

[0127] Next, image analysis is performed for each unit pixel of the multiple polarized images captured in the same direction (step S14). Finally, a first image is obtained from each polarized image (step S15). Note that the polarized images excluded in step S13 are not included in the first image.

[0128] By eliminating polarization images that are not suitable for the first image, a high-quality first image can be obtained, and therefore a high-quality 3D model can be obtained from the high-quality first image.

[0129] 10 , when a polarization image to be excluded from the target of the first image is determined, the computer 20 can also prompt the camera 10 to retake the image. Information about the position and attitude of the camera 10 that took the polarization image can be obtained from information accompanying the polarization image when it is acquired. The computer 20 can notify the user of the position and attitude of the camera 10. The user can easily know the location of the object 30 for which a polarization image needs to be retaken.

[0130] 11 shows a flowchart for identifying first images to be excluded when generating a 3D model. As shown in FIG. 11, first images are acquired (step S21). Specifically, the image acquisition unit 220 acquires multiple first images.

[0131] Next, feature points are extracted from the plurality of first images (step S22). The feature points are detected as corresponding points from the plurality of first images (step S23). Specifically, the point cloud data generation unit 221 extracts feature points from the plurality of first images and compares the feature points of the plurality of first images to detect matching feature points as corresponding points.

[0132] Next, the first image is excluded from the target of the 3D model (step S24). Specifically, the point cloud data generation unit 221 can determine not to apply a first image from which feature points cannot be extracted in step S22 or from which corresponding points cannot be detected in step S23 to the 3D model. Furthermore, the point cloud data generation unit 221 can determine that the first image is an image to be excluded from the target of the 3D model based on high-frequency components, etc. of the first image. The point cloud data generation unit 221 associates information indicating that the first image will not be applied to the generation of the 3D model with the first image and stores the information in the auxiliary storage device 206.

[0133] Next, a 3D model is generated from the first image (step S25). Specifically, the point cloud data generation unit 221 generates point cloud data from the first image by combining the camera parameters, the shooting position and orientation, and the three-dimensional coordinates of the estimated feature points, and the 3D patch model generation unit 222 and the 3D model generation unit 223 generate a 3D model. However, the first image does not include first images that have been excluded from the target of the 3D model. Because first images that are not suitable for generating a 3D model have been excluded, a high-quality 3D model can be generated.

[0134] 11 , when a first image to be excluded from the target of the 3D model is determined, the computer 20 can also prompt the camera 10 to retake an image. Information about the position and orientation of the camera 10 that captured the polarized image applied to the first image can be obtained from information accompanying the first image. The computer 20 can notify the user of the position and orientation of the camera 10. The user can easily know the location of the object 30 for which a polarized image required to generate the first image needs to be retaken.

[0135] <Setting Conditions for Polarized Image Capture> The following describes how to set conditions for capturing a polarized image. Setting conditions includes pre-photographing the object 30 before acquiring a polarized image.

[0136] Next, a description will be given of a case where the object 30 is pre-photographed by the camera 10. Fig. 12 is a flowchart showing a method for generating a three-dimensional model, including pre-photographing.

[0137] Prepare a camera 10 capable of capturing a polarized image, a computer 20 capable of acquiring a first image and creating a 3D model, and an object 30. As shown in Fig. 12, the user places the object 30 on an imaging stand 40 (step S41).

[0138] The user holds the camera 10 and moves to a shooting position for starting pre-shooting. The user fixes the object 30 to the shooting stand 40. The user holds the camera 10 and moves around the object 30, taking pre-shooting images of the object 30 from each shooting direction (step S42). Pre-shooting refers to capturing images of the object 30 before capturing a polarized image (step S45). The pre-shooting may be a still image or a moving image of the object 30. It may also be a live view image. The distance between the object 30 and the camera 10 can also be obtained.

[0139] Next, computer 20 acquires the preliminary images captured by camera 10 and acquires information for determining the conditions for capturing the polarized image (step S43). The information that can be acquired is sufficient to grasp the approximate size of object 30, the surface properties of object 30 (specular, rough, or degree of reflection), the polarization direction of polarizing filter 50, the lighting environment, etc. There is no particular limitation on the information that can be used for capturing the polarized image in step S45.

[0140] The user takes the camera 10 and moves to a shooting position to start shooting (step S44). At the shooting position, the user uses the camera 10 to shoot a polarized image of the object 30 (step S45). The polarized image shooting is performed based on the information acquired in the pre-shooting in step S43.

[0141] The conditions for capturing a polarized image are set by the system control unit 146 (CPU) of the camera 10 or the CPU 200 of the computer 20 .

[0142] FIG. 13 is a diagram illustrating an example of setting conditions for polarized image capture. FIG. 13 shows the results of pre-photographing an object 30 from a certain direction. 13-1 in FIG. 13 focuses on pixel values ​​at position P1 of the object 30. On the other hand, 13-2 in FIG. 13 focuses on pixel values ​​at position P2 of the object 30. Positions P1 and P2 are at the same position on the X axis, and position P1 is located above position P2 on the Y axis.

[0143] The camera 10 captures images of positions P1 and P2 through the polarizing filter 50 while continuously changing the polarization direction of the polarizing filter 50. The graphs shown in 13-1 and 13-2 are graphs with the vertical axis representing the intensity of pixel values ​​and the horizontal axis representing the polarization direction φ. Graph 13-1 is a graph in which the pixel values ​​for each polarization direction at position P1 are decomposed into RGB and plotted. Graph 13-2 is a graph in which the pixel values ​​for each polarization direction at position P2 are decomposed into RGB and plotted.

[0144] It can be seen from 13-1 and 13-2 that when the polarization direction is between 3 degrees and 4 degrees and between 24 degrees and 25 degrees, the pixel value is low and reflection from the object 30 is suppressed.

[0145] From the graphs shown in 13-1 and 13-2, it can be seen that at positions where the X axis is common, the pixel values ​​are low for the same polarization direction φ. In other words, it can be understood that it is not necessary to pay attention to all pixel values.

[0146] In step S45 described later, by using information (for example, polarization angle) obtained in the preliminary photographing, the accuracy of the polarization image can be improved and the reflectance can be reduced.

[0147] Furthermore, in graphs 13-1 and 13-2, pixel values ​​are acquired and plotted for all polarization directions. Therefore, it is expected that determining the polarization direction with low pixel values ​​will require a large amount of data and take a long time. Therefore, in the pre-photography, the polarizing filter 50 is rotated intermittently (for example, at 5-degree or 10-degree intervals) to acquire pixel values. The acquired pixel values ​​are plotted on a graph.

[0148] Although there are gaps in the graphs, these graphs have certain patterns. Therefore, by interpolating the gaps in the graphs (for example, using spline interpolation), it is possible to estimate the polarization direction that results in low pixel values. The estimated polarization direction can be applied to polarization image capture.

[0149] 12 , after step S45, it is determined whether or not image capture of the object 30 has been completed (step S46). If this determination indicates that image capture has not been completed (step S46: No), the camera 10 is moved to the next image capture position (step S47). At the next image capture position, the user uses the camera 10 to capture a polarized image of the object 30 in the next image capture direction (step S45). The processes from step S45 to step S47 are repeated until image capture of the object 30 is completed.

[0150] If the image capturing of the object 30 has been completed (step S46: Yes), the camera 10 ends the polarized image capturing.

[0151] Next, the computer 20 acquires a plurality of polarized images in the shooting direction of the object 30 acquired by the camera 10 (step S48).

[0152] Next, the computer 20 acquires a first image from two or more polarization images in the shooting direction (step S49).

[0153] Next, it is determined whether or not to generate a 3D model (step S50). If it is determined in step S50 that a 3D model will not be generated (step S50: No), the information processing ends.

[0154] In step S50, if it is determined that a 3D model is to be generated (step S50: Yes), the process proceeds to step S51.

[0155] Next, the computer 20 generates a 3D model from the plurality of first images (step S51). Once the 3D model has been generated from the first images, the information processing ends.

[0156] In the above embodiment, systems 1 and 2 constituting an image processing device including a camera 10 and a computer 20 have been described. The camera 10 includes a system control unit 146, and the computer 20 includes a CPU, and the functions of the system control unit 146 and the CPU 200 are not particularly limited. For example, the determination of conditions for capturing a polarized image may be performed by either the CPU 200 or the system control unit 146. In other words, as long as the camera 10 and the computer 20 can generate a first image, it is sufficient for either the CPU 200 or the system control unit 146 to fulfill its functions.

[0157] The present invention also includes an information processing system configured with a computer 20 that generates a first image. It also includes an optical device that uses only the camera 10 to capture a polarized image, obtain a first image from the polarized image, and generate a 3D model from the first image. While the example shows a case in which a polarized image is captured using one camera 10, polarized images can also be captured using multiple cameras 10.

[0158] [Configuration of the system control unit 146 and CPU 200] The functions of the system control unit 146 and CPU 200 are realized by various processors. The various processors include a CPU, which is a general-purpose processor that executes a program and functions as various processing units, a GPU (Graphic Processing Unit), a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically to execute specific processing. The term "program" is synonymous with "software."

[0159] A single processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same or different types. For example, a single processing unit may be composed of multiple FPGAs, or a combination of a CPU and an FPGA. Furthermore, multiple processing units may be composed of a single processor. Examples of multiple processing units composed of a single processor include: a first configuration, as typified by computers used as clients or servers, in which a single processor is composed of a combination of one or more CPUs and software, and this processor functions as multiple processing units; a second configuration, as typified by system-on-chip (SoC), in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip; and a hardware structure, in which the various processing units are composed of one or more of the above-mentioned various processors.

[0160] Although the present invention has been described above, the present invention is not limited to the above examples, and various improvements and modifications may be made without departing from the spirit of the present invention.

[0161] 1...System 2...System 10...Camera 20...Computer 30...Object 40...Photographing stand 41...Mark 42...Photographing stand 43...Mark 44...Tripod 50...Polarizing filter 100...Lens device 102...Photographing optical system 104...Lens group 106...Aperture 110...Lens driving unit 112...Aperture driving unit 130...Image sensor 132...Shutter 134...Shutter driving unit 136...Memory 138...Digital signal processing unit 140...Input / output interface 142...Display unit 144...Operation unit 146...System control unit 150...Correction mechanism 152...Control unit 154...Image sensor driving unit 156...Detection unit 158...Position detection unit 160...Filter driving unit 162...Polarization direction detection unit 200...CPU 202...RAM 204...ROM 206: Auxiliary storage device 207: Input / output interface 208: Input device 209: Display device 210: Image acquisition unit 211: Alignment unit 212: Image analysis unit 213: Synthesis unit 220: Image acquisition unit 221: Point cloud data generation unit 222: 3D patch model generation unit 223: 3D model generation unit

Claims

1. An information processing system having one or more processors, wherein the one or more processors acquire a plurality of polarized images obtained by photographing an object in a plurality of shooting directions with different polarization directions, and acquire a first image from two or more of the polarized images.

2. The information processing system according to claim 1, wherein the one or more processors acquire supplementary information including information on the polarization direction of the polarized image.

3. The information processing system according to claim 1 or 2, wherein the one or more processors acquire the first image generated based on an analysis of two or more images having different polarization directions.

4. The information processing system according to claim 3, wherein the analysis is a pixel-by-pixel analysis.

5. The information processing system according to claim 4, wherein the pixel-by-pixel analysis of corresponding portions of the two or more polarization images having different polarization directions that correspond to the same position is comparative dark synthesis.

6. An information processing system as described in claim 1 or 2, wherein when generating the first image, the one or more processors convert at least one of the two or more images having different polarization directions into the coordinate system of at least one other image by geometric transformation based on information between the two or more images having different polarization directions.

7. An information processing system as described in claim 1 or 2, wherein the one or more processors determine whether or not it is appropriate to apply the first image to the generation of the plurality of polarization images.

8. The information processing system according to claim 2, wherein the additional information includes location information of an imaging device at the time of shooting the image.

9. The information processing system according to claim 2 or 8, wherein the supplementary information is associated with the first image.

10. An information processing system as described in claim 1 or 2, wherein the one or more processors generate a 3D model of the object based on the first images generated in the multiple shooting directions.

11. An information processing system as described in claim 1 or 2, wherein the one or more processors analyze two or more images having different polarization directions acquired in one of the multiple shooting directions, and determine a polarization direction in a shooting direction other than the one of the multiple shooting directions.

12. The information processing system of claim 1, wherein the one or more processors determine conditions for capturing two or more images having different polarization directions based on environmental conditions for capturing an image of the object and a capturing position for capturing an image of the object.

13. An information processing system according to claim 1 or 2, wherein the one or more processors receive a shooting mode and determine conditions for shooting two or more images having different polarization directions based on the shooting mode.

14. An information processing system as described in claim 1 or 2, wherein the one or more processors calculate the specular reflectance or diffuse reflectance at coordinate positions in two or more images having different polarization directions based on information on the shooting direction and the polarization direction.

15. An information processing system as described in claim 1 or 2, wherein the one or more processors determine conditions for capturing the multiple polarized images based on a pre-image of the object taken before obtaining the multiple polarized images.

16. An information processing system according to claim 1 or 2, wherein the two or more polarized images are time frame images contained in a moving image.

17. An image processing device including an optical device equipped with a polarizing filter capable of capturing an image of an object, and an information processing device, the image processing device having one or more processors included in either the optical device or the information processing device, obtaining a plurality of polarized images obtained by capturing an object in a plurality of shooting directions with different polarization directions, and obtaining a first image from two or more of the polarized images.

18. The image processing device according to claim 17, wherein the one or more processors acquire supplementary information including information on the polarization direction of the polarized image.

19. The image processing device according to claim 17 or 18, wherein the one or more processors generate a 3D model of the object based on the first images generated in the multiple shooting directions.

20. An image processing device as described in claim 17 or 18, wherein the one or more processors determine conditions for capturing the multiple polarized images based on a pre-image of the object captured before acquiring the multiple polarized images.

21. An image processing device according to claim 17 or 18, wherein the one or more processors perform control to move the optical device or the object.

22. An image processing device according to claim 17 or 18, wherein the one or more processors perform control to change the polarization direction of the polarizing filter.

23. An optical device capable of capturing an image of an object, comprising one or more processors and a polarizing filter, wherein the one or more processors acquire a plurality of polarized images obtained by capturing images of the object in a plurality of shooting directions with different polarization directions, and acquire a first image from the two or more polarized images.

24. The optical device according to claim 23, wherein the one or more processors obtain supplementary information including information on the polarization direction of the polarized image.

25. An optical device as described in claim 23 or 24, wherein the one or more processors determine conditions for capturing the multiple polarized images based on a pre-image taken of the object before acquiring the multiple polarized images.

26. The optical device of claim 23 or 24, wherein the one or more processors generate a 3D model of the object based on the first images generated in the multiple shooting directions.

27. An information processing method executed by an information processing system having one or more processors, wherein the one or more processors acquire a plurality of polarized images obtained by photographing an object in a plurality of shooting directions with different polarization directions, and acquire a first image from two or more of the polarized images.

28. A program for causing an information processing method to be executed by an information processing system having one or more processors, the program causing the one or more processors to acquire a plurality of polarized images obtained by photographing an object in a plurality of shooting directions with different polarization directions, and to acquire a first image from two or more of the polarized images.

29. A non-transitory computer-readable recording medium having the program according to claim 28 recorded thereon.

Images