Information processing system, image processing apparatus, optical apparatus, information processing method, and program

Abstract

Provided are an information processing system, an image processing device, an optical device, an information processing method, and a program with which it is possible to acquire a good image from a plurality of polarized images obtained by capturing an object. This information processing system is provided with one or more processors that perform: a process for acquiring a plurality of polarized images captured in a plurality of imaging directions of an object in a state in which the polarization directions differ; and acquiring a first image from the two or more polarized images.

Description

Technical Field

[0001] This invention relates to an information processing system, an image processing device, an optical device, an information processing method, and a program. Background Technology

[0002] Patent document 1 discloses a camera comprising: an optical lens; a rotatable polarizing filter; and a control component that detects the rotation angle of the polarizing filter, captures multiple images, and associates and records the rotation angle information with each image.

[0003] Previous technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2014-182328 Summary of the Invention

[0006] -The technical problem that the invention aims to solve-

[0007] One embodiment of the present invention provides an information processing system, image processing apparatus, optical apparatus, information processing method, and program capable of acquiring good images from multiple polarized images of a photographed object.

[0008] -Means used to solve technical problems-

[0009] The information processing system of the first method has one or more processors, wherein the one or more processors perform the following processing: acquiring multiple polarized images acquired by shooting the object in multiple photographic directions with different polarization directions; and acquiring a first image from two or more polarized images.

[0010] The information processing system of the second method, in the first method, involves one or more processors performing the following processing: acquiring incidental information including information about the polarization direction of the polarization image.

[0011] In the third method of information processing system, in the first or second method, one or more processors perform the following processing: acquiring a first image generated by analyzing two or more images with different polarization directions.

[0012] The fourth type of information processing system, in the third type, analyzes data at the pixel level.

[0013] The fifth method of information processing system, in the fourth method, analyzes the corresponding pixel units of two or more polarized images with different polarization directions at the same position as darkening synthesis.

[0014] In any one of the methods from method 1 to method 5, the information processing system of method 6 performs the following processing by one or more processors: when generating the first image, based on the information between two or more images with different polarization directions, transforms at least one of the two or more images with different polarization directions into the coordinate system of at least one other image through geometric transformation.

[0015] In the information processing system of the seventh method, in any one of the methods from the first to the sixth method, one or more processors perform the following processing: for multiple polarization images, determine whether they are appropriate for generating the first image.

[0016] The information processing system of the eighth method, in the second method, includes information on the location of the camera device during photography.

[0017] The information processing system of method 9 associates the accompanying information with the first image in method 2 or method 8.

[0018] In any of the methods 1 to 9, the information processing system of the 10th method involves one or more processors performing the following processing: generating a 3D model of an object based on a first image generated in multiple photographic directions.

[0019] In any of the methods from method 1 to method 10, the information processing system of method 11 performs the following processing by one or more processors: analyzing two or more images with different polarization directions acquired in one of the multiple photographic directions, and determining the polarization direction in the photographic direction that is different from the one of the multiple photographic directions.

[0020] In any of the methods 1 to 11, the information processing system of method 12 performs the following processing by one or more processors: determining the conditions for capturing two or more images with different polarization directions based on the environmental conditions of the object being photographed and the photographic position of the object being photographed.

[0021] In any of the methods 1 to 11, the information processing system of method 13 performs the following processing by one or more processors: accepting the photography mode and determining, based on the photography mode, the conditions for capturing two or more images with different polarization directions.

[0022] In any of the methods 1 to 13, the information processing system of method 14 performs the following processing by one or more processors: calculating the specular reflectance or diffuse reflectance at coordinate positions in two or more images with different polarization directions based on the information of the photographic direction and polarization direction.

[0023] In any of the methods 1 to 14, the information processing system of method 15 performs the following processing by one or more processors: before acquiring multiple polarization images, it determines the conditions for acquiring multiple polarization images based on a pre-image of the object being photographed.

[0024] In the information processing system of the 16th method, in any one of the methods from the 1st to the 15th method, two or more polarization images are time frame images contained in the dynamic image.

[0025] The image processing apparatus of the 17th embodiment includes an optical device having a polarizing filter capable of capturing an image of an object and an information processing device. The image processing device includes one or more processors included in either the optical device or the information processing device, acquires multiple polarized images acquired by capturing images of the object in multiple photographic directions with different polarization directions, and acquires a first image from two or more polarized images.

[0026] In the 17th embodiment, the image processing apparatus of the 18th embodiment involves one or more processors performing the following processing: acquiring incidental information including information about the polarization direction of a polarization image.

[0027] In the 17th or 18th method, the image processing apparatus of the 19th method performs the following processing by one or more processors: generating a 3D model of an object based on a first image generated in multiple photographic directions.

[0028] In any one of the 17th to 19th modes, the image processing apparatus of the 20th mode performs the following process: before acquiring multiple polarization images, it determines the conditions for acquiring multiple polarization images based on a pre-image of the object being photographed.

[0029] In any one of the 17th to 20th modes of the image processing apparatus of the 21st mode, one or more processors control the movement of the optical device or the object.

[0030] In any one of the 17th to 21st modes, the image processing apparatus of mode 22 uses one or more processors to control the polarization direction of the polarization filter.

[0031] The optical device of the 23rd method includes one or more processors and polarizing filters and is capable of capturing images of an object. The one or more processors perform the following processing: acquiring multiple polarized images by capturing images of the object in multiple photographic directions with different polarization directions; and acquiring a first image from two or more polarized images.

[0032] In the 23rd method, the optical device of the 24th method involves one or more processors performing the following processing: acquiring incidental information including information about the polarization direction of the polarization image.

[0033] In the optical device of the 25th method, in the 23rd or 24th method, one or more processors perform the following process: before acquiring multiple polarization images, determine the conditions for acquiring multiple polarization images based on a pre-image of the object being photographed.

[0034] In any one of the 23rd to 25th modes, the optical device of the 26th mode has one or more processors performing the following processing: generating a 3D model of the object based on a first image generated in multiple photographic directions.

[0035] The information processing method of the 27th method is executed by an information processing system having one or more processors, wherein the one or more processors perform the following processing: acquiring multiple polarized images acquired by taking pictures of an object in multiple photographic directions with different polarization directions; and acquiring a first image from two or more polarized images.

[0036] The program of the 28th method causes an information processing system having one or more processors to execute an information processing method, wherein the program causes one or more processors to perform the following processing: acquiring multiple polarized images acquired by taking pictures of an object in multiple photographic directions with different polarization directions; and acquiring a first image from two or more polarized images. Attached Figure Description

[0037] Figure 1 This is a diagram showing an overview of an image processing device.

[0038] Figure 2 It means and Figure 1 Diagrams of different image processing devices.

[0039] Figure 3 It is a block diagram representing the general structure of a camera.

[0040] Figure 4 It is a block diagram that represents the general structure of a computer.

[0041] Figure 5 It is a flowchart used to illustrate information processing methods.

[0042] Figure 6 This is a diagram used to illustrate an example of polarized image photography.

[0043] Figure 7 This is a diagram used to illustrate an example of polarized image photography.

[0044] Figure 8This is a block diagram representing the functions associated with the acquisition of the first image.

[0045] Figure 9 This is a block diagram representing the functions associated with 3D model generation.

[0046] Figure 10 This is a flowchart illustrating the method for determining polarization images that should be excluded.

[0047] Figure 11 This is a flowchart illustrating the method for determining the first image to be excluded.

[0048] Figure 12 It is a flowchart used to illustrate information processing methods, including pre-photography.

[0049] Figure 13 This diagram illustrates an example of the conditions set for polarized image photography. Detailed Implementation

[0050] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

[0051] [summary]

[0052] In photogrammetric 3D modeling, the accuracy of the generated 3D model may be compromised if the image used for modeling includes reflective elements. In scenes where external lighting or other environmental features cannot be constructed, it is necessary to minimize the effects of reflections when acquiring images for 3D modeling.

[0053] Therefore, in the implementation, the image with suppressed reflection can be obtained by utilizing the polarization component, so as to improve the quality of the 3D model.

[0054] <Implementation Method>

[0055] Figure 1 This is a diagram illustrating system 1 as described in the implementation method. System 1 includes a camera 10 and a computer 20. The photographic object, i.e., the subject 30, is positioned on a photographic platform 40.

[0056] Camera 10 is movable around object 30. Camera 10 is configured to capture images of object 30 from multiple directions when moving around it. Camera 10 includes a rotatable polarizing filter 50. When capturing the entire object 30, camera 10 moves around object 30 (e.g., 360° or more). When capturing only a portion of object 30, camera 10 moves within a specific range of object 30 (less than 360°). Camera 10 can be moved around object 30 by a user holding camera 10 or by a movable body supporting camera 10. The movable body is, for example, an arm mounted on camera stand 40. This arm is configured to move around camera stand 40 via a motor or the like. Furthermore, if object 30 is large, the movable body can be a vehicle or a drone. Polarizing filter 50 is an example of a polarizing component of the present invention. Polarizing components are not limited to polarizing filter 50 as long as they can control the polarization of light vibrating in a specific direction and remove components of light from that specific direction (reflected light components).

[0057] The camera 10 is configured to capture images of the object 30 in the photographic direction from various positions of the camera 10 while it moves around the object 30. Furthermore, the camera 10 can acquire multiple polarized images captured with different polarization directions in the photographic direction of the object 30 via a polarizing filter 50, which allows the polarization direction to be set to an arbitrary angle. As described later, the polarization direction is the rotation angle relative to a reference angle of the polarizing filter 50 when the camera 10 captures the object 30. The photographic direction is the angle representing the relative positional relationship between the camera 10 and the object 30 relative to the reference position when the camera 10 captures the object 30. The polarization direction of the polarizing filter 50 can be changed manually by the user or automatically according to the user's instructions. The polarized image is an image captured via the polarizing filter 50.

[0058] The camera 10 is configured to store polarized images of the object 30 obtained by photographing it through the polarizing filter 50. While the camera 10 is moved around the object 30, the camera 10 can acquire multiple polarized images obtained by manual or automatic shooting based on user operation, with different polarization directions in the photographing direction of the object 30 (polarized image photography).

[0059] Computer 20 has a display and a keyboard. The display is an example of a display device that shows various information. The keyboard is an example of an input device that allows a user to input instructions. Computer 20 is configured to process various types of data, including images, input and output various types of data, and store various types of data. Computer 20 is configured to store programs for performing its functions and is capable of executing programs.

[0060] Computer 20, by executing a program, is able to acquire multiple polarization images in the photographic direction acquired by camera 10. Furthermore, computer 20 is able to acquire a first image from multiple (more than two) polarization images in the photographic direction. As a result, computer 20 is able to acquire multiple first images.

[0061] The computer 20 is configured to generate a 3D model of an object 30 from multiple first images by executing a program using photogrammetry. Photogrammetry is a technique that analyzes multiple images of an object 30 taken from different angles and synthesizes these images to generate (restore) a three-dimensional shape or structure (3D model). In this embodiment, the multiple images are multiple first images.

[0062] Object 30 is any object for which a 3D model is to be generated; its shape and size are not particularly limited. Object 30 simply needs to be an object with a physical shape.

[0063] The camera stand 40 has multiple markings 41 on its mounting surface (upper surface). These markings 41 serve as indicators of the photographic direction. Furthermore, the distance between two markings 41 becomes a reference for the size of the 3D model. The camera stand 40 has a cylindrical upper surface with a flat surface. The shape of the camera stand 40 is not limited as long as it can accommodate the object 30. The camera stand 40 can be a cuboid. However, considering factors such as the size of the object 30 and its placement, the camera stand 40 is not essential.

[0064] Figure 2 This is a diagram illustrating a summary of System 2, which generates 3D models differently from System 1. Figure 2 In the text, the parts that are the same as those in System 1 above are marked with the same symbols, and their descriptions are omitted. Figure 2 System 2 includes a camera 10 and a computer 20. The camera 10 is fixed to a tripod 44 and is in a stationary state. The tripod 44 is an example of a support component that keeps the camera 10 stationary.

[0065] The object 30 is positioned on a camera stage 42 with a marking 43. Unlike the camera stage 40 in System 1, the camera stage 42 is rotatable. The camera stage 42 allows the object 30 to rotate at any angle.

[0066] When photographing the entire object 30, the camera stage 42 rotates the object 30 (e.g., 360° or more). When photographing a portion of the object 30, the camera stage 42 rotates the object 30 within a specific range (less than 360°). The camera stage 42 includes a motor or the like, and is capable of rotating at any speed.

[0067] The camera 10 is configured, similar to system 1, to acquire multiple polarized images taken in states where the polarization directions of the object 30 differ in the photographic direction of the object 30, while the object 30 is rotated via the photographing stage 42. The camera 10 can manually or automatically capture images of the object 30 from multiple photographic directions. During the rotation of the object 30, the camera 10 can acquire multiple polarized images, taken manually or automatically based on user operation, in states where the polarization directions of the object 30 differ in the photographic direction of the object 30.

[0068] The camera 10 and computer 20 of System 2 are basically the same as those of System 1 and its camera 10 and computer 20.

[0069] [camera]

[0070] Figure 3 This is a block diagram showing the general structure of camera 10. For example... Figure 3 As shown, the camera 10 includes a lens assembly 100. The lens assembly 100 includes an imaging optical system 102 comprising a lens group 104 and an aperture 106, a lens drive unit 110, an aperture drive unit 112, a polarizing filter 50, a filter drive unit 160, a polarization direction detection unit 162, etc.

[0071] The lens device 100 can be a device that can be attached to or detached from the camera 10, or it can be an integrated device with the camera 10.

[0072] The lens group 104 includes at least a focusing lens that can move along the optical axis. This focusing lens is a focusing lens. The imaging optical system 102 focuses by moving the focusing lens back and forth along the optical axis. The focusing lens is driven by the lens drive unit 110. The focusing lens can be moved to a focusing position that is the focus position for focusing on the object 30.

[0073] The aperture 106 is, for example, a variable aperture. The amount of light passing through the camera optical system 102 is adjusted by the aperture 106. The aperture 106 is driven by the aperture drive unit 112.

[0074] The polarizing filter 50 is an optical filter used to control the polarization of light vibrating in a specific direction and remove components of light from that specific direction (reflected light components). The polarizing filter 50 can change its polarization direction by rotation. The polarizing filter 50 is driven by a filter drive unit 160. The filter drive unit 160 is controlled according to instructions from the system control unit 146. The user can change the polarization direction by rotating the polarizing filter 50 without using the filter drive unit 160. The polarizing filter 50 can be detachably mounted on the lens assembly 100. The camera 10 can photograph the object 30 without using the polarizing filter 50. The filter drive unit 160 may include a motor for rotating the polarizing filter 50.

[0075] The polarization direction detection unit 162 detects the rotation angle of the polarization filter 50 relative to a reference angle of the polarization filter 50 during image capture. The polarization direction detection unit 162 may include, for example, an encoder.

[0076] like Figure 3 As shown, the camera 10 includes an imaging element 130, a shutter 132, a shutter drive unit 134, a memory 136, a digital signal processing unit 138, an input / output interface 140, a display unit 142, an operation unit 144, and a system control unit 146.

[0077] The imaging element 130 is, for example, a CMOS (Complementary Metal-Oxide Semiconductor) type image sensor having a predetermined color filter arrangement (e.g., Bayer arrangement). In the camera 10 of the embodiment, the imaging element 130 is configured to include a driving unit, an ADC (Analog to Digital Converter), and a signal processing unit. The imaging element 130 is driven by the built-in driving unit. Furthermore, the signal of each pixel is converted into a digital signal by the built-in ADC. Moreover, the signal of each pixel is subjected to correlation double sampling processing, gain processing, correction processing, etc., as needed by the built-in signal processing unit. The signal processing can be configured to process either the analog signal of each pixel or the digital signal of each pixel.

[0078] In addition to CMOS image sensors, the imaging element 130 can also be composed of organic thin-film imaging elements, XY address type, CCD (Charged Coupled Device) type image sensors.

[0079] The shutter 132 is positioned between the aperture 106 and the imaging element 130. The shutter 132 is driven by the shutter drive unit 134. The shutter drive unit 134 controls the opening and closing of the shutter 132 and controls the exposure time (shutter speed) in the imaging element 130.

[0080] The memory 136 includes flash memory, ROM (Read-only Memory), RAM (Random Access Memory), auxiliary storage devices, etc. The flash memory and ROM store camera control programs and various data required for executing focus bracketing shooting programs, image processing programs, camera control, etc., in focus bracketing shooting mode. The RAM temporarily stores photographic data and functions as a working area processed by the system control unit 146. Furthermore, it temporarily stores camera control programs, image processing programs, etc., stored in the flash memory, etc. Additionally, the system control unit 146 may have a portion of the memory 136 (RAM) built into it.

[0081] The digital signal processing unit 138 performs signal processing such as offset processing, gamma correction processing, de-mosaic processing, and RGB / YCrCb conversion processing on the image obtained by shooting, and generates image data.

[0082] The input / output interface 140 includes a connection section for connecting to an external display device, a connection section for connecting to an external recording device, a card connection section for inserting and removing a memory card, and a communication section for connecting to a network. For example, the input / output interface 140 can be compatible with USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface) (HDMI is a registered trademark), etc.

[0083] The display unit 142 serves as a playback monitor for displaying captured images and a live view monitor for displaying live view images during shooting. It also functions as a setting monitor when making various settings. The display unit 142 is composed of displays such as LCD (Liquid Crystal Display) or OLED (Organic Light Emitting Diode).

[0084] The operation unit 144 is configured to include various operating components for operating the camera 10. These operating components include a power button, a shutter button, and various types of operation buttons. Among these operation buttons are buttons for turning the image correction mechanism 150 on and off. Furthermore, if the display unit 142 is configured as a display unit with a touch panel, the operation components constituting the operation unit 144 include a touch panel. The operation unit 144 outputs signals corresponding to the operation of each operating component to the system control unit 146. For example, the user can set the photography conditions (polarized image photography) utilizing the polarizing filter 50 from the operation unit 144.

[0085] The system control unit 146 centrally controls the entire camera 10. Furthermore, the system control unit 146 calculates various physical quantities required for control. The system control unit 146 may be configured as, for example, a microcomputer equipped with a processor and memory. The processor may be, for example, a CPU (Central Processing Unit). The system control unit 146 is capable of executing programs for polarized image photography. The system control unit 146 includes control of the shake correction mechanism 150.

[0086] The camera 10 is equipped with a shake correction mechanism 150. The shake correction mechanism 150 is a BIS (Body Image Stabilizer) control method shake correction mechanism that performs shake correction by displacing (including rotating) the imaging element 130 in a plane orthogonal to the optical axis in the opposite direction to the shake direction. The shake correction mechanism 150 includes a shake control unit 152, an imaging element drive unit 154, a shake detection unit 156, and a position detection unit 158.

[0087] The imaging element driving unit 154 may include an actuator for moving the imaging element 130. The jitter detection unit 156 may include an accelerometer and / or a gyroscope sensor, etc. The position detection unit 158 ​​may include a Hall element that generates a voltage signal corresponding to the position of the imaging element 130. The jitter control unit 152 controls the imaging element driving unit 154 based on the signals from the jitter detection unit 156 and the position detection unit 158, causing the imaging element 130 to be displaced in a plane perpendicular to the optical axis to counteract the jitter of the camera 10. The jitter correction mechanism may be provided in the lens assembly 100. The camera 10 is an example of the optical device of the present invention.

[0088] [computer]

[0089] Figure 4 This is a block diagram illustrating an example of the hardware structure of computer 20. For example... Figure 4 As shown, the computer 20 is configured to include a CPU 200, RAM 202, ROM 204, auxiliary storage device 206, input / output interface (IF) 207, input device 208, and display device 209. The ROM 204 and / or auxiliary storage device 206 store programs and various data executed by the CPU 200. The auxiliary storage device 206 is, for example, a HDD (hard disk drive) or an SSD (solid state drive). The input device 208 is, for example, a keyboard, mouse, or touch panel. The display device 209 is, for example, an LCD or OLED. Images captured by the camera 10 are input to the computer 20 via the input / output interface 207 and stored, for example, in the auxiliary storage device 206.

[0090] Computer 20 includes a program for creating a first image based on two or more polarization images, and further includes a program for generating a three-dimensional (3D) model based on multiple first images. CPU 200 generates the first image by executing the program, and is thus able to generate the 3D model. Computer 20 is an example of the information processing apparatus of the present invention.

[0091] [Information Processing Methods]

[0092] Next, the information processing method using camera 10, computer 20 and camera stand 40 will be explained. Figure 5 It is a flowchart representing an information processing method. Figure 6 and Figure 7 This is a diagram used to illustrate polarized image photography.

[0093] like Figure 5 As shown, the user prepares a camera 10 equipped with a polarizing filter 50 capable of polarized image photography, a computer 20 capable of acquiring a first image and creating a 3D model, and an object 30. The user places the object 30 on the photography stage 40 (step S1).

[0094] The user moves the camera 10 to a shooting position to begin photography (step S2). The user fixes the object 30 on the photography platform 40 and moves the camera 10 around the object 30, taking polarized image photographs in multiple shooting directions to acquire polarized images. Therefore, the user moves to any reference shooting position as the shooting position to begin photography. At the reference shooting position, the shooting direction is determined as the reference for the relative position between the camera 10 and the object 30.

[0095] The user takes a polarized image of the object 30 from the camera position using the camera 10 in a certain shooting direction (step S3).

[0096] like Figure 6 As shown in Figure 6-1, the user moves the camera 10 to the shooting position (here, the reference shooting position) and determines the settings (shooting position and camera focal length, etc.) to include the entire object 30 within the shooting range in a certain shooting direction. When starting polarized image photography, the shooting direction θ, representing the relative positional relationship between the camera 10 and the object 30, is determined. In Figure 6-1, the starting position for polarized image photography is [position name], therefore the shooting direction θ = 0 degrees is determined.

[0097] Next, the user switches camera 10 to polarized image photography mode. For example... Figure 6As shown in Figure 6-2, the system control unit 146 controls the filter drive unit 160, which moves the polarizing filter 50 to the polarization direction φ (=0 degrees), which becomes the reference angle.

[0098] Next, the user points the camera 10 at the object 30 and presses the shutter button. For example... Figure 6 As shown in Figure 6-2, the camera 10 is controlled by the system control unit 146. It captures multiple polarized images while changing the polarization direction of the polarization filter 50 using the continuous shooting function. Thus, it is possible to acquire multiple polarized images of the same object 30 from the same shooting position (θ=0 degrees) and the same viewing angle, differing only in polarization direction.

[0099] like Figure 6 As shown in Figure 6-3, each polarization image can establish a correlation between the photographic direction θ and the polarization direction φ, and is represented by Img(θ1,φ1), Img(θ1,φ2), Img(θ1,φ3), Img(θ1,φ4)...Img(θn,φm). As previously described, θ is the photographic direction, and φ is the polarization direction. m is the parameter for identifying the photographic direction, and n is the parameter used to identify the polarization direction.

[0100] Each time a polarization image is captured, the polarization image is stored in memory 136 as Img(θm,φn). Camera 10 is capable of acquiring multiple polarization images.

[0101] Specifically, the polarization image can be stored in memory 136 in the form of Img(0,0), Img(0,30), Img(0,90)...Img(0,175). Furthermore, the position information of camera 10, such as (X,Y,Z,Pitch,Yaw,Roll)=(1,1,1.2,3,3,5), can be associated with and stored as supplementary information in relation to the polarization image. Additionally, the output value of the shake detection unit 156 can be associated with and stored.

[0102] The polarizing filter 50 can move smoothly (continuously). Furthermore, the polarizing filter 50 can perform discrete movements, such as repeatedly rotating by a set angle, stopping, and then rotating further. While the polarizing filter 50 rotates, the camera 10 captures polarized images according to each set polarization direction.

[0103] Camera 10, for example, can capture 12 polarization images (180 degrees each) by changing the polarization direction φ by 15 degrees each time, under the control of system control unit 146. When polarization image capture ends, system control unit 146 can rotate polarization filter 50 to a reference angle to perform the next polarization image capture. The captured polarization images can contain data on the shooting direction θ and polarization direction φ in the tag area of ​​the same file, such as EXIF. The polarization images and the data on the shooting direction θ and polarization direction φ can be in different files.

[0104] Furthermore, if the user half-presses the shutter button, the camera 10 analyzes the live view image and determines the appropriate setting for the polarizing filter 50 to acquire a polarized image. This determination allows the camera to determine the conditions for polarized image photography. For example, if the object 30 is determined to have a large mirror-like area, the camera 10 can subdivide the polarization direction (in 5-degree increments). Therefore, the shooting direction for polarized image photography can differ for each shooting direction. For instance, the polarization direction of the polarizing filter 50 can be changed in 15-degree increments from one shooting direction and in 5-degree increments from another. This structure reduces the user's workload and allows for an appropriate number of polarized images to be acquired.

[0105] Furthermore, lighting conditions (e.g., position, direction, intensity, etc. of the lighting) can be input into the camera 10 or computer 20 in advance. The camera 10 or computer 20 can use this information, along with information about the current position of the camera 10, to determine the conditions for polarized image photography, such as the polarization direction of the polarization filter 50.

[0106] Users can manually or automatically set the reflection suppression rate (low, medium, high) as a mode using the camera 10. Based on this setting, the number of polarized images can be increased or decreased in a certain shooting direction θi.

[0107] Next, after step S3, a determination is made as to whether the shooting of object 30 has been completed (step S4). If the result of this determination is that the shooting has not been completed (step S4: No), the camera 10 is moved to the next shooting position (step S5). Then, returning to step S3, the user takes a polarized image of object 30 through the camera 10 in the next shooting direction from the next shooting position (step S3).

[0108] like Figure 7 As shown in Figure 7-1, the user moves the camera 10 to the next shooting position and sets the object 30 to be included in the shooting range in the next shooting direction (θ=15 degrees).

[0109] Next, the user points the camera 10 at the object 30 and presses the shutter button. For example... Figure 7As shown in Figure 7-2, the camera 10 is controlled by the system control unit 146, and captures multiple polarized images while changing the polarization direction of the polarization filter 50 through continuous shooting. Therefore, multiple polarized images of the object 30 can be acquired from the same shooting position (θ=15 degrees). Specifically, as... Figure 7 As shown in Figure 7-3, polarized images obtained by taking polarized image photographs in the photographic direction θ (=15 degrees) are stored in memory 136 in the format of Img(15,0), Img(15,30), Img(15,90)...Img(15,175). Figure 6 Similarly, the accompanying information can be associated with and stored in the polarization image. Furthermore, the output value of the jitter detection unit 156 can be associated with and stored.

[0110] Furthermore, the camera 10 or computer 20 can analyze information such as the polarization direction φi and polarized image of another shooting direction θi that has already completed polarized image photography, and determine the polarization direction of a certain shooting direction θk. When the shooting direction θi is close to the shooting direction θk, the polarization direction φi is applied to the polarization direction of the shooting direction θk as a condition for polarized image photography. Polarized image photography can be performed in the shooting direction θk without rotating the polarizing filter 50. This reduces the user's workload and allows for an appropriate number of shots.

[0111] As the user moves around the object 30 (e.g., θ = 360 degrees), steps S3 to S5 are repeated until polarized image capture from each shooting direction is completed. While the user moves around the object 30, the current position can be determined based on the accelerometer and / or gyroscope sensor of the camera 10's shake detection unit 156, and the shooting direction can be provided to the user as auxiliary information.

[0112] Furthermore, while the above method describes acquiring a polarization image determined by Img(θ1,φ1) from a still image, it is also possible to acquire one from a moving image. Specifically, the camera 10 captures a moving image of the object 30 while rotating the polarization filter 50 from the reference photographing direction. Multiple time-frame images contained in the captured moving image can be acquired, and these time-frame images can be used as polarization images. When acquiring the time-frame images, the photographing direction θ and the polarization direction φ can be associated and stored as supplementary information.

[0113] In step S4, the system control unit 146 can determine whether the photography action in the set photography direction has been completed. The system control unit 146 can display the completed image on the display unit 142 so that the user can easily make a judgment.

[0114] On the other hand, after the object 30 is photographed (step S4: yes), the camera 10 ends the polarization image photography.

[0115] Next, the computer 20 acquires multiple polarization images in the photographic direction of the object 30 captured by the camera 10 (step S6). The computer 20 acquires multiple polarization images from the memory 136 of the camera 10 via the input / output interface 207 in a wired or wireless manner, or acquires multiple polarization images from a recording medium such as a memory card.

[0116] All polarized images, for example, those with photographic direction θm and polarization direction φn, are associated with each other as shown in Table 1 below and stored in the auxiliary storage device 206 of computer 20. Table 1 includes polarized images taken in polarization directions φ1 to φn within the photographic directions θ1 to θm. The conditions (number of images or polarization direction, etc.) of the included polarized images in the photographic directions θ1 to θm do not need to be completely consistent.

[0117]

[0118] Next, the computer 20 acquires a first image from two or more polarization images in the photographic direction (step S7). That is, the first image is acquired from two or more polarization images that have the same photographic direction θ but differ only in polarization direction φ. Specifically, the first image is acquired by the CPU 200 of the computer 20 performing darkening compositing on the two or more polarization images. Furthermore, the darkening compositing technique itself is a known technique. An outline of this process is as follows.

[0119] Figure 8 The function blocks associated with the generation of the first image from CPU 200 are shown. For example... Figure 8 As shown, the image acquisition unit 210 acquires all polarization images in the photographic direction (1: Img(θ1,φ1), Img(θ1,φ2)...Img(θ1,φn), 2: Img(θ2,φ1), Img(θ2,φ2)...Img(θ2,φn), ..., m: Img(θm,φ1), Img(θm,φ2)...Img(θm,φn)) from the auxiliary storage device 206.

[0120] The alignment unit 211 aligns two or more polarization images used to acquire the first image. For example, in the event of hand shaking, multiple polarization images can be aligned in the same coordinate system through geometric transformations such as affine transformations. Regarding alignment in the same coordinate system, any one of the two or more polarization images can be used as a reference polarization image, and the other polarization images can be aligned to the reference polarization image, or the two or more polarization images can be aligned to another coordinate system. Whether or not hand shaking is present can be determined, for example, based on the output value of the shaking detection unit 156 as one of the accompanying information. However, it is not limited to the output value of the shaking detection unit 156, and hand shaking can be detected by other methods. By accurately aligning the positions of multiple polarization images, unnatural edges or shaking during synthesis can be prevented.

[0121] The image analysis unit 212 of the CPU 200 selects the pixel value with the lowest brightness for pixels at the same coordinate from polarized images (e.g., Img(θ1,φ1), Img(θ1,φ2)...Img(θ1,φn)) along the photographic direction. In other words, the image analysis unit 212 performs image analysis on pixels (pixel units) at the same coordinate in each polarized image with the same photographic direction θ but different polarization directions φ.

[0122] Next, the compositing unit 213 of the CPU 200 replaces the pixel value at the corresponding pixel position in two or more polarization images with the smaller pixel value selected for each pixel, and obtains a new first image.

[0123] This is because, if the polarization angle is set inappropriately relative to the reflected light, the influence of the reflected light cannot be eliminated. As a result, the intensity of the light reaching each pixel becomes higher (i.e., brighter and whiter), thus resulting in a large pixel value. On the other hand, this is because, when reflection is suppressed, the opposite side of the reflective object (water surface or glass) is captured, thus (basically) capturing objects and scenery that are darker than the reflected light.

[0124] Even if the object 30 has a surface with high reflectivity, the computer 20 can acquire a first image in which the effects of reflection are suppressed. As will be described later, even for an object 30 with a surface with high reflectivity, a high-quality three-dimensional (3D) model can be generated.

[0125] Furthermore, based on multiple polarization images and polarization directions, the specular reflectance and diffuse reflectance of each position (at this point, each coordinate in the polarization image) of the object 30 can be determined. The specular reflectance and diffuse reflectance can be obtained using the known technique BRDF (Bidirectional Reflectance Distribution Function).

[0126] By calculating the specular reflectance and diffuse reflectance, different gloss, diffusion, and reflected colors can be represented according to the material of the object 30 when generating a 3D model.

[0127] The compositing unit 213 acquires the first image (Img_syn(θ1), Img_syn(θ2), ..., Img(θ1,φn), 2: Img(θ2,φ1), Img(θ2,φ2), ..., m: Img(θm,φ1), Img(θm,φ2), ..., Img(θm,φn)) from the polarization images (1: Img(θ1,φ1), Img(θm,φ2), ..., Img(θm,φn)). The image analysis and compositing methods in the darkening compositing are not particularly limited, and known techniques can be applied.

[0128] If a first image is acquired in the photographic direction, the CPU 200 stores the first image (Img_syn(θ1), Img_syn(θ2)...Img_syn(θm)) in the auxiliary storage device 206. Furthermore, it can associate and store information such as the polarization image's accompanying data, specular reflectance, or diffuse reflectance with the first image.

[0129] In the above explanation, an example of acquiring a first image from two or more polarization images in the photographic directions θ1 to θm was described. However, this is not a limitation; for example, the polarization direction φ can be determined in relation to a certain photographic direction θ. That is, in the case where multiple polarization images are not acquired in a certain photographic direction θ. Therefore, with respect to the photographic direction θi, one polarization image is acquired, and this polarization image is acquired as the first image.

[0130] Next, it is determined whether to generate a 3D model (step S8). In step S8, if it is determined that a 3D model will not be generated (step S8: No), the information processing ends. The stored multiple first images can, for example, be provided to another information system as images for 3D model generation.

[0131] Furthermore, as an example of acquiring the first image, the case of acquiring the first image in the photographic direction after acquiring all polarization images has been explained. However, it is not limited to this; after acquiring polarization images in a certain photographic direction, the first image can be acquired, the camera can move to the next photographic position, and after acquiring polarization images in the next photographic direction, the first image can be acquired. That is, the acquisition of polarization images and the acquisition of the first image can be performed sequentially.

[0132] In step S8, if it is determined that a 3D model needs to be generated (step S8: yes), proceed to step S9.

[0133] Next, computer 20 generates a 3D model based on multiple first images (step S9). Furthermore, the technique of generating a 3D model using photogrammetry is a well-known technique. A summary of this process is as follows.

[0134] Figure 9 The diagram shows the function blocks associated with the 3D model generation of the CPU200. For example... Figure 9 As shown, the image acquisition unit 220 acquires multiple first images by acquiring first images (Img_syn(θ1), Img_syn(θ2)...Img_syn(θm)) in the photographic direction from the auxiliary storage device 206.

[0135] The point cloud data generation unit 221 analyzes multiple first images and generates 3D point cloud data containing feature points. The point cloud data generation unit 221 extracts feature points from each first image. Next, the point cloud data generation unit 221 uses corresponding feature points between different first images as corresponding points and matches these corresponding points with each other. The point cloud data generation unit 221 estimates the camera parameters (e.g., fundamental matrix, basic matrix, and intrinsic parameters) and estimates the shooting position and pose based on the estimated camera parameters. Then, the 3D position of the feature points of the object 30 is calculated. Clustering adjustment is performed as needed. The estimated 3D coordinates of the feature points are combined to generate point cloud data (point cloud).

[0136] The 3D surface model generation unit 222 processes the data to generate a 3D surface model of the subject based on the 3D point cloud data of the object 30 generated by the point cloud data generation unit 221. Specifically, it generates surface patches (mesh) based on the generated 3D point cloud and then generates a 3D surface model. As a result, the surface undulations can be represented with a small number of points.

[0137] The 3D model generation unit 223 generates a textured 3D model (three-dimensional model) by performing texture mapping on the 3D patch model generated by the 3D patch model generation unit 222. The 3D model generation unit 223 gives the object 30 a realistic appearance by mapping textures onto the mesh. By adding specular reflectivity and diffuse reflectivity, the material properties of the object 30 can be represented when generating the 3D model.

[0138] The generated 3D model data is stored in auxiliary storage device 206, etc. Furthermore, the 3D model data is displayed on display device 209 as needed.

[0139] In this embodiment, system 1, in which the object 30 is placed on the camera platform 40 and the camera 10 moves around the object 30, has been described. However, this is not a limitation; 3D models can also be generated in system 2.

[0140] In this implementation, a 3D model is generated by photogrammetry using multiple first images obtained from the object 30 with suppressed reflections, thus enabling the generation of a high-precision 3D model.

[0141] When creating a 3D model, the distance information between the two markers 41 on the camera platform 40 can be used as a reference for the size of the 3D model. Based on the multiple markers 41 on the camera platform 40, the positional relationship between the camera 10 and the object 30 can be determined, and a 3D model can be generated.

[0142] [Preferred Implementation]

[0143] Next, a preferred embodiment will be described. In the preferred embodiment, the computer 20 determines the polarization image that should be excluded when acquiring the first image. Furthermore, the computer 20 determines the first image that should be excluded when generating the 3D model.

[0144] Figure 10 A flowchart is shown to determine which polarized images should be excluded when acquiring the first image. For example... Figure 10 As shown, multiple polarization images are acquired in the photographic direction (step S11). Specifically, the image acquisition unit 210 acquires multiple polarization images.

[0145] Next, the alignment of multiple polarization images is performed in the photographic direction (step S12). Specifically, the alignment unit 211 performs the alignment. Regarding alignment, for example, a reference image is selected from multiple polarization images, and feature points are extracted from the reference image. The position of the corresponding point in the remaining polarization image is tracked, and the corresponding point in the reference image is moved to. Based on the result, the remaining polarization image is subjected to parallel translation, rotation, and magnification / reduction processing through affine transformations, etc. The alignment of multiple polarization images is performed. Even if there is some hand tremor, alignment can be performed on the polarization images.

[0146] Next, inappropriate polarization images are excluded from the objects acquired when the first image is obtained (step S13). Specifically, the alignment unit 211 excludes polarization images from the objects of the first image. In step S12, the alignment unit 211 is able to determine that polarization images that cannot be aligned with the reference image are not suitable for acquiring the first image.

[0147] Furthermore, it can be determined that a polarization image with significant hand tremor is not applicable to the first image. The presence or absence of hand tremor can be determined based on the high-frequency components of the polarization image or the output value of the hand tremor detection unit 156. The alignment unit 211 associates the information indicating that the image is not applicable to the first image with the polarization image and stores it in the auxiliary storage device 206.

[0148] When excluding a polarization image from an object, you can exclude the polarization image itself or only exclude a specific region within the polarization image.

[0149] Next, image analysis is performed on multiple polarization images with the same shooting direction for each unit pixel (step S14). Finally, the first image is obtained from each polarization image (step S15). In addition, the polarization images excluded in step S13 are not included in the polarization images.

[0150] By excluding polarization images that are unsuitable for the first image, a high-quality first image can be obtained. Therefore, a high-quality 3D model can be obtained based on the high-quality first image.

[0151] In addition, Figure 10 In the illustrated process, when a polarized image is determined to be excluded from the object in the first image, the computer 20 can prompt the camera 10 to retake the image. Information such as the position and orientation of the camera 10 that captured the polarized image can be obtained based on accompanying information such as the information accompanying the acquisition of the polarized image. The computer 20 can then notify the user of the position and orientation of the camera 10. The user can easily determine the position of the object 30 that needs to be retaken as a polarized image.

[0152] Figure 11 A flowchart is shown to determine the first image that should be excluded when generating the 3D model. (See attached flowchart.) Figure 11 As shown, the first image is acquired (step S21). Specifically, the image acquisition unit 220 acquires a plurality of first images.

[0153] Next, feature points are extracted from multiple first images (step S22). Feature points are detected from multiple first images as corresponding points (step S23). Specifically, the point cloud data generation unit 221 extracts feature points from multiple first images, and detects consistent feature points as corresponding points by comparing the feature points of each feature point in the multiple first images.

[0154] Next, the first image is excluded from the objects of the 3D model (step S24). Specifically, the point cloud data generation unit 221 can determine that the first image from which feature points cannot be extracted in step S22 or from which corresponding points cannot be detected in step S23 is not suitable for the 3D model. Furthermore, the point cloud data generation unit 221 can determine the images excluded from the objects of the 3D model based on high-frequency components, etc., of the first image. The point cloud data generation unit 221 associates the information indicating that the image is not suitable for the 3D model with the first image and stores it in the auxiliary storage device 206.

[0155] Next, a 3D model is generated based on the first image (step S25). Specifically, the point cloud data generation unit 221 combines the camera parameters, shooting position and pose of the camera, and the estimated 3D coordinates of the feature points to generate point cloud data based on the first image, while the 3D patch model generation unit 222 and the 3D model generation unit 223 generate the 3D model. However, the first image does not include images excluded from the objects in the 3D model. Since the first image, which is unsuitable for generating the 3D model, is excluded, a high-quality 3D model can be generated.

[0156] In addition, Figure 11 In the illustrated process, when a first image is determined to be excluded from the 3D model object, the computer 20 can also prompt the camera 10 to retake the image. Information such as the position and orientation of the camera 10 that captured the polarized image suitable for the first image can be obtained based on accompanying information when the first image was acquired. The computer 20 can then notify the user of the position and orientation of the camera 10. The user can easily determine the position of the object 30 for which the polarized image needed to be retaken for generating the first image.

[0157] <Conditions for Polarized Image Photography>

[0158] The determination of the conditions for polarized image photography is explained. The conditions include pre-photographing the object 30 before acquiring the polarized image.

[0159] Next, the situation of pre-photographing the object 30 by camera 10 will be explained. Figure 12 This is a flowchart illustrating a method for generating a 3D model, including pre-photographed images.

[0160] Prepare a camera 10 capable of polarized image photography, a computer 20 capable of acquiring the first image and creating a 3D model, and an object 30. For example... Figure 12 As shown, the user places the object 30 on the camera platform 40 (step S41).

[0161] The user moves the camera 10 to a designated shooting position to begin pre-shooting. The object 30 is fixed on the shooting platform 40. While moving the camera 10 around the object 30, the user takes pre-shooting images of the object 30 from various shooting directions (step S42). Pre-shooting refers to taking pictures of the object 30 before performing polarized image shooting (step S45). The pre-shooting can be a still image of the object 30 or a moving image. It can also be a live view image. The distance between the object 30 and the camera 10 can also be obtained.

[0162] Next, the computer 20 acquires the pre-image obtained by the camera 10 and acquires information for determining the conditions for polarized image photography (step S43). The information that can be acquired is simply the approximate size of the object 30, the surface characteristics of the object 30 (specular, rough, or reflective), the polarization direction of the polarization filter 50, and the lighting environment. There are no particular limitations on the information that can be used in the polarized image photography in step S45.

[0163] The user moves the camera 10 to a shooting position to begin photography (step S44). At the shooting position, the user takes a polarized image photograph of the object 30 using the camera 10 (step S45). The polarized image photograph is taken based on the information obtained in the pre-photographing process in step S43.

[0164] The conditions for polarized image photography are set by the system control unit 146 (CPU) of camera 10 or the CPU 200 of computer 20.

[0165] Figure 13 This diagram illustrates an example of the conditions set for polarized image photography. Figure 13 The result of a pre-photograph of object 30 from a certain direction is shown. Figure 13 In section 13-1, the pixel value of position P1 of object 30 is of interest. On the other hand, in... Figure 13 In section 13-2, we focus on the pixel value of position P2 of object 30. Positions P1 and P2 are at the same location on the X-axis, while on the Y-axis, position P1 is located higher than position P2.

[0166] While continuously changing the polarization direction of the polarization filter 50, the camera 10 captures images of positions P1 and P2 through the polarization filter 50. Figure 13-1 and Figure 13-2 The graph shown is a graph with the pixel value intensity on the vertical axis and the polarization direction φ on the horizontal axis. Figure 13-1 The curve is a graph created by decomposing the pixel value in each polarization direction at position P1 into RGB values ​​and then plotting them. Figure 13-2 The curve is a graph created by decomposing the pixel value in each polarization direction at position P2 into RGB values ​​and then plotting them.

[0167] As can be understood from 13-1 and 13-2, when the polarization direction is 3 to 4 degrees and 24 to 25 degrees, the pixel value is low and the reflection from the object 30 is suppressed.

[0168] As shown in graphs 13-1 and 13-2, at the common position on the X-axis, with the same polarization direction φ, the pixel value is low. That is, it can be understood that there is no need to focus on all pixel values.

[0169] In step S45, described later, the accuracy of the polarization image is improved by utilizing information obtained from the pre-photograph (e.g., polarization angle), which enables the suppression of reflectivity.

[0170] Furthermore, in the graphs 13-1 and 13-2, pixel values ​​were acquired and plotted in all polarization directions. Therefore, it is anticipated that determining the polarization directions with low pixel values ​​would require a large amount of data and be time-consuming. Therefore, during pre-photographing, the polarizing filter 50 was rotated intermittently (e.g., at 5-degree or 10-degree intervals), and pixel values ​​were acquired. The acquired pixel values ​​were then plotted on the graphs.

[0171] The graphs contain missing portions, but these graphs also exhibit a certain pattern. Therefore, by interpolating the missing portions in the graphs (e.g., spline interpolation), the polarization direction of the lower pixel values ​​can be estimated. The estimated polarization direction can then be applied to polarized image photography.

[0172] Return to Figure 12 After step S45, a determination is made as to whether the photographing of object 30 has been completed (step S46). If the result of this determination is that the photographing has not been completed (step S46: No), the camera 10 is moved to the next photographing position (step S47). At the next photographing position, the user takes a polarized image photograph of object 30 using the camera 10 in the next photographing direction (step S45). The process from step S45 to step S47 is repeated until the photographing of object 30 is completed.

[0173] Upon completion of the photographing of object 30 (step S46: Yes), camera 10 terminates polarization image photography.

[0174] Next, the computer 20 acquires multiple polarization images in the photographic direction of the object 30 acquired by the camera 10 (step S48).

[0175] Next, the computer 20 acquires a first image from two or more polarized images in the photographic direction (step S49).

[0176] Next, it is determined whether to generate a 3D model (step S50). If it is determined in step S50 that a 3D model will not be generated (step S50: No), the information processing ends.

[0177] If it is determined in step S50 that a 3D model needs to be generated (step S50: yes), proceed to step S51.

[0178] Next, the computer 20 generates a 3D model based on the multiple first images (step S51). If a 3D model is generated based on the first images, the information processing ends.

[0179] In the above embodiments, systems 1 and 2 constituting an image processing apparatus including a camera 10 and a computer 20 have been described. The camera 10 includes a system control unit 146, and the computer 20 includes a CPU. The functions of the system control unit 146 and the CPU 200 are not particularly limited. For example, the determination of the conditions for polarized image photography can be performed by either the CPU 200 or the system control unit 146. That is, as long as the camera 10 and the computer 20 can generate the first image, either the CPU 200 or the system control unit 146 can perform its function.

[0180] Furthermore, the present invention includes an information processing system comprising a computer 20 for generating a first image. It also includes an optical apparatus for processing up to and including polarization image photography performed solely by the camera 10, acquiring the first image from the polarization image, and generating a 3D model from the first image. An example of polarization image photography performed by a single camera 10 is shown, but polarization image photography can also be performed by multiple cameras 10.

[0181] [Structure of System Control Unit 146 and CPU 200]

[0182] The functions of the system control unit 146 and CPU 200 are implemented by various processors. These processors include general-purpose processors (CPUs and / or GPUs, Graphics Processing Units) that execute programs and function as various processing units; programmable logic devices (PLDs) whose circuit structures can be modified after manufacturing; and application-specific integrated circuits (ASICs) with circuit structures specifically designed to perform specific processes. The terms "program" and "software" have the same meaning.

[0183] A processing unit can be composed of one of these various processors, or it can be composed of two or more processors of the same or different types. For example, a processing unit can be composed of multiple FPGAs or a combination of a CPU and an FPGA. Furthermore, multiple processing units can also be composed of a single processor. As examples of multiple processing units composed of a single processor, firstly, in computers used for clients and servers, a single processor is composed of a combination of one or more CPUs and software, and this processor functions as multiple processing units. Secondly, in systems-on-chips (SoCs), a processor that implements the overall system functionality including multiple processing units is used, represented by a single IC (Integrated Circuit) chip. In this way, various processing units are constructed as hardware structures using one or more of the aforementioned processors.

[0184] The present invention has been described above, but the present invention is not limited to the examples above. Of course, various improvements or modifications can be made without departing from the spirit of the present invention.

[0185] Symbol Explanation

[0186] 1-System, 2-System, 10-Camera, 20-Computer, 30-Object, 40-Camera stand, 41-Marker, 42-Camera stand, 43-Marker, 44-Tripod, 50-Polarizing filter, 100-Lens assembly, 102-Camera optical system, 104-Lens group, 106-Aperture, 110-Lens drive unit, 112-Aperture drive unit, 130-Imaging element, 132-Shutter, 134-Shutter drive unit, 136-Memory, 138-Digital signal processing unit, 140-Input / output interface, 142-Display unit, 144-Operation unit, 146-System control unit, 150- Correction mechanism, 152-Control unit, 154-Imaging element driving unit, 156-Detection unit, 158-Position detection unit, 160-Filter driving unit, 162-Polarization direction detection unit, 200-CPU, 202-RAM, 204-ROM, 206-Auxiliary storage device, 207-Input / output interface, 208-Input device, 209-Display device, 210-Image acquisition unit, 211-Alignment unit, 212-Image analysis unit, 213-Synthesis unit, 220-Image acquisition unit, 221-Point cloud data generation unit, 222-3D patch model generation unit, 223-3D model generation unit.

Claims

1. An information processing system having one or more processors, wherein, The one or more processors perform the following processing: To acquire multiple polarized images obtained by shooting an object from multiple photographic directions with different polarization directions; and The first image is obtained from two or more polarization images.

2. The information processing system according to claim 1, wherein, The one or more processors perform the following processing: Acquire additional information including information about the polarization direction of the polarization image.

3. The information processing system according to claim 1 or 2, wherein, The one or more processors perform the following processing: The first image is generated by analyzing two or more images with different polarization directions.

4. The information processing system according to claim 3, wherein, The analysis is performed at the pixel level.

5. The information processing system according to claim 4, wherein, The analysis of the corresponding pixel units at the same position of two or more polarized images with different polarization directions is called darkening synthesis.

6. The information processing system according to claim 1 or 2, wherein, The one or more processors perform the following processing: When generating the first image, based on the information between two or more images with different polarization directions, at least one of the two or more images with different polarization directions is transformed into the coordinate system of at least one other image through geometric transformation.

7. The information processing system according to claim 1 or 2, wherein, The one or more processors perform the following processing: For the multiple polarization images, determine whether they are suitable for generating the first image.

8. The information processing system according to claim 2, wherein, The accompanying information includes the location information of the camera device when the shooting is performed.

9. The information processing system according to claim 2 or 8, wherein, The accompanying information is associated with the first image.

10. The information processing system according to claim 1 or 2, wherein, The one or more processors perform the following processing: A 3D model of the object is generated based on the first image generated in the plurality of photographic directions.

11. The information processing system according to claim 1 or 2, wherein, The one or more processors perform the following processing: Analyze two or more images with different polarization directions acquired in one of the multiple photographic directions to determine the polarization direction in the photographic direction that is different from the one of the multiple photographic directions.

12. The information processing system according to claim 1, wherein, The one or more processors perform the following processing: Based on the environmental conditions and the photographic position of the object being photographed, determine the conditions for photographing two or more images with different polarization directions.

13. The information processing system according to claim 1 or 2, wherein, The one or more processors perform the following processing: Accept the photography mode, and determine the conditions for capturing two or more images with different polarization directions based on the photography mode.

14. The information processing system according to claim 1 or 2, wherein, The one or more processors perform the following processing: Based on the information of the photographing direction and the polarization direction, calculate the specular reflectance or diffuse reflectance at the coordinate positions in two or more images with different polarization directions.

15. The information processing system according to claim 1 or 2, wherein, The one or more processors perform the following processing: Before acquiring the plurality of polarization images, the conditions for acquiring the plurality of polarization images are determined based on the pre-image obtained by capturing the object.

16. The information processing system according to claim 1 or 2, wherein, The two or more polarization images are time frame images contained in the dynamic image.

17. An image processing apparatus comprising an optical device having a polarizing filter capable of capturing an image of an object and an information processing device, wherein, The image processing apparatus includes one or more processors, one of which is included in either the optical apparatus or the information processing apparatus. To acquire multiple polarized images of an object captured from multiple photographic directions with different polarization directions. The first image is obtained from two or more polarization images.

18. The image processing apparatus according to claim 17, wherein, The one or more processors perform the following processing: Acquire additional information including information about the polarization direction of the polarization image.

19. The image processing apparatus according to claim 17 or 18, wherein, The one or more processors perform the following processing: A 3D model of the object is generated based on the first image generated in the plurality of photographic directions.

20. The image processing apparatus according to claim 17 or 18, wherein, The one or more processors perform the following processing: Before acquiring the plurality of polarization images, the conditions for acquiring the plurality of polarization images are determined based on the pre-image obtained by capturing the object.

21. The image processing apparatus according to claim 17 or 18, wherein, The one or more processors control the movement of the optical device or the object.

22. The image processing apparatus according to claim 17 or 18, wherein, The one or more processors control the change of the polarization direction of the polarization filter.

23. An optical device comprising one or more processors and polarizing filters, and capable of capturing images of an object, wherein, The one or more processors perform the following processing: To acquire multiple polarized images obtained by shooting an object from multiple photographic directions with different polarization directions; and The first image is obtained from two or more polarization images.

24. The optical device according to claim 23, wherein, The one or more processors perform the following processing: Acquire additional information including information about the polarization direction of the polarization image.

25. The optical device according to claim 23 or 24, wherein, The one or more processors perform the following processing: Before acquiring the plurality of polarization images, the conditions for acquiring the plurality of polarization images are determined based on the pre-image obtained by capturing the object.

26. The optical device according to claim 23 or 24, wherein, The one or more processors perform the following processing: A 3D model of the object is generated based on the first image generated in the plurality of photographic directions.

27. An information processing method, executed by an information processing system having one or more processors, wherein, The one or more processors perform the following processing: To acquire multiple polarized images obtained by shooting an object from multiple photographic directions with different polarization directions; and The first image is obtained from two or more polarization images.

28. A program that causes an information processing system having one or more processors to execute an information processing method, wherein, The program causes the one or more processors to perform the following processing: To acquire multiple polarized images obtained by shooting an object from multiple photographic directions with different polarization directions; and The first image is obtained from two or more polarization images.

29. A recording medium that is non-transitory and computer-readable, wherein, The recording medium records the program as described in claim 28.

Images