Three-dimensional model generation system and method for operating three-dimensional model generation system

Abstract

An aspect of the present invention provides a three-dimensional model generation system and a method for operating the three-dimensional model generation system. A three-dimensional model generation system according to one aspect of the present invention comprises: an illumination device having a first polarization element; an imaging device having a second polarization element; a control device that controls rotation of at least one of the first polarization element and the second polarization element; and a three-dimensional model generation device that generates a three-dimensional model of an object from an image captured by the imaging device. The control device comprises a determination unit that determines, in first imaging for capturing an image to be used for generating the three-dimensional model, at least one of a first setting condition that is a setting condition for the first polarization element and a second setting condition that is a setting condition for the second polarization element.

Description

3D model generation system and operation method of the 3D model generation system

[0001] The present invention relates to a three-dimensional model generation system and a method for operating the three-dimensional model generation system, and more particularly to a technique for generating a three-dimensional model from an image captured using a polarizing element.

[0002] Regarding the generation of three-dimensional models, for example, Patent Document 1 describes a technology for converting a real-world subject into subject information (three-dimensional model) that can be handled using CG technology, etc. Furthermore, for example, Patent Document 2 discloses a camera that includes an optical lens, a rotatable polarizing filter, and control means that detects the rotation angle of the polarizing filter, captures multiple images, and records each image by associating it with rotation angle information.

[0003] JP 2022-60641 A JP 2014-182328 A

[0004] One embodiment of the disclosed technique provides a three-dimensional model generation system and a method of operating the three-dimensional model generation system.

[0005] A three-dimensional model generation system according to a first aspect of the present invention is a three-dimensional model generation system comprising: an illumination device having a first polarizing element; an imaging device having a second polarizing element; a control device that controls the rotation of at least one of the first polarizing element and the second polarizing element; and a three-dimensional model generation device that generates a three-dimensional model of an object from an image captured by the imaging device, wherein the control device comprises a determination unit that determines at least one of a first setting condition that is a setting condition of the first polarizing element and a second setting condition that is a setting condition of the second polarizing element in a first imaging step that captures an image to be used to generate the three-dimensional model.

[0006] In the 3D model generation system according to the second aspect of the present invention, in the first aspect, the determination unit determines the first setting conditions and the second setting conditions using images acquired in a second photograph taken prior to the first photograph in the photographing direction in which the first photograph is taken. The "first photograph" refers to photographing for obtaining images for generating a 3D model (main photographing), and the "second photograph" refers to photographing for obtaining images used in determining the setting conditions (preliminary photographing). Note that in the present invention, the terms "first," "second," etc. do not specify the chronological order of processing (the same applies to each of the following aspects).

[0007] In the third aspect of the three-dimensional model generation system, in the second aspect, the photographing device photographs the object at multiple rotation angles for at least one of the first polarizing element and the second polarizing element in one photographing direction in the first photographing, and the determination unit determines the first setting condition and the second setting condition based on the multiple images acquired in the second photographing.

[0008] In the three-dimensional model generation system according to the fourth aspect, in the second or third aspect, the determination unit determines the first setting condition and the second setting condition by performing a second photograph each time the amount of change in the photographing direction in which the first photograph is taken exceeds a predetermined angle.

[0009] In the three-dimensional model generation system according to the fifth aspect, in any one of the second to fourth aspects, if the image captured in the first photographing has changed beyond a predetermined standard after the first setting condition and the second setting condition have been determined, the determination unit performs a second photographing to determine the first setting condition and the second setting condition.

[0010] The three-dimensional model generation system according to the sixth aspect is any one of the second to fifth aspects, in which the determination unit determines the first setting condition and the second setting condition based on the brightness values ​​of pixels in one or more specified areas in the image acquired in the second photograph.

[0011] A three-dimensional model generation system according to a seventh aspect is the sixth aspect, wherein the determination unit specifies the entire area of ​​the object as the one or more areas.

[0012] The three-dimensional model generation system according to an eighth aspect is the sixth or seventh aspect, wherein the determination unit specifies one or more regions based on an external input.

[0013] A three-dimensional model generation system according to a ninth aspect is the eighth aspect, wherein the determination unit specifies one or more regions in response to a user operation.

[0014] The three-dimensional model generation system according to the tenth aspect is any one of the sixth to ninth aspects, in which the determination unit determines the first setting condition and the second setting condition based on the luminance values ​​of pixels whose luminance values ​​are less than a predetermined value in one or more regions.

[0015] The three-dimensional model generation system according to the eleventh aspect is the tenth aspect, in which the determination unit determines, as the first setting condition and the second setting condition, the rotation angle of the first polarizing element and the rotation angle of the second polarizing element that result in the smallest sum, average, or maximum value of the brightness values ​​of pixels in one or more regions.

[0016] A three-dimensional model generation system according to a twelfth aspect is any one of the first to eleventh aspects, wherein the first setting condition includes a first minimum rotation angle, which is the rotation angle of the first polarizing element that minimizes the effect of specular reflection by the object in the image acquired in the first shooting, and the second setting condition includes a second minimum rotation angle, which is the rotation angle of the second polarizing element that minimizes the effect of specular reflection by the object in the image acquired in the first shooting.

[0017] A three-dimensional model generation system according to a thirteenth aspect is the twelfth aspect, wherein the first setting condition further includes a first maximum rotation angle, which is the rotation angle of the first polarizing element that maximizes the influence of specular reflection by the object in the image acquired in the first shooting, and the second setting condition further includes a second maximum rotation angle, which is the rotation angle of the second polarizing element that maximizes the influence of specular reflection by the object in the image acquired in the first shooting.

[0018] The three-dimensional model generation system according to the fourteenth aspect is the twelfth or thirteenth aspect, in which the photographing device photographs the object at a combination of multiple rotation angles including a combination of a first minimum rotation angle and a second minimum rotation angle for one photographing direction in the first photographing operation to obtain multiple images, and the three-dimensional model generation device generates a three-dimensional model using the obtained multiple images.

[0019] A fifteenth aspect of the present invention relates to a method for operating a three-dimensional model generation system including an illumination device having a first polarizing element, an image capture device having a second polarizing element, a control device that controls the rotation of at least one of the first polarizing element and the second polarizing element, and a three-dimensional model generation device that generates a three-dimensional model of an object from an image captured by the image capture device, wherein the control device determines at least one of a first setting condition that is a setting condition for the first polarizing element and a second setting condition that is a setting condition for the second polarizing element in a first image capture that captures an image to be used in generating the three-dimensional model. A program that causes the three-dimensional model generation system to execute the method according to the fifteenth aspect, and a non-transitory tangible computer-readable medium having such a program recorded thereon, can also be cited as aspects of the present invention.

[0020] FIG. 1 is a diagram showing the configuration of a 3D model generation system according to a first embodiment. FIG. 2 is a diagram showing a dolly for changing the shooting direction, etc. FIG. 3 is a diagram showing an example of a 3D model generation system equipped with multiple cameras. FIG. 4 is a diagram showing the configuration of a controller. FIG. 5 is a diagram showing the functional configuration of a processor. FIG. 6 is a diagram showing an example of information recorded in a recording device. FIG. 7 is a diagram showing the configuration of a camera. FIG. 8 is a diagram showing the configuration of a lighting device. FIG. 9 is a flowchart showing an outline of processing in the 3D model generation system. FIG. 10 is a diagram showing how preliminary shooting is performed while changing the rotation angle of a polarizing filter. FIG. 11 is a diagram showing how a brightness measurement area is set in an image of an object. FIG. 12 is a conceptual diagram showing the relationship between the rotation angle of a polarizing filter and the brightness value of a brightness measurement area. FIG. 13 is a diagram showing how the rotation angle of a polarizing filter for main shooting is determined from the brightness value in a preliminary shot image. FIG. 14 is a diagram showing an example of a generation pattern of a reflection information image. FIG. 15 is a diagram showing how image synthesis is performed. FIG. 16 is a diagram showing how a 3D model is generated. FIG. 17 is a flowchart showing a modified example of processing in the 3D model generation system.

[0021] [Generating 3D Models Using Images with Suppressed Reflections] Typical methods for generating 3D models (hereinafter referred to as "three-dimensional models") include the ToF (Time of Flight) method, structured light method, and photogrammetry method. Among these three methods, photogrammetry generates 3D models based on photographed images, and it is known that high-quality 3D models can be generated if the resolution and image quality of the photographs are high. However, when generating 3D models using photogrammetry, accuracy is known to deteriorate if the images used to create the model contain reflective areas (areas with a high specular reflection component within the object area in the image). To improve the quality of the 3D model, it is necessary to obtain images used to generate the 3D model while minimizing the effects of reflections.

[0022] In light of these circumstances, the inventors of the present application conducted extensive research and discovered that "images with reduced reflections can be obtained using an apparatus equipped with polarizing elements in each of multiple cameras and lighting devices, and a three-dimensional model can be generated using the obtained images. By determining and following the optimal setting conditions for the polarizing elements when taking images, it becomes possible to obtain images with reduced reflections, and a high-quality three-dimensional model can be generated from these images." The present invention was created based on this discovery, and specific aspects of the three-dimensional model generation system and operating method of the three-dimensional model generation system according to the present invention will be described below.

[0023] [First Embodiment] [Overall Configuration of 3D Model Generation System] Fig. 1 is a diagram showing the configuration of a 3D model generation system 10 (3D model generation system) according to the first embodiment. The 3D model generation system 10 includes a camera 100 (imaging device), an illumination device 200 (illumination device), and a controller 300 (control device, 3D model generation device). An object 99 to be photographed is placed on an imaging stand 400.

[0024] The camera 100 is configured to be able to move around the object 99 and to capture images of the object 99 from multiple directions as it moves around the object 99. The camera 100 includes a rotatable polarizing filter 170.

[0025] When capturing an image of the entire object 99, the camera 100 moves around the object 99 (for example, 360° or more). When capturing an image of only a portion of the object 99, the camera 100 moves within a specific range of the object 99 (a range smaller than 360°). The camera 100 can be moved around the object 99 by a user holding the camera 100 or by a mobile object supporting the camera 100. The mobile object is, for example, an arm or a crane attached to the imaging stand 400. This arm is configured to be able to move around the imaging stand 400 by a motor or the like. Furthermore, when the object 99 is large, the mobile object may be a dolly, vehicle, or drone.

[0026] In the first embodiment, the processor 310 of the controller 300 (such as the imaging table control unit 315; see Figures 4 and 5) may control the arm, the imaging table 400, etc. to change the imaging direction, or the user may move the arm, the imaging table 400, etc. to change the imaging direction.

[0027] [Movement of Camera and Lighting Device Using a Dolly] FIG. 2 is a diagram showing the configuration of a dolly 500 for moving the camera 100. The dolly 500 can travel using wheels and can rotate the camera 100 around two axes. Furthermore, the arm can be extended or retracted. This dolly 500 can also be used to change the position and shooting direction of the camera 100. Note that the processor 310 (see FIGS. 4 and 5 ) may control the travel and direction of the dolly 500 and the extension and retraction of the arm, or these may be controlled by the user. Furthermore, this dolly 500 can also be used for the lighting device 200.

[0028] [Multiple Cameras with Different Shooting Directions] In the three-dimensional model generation system 10, instead of moving the camera 100 using an arm, a dolly, or the like, multiple cameras with different shooting directions may be provided. For example, as shown in FIG. 3 , the three-dimensional model generation system may be provided with multiple cameras 100 with different orientations for one shooting direction (θ) (in the example of FIG. 3 , one camera 100 that shoots from a substantially horizontal direction and two cameras 100 that shoot from diagonally above). Similarly, the three-dimensional model generation system may be provided with multiple cameras 100 with different shooting directions (θ). Similarly, the three-dimensional model generation system 10 may be provided with multiple lighting devices 200 with different lighting directions.

[0029] The camera 100 is configured to capture images of the object 99 in a shooting direction at each position of the camera 100 while moving around the object 99. The camera 100 can also capture multiple polarized images of the object 99 at different rotation angles in a single shooting direction through a polarizing filter 170, the rotation angle (polarization direction) of which can be set to any angle. As will be described later, this rotation angle is the rotation angle relative to the reference angle of the polarizing filter 170 when the object 99 is captured by the camera 100. The shooting direction θ is an angle that represents the relative positional relationship between the camera 100 and the object 99 with respect to the reference positions of the camera 100 and the object 99 when the object 99 is captured by the camera 100. The rotation angle of the polarizing filter 170 can be changed manually by the user or automatically in accordance with the user's instructions. A polarized image is an image captured through the polarizing filter 170.

[0030] The camera 100 is configured to store polarized images of the object 99 captured through the polarizing filter 170. While the camera 100 is moving around the object 99, it can acquire multiple polarized images captured (polarized image capture) at different rotation angles in the shooting direction of the object 99, either manually by a user's operation or automatically (under the control of the controller 300).

[0031] The illumination device 200 irradiates illumination light onto the object 99 when photographing the object 99. The illumination device 200 includes a rotatable polarizing filter 202, similar to the camera 100 (the rotation angle of the polarizing filter 202 is hereinafter referred to as ψ). The polarizing filter 202 (polarizing element) is an example of a polarizing element in the present invention, and, similar to the polarizing filter 170 described above, may be a polarizing plate, a wire grid type, a photonic crystal, or a polarizing element based on other principles.

[0032] The controller 300 is configured to execute a program to generate a 3D model of the object 99 by photogrammetry from a plurality of polarized images. Photogrammetry is a technology that analyzes a plurality of images of the object 99 taken from different angles and synthesizes the images to generate (restore) a three-dimensional shape or structure (3D model). As will be described in detail later, in this embodiment, the images used to generate the 3D model are a plurality of composite images.

[0033] The object 99 is not particularly limited in shape or size as long as it is an object for which a 3D model is to be generated. The object 99 may be any object having a physical shape.

[0034] The imaging table 400 has multiple marks 410 on its mounting surface (top surface). The marks 410 serve as indicators for the imaging direction. The distance between two marks 410 (assumed to be known) serves as a reference for the size when creating a 3D model. The imaging table 400 has a cylindrical shape with a flat top surface. The shape of the imaging table 400 is not limited as long as it can accommodate the object 99. The imaging table 400 may be a rectangular parallelepiped. Note that the imaging table 400 is not essential when considering conditions such as the size of the object 99 and the location where the object 99 is located. If the imaging direction can be changed by moving the camera 100 around the object 99, the imaging table 400 may be omitted. The imaging direction may be changed relatively by fixing the position of the camera 100 and rotating the imaging table 400 under the control of the imaging table control unit 315 (controller 300) and a motor (not shown).

[0035] [Configuration of Controller] Fig. 4 is a diagram showing the configuration of the controller 300 (control device, 3D model generating device) in the first embodiment. As shown in Fig. 4, the controller 300 includes a processor 310 (determination unit), a ROM 330 (ROM: Read Only Memory, a non-transitory and tangible recording medium), a RAM 340 (RAM: Random Access Memory), an operation unit 350, a speaker 360 (output device), a display 370 (output device), an input / output interface 380, and a recording device 390, and these components are connected by a bus 395. The controller 300 can communicate with the camera 100, the lighting device 200, and various external devices via the input / output interface 380.

[0036] In this embodiment, the controller 300 may execute necessary processes (such as control of the camera 100, the lighting device 200, and the imaging stand 400, and generation of a 3D model) using a processor, a program, or a combination thereof. The controller 300 may be a system such as a general-purpose computer, a computer for a specific application, or a workstation, or other hardware element capable of executing a program.

[0037] [Processor Configuration] Fig. 5 is a diagram showing the functional configuration of the processor 310. As shown in Fig. 5, the processor 310 includes a lighting device control unit 312, a camera control unit 314, a shooting condition determination unit 316, an image processing unit 318, a three-dimensional model generation unit 320, and an input / output control unit 322.

[0038] The processor 310 may be configured with one or more pieces of hardware, and the type of hardware is not limited. For example, the processor 310 may be configured with hardware such as a programmable logic device such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or an FPGA (Field Programmable Gate Array), a dedicated circuit for executing specific processes such as an ASIC (Application Specific Integrated Circuit), a GPU (Graphic Processing Unit), or an NPU (Neural Processing Unit). The processor 310 also includes various units or means for executing various processes in this embodiment. The hardware may also be a combination of different types of hardware. When multiple pieces of hardware are configured to execute one or more processes of a certain processor, the multiple pieces of hardware may exist in physically separate devices or in the same device. In any of the embodiments, the order of each process performed by the processor is not particularly limited and may be changed as appropriate. The hardware may be configured with an electrical circuit or the like that combines circuit elements such as semiconductor devices.

[0039] Furthermore, in this embodiment, the processor 310 may be implemented by hardware, software, firmware, microcode, or a combination thereof. Software, firmware, and microcode may be configured as a program. A program may also be, for example, a group of program modules, each function of which may be implemented by a processor configured to execute the respective function. The program may be program code or multiple code segments stored in one or more non-transitory, tangible computer-readable media (e.g., storage media or other storage, such as the ROM 330 or the recording device 390 (the same applies hereinafter)). The program may be stored in multiple non-transitory, tangible computer-readable media that are physically separate from each other. The program code or code segment may represent a procedure, a function, a subprogram, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. The program code or code segment may be connected to another code segment or a hardware circuit by sending or receiving information, data, arguments, parameters, or memory contents.

[0040] In this embodiment, the term "non-transitory tangible computer-readable medium" does not include non-tangible recording media such as carrier signals or propagation signals. The processor 310 can use the RAM 340 as a temporary storage area or a working area when processing using a program.

[0041] The functions of the processor 310 described above may be realized by various types of AI (Artificial Intelligence). Such AI may be, for example, an AI that performs image recognition, segmentation, feature extraction, or the like, or an AI that generates an image from given information. These AIs may also be realized by hardware, software, firmware, microcode, or a combination thereof, as described above.

[0042] [Configuration of Operation Unit and Display] The operation unit 350 is composed of devices such as a keyboard, mouse, buttons, and switches (not shown). A user can issue instructions to the controller 300 via these devices, and the processor 310 accepts the instructions and performs processing in accordance with the accepted instructions. The display 370 may be composed of a touch panel device so that the user can issue instructions via the touch panel. The display 370 is composed of such a touch panel device or a device such as a liquid crystal display device, and can display the rotation angle of the polarizing filters 170 and 202, characteristic information of the object, preliminary captured images, actual captured images, generated 3D models, and the like. The display 370 can also display information recorded in the recording device 390.

[0043] [Configuration of Input / Output Interface] The input / output interface 380 is configured by terminals and slots for connecting external devices such as a display, printer, and recording medium, and communication interfaces such as Wi-Fi (registered trademark) and Bluetooth (registered trademark). The controller 300 can acquire data such as image data from external devices (server devices, recording devices, databases, photographing devices, etc.) via the input / output interface 380. The external devices may be connected to the controller 300 by wire or wirelessly. Furthermore, the external devices may be connected via a network such as the Internet.

[0044] [Configuration of the recording device] The recording device 390 (recording device, output device) is composed of recording media (non-transitory, tangible computer-readable media) such as semiconductor memories such as hard disks and SSDs (Solid-State Drives), and various types of magneto-optical recording media, and their control units, and records or saves various types of information.

[0045] FIG. 6 is a diagram showing an example of information recorded in the recording device 390. Data processing conditions 390A include shooting conditions and other data processing conditions. The shooting conditions are shooting conditions for the preliminary shooting and the actual shooting, and may include conditions such as the shooting direction (θ), the polarizing filter rotation angle (φ, ψ) during the preliminary shooting and the actual shooting, and exposure. The shooting conditions may also include lighting conditions provided by the lighting device 200 (e.g., on / off of the illumination light, color, brightness, etc.). Other data processing conditions include, for example, the following. The preliminary image 390B is an image obtained by preliminary shooting (second shooting) performed prior to capturing (first shooting) an image (actually shot image 390C) used to generate a 3D model of the object. The actual shot image 390C is an image used to generate a 3D model of the object. The brightness information 390D is information indicating the brightness values ​​of pixels in one or more areas (brightness measurement areas) specified in the image acquired by the preliminary shooting. The brightness information 390D may include information such as the number and position of brightness measurement areas, brightness thresholds, etc. The three-dimensional model 390E is a three-dimensional model of the object generated using a plurality of images acquired by the actual photography.

[0046] The preliminary captured image 390B and the actual captured image 390C may include additional information (such as the shooting direction, the rotation angle of the polarizing filter, the position information of the camera 100, and the output of the shake detection unit 156), and the additional information may be recorded separately from the image.

[0047] It is preferable that the above-mentioned information be recorded in association with each other. The processor 310 can display the information recorded in the recording device 390 on the display 370 in response to a user instruction via the operation unit 350, or automatically without a user instruction.

[0048] [Camera Configuration] Fig. 7 is a block diagram showing a schematic configuration of the camera 100. As shown in Fig. 7, the camera 100 is equipped with a lens device 102, which includes an imaging optical system 104 including a lens group 106 and an aperture 108, a lens driver 110, an aperture driver 112, a polarizing filter 170, a filter driver 160, a rotation angle detector 162, etc. The lens device 102 may be detachable from the camera 100, or may be integrated with the camera 100.

[0049] The lens group 106 includes at least a focus lens that is movable in the optical axis direction. This focus lens is a lens for focus adjustment. The focus of the imaging optical system 104 is adjusted by moving the focus lens back and forth along the optical axis. The focus lens is driven and operated by a lens driver 110. The focus lens can be moved to a focus position where it focuses on the object 99.

[0050] The diaphragm 108 is configured by, for example, an iris diaphragm. The amount of light passing through the imaging optical system 104 is adjusted by the diaphragm 108. The diaphragm 108 is driven by a diaphragm driver 112 to operate.

[0051] The polarizing filter 170 (second polarizing element) is an optical filter that controls the polarization of light oscillating in a specific direction and removes light components (reflected light components) from a specific direction. It operates (rotates) when driven by the filter driver 160, and the rotation can change the rotation angle (polarization direction). This allows the vibration direction of light from the object 99 that reaches the image sensor 130 of the camera 100 to be selected. The filter driver 160 is controlled by commands from the system controller 146 and includes a motor that rotates the polarizing filter 170. The rotation angle may be changed by a user rotating the polarizing filter 170 without using the filter driver 160. The polarizing filter 170 may include a front frame that holds the filter and a rear frame that is attached to the tip of the lens device 102, and the front frame may be configured to rotate relative to the rear frame.

[0052] The polarizing filter 170 may be of a type that can be detachably attached to the lens device 102. In this case, if the polarizing filter 170 is removed, the camera 100 can photograph the object 99 without using the polarizing filter 170.

[0053] The polarizing filter 170 is an example of a polarizing element according to the present invention. The polarizing element is not limited to the polarizing filter 170, as long as it can control the polarization of light oscillating in a specific direction and remove light components (reflected light components) from a specific direction. Polarizing elements based on a polarizing plate, a wire grid, a photonic crystal, or other principles may also be used. These polarizing elements may be attached to the tip (on the object 99 side) of the lens unit 102 of the camera 100, or may be separate polarizing elements separated from the tip of the lens unit 102. Furthermore, the polarizing element may be disposed between the lenses constituting the lens group 106, or may be disposed after the lens group 106 (on the image sensor 130 side). Furthermore, the polarizing element may not be a type that rotates the polarizing filter itself, but may be a type that arranges multiple polarizing filters with different polarization directions on a sliding or turret-type holding member, and the holding member is translated and / or rotated to insert a polarizing filter with a desired polarization direction into the optical path.

[0054] The rotation angle detection section 162 detects the rotation angle φ of the polarizing filter 170 at the time of image capture relative to a reference angle of the polarizing filter 170. The rotation angle detection section 162 may include, for example, an encoder.

[0055] As shown in FIG. 7, the camera 100 includes an image sensor 130, a shutter 132, a shutter driver 134, a memory 136, a digital signal processor 138, an input / output interface 140 (IF: interface), a display unit 142, an operation unit 144, and a system controller 146.

[0056] The image sensor 130 is, for example, a CMOS (Complementary Metal-Oxide Semiconductor) image sensor having a predetermined color filter array (e.g., a Bayer array). In the camera 100 of this embodiment, the image sensor 130 is configured to include a drive unit, an ADC (Analog to Digital Converter), a signal processing unit, and the like. In this case, the image sensor 130 is driven and operated by the built-in drive unit. The signals of each pixel are converted into digital signals by the built-in ADC. Furthermore, the signals of each pixel are subjected to processes such as correlated double sampling, gain processing, and correction processing by the built-in signal processing unit as needed. The signal processing can be configured to be performed on the analog signals of each pixel, or on the digital signals of each pixel.

[0057] The imaging element 130 may be configured as an organic thin-film imaging element, an XY address type, a CCD (Charged Coupled Device) type, or other image sensor, in addition to a CMOS type image sensor.

[0058] The shutter 132 is disposed between the aperture 108 and the image sensor 130. The shutter 132 is driven to operate by a shutter driver 134. The shutter driver 134 controls the opening and closing of the shutter 132, and controls the exposure time (shutter speed) at the image sensor 130. In the first embodiment, a controller 300 (camera control unit 314, etc.) connected to the camera 100 may control the shutter driver 134 to open and close the shutter 132, or a user may open and close the shutter 132 by operating a shutter button.

[0059] The memory 136 may include a flash memory, a read-only memory (ROM), a random access memory (RAM), an auxiliary storage device, etc. The flash memory and ROM store a camera control program, a focus bracketing shooting program for executing shooting in the focus bracketing shooting mode, an image processing program, various data necessary for camera control, etc. The RAM temporarily stores shooting data and functions as a work area for processing by the system control unit 146. It also temporarily stores the camera control program, image processing program, etc. stored in the flash memory, etc. Note that part of the memory 136 (RAM) may be built into the system control unit 146.

[0060] The digital signal processor 138 performs signal processing such as offset processing, gamma correction processing, demosaic processing, and RGB / YCrCb conversion processing on the image obtained by shooting, thereby generating image data.

[0061] The input / output interface 140 includes a connection unit connectable to an external display device, a connection unit connectable to an external recording device, a card connection unit for inserting and removing a memory card, a communication unit connectable to a network, etc. For example, the input / output interface 140 may be a Universal Serial Bus (USB), a High-Definition Multimedia Interface (HDMI) (HDMI is a registered trademark), etc.

[0062] The display unit 142 is used as a playback monitor when playing back captured images, a live view monitor that displays live view images during capture, and a setting monitor when performing various settings. The display unit 142 is configured with a display such as an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diode), for example.

[0063] The operation unit 144 is configured to include various operation members for operating the camera 100. The operation members include a power button, a shutter button, and various other operation buttons. The various operation buttons include a button for turning the function of the image stabilization mechanism 150 on and off. Furthermore, if the display unit 142 is configured as a display unit with a touch panel, the operation members that make up the operation unit 144 include the touch panel. The operation unit 144 outputs signals corresponding to the operation of each operation member to the system control unit 146. For example, the user can set the conditions for shooting using the polarizing filter 170 (polarized image shooting) from the operation unit 144.

[0064] The system control unit 146 performs overall control of the camera 100. The system control unit 146 also calculates various physical quantities required for control. The system control unit 146 is configured, for example, by a microcomputer equipped with a processor, memory, etc. The processor is configured, for example, by a CPU (Central Processing Unit), etc. The system control unit 146 can execute a program for capturing polarized images. The system control unit 146 also controls the shake correction mechanism 150.

[0065] The camera 100 may include an image stabilization mechanism 150. The image stabilization mechanism 150 is an in-body image stabilizer (IBIS) control type image stabilization mechanism that performs image stabilization by shifting (including rotating) the image sensor 130 in a plane perpendicular to the optical axis in the direction opposite to the direction of shake. The image stabilization mechanism 150 includes a shake control unit 152, an image sensor drive unit 154, a shake detection unit 156, and a position detection unit 158.

[0066] The image sensor driver 154 may include an actuator that moves the image sensor 130. The shake detector 156 may include an acceleration sensor and / or a gyro sensor. The position detector 158 may include a Hall element that generates a voltage signal corresponding to the position of the image sensor 130. The shake controller 152 controls the image sensor driver 154 based on the signals from the shake detector 156 and the position detector 158 to shift the image sensor 130 in a plane perpendicular to the optical axis so as to offset shake of the camera 100. The shake correction mechanism 150 may be provided in the lens device 102.

[0067] 8 is a diagram showing the configuration of an illumination device 200. The illumination device 200 includes a polarizing filter 202 (first polarizing element), a filter driver 208 that controls the rotation of the polarizing filter 202, a rotation angle detector 212, a lens 204, a lens driver 210 that drives the lens 204, and a light source device 206.

[0068] The lens 204 may be a single lens or a lens group consisting of multiple lenses. The controller 300 may drive some or all of the lenses 204 forward and backward to change the spread of light emitted from the lighting device 200. The polarizing filter 202 and filter drive unit 208 may be configured in the same manner as the polarizing filter 170 and filter drive unit 160 described above. The rotation angle detection unit 212 detects the rotation angle (ψ) of the polarizing filter 202 during shooting relative to a reference angle of the polarizing filter 202. The rotation angle detection unit 212 may include, for example, an encoder.

[0069] The configuration of the light source device 206 is not particularly limited, but for example, an illumination device configured with LEDs (Light-Emitting Diodes) of multiple colors (red, blue, green, etc.) can be used. The light source device 206 is preferably a device that can change the brightness and color (color temperature, etc.) of the illumination light in addition to turning it on and off.

[0070] Furthermore, similar to the above-described case of the camera 100, the lighting device 200 may be configured so that its position and / or irradiation direction can be changed by an arm, a crane, a dolly, a vehicle, a drone, or the like. In the first embodiment, the processor 310 may control an arm, a dolly, or the like to change the position and / or irradiation direction of the lighting device 200, or a user may move the arm, or the like to change the position and / or irradiation direction. Alternatively, instead of changing the position or irradiation direction of the lighting device 200, a plurality of lighting devices 200 with different positions and irradiation directions may be provided.

[0071] In addition to the above configuration, the 3D model generation system 10 of this embodiment may be provided with a display device for displaying a background, and photographing (preliminary photographing and main photographing) may be performed with a background image displayed on the display device under the control of the controller 300. In this case, it is preferable that the processor 310 (controller 300) generates or selects a background image according to the characteristics of the object.

[0072] [Processing in the Three-Dimensional Model Generation System] Next, processing in the three-dimensional model generation system 10 configured as described above (including the operating method of the three-dimensional model generation system) will be described. FIG.

[0073] First, the user prepares a camera 100 equipped with a polarizing filter 170 and capable of capturing polarized images (preliminary captured image, actual captured image), a lighting device 200 equipped with a polarizing filter 202, a controller 300 capable of acquiring brightness information, synthesizing images, and generating a 3D model, and an object 99. The user places the object 99 on an imaging stand 400.

[0074] [Setting the Shooting Direction] The user sets the shooting direction of the camera 100 (step S100). The user fixes the object 99 to the shooting stand 400 and moves around the object 99 while holding the camera 100, capturing polarized images of the object 99 in multiple shooting directions, thereby obtaining polarized images. Therefore, the user moves to an arbitrary reference shooting position as a shooting position for starting shooting. At the reference shooting position, the user determines the shooting direction that serves as a reference for the relative positions of the camera 100 and the object 99. The user may rotate the shooting stand 400 when determining the shooting direction.

[0075] Instead of the user moving or rotating the camera 100 or the shooting stand 400, the controller 300 (camera control unit 314, etc.; processor 310) may set the shooting direction by controlling the arm, cart (cart 500, etc.), shooting stand 400, etc. (step S100).

[0076] [Setting the Position and Direction of the Lighting Device] The user sets the lighting direction of the lighting device 200 by moving an arm, a dolly, or the like (step S103). Alternatively, the controller 300 (lighting device control unit 312, etc.; processor 310) may set the position and lighting direction of the lighting device 200 by controlling the arm, the dolly, or the like (step S103).

[0077] The imaging condition determination unit 316 (processor 310) can determine imaging conditions (which may include the position and imaging direction to use, the angular interval of the imaging direction, the position and illumination direction of the lighting device, the color and brightness of the illumination light, the exposure, the shutter speed, etc.) taking into account the characteristics of the object 99 (shape, surface characteristics, etc.) and conditions such as the accuracy required for the 3D model. The imaging condition determination unit 316 can notify the user of the determined imaging conditions via the display 370 or speaker 360, record the imaging conditions in the recording device 390 (data processing conditions 390A; see FIG. 6 ), and control the arm, cart, etc. to set the imaging direction and lighting direction based on the determined imaging conditions. The imaging condition determination unit 316 can also read out previously determined and recorded imaging conditions (data processing conditions 390A) and control the arm, cart, etc.

[0078] [Preliminary Capture] The user captures a preliminary image (second image) of the object 99 in a certain capture direction using the camera 100 at the determined capture position, thereby acquiring a polarized image (step S110). As will be described in detail below, the preliminary image capture is performed prior to the main image capture (first image capture). The image capture condition determination unit 316 (processor 310) uses the image captured in the preliminary image capture to determine the rotation angle φ (second setting condition) of the polarizing filter 170 (second polarizing element) and the rotation angle ψ (first setting condition) of the polarizing filter 202 (first polarizing element) for the main image capture. The preliminary image capture may be a still image or a moving image of the object 99. It may also be a live view image.

[0079] In the following explanation, we will mainly describe a method in which preliminary shooting is performed to determine the rotation angle of the polarizing filter each time the shooting direction of the main shooting changes, but as will be described later, preliminary shooting (second shooting) may also be performed if the image captured in the main shooting (first shooting) changes beyond a predetermined standard.

[0080] [Rotation Angle of Polarizing Filter in Preliminary Capture] In the present invention, during preliminary capture (first capture), the camera 100 can capture multiple images of the object 99 at multiple rotation angles for at least one of the polarizing filter 170 (second polarizing element) and the polarizing filter 202 (first polarizing element) per capture direction. Therefore, preliminary capture may be performed by fixing the rotation angle ψ of the polarizing filter 202 on the illumination device 200 side and varying the rotation angle φ of the polarizing filter 170 on the camera 100 side (Pattern 1). Conversely, preliminary capture may be performed by varying the rotation angle ψ of the polarizing filter 202 and fixing the rotation angle φ of the polarizing filter 170 (Pattern 2). Alternatively, preliminary capture may be performed by varying the rotation angle ψ of the polarizing filter 202 and the rotation angle φ of the polarizing filter 170 (Pattern 3). The following example mainly describes Pattern 1, with Patterns 2 and 3 described later.

[0081] In pattern 1, the first setting condition can be considered to be determined to be the fixed rotation angle ψ, and in pattern 2, the second setting condition can be considered to be determined to be the fixed rotation angle φ.

[0082] 10 is a diagram showing how a polarization image is acquired by preliminary shooting. As shown in part (a) of FIG. 10, camera 100 captures a polarization image of object 99 at a certain shooting position and in a certain shooting direction (θ=0°) (step S110). As described above, the user may operate camera 100 to capture the image, or controller 300 may control the capture (the same applies to the following shooting operations).

[0083] In the preliminary capture, for one such capture direction, multiple images (polarized images) are acquired by changing the rotation angle φ of the polarizing filter 170 of the camera 100, i.e., by changing the polarization direction. For example, as shown in part (b) of Figure 10, capture is performed at φ = 0°, 45°, 90°, and 135° (multiple angles in the range of 0° to 180°). The number and value of φ may be different from those in the above example (for example, 15°, 60°, 105°, or 150°).

[0084] Acquiring multiple images by finely varying the rotation angle of the polarizing filter during preliminary shooting allows for more accurate determination of the rotation angle during the actual shooting (which allows for the generation of a high-quality 3D model). However, this may result in longer shooting times and a higher image processing load. Therefore, the number of rotation angle changes and the angle interval can be determined taking into account the processing load and the accuracy of the 3D model. Note that in the camera 100, the user may rotate the polarizing filter 170 by a desired value or interval, or the controller 300 (such as the camera control unit 314) and the filter drive unit 160 may control the rotation of the polarizing filter 170.

[0085] As shown in part (c) of Fig. 10, each polarization image can be expressed as Img(θ1, φ1), Img(θ1, φ2), Img(θ1, φ3), Img(θ1, φ4) ... Img(θn, φm) by associating the shooting direction θ with the rotation angle φ, where m is a parameter for identifying the shooting direction and n is a parameter for identifying the rotation angle.

[0086] The captured polarization image is stored in memory 136 as Img(θm, φn). Camera 100 can acquire multiple polarization images. Specifically, the polarization images can be stored in memory 136 as, for example, Img(0,0), Img(0,45), Img(0,90), ... Img(0,135). Position information of camera 100 may also be stored as additional information associated with the polarization image. The output value of shake detection unit 156 may also be stored in association with the polarization image. These polarization images and additional information may be recorded or transferred to recording device 390 of controller 300.

[0087] The captured polarization image may include data on the shooting direction θ and the rotation angle φ in a tag area within the same file, such as an EXIF ​​file. The polarization image and the data on the shooting direction θ and the rotation angle φ may be stored in separate files.

[0088] [Designation of Regions for Calculating Brightness Information] The photographing condition determination unit 316 and the image processing unit 318 (processor 310; determination unit) designate one or more regions (regions for calculating brightness information) in the image acquired in the preliminary photographing (step S120). As will be described later, the rotation angle φ (the rotation angle of the polarizing filter 170) during the actual photographing is determined based on the brightness values ​​of the pixels in the regions for calculating brightness information (the rotation angle φ and / or the rotation angle ψ are determined according to the pattern described above).

[0089] FIG. 11 is a diagram illustrating how a luminance information calculation region is designated. Part (a) of FIG. 11 illustrates an example in which a luminance information calculation region 800 is designated near the center of an object 99 in a preliminary captured image 900. Multiple luminance information calculation regions may be designated. For example, as shown in part (b) of FIG. 11 , luminance information calculation regions 810 and 820 may be designated in a preliminary captured image 910. Furthermore, as shown in part (c) of FIG. 11 , the entire region of the object 99 may be designated as a luminance information calculation region 830 in a preliminary captured image 920. When extracting the region of the object 99 from the preliminary captured image, AI (e.g., a trained model constructed using machine learning techniques) that performs region extraction and segmentation may be used. Such AI can be constructed, for example, by providing a neural network with training images with segmentation labels as correct data and training the network, but the AI ​​is not limited to such a method.

[0090] The brightness information calculation region may include only one specific pixel, or may include multiple pixels. The shooting condition determination unit 316 and image processing unit 318 (processor 310) can calculate the rotation angle φ and / or rotation angle ψ based on the brightness values ​​of pixels in the brightness information calculation region set as in the above example that have brightness values ​​less than a predetermined value. That is, the processor 310 (determination unit) can exclude pixels with so-called "blown-out highlights" from the calculation of the polarization filter rotation angle during actual shooting.

[0091] The shooting condition determination unit 316 and the image processing unit 318 may set the brightness information calculation area 800 based on an external input, for example, in response to a user operation. In this case, the camera 100 or the controller 300 displays the pre-captured image on the display unit 142 or the display 370, and the user can perform an operation to set the brightness calculation area in the pre-captured image via these screens.

[0092] [Luminance Information Calculated in the Luminance Information Calculation Region] The shooting condition determination unit 316 and the image processing unit 318 can calculate luminance information in the luminance information calculation region set as described above. This luminance information may be statistical information on the luminance values ​​of each pixel, specifically, for example, the sum, average, or maximum luminance value of each pixel, but is not limited to these examples. It may also be a weighted sum or average.

[0093] 12 is a conceptual diagram showing the relationship between the rotation angle of the polarizing filter and the luminance value in the luminance information calculation region (an example of the statistical information of the luminance values ​​described above; it may be a sum, average, maximum value, etc.; the same applies below). Note that while this diagram shows the rotation angle φ of polarizing filter 170, the same applies to the rotation angle ψ of polarizing filter 202.

[0094] The brightness value in the brightness information calculation region varies depending on the rotation angle φ of the polarizing filter 170. Here, the brightness value is maximum at φmax and minimum at φmin. The preliminary captured image (polarized image) acquired using the above-described method contains specular reflection and diffuse reflection components, but the specular reflection component is largest at φmax and smallest (close to zero) at φmin, meaning that the image is almost entirely diffuse reflection. Therefore, the magnitude of the specular reflection component can be determined from the image at rotation angle φmax and the image at rotation angle φmin, and an image with the smallest specular reflection component (close to zero) can be obtained by setting the rotation angle to φmin.

[0095] However, it may be difficult to predict in advance which rotation angle will maximize or minimize the brightness value. Therefore, in this embodiment, the rotation angle of the polarizing filter for the actual image capture is determined using an image captured in the preliminary image capture, as described below.

[0096] The maximum / minimum of the luminance value or the maximum / minimum of the specular reflection component are not limited to maximum / minimum in the strict mathematical sense, but may be values ​​with a certain range. For example, a rotation angle at which the change in the specular reflection component is within a predetermined amount for a predetermined angle change may be considered to be φmin (the rotation angle of the polarizing filter 170 that minimizes the effect of specular reflection).

[0097] 13 is a diagram showing an example of the rotation angle φ and the luminance value in the luminance information calculation region in the image acquired in the preliminary image capture described above. In this example, as shown in part (a) of the figure, preliminary image capture is performed by setting φ to four values ​​between 0° and 180° (e.g., 0°, 45°, 90°, and 135°). However, the rotation angle in the preliminary image capture does not necessarily match φmax or φmin, and there is a possibility that φmax or φmin will differ from the rotation angle during preliminary image capture, as shown in part (a).

[0098] Therefore, the shooting condition determination unit 316 and the image processing unit 318 (processor) estimate φmax and φmin from the preliminary captured image (step S120). While the estimation method is not particularly limited, for example, under the assumption that "luminance information varies sinusoidally with a period of 180° rotation angle," the amplitude and phase of a sine wave that best matches the luminance information of the preliminary captured image acquired at multiple rotation angles (e.g., that minimizes the sum of squares of the differences with the luminance information corresponding to the rotation angle of the preliminary capture) can be determined, and the rotation angles φmax and φmin at which this sine wave becomes maximum and minimum can be determined. In this case, if the curve of the luminance information has a complex shape, it may be approximated by superimposing multiple sine waves and / or cosine waves. Furthermore, the curve of the luminance information may be approximated by other functions (e.g., superimposing polynomials) instead of a sine wave.

[0099] In the example shown in part (a) of FIG. 13 , four circles indicate brightness information corresponding to the image acquired in the preliminary capture. Assume also that the rotation angles φ in the preliminary capture were 0°, 45°, 90°, and 135°. In this case, the capture condition determination unit 316 and the image processing unit 318 (processor 310; determination unit) calculate the above-mentioned rotation angles φmax / φmin based on the brightness information at these rotation angles (step S120). Assume, for example, that the results are φmax = 30° and φmin = 120°. Note that φmin is an example of the "second minimum rotation angle" (the rotation angle of the polarizing filter 170 that minimizes the effect of specular reflection from the object 99 in the image acquired in the main capture), and φmax is an example of the "second maximum rotation angle" (the rotation angle of the polarizing filter 170 that maximizes the effect of specular reflection from the object 99 in the image acquired in the main capture).

[0100] In this case, as shown in part (b) of FIG. 13 , the main image capture can be performed at multiple rotation angles including φmin and φmax. The image capture condition determination unit 316 and the image processing unit 318 (processor 310; determination unit) can determine to perform the main image capture at, for example, φ=30°, 75°, 120°, and 165° (an example of a second setting condition, which is a setting condition of the second polarizing element). By performing the main image capture at φmin and φmax, the influence of the specular reflection component can be determined from the difference between the obtained images. Furthermore, performing the main image capture at at least φmin can obtain an image with minimal influence of the specular reflection component. Therefore, even when capturing images at such rotation angles, a 3D model in which the influence of specular reflection is suppressed can be obtained. However, in order to generate a highly accurate 3D model in which the influence of specular reflection is suppressed as much as possible, it is preferable to capture images at multiple rotation angles including φmin and φmax. The image capture condition determination unit 316 and the image processing unit 318 can determine the number and interval of rotation angles at which the image capture is performed, taking into account factors such as the effort and processing load required for the image capture, the accuracy required for the 3D model, and the like. At this time, the determination may be made in consideration of the user's operation, or may be made automatically without depending on the user's operation.

[0101] Note that if actual photographs are taken of φmin and φmax, reflection information (specular reflection component information) can be obtained from the difference between the images. However, depending on the characteristics of the luminance information, reflection information may be obtained by other methods. For example, when the luminance value changes significantly as in the example shown in part (a) of FIG. 14, "difference 1" may be used as reflection information, as described above, or "maximum value 1" may be used as reflection information. On the other hand, when the luminance value is generally high and changes little as in the example shown in part (b) of FIG. 14, the ratio of "difference 2" to "maximum value 2" (= difference 2 ÷ maximum value 2) may be used as reflection information. Processor 310 may determine which method to use to calculate reflection information in response to a user operation, or may automatically determine the method without relying on a user operation.

[0102] Depending on the shape of the object, surface characteristics, and other conditions, the brightness information curve (the curves exemplified in FIGS. 12 to 14 ) may deviate significantly from a sine wave. Specifically, for example, the brightness information curve may include a range in which it changes significantly with changes in the rotation angle, and a range in which it barely changes. For this reason, the intervals between rotation angles in the actual image capture do not need to be uniform; the angle intervals can be made dense in the range in which the brightness information curve changes significantly with changes in the rotation angle, and can be made coarse in the range in which it barely changes. The number of rotation angles may also be changed depending on the shape of the curve.

[0103] If the rotation angle for the main photography determined in this way matches or is very close to that for the preliminary photography (i.e., if the preliminary photography results in photography at an optimal or nearly optimal rotation angle), the main photography for that photography direction may be omitted and a 3D model may be generated using the images from the preliminary photography. Whether or not to perform such processing can be determined taking into consideration the effort required for photography, the processing load, the accuracy required for the 3D model, etc.

[0104] [Actual Shooting] Once the rotation angle for the actual shooting has been determined by the above-described process, actual shooting is performed in a certain shooting direction using camera 100 while changing the rotation angle φ of polarizing filter 170 by the determined rotation angle, thereby obtaining images for generating a 3D model (step S130). The actual shooting may be performed by a user operating camera 100, or may be performed by controller 300 (camera control unit 314, etc.) controlling camera 100.

[0105] 13(b), actual imaging is performed while changing the rotation angle φ of the polarizing filter 170. In this example, φmin (30° in the above example) is set as the initial value for the angle change, and the angle is sequentially changed (φ=75°, 120°, 165°), but imaging may also be started from an angle other than φmin (for example, an angle closest to the angle at which the previous actual imaging or the rotation angle at which the preliminary imaging ended).

[0106] In addition, the processor 310 can output to the display 370, the recording device 390, the display unit 142, etc., brightness information curves such as those illustrated in Figures 12 to 14, and information indicating the angle determined as the rotation angle of the polarizing filters 170, 220 during actual shooting.

[0107] The processor 310 determines whether image acquisition in the main shooting is complete (step S140), and repeats the preliminary shooting and main shooting in the above-described procedure while changing the shooting direction until image acquisition is complete for all shooting directions (e.g., θ = 0° to 360°). In step S140, the processor 310 may determine whether the shooting operation for the set shooting direction is complete. The processor 310 may also display the image for which shooting has been completed on the display 370 so that the user can easily determine whether image acquisition is complete.

[0108] In this embodiment, the polarized image may be acquired as a still image or from a moving image. When acquiring a polarized image from a moving image, specifically, the camera 100 captures a moving image of the object 99 while rotating the polarizing filter 202 from a reference shooting direction. A plurality of time frame images included in the captured moving image can be acquired, and the time frame images can be used as the polarized image. When acquiring the time frame images, the shooting direction θ and the rotation angle φ can be associated and stored as supplementary information.

[0109] [Generation of 3D Model] [Acquisition of Polarized Images] The controller 300 acquires the polarized images of the object 99 acquired by the camera 100. The controller 300 acquires the multiple polarized images via the input / output interface 380 from the memory 136 of the camera 100 in a wired or wireless manner, or from a recording medium such as a memory card.

[0110] All of the polarization images are stored in the recording device 390 of the controller 300, with the shooting direction θm and rotation angle φn associated with each other as shown in Table 1 below (actually captured image 390C in FIG. 6). Table 1 includes polarization images captured at shooting directions θ1 to θm and rotation angles φ1 to φn. The conditions (number of images, rotation angle, etc.) of the included polarization images do not all need to be the same for shooting directions θ1 to θm.

[0111]

[0112] [Generation of Composite Image] Next, the image processing unit 318 (processor 310) generates a composite image from two or more polarized images in the shooting direction. That is, the composite image is generated from two or more polarized images that have the same shooting direction θ and only differ in rotation angle φ. Specifically, the three-dimensional model generation unit 320 (processor 310) generates a composite image by performing comparative dark compositing of two or more polarized images. Note that the comparative dark compositing technique itself is a known technique. The outline of the process is as follows.

[0113] 15 shows functional blocks related to generation of a composite image in the image processing unit 318. As shown in Fig. 15, the image acquisition unit 318A acquires from the recording device 390 all polarized images for the shooting direction (1: Img(θ1, φ1), Img(θ1, φ2) ... Img(θ1, φn), 2: Img(θ2, φ1), Img(θ2, φ2) ... Img(θ2, φn), ..., m: Img(θm, φ1), Img(θm, φ2) ... Img(θm, φn)).

[0114] The alignment unit 318B aligns the two or more polarized images used to generate a composite image. For example, if camera shake or the like occurs, the multiple polarized images may be aligned to the same coordinate system using a geometric transformation such as an affine transformation. Aligning to the same coordinate system may involve using one of the two or more polarized images as a reference polarized image and aligning the other polarized images to the reference polarized image, or the two or more polarized images may be aligned to a different coordinate system. The presence or absence of camera shake can be determined, for example, from the output value of the shake detection unit 156, which is one of the additional information. However, camera shake detection is not limited to the output value of the shake detection unit 156, and other methods may also be used. Accurately aligning the positions of the multiple polarized images can prevent unnatural edges and blurring during synthesis.

[0115] The image analysis unit 318C selects the pixel value with the lowest brightness for pixels at the same coordinates from a polarized image taken in a certain shooting direction (for example, Img(θ1, φ1), Img(θ1, φ2), ... Img(θ1, φn)) as the pixel value for that coordinate position. In other words, the image analysis unit 318C performs image analysis on pixels (pixel by pixel) at the same coordinates in each polarization image taken in the same shooting direction θ but with different rotation angles φ.

[0116] Next, the synthesis unit 318D replaces the pixel value at the same pixel position in the two or more polarization images with the smaller pixel value selected for each pixel, and generates a new synthesis image.

[0117] This is because if the polarization angle is set inappropriately for reflected light, the effect of reflected light cannot be eliminated, and as a result, the intensity of light reaching each pixel becomes high (i.e., it becomes bright and white), resulting in a large pixel value.On the other hand, if reflection is suppressed, the other side of the reflecting object (water surface or glass) is photographed, so objects and scenery that are (basically) darker than the reflected light are photographed.

[0118] Even if the object 99 has a highly reflective surface, the controller 300 can obtain a first image in which the influence of reflection is suppressed. As will be described later, a high-quality three-dimensional (3D) model can be generated even for an object 99 with a highly reflective surface.

[0119] Furthermore, specular reflectance and diffuse reflectance may be calculated from a plurality of polarized images and polarization directions at each position (in this case, each coordinate in the polarized images) of the object 99. The specular reflectance and diffuse reflectance can be calculated using the well-known technique of BRDF (Bidirectional Reflectance Distribution Function).

[0120] By calculating the specular reflectance and diffuse reflectance, when a 3D model is generated, it is possible to express the gloss, diffusion, and reflected color, which differ depending on the material of the object 99.

[0121] The synthesis unit 318D generates a synthetic image (Img_syn(θ1), Img_syn(θ2) ... Img_syn(θm)) from polarized images (1: Img(θ1, φ1), Img(θ1, φ2) ... Img(θ1, φn), 2: Img(θ2, φ1), Img(θ2, φ2) ... Img(θ2, φn), ..., m: Img(θm, φ1), Img(θm, φ2) ... Img(θm, φn)). The image analysis and synthesis methods used in comparative dark synthesis are not particularly limited, and known techniques can be applied.

[0122] When a composite image is generated for one shooting direction, the image processing unit 318 stores the composite image (Img_syn(θ1), Img_syn(θ2), ... Img_syn(θm)) in the recording device 390. Furthermore, the composite image may be stored in association with additional information about the polarized image, such as specular reflectance or diffuse reflectance.

[0123] In the above description, an example has been described in which a composite image is generated from two or more polarized images in the imaging directions θ1 to θm. However, this is not limiting, and for example, the rotation angle φ may be determined for a certain imaging direction θ. In other words, this is a case in which multiple polarized images are not acquired in a certain imaging direction θ. Therefore, in the imaging direction θi, one polarized image is acquired, and that polarized image is acquired as a composite image.

[0124] Next, the three-dimensional model generation unit 320 (processor 310) determines whether to generate a 3D model. If it determines not to generate a 3D model, the information processing ends. The stored composite images can be provided to another information system, for example, as images for generating a 3D model.

[0125] As an example of generating a composite image, a case has been described in which all polarized images are acquired and then a composite image is generated for each shooting direction. However, this is not limited to this. After acquiring a polarized image for a certain shooting direction, a composite image may be generated, the camera may move to the next shooting position, and after acquiring a polarized image for the next shooting direction, a composite image may be generated. In other words, acquisition of a polarized image and acquisition of a composite image may be performed sequentially.

[0126] If it is determined that a 3D model is to be generated, the following process is performed.

[0127] [Generation of 3D Model Using Composite Images] The 3D model generation unit 320 generates a 3D model from multiple composite images. Note that the technology for generating a 3D model using photogrammetry is well known. The outline of this process is as follows.

[0128] Fig. 16 shows functional blocks of the three-dimensional model generation unit 320. As shown in Fig. 16, the image acquisition unit 320A acquires composite images (Img_syn(θ1), Img_syn(θ2), ... Img_syn(θm)) for a certain shooting direction from the recording device 390, thereby acquiring multiple composite images.

[0129] The point cloud data generation unit 320B analyzes multiple composite images and generates three-dimensional point cloud data of feature points. The point cloud data generation unit 320B extracts feature points from each composite image. Next, the point cloud data generation unit 320B identifies corresponding feature points between different composite images as corresponding points and matches the corresponding points. The point cloud data generation unit 320B estimates camera parameters (e.g., fundamental matrix, fundamental matrix, internal parameters, etc.) of the camera 100 and estimates the shooting position and orientation based on the estimated camera parameters. Then, the three-dimensional positions of the feature points of the object 99 are determined. Bundle adjustment is performed as necessary. The three-dimensional coordinates of the estimated feature points are combined to generate point cloud data.

[0130] The 3D patch model generation unit 320C performs processing to generate a 3D patch model of the object 99 based on the three-dimensional point cloud data of the object 99 generated by the point cloud data generation unit 320B. Specifically, patches (meshes) are generated from the generated three-dimensional point cloud to generate a 3D patch model. This makes it possible to represent the surface undulations with a small number of points.

[0131] The image synthesis unit 320D generates a textured 3D model (three-dimensional model) by performing texture mapping on the 3D patch model generated by the 3D patch model generation unit 320C. The image synthesis unit 320D maps the texture onto the mesh to give the 3D patch model a realistic appearance of the object 99. By adding specular reflectance and diffuse reflectance, the material, etc. of the object 99 can be expressed when the 3D model is generated.

[0132] The processor 310 may exclude polarized images that could not be aligned with the reference image or polarized images with significant camera shake when generating a composite image or a 3D model.

[0133] [Output of 3D Model] The data of the generated 3D model can be stored in the recording device 390 (step S160; 3D model 390E in FIG. 6). The processor 310 can also display the 3D model on the display 370 as needed. The processor 310 can also print the 3D model using a 3D printer (not shown) via the input / output interface 380.

[0134] In this embodiment, a 3D model is generated by photogrammetry from multiple composite images in which reflections are suppressed, so that a highly accurate 3D model can be generated.

[0135] When creating a 3D model, the size of the 3D model can be determined by providing information about the distance between the two marks 410 on the imaging stand 400. The positional relationship between the camera 100 and the object 99 may be calculated based on the multiple marks 410 on the imaging stand 400, and a 3D model may be generated.

[0136] [Modification] A modification of the above embodiment will be described.

[0137] [Deciding whether to take preliminary photographs] In the first embodiment, preliminary photographs are taken in all photographing directions to determine the rotation angle of the polarizing filter 202 for the actual photographing. However, depending on the shape and surface characteristics of the object 99 or the accuracy required for the 3D model, it may not be necessary to take preliminary photographs in all photographing directions.

[0138] FIG. 17 is a flowchart showing a modified example of the processing in the 3D model generation system 10. This flowchart differs from the flowchart of FIG. 9 in that it includes a process for determining whether or not to perform preliminary photographing (step S105). In step S105, for example, the processor 310 determines that "preliminary photographing (second photographing) should be performed each time the amount of change (Δθ) in the photographing direction (θ) for the main photographing (first photographing) exceeds a predetermined angle" (YES in step S105), and determines the rotation angle of the polarizing filter 170 (second setting condition) and the rotation angle of the polarizing filter 202 (first setting condition) (steps S110 and S120). The processor 310 may determine the angle threshold ("predetermined angle") in response to a user operation, or may determine the angle automatically without relying on a user operation.

[0139] Processor 310 may also determine that "if the image captured in the main shooting (first shooting) has changed beyond a predetermined standard after determining the rotation angle of polarizing filter 170 (second setting condition) and the rotation angle of polarizing filter 202 (first setting condition), a preliminary shooting (second shooting) will be performed." One example of "if the image captured in the main shooting has changed beyond a predetermined standard" is "if the total difference or ratio of the brightness values ​​of corresponding pixels between an image of an object captured from a certain angle and an image captured from another angle exceeds a certain value." Specifically, processor 310 may make the determination, for example, as follows:

[0140] First, a preliminary image is analyzed (pre-shot analysis) at a certain standard shooting direction (θ = 0 degrees) to determine the initial value of the rotation angle of the polarizing filter. Here, assume that the filter rotation angles φ are determined to be 10 degrees, 55 degrees, 100 degrees, and 145 degrees. The image captured at φ = 10 degrees is saved as a key image. Next, preliminary images (pre-shots) are captured at θ = 5 degrees and φ = 10 degrees, and statistics (e.g., sum of absolute values, average value, etc.) of the pixel value differences between the key image and the image are calculated. If these statistics are below a predetermined threshold, the amount of change is considered small (does not exceed the above-mentioned "predetermined standard"), and no initial angle estimation of the polarizing filter is performed. Images are then captured at φ = 10 degrees, 55 degrees, 100 degrees, and 145 degrees. The key image is also not updated. Next, if the statistics of the pixel value differences between the image captured at θ = 10 degrees and φ = 10 degrees and the key image exceed a predetermined threshold, the initial angle estimation of the polarizing filter is performed. In this case, it is assumed that it is determined that images should be taken at φ=15 degrees, 60 degrees, 105 degrees, and 150 degrees. In this case, the main image taken at φ=15 degrees is updated as the key image.

[0141] [Actual Shooting in Other Patterns] In the first embodiment described above, a mode (Pattern 1) in which photographing (preliminary photographing and actual photographing) is performed while changing the rotation angle φ of the polarizing filter 170 on the camera 100 side has been described. However, as described in the section "Rotation Angle of Polarizing Filter in Preliminary Photographing," in the 3D model generation system 10, preliminary photographing may be performed by changing the rotation angle ψ of the polarizing filter 202 and fixing the rotation angle φ of the polarizing filter 170 (Pattern 2), or by changing the rotation angle ψ of the polarizing filter 202 and the rotation angle φ of the polarizing filter 170 (Pattern 3). In this case, in Pattern 2, preliminary photographing provides a luminance information curve, as shown in Figures 12 to 14, with ψ on the horizontal axis, and the value of ψ (ψmax / ψmin: first maximum rotation angle / first minimum rotation angle) that maximizes / minimizes this luminance information is determined. In addition, in pattern 3, preliminary photography can be used to obtain a curved surface with φ as the X axis, ψ as the Y axis, and luminance information as the Z axis, and the combinations of φ and ψ that maximize / minimize this luminance information (φmax / ψmax, φmin / ψmin) can be obtained.

[0142] Therefore, in the case of pattern 2, the rotation angle φ of polarizing filter 170 is fixed, and the rotation angle ψ of polarizing filter 202 is changed by rotation angles determined based on the results of the preliminary photographing (plurality of rotation angles including at least ψmin (first minimum rotation angle), and preferably including ψmin and ψmax (first maximum rotation angle)), while actual photographing is performed in a certain photographing direction θ by camera 100 to obtain images for generating a 3D model (step S130). In the case of pattern 3, actual photographing is performed while the rotation angle φ of polarizing filter 170 and the rotation angle ψ of polarizing filter 202 are changed by rotation angles determined based on the results of the preliminary photographing (plurality of rotation angles including at least a combination of φmin and ψmin, and preferably including a combination of φmin and ψmin and a combination of φmax and ψmax).

[0143] In patterns 2 and 3, the determination of which rotation angle to use as the initial value and in what cases preliminary imaging should be performed can be made in the same manner as in pattern 1.

[0144] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described aspects and various modifications are possible.

[0145] 10 3D model generation system 30 Object 99 Object 100 Camera 102 Lens device 104 Imaging optical system 106 Lens group 110 Lens driving unit 112 Driving unit 130 Imaging element 132 Shutter 134 Shutter driving unit 136 Memory 138 Digital signal processing unit 140 Input / output interface 142 Display unit 144 Operation unit 146 System control unit 150 Correction mechanism 152 Control unit 154 Imaging element driving unit 156 Shake detection unit 158 ​​Position detection unit 160 Filter driving unit 162 Rotation angle detection unit 170 Polarizing filter 180 Rotation angle 200 Illumination device 202 Polarizing filter 204 Lens 206 Light source device 208 Filter driving unit 210 Lens driving unit 212 Rotation angle detection unit 220 Polarizing filter 300 Controller 310 Processor 312 Illumination device control unit 314 Camera control unit 315 Photography stand control unit 316 Photography condition determination unit 318 Image processing unit 318A Image acquisition unit 318B Alignment unit 318C Image analysis unit 318D Synthesis unit 320 3D model generation unit 320A Image acquisition unit 320B Point cloud data generation unit 320C 3D patch model generation unit 320D Image synthesis unit 322 Input / output control unit 350 Operation unit 360 Speaker 370 Display 380 Input / output interface 390 Recording device 390A Data processing conditions 390B Preliminary captured image 390C Main captured image 390D Brightness information 390E 3D model 395 Bus 400 Photography stand 410 Mark 500 Cart 800 Brightness information calculation area 810 Brightness information calculation area 820 Brightness information calculation area 830 Brightness information calculation area 900 Preliminary captured image 910 Preliminary captured image 920 Preliminary captured image S100 to S160 Steps in the operation method of the 3D model generation system

Claims

1. A three-dimensional model generation system comprising: an illumination device having a first polarizing element; an imaging device having a second polarizing element; a control device that controls the rotation of at least one of the first polarizing element and the second polarizing element; and a three-dimensional model generation device that generates a three-dimensional model of an object from an image captured by the imaging device, wherein the control device has a determination unit that determines at least one of a first setting condition that is a setting condition of the first polarizing element and a second setting condition that is a setting condition of the second polarizing element in a first imaging step that captures an image to be used in generating the three-dimensional model.

2. A three-dimensional model generation system as described in claim 1, wherein the determination unit determines the first setting conditions and the second setting conditions using an image acquired in a second photograph taken prior to the first photograph in the photographing direction in which the first photograph is taken.

3. The three-dimensional model generation system described in claim 2, wherein the photographing device photographs the object at a plurality of rotation angles for at least one of the first polarizing element and the second polarizing element in one photographing direction in the first photographing, and the determination unit determines the first setting condition and the second setting condition based on a plurality of images acquired in the second photographing.

4. A three-dimensional model generation system as described in claim 2 or 3, wherein the determination unit determines the first setting conditions and the second setting conditions by performing the second photographing each time the change in the photographing direction in which the first photographing is performed exceeds a predetermined angle.

5. A three-dimensional model generation system as described in claim 2 or 3, wherein the determination unit, after determining the first setting conditions and the second setting conditions, performs the second photographing if the image captured in the first photographing changes beyond a predetermined standard, and determines the first setting conditions and the second setting conditions.

6. A three-dimensional model generation system as described in claim 2 or 3, wherein the determination unit determines the first setting conditions and the second setting conditions based on the brightness values ​​of pixels in one or more specified areas in the image acquired in the second photograph.

7. The three-dimensional model generation system according to claim 6, wherein the determination unit designates the entire area of ​​the object as the one or more areas.

8. The three-dimensional model generating system according to claim 6, wherein the determining unit specifies the one or more regions based on an external input.

9. The three-dimensional model generating system according to claim 8, wherein the determining unit specifies the one or more regions in response to a user operation.

10. A three-dimensional model generation system as described in claim 6, wherein the determination unit determines the first setting condition and the second setting condition based on the brightness values ​​of pixels in the one or more regions whose brightness values ​​are less than a predetermined value.

11. A three-dimensional model generation system as described in claim 10, wherein the determination unit determines, as the first setting condition and the second setting condition, the rotation angle of the first polarizing element and the rotation angle of the second polarizing element that minimizes the sum, average, or maximum value of the brightness values ​​of the pixels in the one or more regions.

12. A three-dimensional model generation system as described in any one of claims 1 to 3, wherein the first setting condition includes a first minimum rotation angle, which is the rotation angle of the first polarizing element that minimizes the effect of specular reflection by the object in the image acquired in the first photograph, and the second setting condition includes a second minimum rotation angle, which is the rotation angle of the second polarizing element that minimizes the effect of specular reflection by the object in the image acquired in the first photograph.

13. A three-dimensional model generation system as described in claim 12, wherein the first setting condition further includes a first maximum rotation angle, which is the rotation angle of the first polarizing element that maximizes the effect of specular reflection by the object in the image acquired in the first shooting, and the second setting condition further includes a second maximum rotation angle, which is the rotation angle of the second polarizing element that maximizes the effect of specular reflection by the object in the image acquired in the first shooting.

14. The three-dimensional model generation system described in claim 12, wherein the photographing device photographs the object at a combination of multiple rotation angles including the combination of the first minimum rotation angle and the second minimum rotation angle for one photographing direction in the first photographing to obtain multiple images, and the three-dimensional model generation device generates the three-dimensional model using the multiple obtained images.

15. A method for operating a three-dimensional model generation system comprising: an illumination device having a first polarizing element; an image capture device having a second polarizing element; a control device that controls the rotation of at least one of the first polarizing element and the second polarizing element; and a three-dimensional model generation device that generates a three-dimensional model of an object from an image captured by the image capture device, wherein the control device determines at least one of a first setting condition that is a setting condition for the first polarizing element and a second setting condition that is a setting condition for the second polarizing element in a first image capture that captures an image to be used in generating the three-dimensional model.

Images