Image capture processing system and 3D model generation method

Abstract

The present disclosure relates to an image capturing processing system and a 3D model generation method for generating a 3D model with improved accuracy. An image capturing processing system for generating a 3D model of an object using a plurality of captured images obtained by imaging the object is provided, the image capturing processing system being provided with: a plurality of polarized illumination devices each having a polarizer and irradiating the object from mutually different positions with polarized light obtained when light emitted from a light emitter is transmitted through the polarizer; and a plurality of polarized image capturing devices each having a polarizer and generating captured images at mutually different positions using polarized light obtained when external light is transmitted through the polarizer, at which mutually different positions at least one of the polarized illumination devices and the object are within a field of view. The polarization direction of the polarizer of the polarized image capturing devices is different from the polarization direction of the polarizers of the polarized illumination devices. The present disclosure can be applied to, for example, an image capturing processing system and a 3D model generation method.

Description

Technical Field

[0001] This disclosure relates to imaging processing systems and 3D model generation methods, and more particularly, to imaging processing systems and 3D model generation methods capable of generating more accurate 3D models. Background Technology

[0002] The following techniques exist: generating a 3D model from motion images captured from multiple viewpoints, and generating a free-viewpoint motion image based on the 3D model, where the 3D model is a model with 3D information of the subject, and the free-viewpoint motion image is a motion image based on an arbitrary viewpoint position. Such techniques are also known as volumetric capture techniques, etc.

[0003] For example, techniques have been proposed for generating 3D models using methods such as visual hulls, in which the three-dimensional shape of the subject is cut out based on multiple captured images obtained by imaging from different directions (see, for example, Patent Document 1).

[0004] When imaging a subject using such volumetric capture technology, an illumination device is typically used to project light onto the subject and its surroundings to ensure brightness.

[0005] Reference List

[0006] Patent documents

[0007] Patent document 1: WO 2018 / 150933 Summary of the Invention

[0008] The problem to be solved by the present invention

[0009] However, in the case of volumetric capture technology, in order to capture images of the subject from more different directions, imaging is performed by multiple imaging devices arranged around the subject, thus resulting in fewer blind spots, and the illumination devices are easily adapted to the field of view. When high-brightness illumination devices are included in the field of view, optical phenomena such as so-called light spots, ghosting, and halos may occur. When such optical phenomena occur, the accuracy of the 3D model generated from the captured image may be reduced.

[0010] This disclosure is made in view of the circumstances and its purpose is to generate more accurate 3D models.

[0011] Solution to the problem

[0012] An imaging processing system according to one aspect of the present technology is an imaging processing system that generates a three-dimensional (3D) model of an object using multiple captured images obtained by imaging the object. The imaging processing system includes: multiple polarized illumination devices, each including a polarizer, and illuminating the object from different positions using polarized light obtained by transmitting light emitted from a light-emitting unit through the polarizer; and multiple polarized imaging devices, each including a polarizer, and generating captured images at different positions using polarized light obtained by transmitting light from the outside through the polarizer, wherein at least one of the polarized illumination devices and the object are within a viewing angle at these different positions, and wherein the polarization direction of the polarizer of the polarized imaging device is different from the polarization direction of the polarizer of the polarized illumination device.

[0013] According to another aspect of the present technology, a three-dimensional (3D) model generation method includes the following steps: generating captured images of an object at different locations using polarized light whose polarization direction is different from that of polarized light emitted from a polarized illumination device within the viewpoint; and generating a 3D model of the object using multiple captured images obtained at these different locations.

[0014] According to another aspect of the present technology, an imaging processing system includes: a plurality of polarizing illumination devices, each including a polarizer, and illuminating an object from different positions using polarized light obtained by transmitting light emitted from a light-emitting unit through the polarizer; and a plurality of polarizing imaging devices, each including a polarizer, and generating a captured image of the object at different positions using polarized light obtained by transmitting light from the outside through the polarizer, wherein at least one of the polarizing illumination devices and the object are within a viewing angle at these different positions, wherein the polarization direction of the polarizer of the polarizing imaging device is different from the polarization direction of the polarizer of the polarizing illumination device.

[0015] In an imaging processing system that generates a 3D model of an object by using multiple captured images obtained by imaging an object according to one aspect of the present technology, a plurality of polarized illumination devices including polarizers illuminate the object from different positions by means of polarized light obtained by transmitting light emitted from a light-emitting unit through a polarizer, and the polarized light obtained by transmitting light from the outside through a polarizer is used by a plurality of polarized imaging devices including polarizers to generate captured images at different positions, at which at least one of the polarized illumination devices and the object are within a viewing angle, and the polarization direction of the polarizer of the polarized imaging device is different from the polarization direction of the polarizer of the polarized illumination device.

[0016] In another aspect of the 3D model generation method according to the present technology, a captured image of the object is generated at different positions by using polarized light with a polarization direction different from that of polarized light emitted from a polarized illumination device within the viewpoint, and a 3D model of the object is generated by using multiple captured images obtained at different positions.

[0017] An imaging processing system according to another aspect of the present technology includes: a plurality of polarization illumination devices, each including a polarizer, and illuminating an object from different positions using polarized light obtained by transmitting light emitted from a light-emitting unit through the polarizer; and a plurality of polarization imaging devices, each including a polarizer whose polarization direction is different from that of the polarizer of the polarization illumination device, and generating a captured image at different positions using polarized light obtained by transmitting light from the outside through the polarizer, at which at least one of the polarization illumination devices and the object are within a viewing angle. Attached Figure Description

[0018] Figure 1 This is a block diagram illustrating a primary configuration example of an information processing system.

[0019] Figure 2 This is a flowchart illustrating an example of the system's processing flow.

[0020] Figure 3 This is a block diagram showing a main configuration example of the data acquisition unit.

[0021] Figure 4 This is a block diagram showing a main configuration example of the lighting unit.

[0022] Figure 5 This is a block diagram showing a main configuration example of the imaging unit.

[0023] Figure 6 This is a diagram showing an example configuration of the imaging and illumination units.

[0024] Figure 7 This is a diagram showing an example of the arrangement of the imaging and illumination units.

[0025] Figure 8 This is a diagram showing an example of the arrangement of the imaging and illumination units.

[0026] Figure 9 This is a diagram showing an example of the arrangement of the imaging and illumination units.

[0027] Figure 10 This is a diagram showing an example of the arrangement of the imaging and illumination units.

[0028] Figure 11This is a diagram showing an example of the arrangement of the imaging and illumination units.

[0029] Figure 12 This is a diagram showing an example of a captured image.

[0030] Figure 13 This is a block diagram illustrating another configuration example of the data acquisition unit.

[0031] Figure 14 This is a flowchart illustrating an example of the calibration process.

[0032] Figure 15 This is a block diagram illustrating a typical configuration example of a computer. Detailed Implementation

[0033] The following describes the manner in which this disclosure is carried out (hereinafter referred to as implementation). Note that the description will proceed in the following order.

[0034] 1. First Implementation Method (Information Processing System)

[0035] 2. Second Implementation Method (Calibration)

[0036] 3. Application Examples

[0037] 4. Appendix

[0038] <1. First Implementation Method>

[0039] Information Processing System

[0040] There exists a volumetric capture technique in which a three-dimensional model (3D model) is generated from motion images captured from multiple viewpoints, and a free-viewpoint motion image is generated based on the 3D model. The 3D model is a model with three-dimensional information of the subject, and the free-viewpoint motion image is a motion image based on an arbitrary viewpoint position. Figure 1 The information processing system 100 is a system that uses volumetric capture technology to image a subject from multiple viewpoints, generates a 3D model of the subject based on the captured images, and generates a free viewpoint image based on the 3D model at any viewpoint position.

[0041] like Figure 1 As shown, the information processing system 100 includes a data acquisition unit 101, a 3D model generation unit 102, a formatting unit 103, a sending unit 104, a receiving unit 105, a rendering unit 106, and a display unit 107.

[0042] The data acquisition unit 101 acquires image data for generating a 3D model of the subject. For example, the data acquisition unit 101 acquires multiple viewpoint images captured by multiple imaging devices arranged around the subject as image data. In this case, preferably, the multiple viewpoint images are images obtained by multiple imaging devices that perform imaging synchronously.

[0043] Note that the data acquisition unit 101 can perform calibration based on image data and acquire the internal and external parameters of each imaging device. Furthermore, the data acquisition unit 101 can acquire, for example, multiple depth information values ​​indicating the distance from viewpoints at multiple locations to the subject.

[0044] The data acquisition unit 101 provides the acquired image data to the 3D model generation unit 102.

[0045] The 3D model generation unit 102 generates a 3D model based on image data provided by the data acquisition unit 101. This 3D model is a model with three-dimensional information of the subject. The 3D model generation unit 102 generates the 3D model of the subject by sculpting the three-dimensional shape of the subject using images from multiple viewpoints (e.g., contour images from multiple viewpoints), for example, using a so-called visual hull.

[0046] Here, for example, a contour image is an image that only represents the outline (external shape) of the subject, and the area within the outline is represented by filling it with a single color, like a shadow image. That is, the 3D model generation unit 102 generates such a contour image based on the image data (captured image) provided from the data acquisition unit 101. Note that the image data of the contour image can be provided to the 3D model generation unit 102 from the data acquisition unit 101.

[0047] The 3D model generation unit 102 can also use multiple depth information lines indicating the distance from viewpoints at multiple locations to the subject to further deform the 3D model generated with high accuracy using a visual shell.

[0048] The 3D model generated by the 3D model generation unit, which generates 3D models on a time-series frame basis, can also be referred to as a moving image of a 3D model. Furthermore, since the 3D model is generated using images captured by the imaging device of the data acquisition unit 101, it can also be referred to as a 3D model of a real image. The shape information of the 3D model can be represented in the form of mesh data (referred to as a polygon mesh), which represents the 3D model through the connection of vertices, and this shape information represents the surface shape of the subject. The method of representing a 3D model is not limited to this, and the 3D model can also be described by a so-called point cloud representation method that represents the 3D model through the positional information of points.

[0049] It also generates color information data as a texture associated with the 3D shape data. For example, there are cases where the color is constant when viewed from any direction, and cases where the color changes depending on the viewing direction, and cases where the color is view-dependent.

[0050] The 3D model generation unit 102 provides the generated 3D model data to the formatting unit 103.

[0051] The formatting unit 103 converts the 3D model data provided by the 3D model generation unit 102 into a format suitable for transmission and accumulation. For example, the formatting unit 103 can convert the 3D model generated by the 3D model generation unit 102 into multiple two-dimensional images by performing perspective projection from multiple directions. Furthermore, the formatting unit 103 can generate depth information from the 3D model, which is a two-dimensional depth image from multiple viewpoints. In this case, the formatting unit 103 can encode (compress) the depth and color information in the state of the two-dimensional image. In this case, the formatting unit 103 can encode the depth and color information side-by-side into one image or two separate images. Moreover, since the depth and color information are in the form of two-dimensional image data, the formatting unit 103 can use a two-dimensional compression technique such as Advanced Video Coding (AVC) to encode (compress) the depth and color information.

[0052] In the above case, the formatting unit 103 provides the 3D model data to the sending unit 104 as transmission data including 2D data (or its encoded data).

[0053] Furthermore, for example, the formatting unit 103 can convert the 3D data of the mesh data into a point cloud format and provide this data as transmission data including the 3D data to the transmission unit 104. In this case, the formatting unit 103 can use, for example, a geometrically based 3D compression technique discussed in MPEG to encode (compress) the 3D data.

[0054] The sending unit 104 sends the transmission data generated by the formatting unit 103 to the receiving unit 105. The sending unit 104 performs a series of processes offline, including those of the data acquisition unit 101, the 3D model generation unit 102, and the formatting unit 103, and then sends the transmission data to the receiving unit 105. Furthermore, the sending unit 104 can also send the transmission data generated according to the aforementioned series of processes to the receiving unit 105 in real time.

[0055] The receiving unit 105 receives the transmission data sent from the sending unit 104 and provides the transmission data to the rendering unit 106.

[0056] The rendering unit 106 performs rendering using the transmitted data received by the receiving unit 105. For example, the rendering unit 106 projects the mesh of the 3D model from the viewpoint of the camera device that draws the mesh of the 3D model, and performs texture mapping to paste textures representing colors or patterns. The drawing at this time can be arbitrarily set and viewed from a free viewpoint, regardless of the position of the camera device during imaging.

[0057] For example, rendering unit 106 performs texture mapping to paste textures representing the color, pattern, or texture of the mesh based on the position of the mesh in the 3D model. Texture mapping includes so-called view-dependent methods that take into account the user's viewing viewpoint and view-independent methods that do not. Since view-dependent methods change the texture to be pasted onto the 3D model based on the position of the viewing viewpoint, they have the advantage of achieving higher quality rendering compared to view-independent methods. On the other hand, view-independent methods do not consider the position of the viewing viewpoint, thus having the advantage of reducing processing load compared to view-dependent methods. Note that after display unit 107 detects the user's viewpoint (region of interest), the viewing viewpoint data is input from display unit 107 to rendering unit 106. Furthermore, rendering unit 106 may employ, for example, billboard rendering, which renders objects so that the objects maintain an upright posture relative to the viewing viewpoint. For example, when rendering multiple objects, rendering unit 106 can render objects of low interest to the viewer using billboard rendering and render other objects using another rendering method.

[0058] The rendering unit 106 provides the rendering result data to the display unit 107.

[0059] Display unit 107 displays the result rendered by rendering unit 106 on the display unit of the display device. The display device may be, for example, a 2D monitor or a 3D monitor, such as a head-mounted display, a spatial display, a mobile phone, a television, or a personal computer (PC).

[0060] <System Processing Flow>

[0061] Reference Figure 2 The flowchart describes an example of the system processing flow performed by the information processing system 100.

[0062] When the processing begins, in step S101, the data acquisition unit 101 acquires image data for generating a 3D model of the subject.

[0063] In step S102, the 3D model generation unit 102 generates a 3D model based on the image data obtained in step S101. The 3D model is a model with three-dimensional information of the subject.

[0064] In step S103, the formatting unit 103 encodes the shape and texture data of the 3D model generated in step S102 into a format suitable for transmission and accumulation.

[0065] In step S104, the transmitting unit 104 transmits the encoded data generated in step S103.

[0066] In step S105, the receiving unit 105 receives the data sent in step S104.

[0067] In step S106, the rendering unit 106 performs decoding processing and converts the data into data for the shape and texture required for display. Furthermore, the rendering unit 106 uses the shape and texture data to perform rendering.

[0068] In step S107, the display unit 107 displays the rendering result.

[0069] When the processing in step S107 is completed, the system processing ends.

[0070] By performing each of the processes described above, the information processing system 100 can generate a 3D model of the subject and generate and display an image of the subject viewed from a free viewpoint. Therefore, a user, as the viewer, can view the subject from a free viewpoint.

[0071] <Modification Example>

[0072] The information processing system 100 has been described above, outlining a series of processes from the data acquisition unit 101, which acquires captured images (which are materials used to generate content), to the display unit 107, which displays images viewed by the user. However, this does not mean that all functional blocks are required to implement the invention; the invention can be implemented for each functional block or a combination of multiple functional blocks. For example, in Figure 1 The sending unit 104 and receiving unit 105 are provided to illustrate a series of processes from the content creation side to the content viewing side through the distribution of content data. However, the processing from content creation to content viewing can also be performed by the same information processing device (e.g., a personal computer). In this case, the formatting unit 103, sending unit 104, and receiving unit 105 can be omitted.

[0073] Furthermore, in implementing the information processing system 100, the same implementer can implement all functional blocks, or different implementers can implement each functional block. For example, business operator A can implement a data acquisition unit 101, a 3D model generation unit 102, and a formatting unit 103 for generating 3D content; business operator B can implement a sending unit 104 (platform) for distributing 3D content; and business operator C can implement a receiving unit 105, a rendering unit 106, and a display unit 107 for receiving, rendering, and controlling the display of 3D content.

[0074] Furthermore, each functional block can be implemented in the cloud. For example, the rendering unit 106 can be implemented in the display device or in the server. In this case, information is exchanged between the display device and the server.

[0075] exist Figure 1 In this example, the data acquisition unit 101, the 3D model generation unit 102, the formatting unit 103, the sending unit 104, the receiving unit 105, the rendering unit 106, and the display unit 107 are collectively described as an information processing system 100. However, the configuration of the information processing system 100 is not limited to this example, and it is only required to include at least the data acquisition unit 101. For example, in Figure 1 In the configuration shown, any one or more of the 3D model generation unit 102 to the display unit 107 may be omitted. Furthermore, the information processing system 100 may have a configuration (functional block) different from the one described above.

[0076] Furthermore, each of the above functional blocks (data acquisition unit 101 to display unit 107) can be implemented through arbitrary configuration. For example, each functional block can be implemented by one or more devices. Alternatively, multiple functional blocks can be implemented by one device.

[0077] <Data Acquisition Unit>

[0078] Figure 3 It is shown Figure 1 A block diagram illustrating the main configuration example of the data acquisition unit 101 in the diagram. (See diagram below.) Figure 3 As shown, the data acquisition unit 101, which is an embodiment of the imaging processing system applying this technology, includes an imaging illumination unit 121 and a transmission unit 122.

[0079] The imaging illumination unit 121 images and illuminates the subject. The imaging illumination unit 121 includes imaging units 131-1 to 131-M and illumination units 132-1 to 132-N (M and N are integers of 2 or greater). For the sake of illustration, imaging units 131-1 to 131-M are referred to as imaging unit 131. For the sake of illustration, illumination units 132-1 to 132-N are referred to as illumination unit 132.

[0080] That is, the imaging illumination unit 121 includes a plurality of imaging units 131 and a plurality of illumination units 132. Note that the number of imaging units 131 and the number of illumination units 132 included in the imaging illumination unit 121 may be the same as each other (i.e., M=N), or they may be different from each other.

[0081] Imaging unit 131 includes one or more image acquisition devices and images the subject to generate a captured image for 3D model generation. Specifically, imaging unit 131 generates a captured image for extracting the contours and textures of the subject. Imaging unit 131 provides the data of the generated captured image to transmitting unit 122.

[0082] The wavelength range of the light received by the image pickup device of the imaging unit 131 is arbitrary and can be visible or invisible light. For example, the imaging unit 131 can receive visible light (RGB light) and generate a captured image of visible light, or it can receive infrared light (infrared (IR) light) and generate a captured image of infrared light.

[0083] The illumination unit 132 includes one or more illumination devices and illuminates the subject imaged by the imaging unit 131. The wavelength range of the light emitted by the illumination devices of the illumination unit 132 is arbitrary and can be visible or invisible light. For example, the illumination unit 132 can illuminate the object using visible light (RGB light) or infrared light (IR light).

[0084] The transmitting unit 122 transmits the captured image data provided by the imaging unit 131 to the 3D model generation unit 102. At this time, the transmitting unit 122 may provide the captured image data to the 3D model generation unit 102 without encoding the data, or it may encode the captured image data and provide the encoded data to the 3D model generation unit 102. Furthermore, the transmitting unit 122 may perform arbitrary image processing on the captured image. For example, the transmitting unit 122 may extract contours or textures from the captured image and provide the extracted contour or texture data to the 3D model generation unit 102.

[0085] Typically, it is difficult to image a subject when the brightness is too low (too dark). Therefore, by illuminating the subject with the illumination unit 132, the imaging unit 131 can capture an image of the subject with sufficient brightness and obtain a captured image with adequate brightness.

[0086] However, in the case of the data acquisition unit 101 acquiring captured images for 3D model generation, multiple imaging units 131 are arranged around the subject to reduce blind spots. Therefore, the possibility of the illumination unit 132 being included within the field of view of the imaging unit 131 is very high. In other words, it is difficult to arrange the illumination unit 132 so that it is not included within the field of view of the imaging unit 131.

[0087] Because the illumination unit 132 (the illumination device) is a high-brightness light source, when the illumination unit 132 is included within the viewing angle, light leakage into dark areas may occur, resulting in phenomena such as so-called light spots, ghosting, and halos. When such phenomena occur, it may be difficult to extract the accurate outline of the subject from the captured image. Furthermore, it may also be difficult to extract the texture of the subject. Therefore, the accuracy of the 3D model generated from the captured image may be reduced.

[0088] <Applications of Polarizers>

[0089] Therefore, polarizers are provided in the imaging unit 131 (image pickup device) and the illumination unit 132 (illumination device). The polarizers generate linearly polarized light or circularly polarized light from natural light (unpolarized light). The illumination unit 132 illuminates the subject using polarized light, and the imaging unit 131 receives the polarized light to generate a captured image. Then, the polarization direction of the polarized light emitted by the illumination unit 132 (i.e., the polarization direction of the polarizer included in the illumination unit 132) is made different from the polarization direction of the polarized light received by the imaging unit 131 (i.e., the polarization direction of the polarizer included in the imaging unit 131).

[0090] Note that in this specification, a light beam primarily comprising vibrational components in a predetermined direction will be referred to as polarized light, and the principal vibrational direction of the polarized light will be referred to as the polarization direction (or polarization angle). Furthermore, a polarizer generates polarized light in a predetermined polarization direction, and this polarization direction will also be referred to as the polarization direction (or polarization angle) of the polarizer.

[0091] For example, an imaging processing system that generates a 3D model of an object using multiple captured images obtained by imaging the object includes: multiple polarized illumination devices (e.g., illumination unit 132) that include polarizers and illuminate the object from different positions using polarized light obtained by transmitting light emitted from a light-emitting unit through the polarizers; and multiple polarized imaging devices (e.g., imaging unit 131) that include polarizers and generate captured images at different positions using polarized light obtained by transmitting light from the outside through the polarizers, at which at least one of the polarized illumination devices and the object are within a viewing angle, wherein the polarization direction of the polarizer of the polarized imaging device is different from the polarization direction of the polarizer of the polarized illumination device.

[0092] For example, by using polarized light with a polarization direction different from that emitted from a polarized illumination device (e.g., illumination unit 132) within the viewpoint, captured images of the object are generated at different locations, and a 3D model of the object is generated by using multiple captured images obtained at these different locations.

[0093] For example, the imaging processing system includes: a plurality of polarized illumination devices (e.g., illumination unit 132), the plurality of polarized illumination devices including polarizers, and illuminating an object from different positions using polarized light obtained by transmitting light emitted from a light-emitting unit through a polarizer; and a plurality of polarized imaging devices (e.g., illumination unit 132), the plurality of polarized imaging devices including polarizers, and generating captured images at different positions using polarized light obtained by transmitting light from the outside through a polarizer, at which at least one of the polarized illumination devices and the object are within a viewing angle, wherein the polarization direction of the polarizer of the polarized imaging device is different from the polarization direction of the polarizer of the polarized illumination device.

[0094] When the polarization directions of the polarizer in imaging unit 131 and the polarizer in illumination unit 132 are different from each other, the amount of direct light from illumination unit 132 that passes through the polarizer in imaging unit 131 and enters the sensor is reduced. Therefore, in the captured image generated by imaging unit 131, the brightness value of a portion of illumination unit 132 within the viewing angle can be reduced, thus suppressing the occurrence of so-called light spots, ghosting, halos, etc. Consequently, contours and textures can be extracted more accurately from the captured image, and therefore the 3D model generation unit 102 can generate a more accurate 3D model (suppressing the reduction in the accuracy of the 3D model).

[0095] Note that the reduction in the amount of direct light from the illumination unit 132, transmitted through the polarizer of the imaging unit 131, and entering the sensor in this manner depends on the relationship (angle) between the polarization directions of the polarizers of the imaging unit 131 and the illumination unit 132. Generally, the closer the angle between them is to 90 degrees, the greater the reduction in the amount of light. That is, when the angle between the polarization directions of the polarizers of the imaging unit 131 and the illumination unit 132 is close to 90 degrees, the occurrence of so-called light spots, ghosting, halos, etc., can be suppressed more strongly.

[0096] <Lighting Unit>

[0097] Figure 4 This is a block diagram illustrating a main configuration example of the lighting unit 132. (See diagram for example.) Figure 4 As shown, the illumination unit 132 includes a polarizing filter 151 and a light-emitting unit 152.

[0098] Polarizing filter 151 is an example of a polarizer, and generates polarized light by transmitting light components that vibrate in a predetermined direction. Light emitting unit 152 is a light source, and emits a light beam (unpolarized light) with a predetermined wavelength in a predetermined direction.

[0099] like Figure 4 As shown, a polarizing filter 151 is arranged in front of the light-emitting unit 152 along the beam emission direction (illumination direction). Unpolarized light 161 emitted from the light-emitting unit 152 is guided to the polarizing filter 151. The polarizing filter 151 transmits the vibration component of the unpolarized light 161 in a predetermined direction. That is, polarized light 162 with a predetermined polarization direction is generated by the polarizing filter 151. This polarized light 162 is emitted from the illumination unit 132. That is, the illumination unit 132 is a polarization illumination device that includes a polarizer and emits polarized light generated by the polarizer using light from the light source.

[0100] The illumination unit 132 is mounted at a position and orientation suitable for illuminating the object (which serves as the subject of the imaging unit 131), so that at least a portion of the polarized light 162 is emitted toward the object. Then, at least a portion of the emitted polarized light 162 is reflected by the object or the like, becoming unpolarized light, and travels toward the imaging unit 131. That is, by performing illumination in this manner through the illumination unit 132, the brightness of the captured image can be increased.

[0101] Note that the wavelength range of the polarized light 162 emitted by the illumination unit 132 is arbitrary. For example, the polarized light 162 can be visible light, invisible light, or both. For example, the polarized light 162 can be infrared light (IR light). Furthermore, the illumination unit 132 may include multiple light-emitting units 152 (light sources) that emit light beams in different wavelength regions, and the imaging illumination unit 121 may include multiple illumination units 132 that emit polarized light 162 in different wavelength regions.

[0102] Furthermore, the polarization direction of the polarization filter 151 (i.e., the polarization direction of the polarized light 162) can be predetermined (it can be fixed) or variable. For example, a polarization direction control mechanism (such as a movable ring) can be provided to control the polarization direction of the polarization filter 151, and the polarization direction of the polarization filter 151 can be made variable by the polarization direction control mechanism.

[0103] <Imaging Unit>

[0104] Figure 5 This is a block diagram illustrating a main configuration example of the imaging unit 131. For example, as shown... Figure 5 As shown in A, the imaging unit 131 includes a polarizing filter 171 and an image sensor 172.

[0105] Polarizing filter 171 is an example of a polarizer and generates polarized light by transmitting light components that vibrate in a predetermined direction. Image sensor 172 includes multiple pixels, in which incident light is photoelectrically converted and a captured image is generated. Image sensor 172 provides data of the generated captured image to transmitting unit 122.

[0106] like Figure 5 As shown in Figure A, a polarization filter 171 is arranged on the beam incident side of the image sensor 172. Unpolarized light 181 incident on the imaging unit 131 is guided to the polarization filter 171. The polarization filter 171 transmits the vibration component of the unpolarized light 181 in a predetermined direction. That is, polarized light 182 with a predetermined polarization direction is generated by the polarization filter 171. The polarized light 182 enters the image sensor 172 and is photoelectrically converted. That is, the image sensor 172 generates a captured image corresponding to the polarized light 182. In other words, the imaging unit 131 is a polarization imaging device that includes a polarizer and uses the polarized light generated by the polarizer to generate a captured image.

[0107] Note that when the illumination unit 132 is within the viewing angle of the imaging unit 131, there is a possibility that direct light from the illumination unit 132 enters the imaging unit 131. That is, the polarized light 162 emitted from the illumination unit 132 can be guided to the polarization filter 171. Here, the polarization direction of the polarization filter 171 is set to a direction different from that of the polarization filter 151. That is, the polarization filter 171 and the polarization filter 151 have polarization directions different from each other. Therefore, at least a portion of the polarized light 162 is blocked by the polarization filter 171. That is, the amount of polarized light 162 incident on the image sensor 172 is reduced.

[0108] That is, in the captured image generated by the imaging unit 131, the brightness value of a portion of the illumination unit 132 within the viewing angle can be reduced, thus suppressing the occurrence of so-called light spots, ghosting, halos, etc. Therefore, contours and textures can be extracted more accurately from the captured image, and thus the 3D model generation unit 102 can generate a more accurate 3D model (suppressing the reduction in the accuracy of the 3D model).

[0109] Note that the wavelength range of the light received and photoelectrically converted by image sensor 172 (i.e., the wavelength range of polarized light 182) is arbitrary. For example, image sensor 172 can photoelectrically convert visible light, photoelectrically convert invisible light, or photoelectrically convert both. That is, imaging unit 131 can generate data of a captured image of visible light, data of a captured image of invisible light, or both captured images. For example, image sensor 172 can photoelectrically convert infrared light (IR light). That is, imaging unit 131 can generate a captured image of infrared light (IR light). Furthermore, imaging unit 131 may include multiple image sensors 172 that photoelectrically convert light beams in different wavelength regions, and imaging illumination unit 121 may include multiple imaging units 131 that generate captured images of light beams in different wavelength regions.

[0110] The captured image generated by imaging unit 131 can be used to extract the contour of an object as a subject. In other words, imaging unit 131 can generate a captured image for extracting the contour of an object. By providing a polarizer (e.g., polarizing filter 171) in imaging unit 131 to generate such a captured image, the contour can be extracted more accurately from the captured image.

[0111] Furthermore, the captured image generated by imaging unit 131 can be used to extract the texture of the object as the subject. In other words, imaging unit 131 can generate a captured image for extracting the texture of the object. By providing a polarizer (e.g., polarizing filter 171) in imaging unit 131 to generate such a captured image, the texture can be extracted more accurately from the captured image.

[0112] Of course, the captured image generated by the imaging unit 131 can be used to extract both the contour and texture of the object as the subject. In other words, the imaging unit 131 can generate captured images for extracting the contour and texture of the object. Furthermore, the imaging unit 131 can generate each of the captured images for extracting the contour and the captured images for extracting the texture of the object.

[0113] Furthermore, the imaging illumination unit 121 may include an imaging unit 131 for generating a capture image for extracting the object's contour and an imaging unit 131 for generating a capture image for extracting the object's texture. In this case, a polarizer (e.g., a polarizing filter 171) may be provided in the imaging unit 131 for generating the capture image for extracting the object's contour, the imaging unit 131 for generating the capture image for extracting the object's texture, or both of the imaging units 131.

[0114] Furthermore, the polarization direction of the polarization filter 171 (i.e., the vibration direction of the polarized light 182) can be predetermined (it can be fixed) or variable. For example, a polarization direction control mechanism (such as a movable ring) can be provided to control the polarization direction of the polarization filter 171, and the polarization direction of the polarization filter 171 can be made variable through the polarization direction control mechanism.

[0115] In addition, such as Figure 5 As shown in Figure B, the imaging unit 131 may include a polarization sensor 191. The polarization sensor 191 is an image sensor that photoelectrically converts polarized light to generate a captured image. The polarization sensor 191 includes multiple pixels, each pixel having a polarizer that generates polarized light from the incident light, and a light receiving unit in each pixel receives the polarized light generated by the polarizer and performs photoelectric conversion. That is, the polarization sensor 191 polarizes the incident unpolarized light 181 and performs photoelectric conversion to generate its captured image. Note that the polarization direction of the polarizer in each pixel of the polarization sensor 191 is designed to be different from the polarization direction of the polarization filter 151. That is, the polarizer in each pixel of the polarization sensor 191 and the polarization filter 151 have different polarization directions. Therefore, since at least a portion of the polarized light 162 is blocked by the polarizer, the amount of polarized light 162 to be photoelectrically converted (the brightness in the captured image) is reduced.

[0116] Therefore, in the case of polarization filter 171, the brightness value of a portion of the illumination unit 132 within the viewing angle in the captured image generated by imaging unit 131 (polarization sensor 191) can be reduced, and the occurrence of so-called light spots, ghosting, halos, etc., can be suppressed. Thus, contours and textures can be extracted more accurately from the captured image, and therefore the 3D model generation unit 102 can generate a more accurate 3D model (the reduction in the accuracy of the 3D model can be suppressed).

[0117] Note that the imaging unit 131 and the illumination unit 132 can be configured as a time-of-flight (ToF) sensor. That is, the imaging unit 131 and the illumination unit 132 can be configured as a distance measurement sensor, wherein the illumination unit 132 illuminates the subject, the imaging unit 131 receives the reflected light, and measures the distance to the subject based on its light reception timing. In other words, this technology can also be applied to optical distance measurement sensors such as ToF sensors.

[0118] <Imaging and Illumination Unit>

[0119] Imaging unit 131 and illumination unit 132 can be arranged close to each other. Furthermore, imaging unit 131 and illumination unit 132 can be arranged such that the light illumination direction of illumination unit 132 and the imaging direction of imaging unit 131 (e.g., the direction of the center of the viewing angle) are the same. In other words, each of the illumination units 132 can be positioned close to any one of the imaging units 131, and can be positioned such that the direction of polarized light illumination is the same as the imaging direction of the imaging unit 131 near the illumination unit. With this configuration, when viewed from imaging unit 131, illumination unit 132 can perform illumination from the front of the object being photographed. Therefore, imaging unit 131 can generate a captured image with few unwanted shadows and dark areas on the subject, and the subject has sufficient brightness.

[0120] For example, the imaging and illumination units can be formed by imaging unit 131 and illumination unit 132 arranged close to each other. Figure 6 This is a diagram showing an example of an imaging and illumination unit.

[0121] exist Figure 6 In the example, the imaging and illumination unit 210 includes an RGB camera 211, an IR camera 212, and an IR light 213.

[0122] RGB camera device 211 is an imaging unit 131 that receives visible light and generates a captured image in the wavelength region of visible light. IR camera device 212 is an imaging unit 131 that receives infrared light and generates a captured image in the wavelength region of infrared light. IR light 213 is an illumination unit 132 that emits infrared light.

[0123] For example, it is highly likely that the source of visible light changes drastically outdoors or at a live concert venue. For instance, it is conceivable that the subject is illuminated by spotlights or laser beams at a live concert venue. When the imaging processing system described above performs imaging in such an environment, the captured image in the visible light wavelength range may be affected by such illumination, and optical phenomena such as so-called spotting, ghosting, and halos may occur. Therefore, it may be difficult to accurately extract the outline of the subject using such captured images.

[0124] Therefore, the imaging and illumination unit 210 uses the IR imaging device 212 to generate a captured image in the wavelength region of infrared light as a captured image for extracting the outline of the subject. That is, the outline of the subject is extracted using the captured image in the wavelength region of infrared light. Then, in order for imaging by the IR imaging device 212 (to ensure sufficient brightness in the wavelength region of infrared light), the IR light 213 illuminates the subject using infrared light.

[0125] Note that since the IR camera 212 and the IR light 213 are mounted close to each other and facing the same subject, the IR light 213 can illuminate the front of the subject when viewed from the IR camera 212. Therefore, the IR camera 212 can generate a captured image with few unwanted shadows and dark areas on the subject, and the subject has sufficient brightness. That is, the IR camera 212 can generate a captured image from which a more accurate contour can be extracted. In other words, by using the captured image generated by the IR camera 212, the contour of the subject can be extracted more accurately.

[0126] In addition, the IR camera device 212 has Figure 5 The polarizer in the example. Similarly, IR light 213 has Figure 4 The example uses a polarizer. The polarization directions of the polarizer in the IR camera 212 and the polarizer in the IR light 213 are different from each other. Therefore, as described above, in the captured image generated by the IR camera 212, due to the infrared light emitted by the IR light 213 included within the viewing angle, the occurrence of so-called spotlights, ghosting, halos, etc., can be suppressed. Therefore, contours can be extracted more accurately from the captured image, and thus the 3D model generation unit 102 can generate a more accurate 3D model (capable of suppressing the decrease in the accuracy of the 3D model).

[0127] Since the RGB camera device 211 can generate a captured image in the wavelength region of visible light, it can generate a captured image for extracting the texture of a subject. The RGB camera device 211 and the IR camera device 212 are mounted close to each other, facing the same subject. That is, the viewing angles of the RGB camera device 211 and the IR camera device 212 are the same or similar. Therefore, the captured image generated by the RGB camera device 211 can be used to extract a texture that corresponds to the outline of the subject extracted using the captured image generated by the IR camera device 212.

[0128] <Layout Example>

[0129] An example of the arrangement of imaging unit 131 and illumination unit 132 will be described using imaging and illumination unit 210 as a unit. For example... Figure 7 As shown, multiple imaging and illumination units 210 (i.e., imaging unit 131 and illumination unit 132) can be mounted around the object 231, which is the subject (to surround the object 231). For example, each imaging and illumination unit 210 can be arranged such that the object, which is the subject, is positioned in an area (plane or space) having lines connecting adjacent imaging and illumination units 210 to each other as an outer frame.

[0130] In this configuration, at least one of the illumination units 132 and the object 231 can be arranged to be included within the field of view of each imaging unit 131. Furthermore, another imaging unit 131 can be arranged to be included within the field of view.

[0131] For example, such as Figure 8 As shown, the two imaging and illumination units 210 (imaging and illumination unit 210-1 and imaging and illumination unit 210-2) can be arranged to face each other. Figure 8 In the example case, imaging and illumination units 210-1 and 210-2 are mounted on opposite sides of object 231 along a straight line 241 passing through object 231 and facing object 231. That is, the imaging direction and illumination direction of imaging and illumination units 210-1 and 210-2 are opposite to each other.

[0132] By installing two imaging and illumination units 210 (imaging and illumination unit 210-1 and imaging and illumination unit 210-2) in this manner, a wider range of the object 231 can be imaged (which can reduce blind spots).

[0133] In this arrangement, although the IR light 213 is included within the field of view of the IR camera 212, as described above, the direct light incident from the IR light 213 can be suppressed by using a polarizer, and thus the occurrence of so-called spotting, ghosting, halos, etc., caused by the infrared light emitted by the IR light 213 can be suppressed. Therefore, the outline of the object 231 can be extracted more accurately using the captured image generated by the IR camera 212.

[0134] Note that the number of imaging and illumination units 210 to be installed is arbitrary, as long as there are multiple units. For example, eight imaging and illumination units 210 can be installed. In the case of installing a large number of imaging and illumination units 210 as described above, multiple other imaging units 131 or illumination units 132 may be included within the field of view of the imaging unit 131.

[0135] That is, the imaging and illumination units 210 (imaging unit 131 and illumination unit 132) may be mounted such that multiple illumination units 132 are included within the viewing angle of the imaging unit 131. In this case, the polarization directions of the polarizers of the multiple illumination units 132 can be the same. With this configuration, the incident light from each illumination unit 132 to the imaging unit 131 within the viewing angle can be similarly suppressed. That is, the occurrence of so-called light spots, ghosting, halos, etc., can be further suppressed.

[0136] Furthermore, the imaging and illumination unit 210 can be mounted such that another imaging unit 131 is included within the field of view of the imaging unit 131. Additionally, the imaging and illumination unit 210 can be mounted such that a plurality of other imaging units 131 are included within the field of view of the imaging unit 131.

[0137] Furthermore, multiple polarization illumination devices (e.g., illumination unit 132) may include a first polarization illumination device and a second polarization illumination device, and multiple polarization imaging devices (e.g., imaging unit 131) may include a first polarization imaging device located where the object and the first polarization illumination device are within the viewing angle, and a second polarization imaging device located where the object and the second polarization illumination device are within the viewing angle. The polarization direction of the polarizer of the first polarization imaging device may be different from the polarization direction of the polarizer of the first polarization illumination device, and the polarization direction of the polarizer of the second polarization imaging device may be different from the polarization direction of the polarizer of the second polarization illumination device. That is, multiple imaging units 131 may exist that include a single illumination unit 132 within the viewing angle.

[0138] In this case, the polarization directions of the polarizers of the multiple illumination units 132, which are included in the viewing angles of the imaging units 131 that are different from each other, may be the same or different. That is, the polarization direction of the polarizer of the second polarization imaging device may be different from the polarization direction of the polarizer of the second polarization illumination device.

[0139] That is, the polarization directions of the polarizers of the plurality of illumination units 132 can be different from each other. Similarly, the polarization directions of the polarizers of the plurality of imaging units 131 can be different from each other. For example, if the polarization direction of the polarizer of one of the plurality of imaging units 131 is different from the polarization direction of the polarizer of the illumination unit 132 included in the viewing angle of that imaging unit 131, the effect of this embodiment can be obtained.

[0140] For example, such as Figure 9 As shown in Figure A, multiple imaging and illumination units 210 (imaging unit 131 and illumination unit 132) can be arranged in a circular shape centered on object 231. Figure 9 In example A, eight imaging and illumination units 210 are arranged on a circle 251 centered on object 231. Figure 9 As shown in Figure B, each of the imaging and illumination units 210 (imaging and illumination units 210-1 to 210-8) is mounted facing the object 231. More specifically, imaging and illumination units 210-1 and 210-5 are arranged to face each other on a straight line 252 passing through the object 231. Imaging and illumination units 210-2 and 210-6 are arranged to face each other on a straight line 253 passing through the object 231. Imaging and illumination units 210-3 and 210-7 are arranged to face each other on a straight line 254 passing through the object 231. Imaging and illumination units 210-4 and 210-8 are arranged to face each other on a straight line 255 passing through the object 231.

[0141] Even in such cases, by applying this technology, the occurrence of so-called light spots, ghosting, halos, etc., can be suppressed by using the polarizer described above.

[0142] For example, such as Figure 10 As shown, multiple imaging and illumination units 210 (imaging unit 131 and illumination unit 132) can be arranged in a cylindrical shape with a vertical line 261 passing through the object 231 as the central axis. Even in this case, by applying this technique, the occurrence of so-called light spots, ghosting, halos, etc., can be suppressed by using the polarizer as described above.

[0143] For example, such as Figure 11As shown, multiple imaging and illumination units 210 (imaging unit 131 and illumination unit 132) can be arranged in a spherical (or hemispherical) shape centered on object 231. Even in such a case, by applying this technology, the occurrence of so-called light spots, ghosting, halos, etc., can be suppressed by using the polarizer as described above.

[0144] <Contour Extraction>

[0145] For example, such as Figure 12 As shown in A, in the captured image 301 generated by the RGB imaging device 211, the object 311, as the subject, appears together with the IR light 213 of another imaging and illumination unit 210 (including within the viewing angle). In this case, since the captured image 301 is a captured image in the wavelength region of visible light, no light spots as indicated by ellipse 321 or ellipse 322 will occur due to direct light (infrared light) from the IR light 213.

[0146] On the other hand, such as Figure 12 As shown in Figure B, the IR imaging device 212 generates a captured image 331 in the wavelength region of infrared light. The RGB imaging device 211 and the IR imaging device 212 have substantially the same viewing angle. Therefore, the IR light 213 from the other imaging and illumination unit 210 also appears in the captured image 331 (and is included within the viewing angle). Therefore, without the application of this technique, in the captured image 331, a light spot (ellipse 321 or ellipse 322) occurs due to the direct light (infrared light) from the IR light 213. Therefore, it is difficult to accurately extract the outline of the object 311.

[0147] By applying this technology, the IR camera device 212 can suppress direct light from the IR light 213 by using a polarizer whose polarization direction is different from that of the polarizer of the IR light 213, and can generate light such as... Figure 12 The captured image 331 shown in C. That is, the occurrence of so-called light spots, ghosting, halos, etc. can be suppressed. Therefore, the contour can be extracted more accurately from the captured image 331, and thus the 3D model generation unit 102 can generate a more accurate 3D model (and can suppress the reduction of the accuracy of the 3D model).

[0148] <2. Second Implementation Method>

[0149] <Calibration>

[0150] The polarization directions of the polarizers in imaging unit 131 and illumination unit 132 can be calibrated (adjusted). As described above, the amount of light suppressed by the polarizer of imaging unit 131 varies depending on the relative angle between the polarization directions of the polarizers in imaging unit 131 and illumination unit 132. That is, the degree of suppression of so-called light spots, ghosting, halos, etc., changes. Therefore, for example, the polarization direction of each polarizer can be calibrated so that the relative angle becomes an appropriate angle depending on the position or orientation of the imaging unit 131 or illumination unit 132 (which can further suppress the occurrence of so-called light spots, ghosting, halos, etc.).

[0151] <Data Acquisition Unit>

[0152] Figure 13 This is a block diagram illustrating a main configuration example of the data acquisition unit 101 in this scenario. For example... Figure 13 As shown, except Figure 3 In addition to the configuration, the data acquisition unit 101 also includes a calibration processing unit 401 and a display unit 402.

[0153] The calibration processing unit 401 is an example of a calibration device that calibrates the polarization direction of a polarization filter, acquires a captured image generated by the imaging unit 131, and derives a more suitable polarization direction based on the captured image (a polarization direction that can further suppress the occurrence of so-called spotting, ghosting, halos, etc.). Furthermore, the calibration processing unit 401 generates a display image indicating the obtained polarization direction and provides this display image to the display unit 402.

[0154] Display unit 402 displays the image provided by calibration processing unit 401. The user refers to the image displayed on display unit 402 to determine the polarization direction, which can further suppress the occurrence of so-called spotting, ghosting, halos, etc. The polarization directions of the polarizers in imaging unit 131 and illumination unit 132 are variable, and both units have polarization direction control mechanisms (such as movable rings) to control the polarization direction of the polarizers. The user operates the polarization direction control mechanism to calibrate the polarization direction to the desired direction.

[0155] <Calibration Process>

[0156] Reference Figure 14 The flowchart describes an example of the calibration process performed by such a calibration processing unit 401.

[0157] When the calibration process begins, in step S201, the calibration processing unit 401 acquires the captured image.

[0158] In step S202, the user sets the polarization direction (polarization angle) of the polarizers of the imaging unit 131 and the illumination unit 132 to a predetermined direction (angle) that is different from the previous direction.

[0159] In step S203, the calibration processing unit 401 calculates the brightness value of the acquired captured image.

[0160] In step S204, the calibration processing unit 401 determines whether the polarization direction (polarization angle) has been set to all possible directions (angles). That is, the calibration processing unit 401 acquires captured images in all possible polarization directions (polarization angles) and determines whether the brightness value has been calculated.

[0161] If it is determined that there is an unprocessed direction (angle), the process returns to step S202. That is, imaging is performed in the new polarization direction (polarization angle), and the brightness value of the captured image is calculated. As described above, if it is determined that processing has been performed on all possible directions (angles), the process proceeds to step S205.

[0162] In step S205, the calibration processing unit 401 determines the polarization direction (polarization angle) in which the brightness value is minimized from the polarization direction (polarization angle) calculated from the brightness value of the captured image, and generates a display image indicating the polarization direction (polarization angle). The display unit 402 displays the display image.

[0163] Therefore, the user can calibrate the polarization direction of the polarizers of the imaging unit 131 and the illumination unit 132 to a more appropriate direction based on the display. This further suppresses the occurrence of so-called light spots, ghosting, halos, etc.

[0164] Note that a polarization direction control unit (actuator) may be provided to update the polarization direction of the polarizers of the imaging unit 131 and the illumination unit 132 to the polarization direction obtained by the calibration processing unit 401.

[0165] <3. Application Examples>

[0166] Note that this may also include a device for detecting the tilt of the camera device through camera device calibration. The position of the calibrated camera device is expressed in terms of rotation and translation relative to a certain origin. Furthermore, a device may be provided that changes the angle of the polarizing filter upon detection of rotation of the camera device.

[0167] In addition, in such Figure 5In the case where a polarization sensor is used as imaging unit 131 as in example B, a sensor capable of imaging multiple deflection directions (e.g., four directions such as 0°, 90°, 180°, and 270°) can be used. For example, rotation information of an automatically controlled light source can be acquired, and the polarization sensor can select pixels in the optimal polarization direction based on the open information, and generate a captured image corresponding to the polarized light in the polarization direction.

[0168] Furthermore, a device can be provided that controls the deflection angle of the light source based on the position of the camera device. For example, when an imaging unit whose imaging position can be changed (such as a drone or a boom camera) is introduced into the surrounding volume imaging environment, there is a possibility that the light may change position due to movement. Therefore, the polarization direction (polarization angle) of the light side can be controlled using the position and rotation information of the camera device.

[0169] <4. Appendix>

[0170] <Application Examples of This Technology>

[0171] The technology based on this disclosure can be applied to a variety of products and services.

[0172] (1. Content creation)

[0173] For example, new video content can be created by combining the 3D model of the subject generated in this embodiment with 3D data managed by another server. Furthermore, for example, in the presence of background data acquired by an imaging device such as LiDAR, content that appears as if the subject is located at the position indicated by the background data can be generated by combining the 3D model of the subject generated in this embodiment and the background data. Note that the video content can be three-dimensional video content or two-dimensional video content converted to two dimensions. Note that examples of the 3D model of the subject generated in this embodiment include 3D models generated by a 3D model generation unit and 3D models reconstructed by a rendering unit.

[0174] (2. Experiences in virtual spaces)

[0175] For example, the subject generated in this embodiment (e.g., a performer) can be arranged in a virtual space where the user interacts as an avatar. In this case, the user has an avatar and can view the live images of the subject in the virtual space.

[0176] (3. Applications in remote location communication)

[0177] For example, by sending a 3D model of the subject generated by the 3D model generation unit 102 from the sending unit 104 to a remote location, a user at the remote location can view the 3D model of the subject through a playback device at the remote location. For example, by sending the 3D model of the subject in real time, the subject and the user at the remote location can communicate with each other in real time. For example, it can be assumed that the subject is a teacher and the user is a student, or that the subject is a doctor and the user is a patient.

[0178] (4. Other)

[0179] For example, free-viewpoint videos such as sports can be generated based on the 3D models of multiple subjects generated in this embodiment, or an individual can distribute themselves, using the 3D models generated in this embodiment, to a distribution platform. As described above, the content of the embodiments described in this specification can be applied to various technologies and services.

[0180] Computer

[0181] The above series of processes can be performed either by hardware or by software. In the case where the processes are performed by software, the program constituting the software is installed in the computer. Here, "computer" includes computers integrated with dedicated hardware, such as general-purpose personal computers that can perform various functions by installing various programs.

[0182] Figure 15 This is a block diagram illustrating an example configuration of computer hardware that performs the above series of processes through a program.

[0183] exist Figure 15 In the computer 900 shown, the central processing unit (CPU) 901, read-only memory (ROM) 902 and random access memory (RAM) 903 are interconnected via bus 904.

[0184] The input / output interface 910 is also connected to the bus 904. The input unit 911, output unit 912, storage unit 913, communication unit 914, and driver 915 are connected to the input / output interface 910.

[0185] Input unit 911 includes, for example, a keyboard, mouse, microphone, touchpad, input terminals, etc. Output unit 912 includes, for example, a display, speaker, output terminals, etc. Storage unit 913 includes, for example, a hard disk, RAM disk, non-volatile memory, etc. Communication unit 914 includes, for example, a network interface. Driver 915 drives removable media 921 such as a hard disk, optical disk, magneto-optical disk, or semiconductor memory.

[0186] In the computer configured as described above, the CPU 901 loads a program, for example, stored in storage unit 913, into RAM 903 via input / output interface 910 and bus 904, and executes the program to perform the aforementioned series of processes. RAM 903 also appropriately stores data, etc., required by the CPU 901 to perform various processes.

[0187] For example, a program executed by a computer can be applied by being recorded in a removable medium 921, such as a packaging medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by attaching the removable medium 921 to the drive 915.

[0188] Furthermore, the program can be provided via wired or wireless transmission media such as a local area network, the Internet, or digital satellite broadcasting. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0189] Alternatively, the program can be pre-installed in ROM 902 or storage unit 913.

[0190] <Application Objectives of This Technology>

[0191] Furthermore, although information processing systems and other similar applications have been described above as examples of this technology, this technology can be applied to any configuration.

[0192] For example, this technology can be applied to various electronic devices, such as transmitters and receivers for satellite broadcasting, cable broadcasting such as cable television, distribution over the Internet, and distribution to terminals via cellular communication (e.g., television receivers and mobile phones), or devices for recording images on media such as optical discs, magnetic disks, and flash memory, or for reproducing images from storage media (e.g., hard disk recorders and cameras).

[0193] Furthermore, this technology can also be implemented as a configuration of a device, such as a processor (e.g., a video processor) in a system-wide integrated circuit (LSI), a module (e.g., a video module) using multiple processors, a unit (e.g., a video unit) using multiple modules, or a device obtained by further adding other functions to the unit (e.g., a video device).

[0194] Furthermore, this technology can also be applied to network systems that include multiple devices. For example, this technology can be implemented as cloud computing where multiple devices collaborate and share data via a network. For example, this technology can be implemented in cloud services that provide image (moving image) related services to any terminal such as computers, audiovisual (AV) devices, portable information processing terminals, or Internet of Things (IoT) devices.

[0195] Note that in this specification, "system" means a collection of multiple components (devices, modules (parts), etc.), and it is not important whether all components are housed in the same housing. Therefore, multiple devices housed in different housings and connected via a network, as well as multiple modules housed in one housing, are all systems.

[0196] <Fields and Applications for which this technology is applicable>

[0197] Note that systems, devices, processing units, etc., applying this technology can be used in any field, such as transportation, medical care, crime prevention, agriculture, animal husbandry, mining, beauty, factories, home appliances, weather, nature monitoring, etc. Furthermore, its use is arbitrary.

[0198] <Other>

[0199] The implementation of this technology is not limited to the above-described implementation, and various modifications can be made without departing from the spirit of this technology.

[0200] For example, a configuration described as a single device (or processing unit) can be divided and configured into multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) can be combined and configured into a single device (or processing unit). Furthermore, configurations other than those described above can, of course, be added to the configuration of each device (or each processing unit). Additionally, if the configuration and operation of the entire system are substantially the same, a portion of the configuration of one device (or processing unit) can be included in the configuration of another device (or another processing unit).

[0201] Furthermore, for example, the above procedure can be executed in any device. In this case, it is sufficient as long as the device has the necessary functions (function blocks, etc.) and can obtain the necessary information.

[0202] Furthermore, for example, each step of a flowchart can be executed by a single device, or it can be shared and executed by multiple devices. Additionally, when a step includes multiple processes, these processes can be executed by a single device, or they can be shared and executed by multiple devices. In other words, multiple processes included in a step can be executed as processes of multiple steps. Conversely, processes described as multiple steps can be executed together as a single step.

[0203] Furthermore, for example, in a program executed by a computer, the processes in the steps describing the program can be executed sequentially in the order described in this specification, or they can be executed in parallel or separately at necessary time intervals (e.g., when a call is made). That is, as long as there is no contradiction, the processes in the corresponding steps can be executed in an order different from the above-described order. In addition, the processes in the steps describing the program can be executed in parallel with the processes in another program, or they can be combined with the processes in another program.

[0204] Furthermore, for example, multiple technologies related to this technology can be implemented independently as a single entity, provided there are no contradictions. Of course, any multiple technologies in this technology can also be combined and implemented. For example, part or all of the technology described in any embodiment can be implemented in combination with part or all of the technology described in other embodiments. Furthermore, any part or all of the technologies described above can be implemented by using them in conjunction with other technologies not described above.

[0205] Note that this technology can have the following configurations.

[0206] (1) An imaging processing system that generates a three-dimensional (3D) model of an object using a plurality of captured images obtained by imaging the object, said imaging processing system comprising:

[0207] Multiple polarized illumination devices, each including a polarizer, illuminate the object from different positions using polarized light obtained by transmitting light emitted from a light-emitting unit through the polarizer; and

[0208] Multiple polarization imaging devices, each including a polarizer, generate the captured image at different locations using polarized light obtained by transmitting light from the outside through the polarizer, wherein at least one of the polarization illumination devices and the object are within the viewing angle at these different locations.

[0209] The polarization direction of the polarizer in the polarization imaging device is different from that of the polarization direction of the polarization illumination device.

[0210] (2) The imaging processing system according to (1), wherein,

[0211] Another polarization imaging device among the plurality of polarization imaging devices is located within the field of view of the polarization imaging device.

[0212] (3) The imaging processing system according to (2), wherein,

[0213] The other polarization imaging device faces the polarization imaging device.

[0214] (4) The imaging processing system according to (3), wherein,

[0215] The polarization illumination device is positioned close to any one of the polarization imaging devices, and the direction of the polarization light is the same as the imaging direction of the polarization imaging device.

[0216] (5) The imaging processing system according to any one of (1) to (4), wherein,

[0217] The plurality of polarization illumination devices are located within the field of view of the polarization imaging device.

[0218] (6) The imaging processing system according to any one of (1) to (5), wherein,

[0219] The plurality of polarization illumination devices includes a first polarization illumination device and a second polarization illumination device.

[0220] The plurality of polarization imaging devices include: a first polarization imaging device located at a position where the object and the first polarization illumination device are within the viewing angle; and a second polarization imaging device located at a position where the object and the second polarization illumination device are within the viewing angle;

[0221] The polarization direction of the polarizer of the first polarization imaging device is different from the polarization direction of the polarizer of the first polarization illumination device, and

[0222] The polarization direction of the polarizer of the second polarization imaging device is different from that of the polarizer of the second polarization illumination device.

[0223] (7) The imaging processing system according to (6), wherein,

[0224] The polarization direction of the polarizer of the first polarization illumination device is different from that of the polarizer of the second polarization illumination device.

[0225] (8) The imaging processing system according to any one of (1) to (7), wherein,

[0226] The plurality of polarization imaging devices and the plurality of polarization illumination devices are arranged to surround the object.

[0227] (9) The imaging processing system according to (8), wherein,

[0228] The plurality of polarization imaging devices and the plurality of polarization illumination devices are arranged in a circular shape centered on the object.

[0229] (10) The imaging processing system according to (8), wherein,

[0230] The plurality of polarization imaging devices and the plurality of polarization illumination devices are arranged in a cylindrical shape with a vertical line passing through the object as the central axis.

[0231] (11) The imaging processing system according to (8), wherein,

[0232] The plurality of polarization imaging devices and the plurality of polarization illumination devices are arranged in a spherical shape centered on the object.

[0233] (12) The imaging processing system according to any one of (1) to (11), wherein,

[0234] The polarization imaging device generates a captured image for extracting the contour of the object.

[0235] (13) The imaging processing system according to any one of (1) to (12) further includes:

[0236] An imaging device that images the object and generates a captured image for extracting the texture of the object.

[0237] (14) The imaging processing system according to any one of (1) to (13), wherein,

[0238] The polarized illumination device emits polarized visible light, and

[0239] The polarization imaging device generates the captured image by using polarized light from the visible light source.

[0240] (15) The imaging processing system according to any one of (1) to (13), wherein,

[0241] The polarized illumination device emits invisible polarized light, and

[0242] The polarization imaging device generates the captured image by using polarized light of the invisible light.

[0243] (16) The imaging processing system according to any one of (1) to (15), wherein,

[0244] The polarizer is a polarizing filter.

[0245] (17) The imaging processing system according to (16), wherein,

[0246] The polarization direction of the polarization filter is variable, and

[0247] The polarization imaging device further includes a polarization direction control mechanism, which controls the polarization direction of the polarization filter.

[0248] (18) The imaging processing system according to (17) further includes:

[0249] A calibration device that calibrates the polarization direction of the polarization filter.

[0250] (19) A method for generating a three-dimensional (3D) model, comprising:

[0251] Using polarized light whose polarization direction differs from that of the polarized light emitted from the polarized illumination device within the viewpoint, capture images of the object are generated at different locations; and

[0252] A 3D model of the object is generated using multiple captured images obtained at different locations.

[0253] (20) An imaging processing system, comprising:

[0254] Multiple polarized illumination devices, each including a polarizer, illuminate an object from different positions using polarized light obtained by transmitting light emitted from a light-emitting unit through the polarizer; and

[0255] Multiple polarization imaging devices, each including a polarizer, generate captured images of an object at different locations using polarized light obtained by transmitting light from the outside through the polarizer, wherein at least one of the polarization illumination devices and the object are within the viewing angle at said different locations.

[0256] The polarization direction of the polarizer in the polarization imaging device is different from that of the polarization direction of the polarization illumination device.

[0257] Reference tag list

[0258] 100 Information Processing System

[0259] 101 Data Acquisition Unit

[0260] 102 3D model generation units

[0261] 103 formatted unit

[0262] 104 Transmitting Unit

[0263] 105 receiving unit

[0264] 106 rendering units

[0265] 107 display units

[0266] 121 Imaging Illumination Unit

[0267] 122 Transmitting Unit

[0268] 131 imaging units

[0269] 132 lighting units

[0270] 151 polarization filter

[0271] 152 light-emitting units

[0272] 171 polarization filter

[0273] 172 image sensor

[0274] 191 polarization sensor

[0275] 210 Imaging and Illumination Unit

[0276] 211RGB camera device

[0277] 212IR camera device

[0278] 213IR light

[0279] 231 object

[0280] 401 Calibration Processing Unit

[0281] 402 Display Unit

Claims

1. An imaging processing system that generates a three-dimensional (3D) model of an object using a plurality of captured images obtained by imaging the object, the imaging processing system comprising: Multiple polarized illumination devices, each including a polarizer, illuminate the object from different positions using polarized light obtained by transmitting light emitted from the light-emitting unit through the polarizer; as well as Multiple polarization imaging devices, each including a polarizer, generate the captured image at different locations using polarized light obtained by transmitting light from the outside through the polarizer, wherein at least one of the polarization illumination devices and the object are within the field of view at said different locations. The plurality of polarization illumination devices includes a first polarization illumination device and a second polarization illumination device. The plurality of polarization imaging devices includes: a first polarization imaging device located at a position where the object and the first polarization illumination device are within the viewing angle, and a second polarization imaging device located at a position where the object and the second polarization illumination device are within the viewing angle. The polarization direction of the polarizer in the first polarization imaging device is different from the polarization direction of the polarizer in the first polarization illumination device. The polarization direction of the polarizer in the second polarization imaging device is different from the polarization direction of the polarizer in the second polarization illumination device, and The polarization direction of the polarizer of the first polarization illumination device is different from that of the polarizer of the second polarization illumination device.

2. The imaging processing system according to claim 1, wherein, Another polarization imaging device among the plurality of polarization imaging devices is located within the field of view of the polarization imaging device.

3. The imaging processing system according to claim 2, wherein, The other polarization imaging device faces the polarization imaging device.

4. The imaging processing system according to claim 3, wherein, The polarization illumination device is positioned close to any one of the polarization imaging devices, and the direction of the polarization light is the same as the imaging direction of the polarization imaging device.

5. The imaging processing system according to claim 1, wherein, The plurality of polarization illumination devices are located within the field of view of the polarization imaging device.

6. The imaging processing system according to claim 1, wherein, The plurality of polarization imaging devices and the plurality of polarization illumination devices are arranged to surround the object.

7. The imaging processing system according to claim 6, wherein, The plurality of polarization imaging devices and the plurality of polarization illumination devices are arranged in a circular shape centered on the object.

8. The imaging processing system according to claim 6, wherein, The plurality of polarization imaging devices and the plurality of polarization illumination devices are arranged in a cylindrical shape with a vertical line passing through the object as the central axis.

9. The imaging processing system according to claim 6, wherein, The plurality of polarization imaging devices and the plurality of polarization illumination devices are arranged in a spherical shape centered on the object.

10. The imaging processing system according to claim 1, wherein, The polarization imaging device generates a captured image for extracting the contour of the object.

11. The imaging processing system according to claim 1, further comprising: An imaging device that images the object and generates a captured image for extracting the texture of the object.

12. The imaging processing system according to claim 1, wherein, The polarized illumination device emits polarized visible light, and The polarization imaging device generates the captured image by using polarized light from the visible light source.

13. The imaging processing system according to claim 1, wherein, The polarized illumination device emits invisible polarized light, and The polarization imaging device generates the captured image by using polarized light of the invisible light.

14. The imaging processing system according to claim 1, wherein, The polarizer is a polarizing filter.

15. The imaging processing system according to claim 14, wherein, The polarization direction of the polarization filter is variable, and The polarization imaging device further includes a polarization direction control mechanism, which controls the polarization direction of the polarization filter.

16. The imaging processing system according to claim 15, further comprising: A calibration device that calibrates the polarization direction of the polarization filter.

17. A method for generating a three-dimensional (3D) model, comprising: An object is illuminated by multiple polarized illumination devices, each of which includes a polarizer, and the object is illuminated from different positions by means of polarized light obtained by transmitting light emitted from a light-emitting unit through the polarizer; Multiple captured images of the object are generated by multiple polarization imaging devices, each including a polarizer, and the captured images are generated at different locations using polarized light obtained by transmitting light from the outside through the polarizer, wherein at least one of the polarization illumination devices and the object are within the field of view at said different locations; and A 3D model of the object is generated using the plurality of captured images obtained at the different locations they are located at, wherein... The plurality of polarization illumination devices includes a first polarization illumination device and a second polarization illumination device. The plurality of polarization imaging devices includes: a first polarization imaging device located at a position where the object and the first polarization illumination device are within the viewing angle, and a second polarization imaging device located at a position where the object and the second polarization illumination device are within the viewing angle. The polarization direction of the polarizer in the first polarization imaging device is different from the polarization direction of the polarizer in the first polarization illumination device. The polarization direction of the polarizer in the second polarization imaging device is different from the polarization direction of the polarizer in the second polarization illumination device, and The polarization direction of the polarizer of the first polarization illumination device is different from that of the polarizer of the second polarization illumination device.

18. An imaging processing system, comprising: Multiple polarized illumination devices, each including a polarizer, illuminate an object from different positions using polarized light obtained by transmitting light emitted from a light-emitting unit through the polarizer; as well as Multiple polarization imaging devices, each including a polarizer, generate captured images of the object at different locations using polarized light obtained by transmitting light from the outside through the polarizer, wherein at least one of the polarization illumination devices and the object are within the viewing angle at said different locations. The plurality of polarization illumination devices includes a first polarization illumination device and a second polarization illumination device. The plurality of polarization imaging devices includes: a first polarization imaging device located at a position where the object and the first polarization illumination device are within the viewing angle, and a second polarization imaging device located at a position where the object and the second polarization illumination device are within the viewing angle. The polarization direction of the polarizer in the first polarization imaging device is different from the polarization direction of the polarizer in the first polarization illumination device. The polarization direction of the polarizer in the second polarization imaging device is different from the polarization direction of the polarizer in the second polarization illumination device, and The polarization direction of the polarizer of the first polarization illumination device is different from that of the polarizer of the second polarization illumination device.

Images