Image processing system and 3D model generation method

Abstract

The present disclosure relates to an image-capture processing system and a 3D model generating method for generating a 3D model with increased accuracy. Provided is an image-capture processing system for generating a 3D model of an object using a plurality of captured images obtained by imaging the object, the image-capture processing system being provided with: a plurality of polarized illumination devices each having a polarizer and irradiating, from mutually different positions, the object with polarized light obtained as light emitted from a light emitter is transmitted through the polarizer; and a plurality of polarized image-capture devices each having a polarizer and generating, at mutually different positions with the object and at least one of the polarized illumination devices being located within a field angle, a captured image using polarized light obtained as external light is transmitted through the polarizer. The direction of polarization of the polarizers of the polarized image-capture devices is different from the direction of polarization of the polarizers of the polarized illumination devices. The present disclosure may be applied to, for example, image-capture processing systems and 3D model generating methods.

Description

[Technical Field]

[0001] The present disclosure relates to an image processing system and a 3D model generation method, and more particularly to an image processing system and a 3D model generation method that enable more accurate 3D models to be generated. [Background technology]

[0002] There is a technology that generates a 3D model (a model containing three-dimensional information about a subject) from video images captured from multiple viewpoints, and then generates free viewpoint video, which is video corresponding to any viewpoint position, based on the 3D model. This technology is also called volumetric capture technology.

[0003] For example, a technique for generating a 3D model using a method such as Visual Hull, which carves out the three-dimensional shape of a subject based on multiple captured images taken from different directions, has been proposed (see, for example, Patent Document 1).

[0004] When capturing an image of a subject using such volumetric capture technology, a lighting device is generally used to irradiate the subject and its surroundings with light in order to ensure brightness. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2018 / 150933 Summary of the Invention [Problem to be solved by the invention]

[0006] However, with volumetric capture technology, multiple image capture devices are placed around the subject to capture images from a wider variety of directions, resulting in fewer blind spots and making it easier for lighting devices to fit within the field of view. When a high-intensity lighting device fits within the field of view, optical phenomena such as flare, ghosting, and halation are likely to occur. The occurrence of such optical phenomena could reduce the accuracy of the 3D model generated from the captured images.

[0007] The present disclosure has been made in light of these circumstances, and aims to generate more accurate 3D models. [Means for solving the problem]

[0008] An image processing system according to one aspect of the present technology is an image processing system that generates a 3D model of an object using a plurality of captured images obtained by capturing an image of the object, the image processing system including a polarizer, and irradiating the object with polarized light obtained by transmitting light emitted from a light emitting unit through the polarizer from different positions. a first polarized illumination device and a second polarized illumination device; a polarized illumination device and a polarizer, The first The polarized lighting device is within the field of view. position In the method, the captured image is generated using polarized light obtained by transmitting external light through the polarizer. a second polarized light imaging device that includes a first polarized light imaging device and a polarizer, and that generates the captured image using polarized light obtained by transmitting external light through the polarizer at a position where the object and the second polarized light imaging device are within an angle of view; a polarization imaging device, The first The polarization direction of the polarizer of the polarization imaging device is The polarization direction of the polarizer of the first polarized illumination device is different from the polarization direction of the polarizer of the second polarized illumination device, and the polarization direction of the polarizer of the first polarized illumination device is different from the polarization direction of the polarizer of the second polarized illumination device. The polarization direction of the polarizer of the polarized illumination device is different from that of the imaging processing system.

[0009] Another aspect of the present technology is a 3D model generation method, Angle of view Within No. 1 Illuminated by a polarized lighting device First polarization and a polarization direction different from Second Polarization Use the object First captured image Generate a second captured image of the object is generated at a position different from the position where the first captured image is generated, using fourth polarized light irradiated from a second polarized lighting device within an angle of view and having a polarization direction different from that of the third polarized light; Generate a 3D model of the object using The polarization direction of the third polarized light is different from the polarization direction of the first polarized light. A method for generating 3D models.

[0010] According to yet another aspect of the present technology, there is provided an image processing system including a polarizer, and irradiating an object with polarized light obtained by transmitting light emitted from a light emitting unit through the polarizer from different positions. a first polarized illumination device and a second polarized illumination device; a polarized illumination device and a polarizer, The first The polarized lighting device is within the field of view. position In the method, an image of the object is generated using polarized light obtained by transmitting external light through the polarizer. a second polarized light imaging device that includes a first polarized light imaging device and a polarizer, and that generates the captured image using polarized light obtained by transmitting external light through the polarizer at a position where the object and the second polarized light imaging device are within an angle of view; a polarization imaging device, The first The polarization direction of the polarizer of the polarization imaging device is The polarization direction of the polarizer of the first polarized illumination device is different from the polarization direction of the polarizer of the second polarized illumination device, and the polarization direction of the polarizer of the first polarized illumination device is different from the polarization direction of the polarizer of the second polarized illumination device. The polarization direction of the polarizer of the polarized illumination device is different from that of the imaging processing system.

[0011] According to one aspect of the present technology, an imaging processing system for generating a 3D model of an object using a plurality of captured images of the object includes a polarizer. a first polarized illumination device and a second polarized illumination device; The polarized lighting device illuminates the light emitted from the light-emitting unit. Different polarization directions The polarized light obtained by passing through the polarizer is irradiated onto the object from different positions, No. 1 The polarization imaging device detects the object and No. 1 The polarized lighting device is within the field of view. At this position, the captured image is generated using polarized light obtained by transmitting external light through a polarizer having a polarization direction different from that of the polarizer of the first polarized illumination device, and the object and the second polarized illumination device are within the angle of view by a second polarization imaging device equipped with a polarizer. At this position, external light The polarization direction is different from that of the polarizer of the second polarized illumination device The polarized light obtained by passing through the polarizer is used to generate the captured image.

[0012] In another aspect of the present technology, a 3D model generation method includes: Angle of view Within No. 1 Illuminated by a polarized lighting device First polarization and a polarization direction different from Second Polarization Use the object First captured image is generated, a second captured image of the object is generated at a position different from the position at which the first captured image was generated, using fourth polarized light irradiated from a second polarized illumination device within the angle of view and having a polarization direction different from that of the third polarized light; the First captured image and second captured image is used to generate a 3D model of the object. The polarization direction of the third polarized light and the polarization direction of the first polarized light are different from each other.

[0013] According to yet another aspect of the present technology, there is provided an imaging processing system including a polarizer, and light emitted from a light emitting unit is transmitted through the polarizer from different positions, and the polarized light thus obtained is irradiated onto an object. a first polarized illumination device and a second polarized illumination device; Polarized lighting device and polarization direction No. 1 The object is provided with a polarizer different from the polarizer of the polarized lighting device. No. 1 The polarized lighting device is within the field of view. position In this case, an image of the object is generated using polarized light obtained by transmitting external light through the polarizer. a second polarized light imaging device that includes a first polarized light imaging device and a polarizer whose polarization direction is different from that of the polarizer of the second polarized light imaging device, and that generates a captured image of the object using polarized light obtained when external light passes through the polarizer at a position where the object and the second polarized light imaging device are within the angle of view; and a polarization imaging device. The polarization direction of the polarizer of the first polarized illumination device and the polarization direction of the polarizer of the second polarized illumination device are different from each other. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a block diagram illustrating an example of a main configuration of an information processing system. [Figure 2] 10 is a flowchart illustrating an example of the flow of system processing. [Figure 3] FIG. 2 is a block diagram illustrating an example of the main configuration of a data acquisition unit. [Figure 4] FIG. 2 is a block diagram showing an example of the main configuration of an illumination unit. [Figure 5] FIG. 2 is a block diagram illustrating an example of the main configuration of an imaging unit. [Figure 6] FIG. 2 is a diagram illustrating an example of the configuration of an imaging illumination unit. [Figure 7] FIG. 10 is a diagram illustrating an example of the arrangement of an imaging illumination unit. [Figure 8] FIG. 10 is a diagram illustrating an example of the arrangement of an imaging illumination unit. [Figure 9] FIG. 10 is a diagram illustrating an example of the arrangement of an imaging illumination unit. [Figure 10] FIG. 10 is a diagram illustrating an example of the arrangement of an imaging illumination unit. [Figure 11] FIG. 10 is a diagram illustrating an example of the arrangement of an imaging illumination unit. [Figure 12] FIG. 10 is a diagram illustrating an example of a captured image. [Figure 13] FIG. 10 is a block diagram showing another example of the configuration of the data acquisition unit. [Figure 14]10 is a flowchart illustrating an example of the flow of a calibration process. [Figure 15] FIG. 1 is a block diagram illustrating an example of the main configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described in the following order. 1. First embodiment (information processing system) 2. Second embodiment (calibration) 3. Application Examples 4. Notes

[0016] <1. First embodiment> <Information Processing System> There is a volumetric capture technology that generates a 3D model (three-dimensional model) that is a model having three-dimensional information of a subject from video images captured from multiple viewpoints, and generates free viewpoint video images that are video images corresponding to any viewpoint position based on the 3D model. The information processing system 100 in Figure 1 is a system that uses such volumetric capture technology to capture video images of a subject from multiple viewpoints, generate a 3D model of the subject from the captured images, and generate free viewpoint images corresponding to any viewpoint position based on the 3D model.

[0017] As shown in FIG. 1, the information processing system 100 includes a data acquisition unit 101, a 3D model generation unit 102, a formatting unit 103, a transmission unit 104, a reception unit 105, a rendering unit 106, and a display unit 107.

[0018] The data acquisition unit 101 acquires image data for generating a 3D model of a subject. For example, the data acquisition unit 101 acquires, as image data, a plurality of viewpoint images captured by a plurality of imaging devices arranged to surround the subject. In this case, it is preferable that the plurality of viewpoint images are images obtained by synchronously capturing images by a plurality of imaging devices.

[0019] The data acquisition unit 101 may perform calibration based on image data to acquire internal and external parameters of each image capture device. The data acquisition unit 101 may also acquire, for example, multiple pieces of depth information indicating the distances from multiple viewpoints to the subject.

[0020] The data acquisition unit 101 supplies the acquired image data to the 3D model generation unit 102 .

[0021] The 3D model generation unit 102 generates a 3D model, which is a model having three-dimensional information of the subject, based on the image data supplied from the data acquisition unit 101. The 3D model generation unit 102 generates the 3D model of the subject by using, for example, a so-called visual hull to carve out the three-dimensional shape of the subject using images from multiple viewpoints (for example, silhouette images from multiple viewpoints).

[0022] Here, a silhouette image is an image that represents only the outline (outline) of a subject, and the area inside the outline is represented by being filled in with a single color, for example, like a shadow puppet. In other words, the 3D model generation unit 102 generates such a silhouette image from image data (captured image) supplied from the data acquisition unit 101. Note that the image data of the silhouette image may be supplied from the data acquisition unit 101 to the 3D model generation unit 102.

[0023] The 3D model generation unit 102 can further deform the 3D model generated using the visual hull with high precision using a plurality of pieces of depth information indicating the distance from a plurality of viewpoints to the subject.

[0024] The 3D model generated by the 3D model generation unit can also be called a video of the 3D model because it is generated frame by frame in a time series. Furthermore, since the 3D model is generated using images captured by the imaging device of the data acquisition unit 101, it can also be called a live-action 3D model. The 3D model can be expressed in the form of mesh data, which expresses shape information representing the surface shape of the subject using connections between vertices, known as a polygon mesh. The method of expressing the 3D model is not limited to these, and it may also be described using a so-called point cloud expression method, which expresses positional information of points.

[0025] Color information data is also generated as textures linked to these 3D shape data. For example, there are view-independent textures, which have a constant color no matter what direction you look at them from, and view-dependent textures, which change color depending on the viewing direction.

[0026] The 3D model generation unit 102 supplies the data of the generated 3D model to the formatting unit 103.

[0027] The formatting unit 103 converts the 3D model data supplied from the 3D model generation unit 102 into a format suitable for transmission and storage. For example, the formatting unit 103 may convert the 3D model generated by the 3D model generation unit 102 into multiple 2D images by perspectively projecting the 3D model from multiple directions. Furthermore, the formatting unit 103 may generate depth information, which is a 2D depth image from multiple viewpoints, using the 3D model. In this case, the formatting unit 103 may encode (compress) the depth information and color information in the state of this 2D image. In this case, the formatting unit 103 may encode the depth information and color information side by side as a single image, or may encode them as two separate images. Furthermore, since the depth information and color information are in the form of 2D image data, the formatting unit 103 may encode (compress) them using a 2D compression technology such as AVC (Advanced Video Coding).

[0028] In the above cases, the formatting unit 103 supplies the 3D model data to the transmitting unit 104 as transmission data consisting of 2D data (or encoded data thereof).

[0029] Furthermore, for example, the formatting unit 103 may convert 3D data of mesh data into a point cloud format and supply the 3D data as transmission data to the transmitting unit 104. In this case, the formatting unit 103 may encode (compress) the 3D data using, for example, a three-dimensional compression technique of the Geometry-based Approach discussed in MPEG.

[0030] The transmitting unit 104 transmits the transmission data formed by the formatting unit 103 to the receiving unit 105. The transmitting unit 104 performs a series of processes by the data acquiring unit 101, the 3D model generating unit 102, and the formatting unit 103 offline, and then transmits the transmission data to the receiving unit 105. The transmitting unit 104 may also transmit the transmission data generated by the above-mentioned series of processes to the receiving unit 105 in real time.

[0031] The receiving unit 105 receives the transmission data transmitted from the transmitting unit 104 and supplies it to the rendering unit 106 .

[0032] The rendering unit 106 performs rendering using the transmission data received by the receiving unit 105. For example, the rendering unit 106 projects the mesh of the 3D model from the viewpoint of the camera that renders it, and performs texture mapping to apply textures that represent colors and patterns. The rendering at this time can be set arbitrarily, regardless of the camera position at the time of shooting, and can be viewed from any viewpoint.

[0033] The rendering unit 106 performs texture mapping, which applies a texture representing the color, pattern, and texture of a mesh according to the position of the mesh of a 3D model. Texture mapping can be performed using a method called "view dependent," which takes into account the user's viewing viewpoint, or a method called "view independent," which does not. The view dependent method has the advantage of achieving higher quality rendering than the view independent method because it changes the texture applied to the 3D model depending on the position of the viewing viewpoint. On the other hand, the view independent method has the advantage of requiring less processing than the view dependent method because it does not take into account the position of the viewing viewpoint. Note that the viewing viewpoint data is detected by the display unit 107, where the user's viewing position (region of interest) is input from the display unit 107 to the rendering unit 106. The rendering unit 106 may also employ billboard rendering, which renders objects so that they maintain a vertical orientation relative to the viewing viewpoint. For example, when rendering multiple objects, the rendering unit 106 can render objects of low viewer interest using billboards and other objects using a different rendering method.

[0034] The rendering unit 106 supplies the data of the rendering result to the display unit 107 .

[0035] The display unit 107 displays the result of rendering by the rendering unit 106 on the display unit of the display device. The display device may be, for example, a 2D monitor or a 3D monitor such as a head-mounted display, a spatial display, a mobile phone, a television, or a PC (Personal Computer).

[0036] <System processing flow> An example of the flow of system processing executed by the information processing system 100 will be described with reference to the flowchart of FIG.

[0037] When the process starts, in step S101, the data acquisition unit 101 acquires image data for generating a 3D model of a subject.

[0038] In step S102, the 3D model generating unit 102 generates a 3D model, which is a model having three-dimensional information of the subject, based on the image data acquired in step S101.

[0039] In step S103, the formatting unit 103 encodes the shape and texture data of the 3D model generated in step S102 into a format suitable for transmission and storage.

[0040] In step S104, the transmitting unit 104 transmits the coded data generated in step S103.

[0041] In step S105, the receiving unit 105 receives the data transmitted in step S104.

[0042] In step S106, the rendering unit 106 performs a decoding process to convert the data into shape and texture data required for display, and then performs rendering using the shape and texture data.

[0043] In step S107, the display unit 107 displays the rendering result.

[0044] When the process of step S107 ends, the system process ends.

[0045] By performing each process as described above, the information processing system 100 can generate a 3D model of the subject and generate and display an image of the subject viewed from any viewpoint, allowing the viewer, or user, to view the subject from any viewpoint.

[0046] <Modification> The above description has shown a series of processes in the information processing system 100, from the data acquisition unit 101 that acquires captured images, which are materials for generating content, to the display unit 107 that displays the images viewed by the user. However, this does not mean that all functional blocks are required to implement the present invention; the present invention can be implemented for each functional block or a combination of multiple functional blocks. For example, in FIG. 1, the transmitter 104 and receiver 105 are provided to show a series of processes from the side that creates content to the side that views the content through the distribution of content data, but the entire process from content creation to viewing can also be implemented on the same information processing device (e.g., a personal computer). In this case, the formatter 103, transmitter 104, and receiver 105 can be omitted.

[0047] Furthermore, when implementing the information processing system 100, the same implementer may implement all of the functions, or different implementers may implement each functional block. For example, an operator A may implement the data acquisition unit 101, the 3D model generation unit 102, and the formatting unit 103 to generate 3D content, an operator B may implement the transmission unit 104 (platform) to distribute the 3D content, and an operator C may implement the reception unit 105, the rendering unit 106, and the display unit 107 to receive, render, control the display of, and the like of the 3D content.

[0048] Furthermore, each functional block can be implemented on the cloud. For example, the rendering unit 106 may be implemented within the display device or on a server. In this case, information is exchanged between the display device and the server.

[0049] 1, the data acquisition unit 101, the 3D model generation unit 102, the formatting unit 103, the transmission unit 104, the reception unit 105, the rendering unit 106, and the display unit 107 are collectively described as the information processing system 100. However, the configuration of the information processing system 100 is not limited to this example, and it is sufficient if the information processing system 100 includes at least the data acquisition unit 101. For example, among the configuration shown in FIG. 1, any one or more of the 3D model generation unit 102 to the display unit 107 may be omitted. Furthermore, the information processing system 100 may have a configuration (functional block) other than the above-described configuration.

[0050] Furthermore, each of the above-described functional blocks (data acquisition unit 101 to display unit 107) may be realized by any configuration. For example, each functional block may be realized by one or more devices (apparatuses). Furthermore, multiple functional blocks may be realized by one device (apparatus).

[0051] <Data Acquisition Section> Fig. 3 is a block diagram showing an example of the main configuration of the data acquisition unit 101 in Fig. 1. The data acquisition unit 101, which is an embodiment of an imaging processing system to which the present technology is applied, includes an imaging illumination unit 121 and a transmission unit 122, as shown in Fig. 3.

[0052] The image capturing and lighting unit 121 captures an image of a subject and illuminates the subject. The image capturing and lighting unit 121 has image capturing units 131-1 to 131-M and lighting units 132-1 to 132-N (M and N are integers equal to or greater than 2). When it is not necessary to distinguish between the image capturing units 131-1 to 131-M, they will be referred to as the image capturing unit 131. When it is not necessary to distinguish between the lighting units 132-1 to 132-N, they will be referred to as the lighting unit 132.

[0053] That is, the image capturing and lighting section 121 has a plurality of image capturing sections 131 and a plurality of lighting sections 132. Note that the number of image capturing sections 131 and the number of lighting sections 132 included in the image capturing and lighting section 121 may be the same (that is, M=N) or may be different from each other.

[0054] The imaging unit 131 is configured with one or more imaging devices, captures an image of a subject, and generates a captured image for generating a 3D model. That is, the imaging unit 131 generates a captured image used to extract the silhouette and texture of the subject. The imaging unit 131 supplies data of the generated captured image to the transmission unit 122.

[0055] The wavelength band of light received by the imaging device of the imaging unit 131 is arbitrary, and may be visible light or invisible light. For example, the imaging unit 131 may receive visible light (RGB light) and generate a captured image of visible light, or may receive infrared light (IR (InfraRed) light) and generate a captured image of infrared light.

[0056] The illumination unit 132 is composed of one or more illumination devices, and illuminates the subject to be imaged by the imaging unit 131. The wavelength band of light emitted by the illumination devices of the illumination unit 132 is arbitrary, and may be visible light or invisible light. For example, the illumination unit 132 may illuminate the subject with visible light (RGB light) or infrared light (IR light).

[0057] The transmission unit 122 transmits the captured image data supplied from the imaging unit 131 to the 3D model generation unit 102. At this time, the transmission unit 122 may supply the captured image data to the 3D model generation unit 102 without encoding it, or may encode the captured image data and supply it to the 3D model generation unit 102 as encoded data. The transmission unit 122 may also perform any image processing on the captured image. For example, the transmission unit 122 may extract a silhouette or texture from the captured image and supply the extracted silhouette or texture data to the 3D model generation unit 102.

[0058] Generally, it is difficult to capture an image of a subject when the brightness is too low (too dark). Therefore, by having the illumination unit 132 illuminate the subject, the imaging unit 131 can capture the subject under sufficient brightness, and obtain a captured image with sufficient brightness.

[0059] However, in the case of the data acquisition unit 101 that acquires captured images for generating a 3D model, the multiple image capture units 131 are arranged around the subject so as to reduce blind spots. Therefore, it is highly likely that the illumination unit 132 will fall within the angle of view of the image capture units 131. In other words, it has been difficult to arrange the illumination unit 132 so that it does not fall within the angle of view of the image capture units 131.

[0060] Because the illumination unit 132 (its illumination device) is a high-brightness light source, when the illumination unit 132 is within the angle of view, the light leaks into dark areas, easily causing phenomena such as flare, ghosting, and halation. When such phenomena occur, it may be difficult to accurately extract the silhouette of the subject from the captured image. It may also be difficult to extract the texture of the subject. As a result, there is a risk that the accuracy of the 3D model generated from the captured image may be reduced.

[0061] <Application of polarizers> Therefore, polarizers that generate linearly polarized light from natural light (unpolarized) or circularly polarized light are provided in the imaging unit 131 (imaging device) and the illumination unit 132 (illumination device), so that the illumination unit 132 illuminates the subject with polarized light, and the imaging unit 131 receives the polarized light to generate a captured image. The polarization direction of the polarized light emitted by the illumination unit 132 (i.e., the polarization direction of the polarizer of the illumination unit 132) and the polarization direction of the polarized light received by the imaging unit 131 (i.e., the polarization direction of the polarizer of the imaging unit 131) are made to differ from each other.

[0062] In this specification, a light beam consisting mainly of vibration components in a predetermined direction is referred to as polarized light, and the main vibration direction of the polarized light is referred to as the polarization direction (or polarization angle). A polarizer generates polarized light in a predetermined polarization direction, and this polarization direction is also referred to as the polarization direction (or polarization angle) of the polarizer.

[0063] For example, an imaging processing system that generates a 3D model of an object using multiple captured images obtained by capturing an image of the object may include multiple polarized lighting devices (e.g., lighting unit 132) that are equipped with polarizers and that irradiate the object with polarized light obtained when light emitted from a light emitting unit passes through the polarizer from different positions, and multiple polarized imaging devices (e.g., imaging unit 131) that are equipped with polarizers and that generate captured images using polarized light obtained when light from outside passes through the polarizer at different positions where the object and at least one of the polarized lighting devices are within the angle of view, and the polarization direction of the polarizer of the polarized imaging devices is made different from the polarization direction of the polarizer of the polarized lighting device.

[0064] For example, captured images of an object are generated at different positions using polarized light with a polarization direction different from the polarized light emitted from a polarized lighting device (e.g., lighting unit 132) within the angle of view, and a 3D model of the object is generated using the multiple captured images obtained at different positions.

[0065] For example, an image capturing processing system may include a plurality of polarized lighting devices (e.g., lighting unit 132) each equipped with a polarizer, which illuminates an object with polarized light obtained when light emitted from a light emitting unit passes through the polarizer from different positions, and a plurality of polarized imaging devices (e.g., imaging unit 131) each equipped with a polarizer, which generate an image of an object using polarized light obtained when light from outside passes through the polarizer at different positions where the object and at least one of the polarized lighting devices are within the angle of view, and the polarization direction of the polarizer of the polarized imaging devices is different from the polarization direction of the polarizer of the polarized lighting device.

[0066] When the polarization direction of the polarizer of the imaging unit 131 and the polarization direction of the polarizer of the illumination unit 132 differ from each other, the amount of direct light from the illumination unit 132 that passes through the polarizer of the imaging unit 131 and enters the sensor is reduced. Therefore, in the captured image generated by the imaging unit 131, the luminance value of the part of the illumination unit 132 that falls within the angle of view can be reduced, and the occurrence of so-called flare, ghost, halation, etc. can be suppressed. Therefore, it becomes possible to more accurately extract silhouettes and textures from the captured image, and the 3D model generation unit 102 can generate a more accurate 3D model (a reduction in the accuracy of the 3D model can be suppressed).

[0067] The degree to which the amount of direct light from the illumination unit 132 that passes through the polarizer of the imaging unit 131 and enters the sensor is reduced depends on the relationship (angle) between the polarization direction of the polarizer of the imaging unit 131 and the polarization direction of the polarizer of the illumination unit 132. Generally, the closer the angle between them is to 90 degrees, the greater the reduction in the amount of light. In other words, the closer the angle between the polarization direction of the polarizer of the imaging unit 131 and the polarization direction of the polarizer of the illumination unit 132 is to 90 degrees, the more effectively the occurrence of so-called flare, ghost, halation, etc. can be suppressed.

[0068] <Lighting Department> 4 is a block diagram showing an example of the main configuration of the illumination unit 132. As shown in FIG.

[0069] Polarizing filter 151 is an example of a polarizer, and generates polarized light by transmitting light components that vibrate in a predetermined direction. Light emitting unit 152 is a light source, and emits light rays (unpolarized) of a predetermined wavelength in a predetermined direction.

[0070] As shown in Fig. 4, polarizing filter 151 is disposed in front of the emission direction (irradiation direction) of the light beam from light emitter 152. Unpolarized light 161 emitted from light emitter 152 travels toward polarizing filter 151. Polarizing filter 151 transmits vibration components of unpolarized light 161 in a predetermined direction. In other words, polarized light 162, whose polarization direction is the predetermined direction, is generated by polarizing filter 151. This polarized light 162 is emitted from illumination unit 132. In other words, illumination unit 132 is a polarized lighting device that includes a polarizer and irradiates polarized light generated by the polarizer using light from a light source.

[0071] The illumination unit 132 is installed at a position and orientation such that it illuminates an object that is the subject of the imaging unit 131, and at least a portion of this polarized light 162 is irradiated onto the object. Then, at least a portion of the irradiated polarized light 162 is reflected by the object or the like, becomes unpolarized, and travels toward the imaging unit 131. In other words, by illuminating in this manner by the illumination unit 132, it is possible to increase the brightness of the captured image.

[0072] The wavelength band of the polarized light 162 emitted by the illumination unit 132 is arbitrary. For example, the polarized light 162 may be visible light, invisible light, or both. For example, the polarized light 162 may be infrared light (IR light). Furthermore, the illumination unit 132 may have a plurality of light-emitting units 152 (light sources) that emit light rays in different wavelength ranges, or the image capture illumination unit 121 may have a plurality of illumination units 132 that emit polarized light 162 in different wavelength ranges.

[0073] Furthermore, the polarization direction of polarizing filter 151 (i.e., the polarization direction of polarized light 162) may be predetermined (fixed) or may be variable. For example, a polarization direction control mechanism (such as a movable ring) that controls the polarization direction of polarizing filter 151 may be provided, and the polarization direction of polarizing filter 151 may be variable by the polarization direction control mechanism.

[0074] <Image capture unit> 5 is a block diagram showing an example of the main configuration of the imaging unit 131. For example, as shown in A of FIG.

[0075] The polarizing filter 171 is an example of a polarizer, and generates polarized light by transmitting light components that vibrate in a predetermined direction. The image sensor 172 has a plurality of pixels, and each pixel performs photoelectric conversion on incident light to generate a captured image. The image sensor 172 supplies data of the generated captured image to the transmission unit 122.

[0076] As shown in A of FIG. 5, polarizing filter 171 is disposed on the light incident side of image sensor 172. Unpolarized light 181 incident on imaging unit 131 is directed toward polarizing filter 171. Polarizing filter 171 transmits vibration components of unpolarized light 181 in a predetermined direction. That is, polarized light 182, whose polarization direction is the predetermined direction, is generated by polarizing filter 171. This polarized light 182 is incident on image sensor 172 and photoelectrically converted. That is, image sensor 172 generates a captured image corresponding to this polarized light 182. That is, imaging unit 131 is a polarization imaging device that includes a polarizer and generates a captured image using polarized light generated by the polarizer.

[0077] When the illumination unit 132 is located within the angle of view of the imaging unit 131, direct light from the illumination unit 132 may enter the imaging unit 131. That is, polarized light 162 emitted from the illumination unit 132 may be directed toward the polarizing filter 171. Here, the polarization direction of the polarizing filter 171 is set to a direction different from the polarization direction of the polarizing filter 151. That is, the polarization directions of the polarizing filter 171 and the polarizing filter 151 are different from each other. Therefore, at least a portion of the polarized light 162 is blocked by the polarizing filter 171. That is, the amount of polarized light 162 entering the image sensor 172 is reduced.

[0078] That is, in the captured image generated by the imaging unit 131, the luminance value of the part of the illumination unit 132 that falls within the angle of view can be reduced, thereby suppressing the occurrence of so-called flare, ghost, halation, etc. Therefore, it becomes possible to more accurately extract silhouettes and textures from the captured image, and the 3D model generation unit 102 can generate a more accurate 3D model (can suppress a decrease in the accuracy of the 3D model).

[0079] The wavelength band of light that the image sensor 172 receives and photoelectrically converts (i.e., the wavelength band of the polarized light 182) is arbitrary. For example, the image sensor 172 may photoelectrically convert visible light, may photoelectrically convert invisible light, or may photoelectrically convert both. That is, the imaging unit 131 may generate data of a captured image of visible light, may generate data of a captured image of invisible light, or may generate captured images of both. For example, the image sensor 172 may photoelectrically convert infrared light (IR light). That is, the imaging unit 131 may generate a captured image of infrared light (IR light). Furthermore, the imaging unit 131 may have multiple image sensors 172 that photoelectrically convert light in different wavelength ranges, and the imaging illumination unit 121 may have multiple imaging units 131 that generate captured images of light in different wavelength ranges.

[0080] The captured image generated by the imaging unit 131 may be used to extract the silhouette of an object that is a subject. In other words, the imaging unit 131 may generate a captured image for extracting the silhouette of the object. By providing a polarizer (e.g., polarizing filter 171) in the imaging unit 131 that generates such a captured image, it becomes possible to extract the silhouette from the captured image more accurately.

[0081] Furthermore, the captured image generated by the imaging unit 131 may be used to extract the texture of an object that is a subject. In other words, the imaging unit 131 may generate a captured image for extracting the texture of the object. By providing a polarizer (e.g., polarizing filter 171) in the imaging unit 131 that generates such a captured image, it becomes possible to extract the texture from the captured image more accurately.

[0082] Of course, the captured image generated by the imaging unit 131 may be used to extract both the silhouette and texture of the object that is the subject. In other words, the imaging unit 131 may generate a captured image for extracting the silhouette and texture of the object. Furthermore, the imaging unit 131 may generate both a captured image for extracting the silhouette of the object and a captured image for extracting the texture of the object.

[0083] Furthermore, the image capturing and lighting unit 121 may have an image capturing unit 131 that generates a captured image used to extract the silhouette of an object, and an image capturing unit 131 that generates a captured image used to extract the texture of the object. In this case, the polarizer (e.g., polarizing filter 171) may be provided in the image capturing unit 131 that generates the captured image used to extract the silhouette of the object, or in the image capturing unit 131 that generates the captured image used to extract the texture of the object, or in both of the image capturing units 131.

[0084] Furthermore, the polarization direction of polarizing filter 171 (i.e., the vibration direction of polarized light 182) may be predetermined (fixed) or may be variable. For example, a polarization direction control mechanism (such as a movable ring) that controls the polarization direction of polarizing filter 171 may be provided, and the polarization direction of polarizing filter 171 may be variable by the polarization direction control mechanism.

[0085] 5B, the imaging unit 131 may be configured with a polarization sensor 191. The polarization sensor 191 is an image sensor that photoelectrically converts polarized light to generate a captured image. The polarization sensor 191 has a plurality of pixels, each of which is provided with a polarizer that generates polarized light from incident light. A light receiving unit provided in each pixel receives the polarized light generated by the polarizer and photoelectrically converts it. That is, the polarization sensor 191 polarizes the incident unpolarized light 181, photoelectrically converts it, and generates a captured image. The polarization direction of the polarizer provided in each pixel of the polarization sensor 191 is designed to be different from the polarization direction of the polarizing filter 151. That is, the polarization direction of the polarizer provided in each pixel of the polarization sensor 191 and the polarization filter 151 are different from each other. Therefore, at least a portion of the polarized light 162 is blocked by the polarizer, and the amount of light (brightness in the captured image) of the polarized light 162 that is photoelectrically converted is reduced.

[0086] Therefore, similar to the case of the polarizing filter 171, it is possible to reduce the luminance value of the portion of the illumination unit 132 that falls within the angle of view in the captured image generated by the imaging unit 131 (polarization sensor 191), thereby suppressing the occurrence of so-called flare, ghost, halation, etc. Therefore, it becomes possible to more accurately extract silhouettes and textures from the captured image, and the 3D model generation unit 102 can generate a more accurate 3D model (it is possible to suppress a reduction in the accuracy of the 3D model).

[0087] The imaging unit 131 and the illumination unit 132 may be configured as a ToF (Time of Flight) sensor. That is, the illumination unit 132 may illuminate a subject, the imaging unit 131 may receive the reflected light, and the imaging unit 131 may be configured as a distance measuring sensor that measures the distance to the subject based on the timing of the light reception. In other words, the present technology can also be applied to an optical distance measuring sensor such as a ToF sensor.

[0088] <Imaging lighting unit> The imaging unit 131 and the illumination unit 132 may be disposed in close proximity to each other. Furthermore, the imaging unit 131 and the illumination unit 132 may be disposed so that the direction of light emitted by the illumination unit 132 and the imaging direction of the imaging unit 131 (for example, the direction of the center of the angle of view) are the same. In other words, each illumination unit 132 may be disposed in the vicinity of one of the imaging units 131 and may be oriented so that the direction of irradiation of polarized light is the same as the imaging direction of the adjacent imaging unit 131. In this manner, the illumination unit 132 can illuminate the object, which is the subject, from the front as viewed from the imaging unit 131. Therefore, the imaging unit 131 can generate a captured image in which the subject has little unnecessary shadow and the subject is sufficiently bright.

[0089] For example, the imaging section 131 and the illumination section 132, which are arranged close to each other, may form an imaging illumination unit. Fig. 6 is a diagram showing an example of such an imaging illumination unit.

[0090] In the example of FIG. 6, the imaging and lighting unit 210 includes an RGB camera 211, an IR camera 212, and an IR light 213.

[0091] The RGB camera 211 is the imaging unit 131 that receives visible light and generates a captured image in the wavelength range of visible light. The IR camera 212 is the imaging unit 131 that receives infrared light and generates a captured image in the wavelength range of infrared light. The IR light 213 is the illumination unit 132 that irradiates infrared light.

[0092] For example, outdoors or at a live music venue, the light source of visible light is likely to change dramatically. For example, at a live music venue, spotlights or laser light may be irradiated onto the subject. When capturing images using the imaging processing system described above in such an environment, captured images in the visible light wavelength range are susceptible to the effects of such lighting, and optical phenomena such as flare, ghosting, and halation are likely to occur. Therefore, it may be difficult to accurately extract the silhouette of the subject using such captured images.

[0093] Therefore, the imaging illumination unit 210 uses the IR camera 212 to generate an image in the infrared wavelength range as an image for extracting the silhouette of the subject. That is, the image in the infrared wavelength range is used to extract the silhouette of the subject. Then, for imaging by the IR camera 212 (to ensure sufficient brightness in the infrared wavelength range), the IR light 213 illuminates the subject with infrared light.

[0094] Note that, since IR camera 212 and IR light 213 are installed in positions close to each other and facing the same subject, IR light 213 can illuminate the subject from the front as seen from IR camera 212. Therefore, IR camera 212 can generate a captured image in which unnecessary shadows on the subject are reduced and the subject is sufficiently bright. In other words, IR camera 212 can generate a captured image from which a silhouette can be extracted more accurately. In other words, by using the captured image generated by IR camera 212, the silhouette of the subject can be extracted more accurately.

[0095] Furthermore, the IR camera 212 has a polarizer as in the example of FIG. 5. Similarly, the IR light 213 has a polarizer as in the example of FIG. 4. The polarization direction of the polarizer of the IR camera 212 and the polarization direction of the polarizer of the IR light 213 are different from each other. Therefore, as described above, in the captured image generated by the IR camera 212, it is possible to suppress the occurrence of so-called flare, ghost, halation, and the like caused by the infrared light irradiated by the IR light 213 that falls within the angle of view of the captured image. Therefore, it becomes possible to more accurately extract a silhouette from the captured image, and the 3D model generation unit 102 can generate a more accurate 3D model (it is possible to suppress a decrease in the accuracy of the 3D model).

[0096] The RGB camera 211 can generate captured images in the wavelength range of visible light, and can therefore generate captured images used to extract the texture of a subject. The RGB camera 211 and the IR camera 212 are installed in positions close to each other and facing the same subject. That is, the angles of view of the RGB camera 211 and the IR camera 212 are the same or similar. Therefore, the captured images generated by the RGB camera 211 can be used to extract texture corresponding to the silhouette of the subject extracted using the captured images generated by the IR camera 212.

[0097] <Layout example> An example of the arrangement of the imaging section 131 and the lighting section 132 will be described using the imaging lighting unit 210 as a unit. A plurality of imaging lighting units 210 (i.e., imaging sections 131 and lighting sections 132) may be installed around (so as to surround) an object 231 that is a subject, as shown in Fig. 7. For example, the imaging lighting units 210 may be arranged so that the object that is a subject is located in an area (plane or space) whose outer frame is a line connecting adjacent imaging lighting units 210.

[0098] In this case, the object 231 and at least one of the illumination units 132 may be arranged so as to fall within the angle of view of each imaging unit 131. Furthermore, the other imaging units 131 may also be arranged so as to fall within the angle of view.

[0099] For example, as shown in Fig. 8, two imaging lighting units 210 (imaging lighting unit 210-1 and imaging lighting unit 210-2) may be arranged to face each other. In the example of Fig. 8, the imaging lighting unit 210-1 and the imaging lighting unit 210-2 are installed on opposite sides of the object 231 from each other on a straight line 241 that passes through the object 231, facing the object 231. In other words, the imaging directions and lighting directions of the imaging lighting unit 210-1 and the imaging lighting unit 210-2 are opposite to each other.

[0100] By installing the two imaging lighting units 210 (imaging lighting unit 210-1 and imaging lighting unit 210-2) in this manner, it is possible to image a wider range of the object 231 (reducing blind spots).

[0101] In such an arrangement, the IR light 213 falls within the angle of view of the IR camera 212, but as described above, the polarizer can be used to suppress the incidence of direct light from the IR light 213, thereby suppressing the occurrence of so-called flare, ghost, halation, etc. caused by the infrared light emitted by the IR light 213. Therefore, the silhouette of the object 231 can be extracted more accurately using the captured image generated by the IR camera 212.

[0102] Note that any number of image capturing illumination units 210 may be installed as long as there is more than one. For example, eight image capturing illumination units 210 may be installed. When a large number of image capturing illumination units 210 are installed in this manner, it is possible that a plurality of other image capturing units 131 and illumination units 132 may fit within the angle of view of the image capturing unit 131.

[0103] That is, the imaging illumination unit 210 (imaging section 131 and illumination section 132) may be installed so that multiple illumination sections 132 fit within the angle of view of the imaging section 131. In this case, the polarization directions of the polarizers of the multiple illumination sections 132 may be the same. By doing so, it is possible to similarly suppress the incidence of direct light from each illumination section 132 that fits within the angle of view onto the imaging section 131. In other words, it is possible to further suppress the occurrence of so-called flare, ghost, halation, etc.

[0104] The imaging lighting unit 210 may also be installed so that another imaging unit 131 fits within the angle of view of the imaging unit 131. Furthermore, the imaging lighting unit 210 may also be installed so that a plurality of other imaging units 131 fit within the angle of view of the imaging unit 131.

[0105] Furthermore, the multiple polarized illumination devices (e.g., illumination units 132) include a first polarized illumination device and a second polarized illumination device, and the multiple polarization imaging devices (e.g., imaging units 131) include a first polarization imaging device located at a position where the object and the first polarized illumination device are within the angle of view, and a second polarization imaging device located at a position where the object and the second polarized illumination device are within the angle of view, wherein the polarization direction of the polarizer of the first polarization imaging device may be different from the polarization direction of the polarizer of the first polarization illumination device, and the polarization direction of the polarizer of the second polarization imaging device may be different from the polarization direction of the polarizer of the second polarization illumination device. In other words, there may be multiple imaging units 131, each with a single illumination unit 132 within its angle of view.

[0106] In such a case, the polarization directions of the polarizers of the multiple illumination units 132 that fall within the angles of view of the different imaging units 131 may or may not be the same. In other words, the polarization direction of the polarizer of the second polarization imaging device may be different from the polarization direction of the polarizer of the second polarization illumination device.

[0107] That is, the polarization directions of the polarizers of the multiple illumination units 132 do not have to be the same. Similarly, the polarization directions of the polarizers of the multiple image capture units 131 do not have to be the same. For example, the effect of this embodiment can be obtained as long as the polarization direction of the polarizer of one of the multiple image capture units 131 is different from the polarization direction of the polarizer of an illumination unit 132 that falls within the angle of view of that image capture unit 131.

[0108] 9A, a plurality of imaging lighting units 210 (imaging sections 131 and lighting sections 132) may be arranged in a circle with the object 231 at the center. In the example of FIG. 9A, eight imaging lighting units 210 are arranged on a circle 251 with the object 231 at the center. As shown in FIG. 9B, each imaging lighting unit 210 (imaging lighting units 210-1 to 210-8) is installed facing the object 231. More specifically, the imaging lighting unit 210-1 and the imaging lighting unit 210-5 are arranged to face each other on a line 252 that passes through the object 231. The imaging lighting unit 210-2 and the imaging lighting unit 210-6 are arranged to face each other on a line 253 that passes through the object 231. The imaging lighting unit 210-3 and the imaging lighting unit 210-7 are arranged to face each other on a line 254 that passes through the object 231. The imaging lighting unit 210-4 and the imaging lighting unit 210-8 are arranged to face each other on a straight line 255 that passes through the object 231.

[0109] Even in such a case, by applying the present technology, it is possible to suppress the occurrence of so-called flare, ghost, halation, and the like by using a polarizer as described above.

[0110] 10, multiple imaging illumination units 210 (imaging sections 131 and illumination sections 132) may be arranged in a cylindrical shape with a central axis defined by a vertical line 261 passing through an object 231. Even in such a case, by applying the present technology, it is possible to suppress the occurrence of so-called flare, ghost, halation, and the like by using a polarizer as described above.

[0111] 11, multiple imaging illumination units 210 (imaging sections 131 and illumination sections 132) may be arranged in a spherical (or hemispherical) shape with the object 231 at the center. Even in such a case, by applying the present technology, it is possible to suppress the occurrence of so-called flare, ghost, halation, and the like by using a polarizer as described above.

[0112] <Silhouette extraction> 12A, assume that the IR light 213 of another imaging illumination unit 210 is captured (fitted within the angle of view) along with the object 311 that is the subject in a captured image 301 generated by the RGB camera 211. In this case, since the captured image 301 is an image captured in the wavelength range of visible light, no flare such as that shown by the ellipses 321 and 322 due to direct light (infrared light) from the IR light 213 occurs.

[0113] In contrast, the IR camera 212 generates a captured image 331 in the infrared wavelength range as shown in B of Fig. 12. The RGB camera 211 and the IR camera 212 have substantially the same angle of view. Therefore, the IR light 213 of the other imaging illumination unit 210 is also reflected in this captured image 331 (within the angle of view). Therefore, if the present technology is not applied, flare (ellipses 321 and 322) occurs in the captured image 331 due to direct light (infrared light) from the IR light 213. This makes it difficult to accurately extract the silhouette of the object 311.

[0114] By applying this technology, the IR camera 212 can suppress direct light from the IR light 213 using a polarizer with a polarization direction different from the polarization direction of the polarizer of the IR light 213, and generate a captured image 331 as shown in C of Fig. 12. In other words, the occurrence of so-called flare, ghost, halation, etc. can be suppressed. Therefore, it becomes possible to more accurately extract a silhouette from the captured image 331, and the 3D model generation unit 102 can generate a more accurate 3D model (it is possible to suppress a decrease in the accuracy of the 3D model).

[0115] <2. Second embodiment> <Calibration> The polarization directions of the polarizers of the imaging unit 131 and the illumination unit 132 may be calibrated (adjusted). As described above, the amount of light suppressed by the polarizer of the imaging unit 131 changes depending on the relative angle between the polarization directions of the polarizers of the imaging unit 131 and the illumination unit 132. In other words, the degree to which the occurrence of so-called flare, ghost, halation, etc. is suppressed changes. Therefore, for example, the polarization direction of each polarizer may be calibrated so that the relative angle becomes appropriate (so that the occurrence of so-called flare, ghost, halation, etc. can be further suppressed) depending on the position, posture, etc. at which the imaging unit 131 and the illumination unit 132 are installed.

[0116] <Data Acquisition Section> 13 is a block diagram showing an example of the main configuration of the data acquisition unit 101 in this case. As shown in Fig. 13, the data acquisition unit 101 has a calibration processing unit 401 and a display unit 402 in addition to the configuration in Fig. 3.

[0117] The calibration processing unit 401 is an example of a calibration device that calibrates the polarization direction of a polarizing filter, and acquires a captured image generated by the imaging unit 131, and derives a more suitable polarization direction (a polarization direction that can further suppress the occurrence of so-called flare, ghost, halation, etc.) based on the captured image. The calibration processing unit 401 also generates a display image that indicates the derived polarization direction, and supplies the display image to the display unit 402.

[0118] The display unit 402 displays the display image supplied from the calibration processing unit 401. By referring to the display image displayed on the display unit 402, the user can ascertain the polarization direction that can further suppress the occurrence of so-called flare, ghost, halation, and the like. The polarization directions of the polarizers of the image capture unit 131 and the illumination unit 132 are variable, and the image capture unit 131 and the illumination unit 132 have polarization direction control mechanisms (such as movable rings) that control the polarization directions of the polarizers. The user operates the polarization direction control mechanisms to calibrate the polarization direction to the desired direction.

[0119] <Calibration process flow> An example of the flow of the calibration process executed by the calibration processing unit 401 will be described with reference to the flowchart of FIG.

[0120] When the calibration process starts, the calibration processing unit 401 acquires a captured image in step S201.

[0121] In step S202, the user sets the polarization direction (polarization angle) of the polarizers of the image capturing unit 131 and the illumination unit 132 to a predetermined direction (angle) different from the previous direction (angle).

[0122] In step S203, the calibration processing unit 401 calculates the luminance value of the acquired captured image.

[0123] In step S204, the calibration processing unit 401 determines whether the polarization direction (polarization angle) has been set to all candidate directions (angles). That is, the calibration processing unit 401 determines whether captured images have been acquired in all candidate polarization directions (polarization angles) and whether luminance values ​​have been calculated.

[0124] If it is determined that an unprocessed direction (angle) exists, the process returns to step S202. That is, an image is captured in a new polarization direction (polarization angle), and the luminance value of the captured image is calculated. If it is determined that the processing has been performed for all candidate directions (angles) in this way, the process proceeds to step S205.

[0125] In step S205, the calibration processing unit 401 determines the polarization direction (polarization angle) that minimizes the brightness value from among the polarization directions (polarization angles) for which the brightness value of the captured image has been calculated, and generates a display image showing that polarization direction (polarization angle). The display unit 402 displays that display image.

[0126] This allows the user to calibrate the polarization directions of the polarizers of the imaging unit 131 and the illumination unit 132 to more appropriate directions based on the display, thereby making it possible to further suppress the occurrence of so-called flare, ghost, halation, and the like.

[0127] A polarization direction control unit (actuator) that updates the polarization directions of the polarizers of the image capturing unit 131 and the illumination unit 132 to the polarization direction derived by the calibration processing unit 401 may be provided.

[0128] <3. Application Examples> The system may further include a device that detects the tilt of the camera through camera calibration. The calibrated camera position is expressed in terms of rotation and translation relative to a certain origin. The system may also include a device that changes the angle of the polarizing filter when camera rotation is detected.

[0129] 5B, when a polarization sensor is used as the imaging unit 131, a sensor that captures images in multiple polarization directions (for example, four directions of 0°, 90°, 180°, and 270°) may be used. For example, rotation information of an automatically controlled light source may be acquired, and the polarization sensor may select pixels with the optimal polarization direction based on the rotation information, and generate a captured image corresponding to the polarization of that polarization direction.

[0130] In addition, a device may be provided that controls the polarization angle of the light source according to the camera position. For example, when an imaging unit with a variable imaging position, such as a drone or crane camera, is introduced into a volumetric imaging environment, there is a risk that light may enter the opposite position due to its movement. Therefore, the polarization direction (polarization angle) on the light side may be controlled using information on the camera position and rotation.

[0131] <4. Notes> <Application examples of this technology> The technology disclosed herein can be applied to a variety of products and services.

[0132] (1. Content Creation) For example, new video content may be produced by combining the 3D model of the subject generated in this embodiment with 3D data managed by another server. Furthermore, if background data acquired by an imaging device such as Lidar is present, the 3D model of the subject generated in this embodiment may be combined with the background data to produce content that makes the subject appear as if they are in the location indicated by the background data. The video content may be three-dimensional video content, or may be two-dimensional video content converted into two dimensions. The 3D model of the subject generated in this embodiment may be, for example, a 3D model generated by a 3D model generation unit or a 3D model reconstructed by a rendering unit.

[0133] (2. Virtual space experience) For example, a subject (e.g., a performer) generated in this embodiment can be placed in a virtual space where a user can communicate as an avatar. In this case, the user can view a live-action subject in the virtual space as an avatar.

[0134] (3. Application to remote communication) For example, by transmitting the 3D model of the subject generated by the 3D model generation unit 102 from the transmission unit 104 to a remote location, a user at a remote location can view the 3D model of the subject through a playback device at the remote location. For example, by transmitting the 3D model of the subject in real time, the subject and a user at a remote location can communicate in real time. For example, a case can be imagined in which the subject is a teacher and the user is a student, or the subject is a doctor and the user is a patient.

[0135] (4. Other) For example, free viewpoint video of sports or the like can be generated based on 3D models of multiple subjects generated in this embodiment, or an individual can distribute the 3D model of themselves generated in this embodiment to a distribution platform. In this way, the content of the embodiments described in this specification can be applied to various technologies and services.

[0136] <Computer> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, etc., that can execute various functions by installing various programs.

[0137] FIG. 15 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0138] In a computer 900 shown in FIG. 15, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0139] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0140] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 912 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 913 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0141] In a computer configured as above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0142] The program executed by the computer can be applied by recording it on removable media 921 such as package media, for example. In this case, the program can be installed in storage unit 913 via input / output interface 910 by inserting removable media 921 into drive 915.

[0143] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0144] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .

[0145] <Applicable targets of this technology> Furthermore, although an information processing system and the like have been described above as an application example of the present technology, the present technology can be applied to any configuration.

[0146] For example, this technology can be applied to various electronic devices, such as transmitters and receivers (e.g., television sets and mobile phones) used in satellite broadcasting, cable TV and other wired broadcasting, distribution over the Internet, and distribution to terminals via cellular communications, or devices (e.g., hard disk recorders and cameras) that record images on media such as optical disks, magnetic disks, and flash memories, or play images from these storage media.

[0147] Furthermore, for example, the present technology can also be implemented as a part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module (e.g., a video module) using multiple processors, a unit (e.g., a video unit) using multiple modules, or a set in which other functions are added to a unit (e.g., a video set).

[0148] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, AV (Audio Visual) equipment, a portable information processing terminal, or an IoT (Internet of Things) device.

[0149] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0150] <Fields and applications where this technology can be applied> Systems, devices, processing units, etc. to which the present technology is applied can be used in any field, such as transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. In addition, the applications thereof are also arbitrary.

[0151] <Other> The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0152] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0153] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0154] Also, for example, each step of a single flowchart may be executed by one device, or may be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes may be executed by one device, or may be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0155] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0156] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0157] The present technology can also be configured as follows. (1) An imaging processing system that generates a 3D model of an object using a plurality of captured images obtained by capturing an object, a plurality of polarized lighting devices each including a polarizer, and configured to irradiate the object with polarized light obtained by transmitting light emitted from a light-emitting unit through the polarizer from different positions; a plurality of polarization imaging devices each including a polarizer, the polarization imaging devices generating the captured image using polarized light obtained by transmitting external light through the polarizer at different positions where the object and at least one of the polarized illumination devices are within an angle of view; Equipped with The polarization direction of the polarizer of the polarization imaging device is different from the polarization direction of the polarizer of the polarized illumination device. Image processing system. (2) Another polarization imaging device among the plurality of polarization imaging devices is located within the angle of view of the polarization imaging device. The imaging processing system according to (1). (3) The other polarization imaging device faces the polarization imaging device. The imaging processing system according to (2). (4) The polarized illumination device is located near one of the polarization imaging devices, and the irradiation direction of the polarized light is the same as the imaging direction of the polarization imaging device. The imaging processing system according to (3). (5) A plurality of the polarized illumination devices are positioned within the angle of view of the polarization imaging device. An imaging processing system according to any one of (1) to (4). (6) The plurality of polarized illumination devices include a first polarized illumination device and a second polarized illumination device; the plurality of polarization imaging devices include a first polarization imaging device located at a position where the object and the first polarization illumination device are within an angle of view, and a second polarization imaging device located at a position where the object and the second polarization illumination device are within an angle of view; the polarization direction of the polarizer of the first polarization imaging device is different from the polarization direction of the polarizer of the first polarization illumination device; The polarization direction of the polarizer of the second polarization imaging device is different from the polarization direction of the polarizer of the second polarization illumination device. An imaging processing system according to any one of (1) to (5). (7) The polarization direction of the polarizer of the first polarized illumination device is different from the polarization direction of the polarizer of the second polarized illumination device. The imaging processing system according to (6). (8) The plurality of polarization imaging devices and the plurality of polarization illumination devices are arranged to surround the object. An imaging processing system according to any one of (1) to (7). (9) The plurality of polarization imaging devices and the plurality of polarization illumination devices are arranged in a circle around the object. The imaging processing system according to (8). (10) The plurality of polarization imaging devices and the plurality of polarization illumination devices are arranged in a cylindrical shape with a central axis being a vertical line passing through the position of the object. The imaging processing system according to (8). (11) The plurality of polarization imaging devices and the plurality of polarization illumination devices are arranged in a sphere with the object at the center. The imaging processing system according to (8). (12) The polarization imaging device generates a captured image for extracting a silhouette of the object. An imaging processing system according to any one of (1) to (11). (13) The image capturing device further includes an image capturing device for capturing an image of the object and generating a captured image for extracting a texture of the object. An imaging processing system according to any one of (1) to (12). (14) The polarized light illumination device irradiates the polarized light of visible light, The polarization imaging device generates the captured image using the polarization of visible light. An imaging processing system according to any one of (1) to (13). (15) The polarized lighting device irradiates the polarized invisible light, The polarization imaging device generates the captured image using the polarized light of invisible light. An imaging processing system according to any one of (1) to (13). (16) The polarizer is a polarizing filter. An imaging processing system according to any one of (1) to (15). (17) The polarization direction of the polarizing filter is variable, The polarization imaging device further includes a polarization direction control mechanism for controlling the polarization direction of the polarizing filter. The imaging processing system according to (16). (18) The optical fiber further includes a calibration device for calibrating the polarization direction of the polarizing filter. The imaging processing system according to (17).

[0158] (19) At different positions, captured images of the object are generated using polarized light having a polarization direction different from that of the polarized light irradiated from ... A 3D model of the object is generated using the plurality of captured images obtained at different positions. 3D model generation method.

[0159] (20) A plurality of polarized lighting devices each including a polarizer, each of which irradiates an object with polarized light obtained by transmitting light emitted from a light-emitting unit through the polarizer from different positions; a plurality of polarization imaging devices each including a polarizer, the polarization imaging devices generating captured images of the object using polarized light obtained by transmitting external light through the polarizer at different positions where the object and at least one of the polarized illumination devices are within an angle of view; Equipped with The polarization direction of the polarizer of the polarization imaging device is different from the polarization direction of the polarizer of the polarized illumination device. Image processing system. [Explanation of symbols]

[0160] 100 Information processing system, 101 Data acquisition unit, 102 3D model generation unit, 103 Formatting unit, 104 Transmission unit, 105 Reception unit, 106 Rendering unit, 107 Display unit, 121 Imaging and lighting unit, 122 Transmission unit, 131 Imaging unit, 132 Lighting unit, 151 Polarizing filter, 152 Light emitting unit, 171 Polarizing filter, 172 Image sensor, 191 Polarization sensor, 210 Imaging and lighting unit, 211 RGB camera, 212 IR camera, 213 IR light, 231 Object, 401 Calibration processing unit, 402 Display unit

Claims

1. 1. An imaging processing system that generates a 3D model of an object using a plurality of captured images obtained by capturing an object, a first polarized illumination device and a second polarized illumination device, each of which includes a polarizer and illuminates the object from different positions with polarized light obtained by transmitting light emitted from a light-emitting unit through the polarizer; a first polarization imaging device including a polarizer, the first polarization imaging device generating the captured image using polarized light obtained when external light passes through the polarizer at a position where the object and the first polarized illumination device are within an angle of view; a second polarization imaging device that includes a polarizer and generates the captured image using polarized light obtained when external light passes through the polarizer at a position where the object and the second polarized illumination device are within an angle of view; Equipped with a polarization direction of the polarizer of the first polarization imaging device is different from a polarization direction of the polarizer of the first polarization illumination device; the polarization direction of the polarizer of the second polarization imaging device is different from the polarization direction of the polarizer of the second polarization illumination device; The polarization direction of the polarizer of the first polarized illumination device is different from the polarization direction of the polarizer of the second polarized illumination device. Image processing system.

2. The second polarization imaging device is located within the angle of view of the first polarization imaging device. The imaging processing system according to claim 1 .

3. The second polarization imaging device faces the first polarization imaging device. The imaging processing system according to claim 2 .

4. The first polarized illumination device is located near the second polarized imaging device, and the irradiation direction of the polarized light is the same as the imaging direction of the second polarized imaging device. The imaging processing system according to claim 3 .

5. The first polarized illumination device and the second polarized illumination device are located within the angle of view of the first polarized imaging device. The imaging processing system according to claim 1 .

6. The first polarized imaging device and the second polarized imaging device, as well as the first polarized illumination device and the second polarized illumination device, are arranged to surround the object. The imaging processing system according to claim 1 .

7. The first polarized imaging device, the second polarized imaging device, and the first polarized illumination device, the second polarized illumination device are arranged in a circle with the object at the center. The imaging processing system according to claim 6 .

8. The first polarized imaging device and the second polarized imaging device, as well as the first polarized illumination device and the second polarized illumination device, are arranged in a cylindrical shape with a central axis that is a vertical line that passes through the position of the object. The imaging processing system according to claim 6 .

9. The first polarized imaging device, the second polarized imaging device, and the first polarized illumination device, the second polarized illumination device are arranged in a spherical shape with the object at the center. The imaging processing system according to claim 6 .

10. The first polarization imaging device and the second polarization imaging device generate an image for extracting a silhouette of the object. The imaging processing system according to claim 1 .

11. The object may further include an image capturing device that captures an image of the object and generates a captured image for extracting a texture of the object. The imaging processing system according to claim 1 .

12. The first polarized illumination device and the second polarized illumination device irradiate the polarized light of visible light, The first polarization imaging device and the second polarization imaging device generate the captured image using the polarization of visible light. The imaging processing system according to claim 1 .

13. The first polarized illumination device and the second polarized illumination device irradiate the polarized light of invisible light, The first polarization imaging device and the second polarization imaging device generate the captured image using the polarized light of invisible light. The imaging processing system according to claim 1 .

14. The polarizer is a polarizing filter The imaging processing system according to claim 1 .

15. The polarization direction of the polarizing filter is variable, The first polarization imaging device and the second polarization imaging device further include a polarization direction control mechanism for controlling the polarization direction of the polarization filter. The imaging processing system according to claim 14.

16. The polarizing filter further includes a calibration device for calibrating the polarization direction of the polarizing filter. The imaging processing system according to claim 15.

17. A first captured image of an object is generated using second polarized light having a polarization direction different from that of first polarized light irradiated from a first polarized lighting device within an angle of view; generating a second captured image of the object at a position different from the position at which the first captured image was generated, using fourth polarized light irradiated from a second polarized illumination device within an angle of view and having a polarization direction different from that of the third polarized light; generating a 3D model of the object using the first captured image and the second captured image; The polarization direction of the third polarized light is different from the polarization direction of the first polarized light. 3D model generation method.

18. a first polarized illumination device and a second polarized illumination device, each of which includes a polarizer and illuminates an object from different positions with polarized light obtained by transmitting light emitted from a light emitting unit through the polarizer; a first polarization imaging device including a polarizer, the first polarization imaging device generating an image of the object using polarized light obtained by transmitting external light through the polarizer at a position where the object and the first polarized illumination device are within an angle of view; a second polarization imaging device that includes a polarizer and generates the captured image using polarized light obtained when external light passes through the polarizer at a position where the object and the second polarized illumination device are within an angle of view; Equipped with a polarization direction of the polarizer of the first polarization imaging device is different from a polarization direction of the polarizer of the first polarization illumination device; the polarization direction of the polarizer of the second polarization imaging device is different from the polarization direction of the polarizer of the second polarization illumination device; The polarization direction of the polarizer of the first polarized illumination device is different from the polarization direction of the polarizer of the second polarized illumination device. Image processing system.

Images