Image processing system and 3D model generation method

Abstract

The present disclosure relates to an image processing system and a 3D model generation method that enable the generation of a more accurate 3D model. An image processing system that generates a 3D model of an object using a plurality of captured images obtained by capturing an object comprises: a plurality of polarizing illumination devices equipped with polarizers, wherein light emitted from a light-emitting unit at different locations irradiates the object with polarized light obtained by passing through the polarizers; and a plurality of polarizing imaging devices equipped with polarizers, wherein the object and at least one polarizing illumination device are at different locations within a field of view, and the polarizing light from the outside passes through the polarizers to generate the image, wherein the polarization direction of the polarizer of the polarizing imaging device is different from the polarization direction of the polarizer of the polarizing illumination device. The present disclosure may be applied, for example, to an image processing system or a 3D model generation method.

Description

Technology Field

[0001] The present disclosure relates to an image processing system and a 3D model generation method, and in particular to an image processing system and a 3D model generation method capable of generating a more accurate 3D model. Background Technology

[0002] There is a technology that generates a 3D model (3D model) containing three-dimensional information of a subject from a video captured from multiple viewpoints, and generates a free-view video based on the 3D model. This technology is also referred to as volumetric capture technology.

[0003] For example, a technique for generating a 3D model has been proposed using a method such as Visual Hull, which performs sculpting of the three-dimensional shape of a subject based on multiple captured images obtained from different directions (see, for example, Patent Document 1).

[0004] In the imaging of a subject in such volumetric capture technology, generally, a lighting device is used to irradiate light onto the subject or the surroundings of the subject in order to secure brightness. Prior art literature

[0005] International Publication No. 2018 / 150933 The problem to be solved

[0006] However, in the case of volumetric capture technology, since imaging is performed by multiple imaging devices arranged to surround the subject in order to capture the subject from a wider variety of directions, there are fewer blind spots, and lighting devices tend to converge within the field of view. When high-intensity lighting devices converge within the field of view, optical phenomena such as so-called flare, ghosting, and halation are likely to occur. When such optical phenomena occur, there was a concern that the accuracy of the 3D model generated from the captured image would be reduced.

[0007] The present disclosure is made in consideration of these circumstances and aims to generate a more accurate 3D model. means of solving the problem

[0008] An imaging processing system of one aspect of the present technology is an imaging processing system that generates a 3D model of an object using a plurality of imaging images obtained by capturing an object, and comprises a plurality of polarizing illumination devices equipped with a polarizer, wherein light emitted from a light-emitting unit at different locations passes through the polarizer to irradiate the object with polarized light obtained therefrom, and a plurality of polarizing imaging devices equipped with a polarizer, wherein the object and at least one of the polarizing illumination devices are at different locations within a field of view, and the polarizing direction of the polarizer of the polarizing imaging device is different from the polarizing direction of the polarizer of the polarizing illumination device.

[0009] A 3D model generation method according to another aspect of the present technology is a method for generating a 3D model of an object by using polarization in a polarization direction different from the polarization irradiated from a polarizing illumination device within a field of view at different locations, and generating a 3D model of the object using a plurality of said images obtained at different locations.

[0010] Another aspect of the present invention is an imaging processing system comprising: a plurality of polarizing illumination devices equipped with polarizers, wherein light emitted from a light-emitting unit at different locations passes through the polarizers to irradiate an object with polarized light obtained therefrom; and a plurality of polarizing imaging devices equipped with polarizers, wherein the object and at least one of the polarizing illumination devices are at different locations within a field of view, and the polarizing imaging system generates an image of the object using polarized light obtained therefrom passing through the polarizers, wherein the polarization direction of the polarizers of the polarizing imaging devices is different from the polarization direction of the polarizers of the polarizing illumination devices.

[0011] In an imaging processing system for generating a 3D model of an object using a plurality of images obtained by capturing an object, the polarization obtained by light emitted from a light-emitting unit passing through a polarizer is irradiated onto the object from different positions by a plurality of polarizing illumination devices equipped with polarizers, and the polarization obtained by light from the outside passing through a polarizer is used by a plurality of polarizing imaging devices equipped with polarizers having a polarization direction different from that of the polarizer of the polarizing illumination device at different positions where the object and at least one polarizing illumination device are within the field of view, thereby generating the image.

[0012] In a method for generating a 3D model of another aspect of the present technology, an image of an object is generated at different positions using polarization in a polarization direction different from the polarization irradiated from a polarizing light device within the field of view, and a plurality of images obtained at those different positions are used to generate a 3D model of the object.

[0013] In another aspect of the imaging processing system of the present technology, a plurality of polarizing illumination devices are provided, each having a polarizer, and irradiating an object with polarized light obtained by light emitted from a light-emitting unit passing through the polarizer at different locations; and a plurality of polarizing imaging devices are provided, each having a polarizer whose polarization direction is different from that of the polarizer of the polarizing illumination device, and generating an image of the object using polarized light obtained by light from the outside passing through the polarizer at different locations where the object and at least one polarizing illumination device are within the field of view. Brief explanation of the drawing

[0014] Figure 1 is a block diagram illustrating a main configuration example of an information processing system. Figure 2 is a flowchart illustrating an example of the flow of system processing. FIG. 3 is a block diagram illustrating a main configuration example of a data acquisition unit. Figure 4 is a block diagram illustrating a main configuration example of a lighting unit. FIG. 5 is a block diagram illustrating a main configuration example of an imaging unit. Figure 6 is a drawing illustrating an example of the configuration of an imaging illumination unit. Figure 7 is a drawing illustrating an example of the arrangement of imaging lighting units. Figure 8 is a drawing illustrating an example of the arrangement of imaging lighting units. Figure 9 is a drawing illustrating an example of the arrangement of imaging lighting units. Figure 10 is a drawing illustrating an example of the arrangement of imaging lighting units. Figure 11 is a drawing illustrating an example of the arrangement of imaging lighting units. Figure 12 is a drawing illustrating an example of an image. FIG. 13 is a block diagram illustrating another configuration example of a data acquisition unit. Figure 14 is a flowchart illustrating an example of the flow of the calibration process. FIG. 15 is a block diagram illustrating a main configuration example of a computer. Specific details for implementing the invention

[0015] Hereinafter, embodiments for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. In addition, the description will be carried out in the following order.

[0016] 1. First embodiment (information processing system)

[0017] 2. Second embodiment (calibration)

[0018] 3. Application Examples

[0019] 4. Bookkeeping

[0020] <1. First Embodiment>

[0021] Information Processing System

[0022] There is a volumetric capture technology that generates a 3D model (3D model) having 3D information of a subject from a moving image captured from multiple views, and generates a free-view moving image according to an arbitrary viewpoint position based on the 3D model. The information processing system (100) of FIG. 1 is a system that captures a subject from multiple views using this volumetric capture technology, generates a 3D model of the subject from the captured image, and generates a free-view moving image according to an arbitrary viewpoint position based on the 3D model.

[0023] As illustrated in FIG. 1, the information processing system (100) has a data acquisition unit (101), a 3D model generation unit (102), a formatting unit (103), a transmission unit (104), a reception unit (105), a rendering unit (106), and a display unit (107).

[0024] The data acquisition unit (101) acquires image data for generating a 3D model of a subject. For example, the data acquisition unit (101) acquires a plurality of viewpoint images captured by a plurality of imaging devices arranged to surround the subject as image data. In this case, it is preferable that the plurality of viewpoint images are images obtained by the plurality of imaging devices capturing images in synchronization.

[0025] Additionally, the data acquisition unit (101) may perform calibration based on image data to acquire internal parameters and external parameters of each imaging device. Additionally, the data acquisition unit (101) may acquire multiple depth information indicating the distance from multiple viewpoints to the subject, for example.

[0026] The data acquisition unit (101) supplies the acquired image data to the 3D model generation unit (102).

[0027] The 3D model generation unit (102) generates a 3D model having three-dimensional information of a subject based on image data supplied from the data acquisition unit (101). The 3D model generation unit (102) generates a 3D model of a subject by using images from multiple viewpoints (e.g., silhouette images from multiple viewpoints) to carve the three-dimensional shape of the subject, for example, using a so-called Visual Hull.

[0028] Here, a silhouette image is an image that represents only the outline (outer shape) of a subject, and the inner area of ​​the outline is expressed by filling it completely with a single color, for example, like a shadow image. That is, the 3D model generation unit (102) generates such a silhouette image from image data (captured image) supplied from the data acquisition unit (101). Additionally, image data of the silhouette image may be supplied to the 3D model generation unit (102) from the data acquisition unit (101).

[0029] The 3D model generation unit (102) can also transform the 3D model generated using Visual Hull with high precision using multiple depth information representing the distance from multiple viewpoints to the subject.

[0030] The 3D model generated by the 3D model generation unit can be described as a moving image of the 3D model by generating it in units of time series frames. Additionally, since the 3D model is generated using an image captured by the imaging device of the data acquisition unit (101), it can also be described as a real-life 3D model. The 3D model can be expressed in the form of mesh data that represents shape information indicating the surface shape of the subject, for example, called a polygon mesh, which is expressed as vertices and connections between vertices. The method of expressing the 3D model is not limited to these, and it may also be described as a so-called point cloud expression method expressed as point position information.

[0031] Color information data is also generated as a texture in a form associated with these 3D shape data. For example, there are View Independent textures, which have a constant color regardless of the viewing direction, and View Dependent textures, which have a color that changes depending on the viewing direction.

[0032] The 3D model generation unit (102) supplies the data of the generated 3D model to the formatting unit (103).

[0033] The formatting unit (103) converts the 3D model data supplied from the 3D model generation unit (102) into a format suitable for transmission or storage. For example, the formatting unit (103) may convert the 3D model generated by the 3D model generation unit (102) into multiple 2D images by projecting it from multiple directions. Additionally, the formatting unit (103) may generate depth information, which is a 2D depth image from multiple viewpoints, using the 3D model. In this case, the formatting unit (103) may encode (compress) the depth information and color information of the 2D image state. In that case, the formatting unit (103) may encode the depth information and color information as a single image by arranging them, or encode them as two separate images. In addition, since the depth information and color information are in the form of two-dimensional image data, the formatting unit (103) may encode (compress) them using a two-dimensional compression technology such as AVC (Advanced Video Coding).

[0034] In the above case, the formatting unit (103) supplies the data of the 3D model to the transmitting unit (104) as transmission data including 2D data (or the encoded data thereof).

[0035] Additionally, for example, the formatting unit (103) may convert the 3D data of the mesh data into a point cloud format and supply it to the transmitting unit (104) as transmission data containing the 3D data. In this case, the formatting unit (103) may encode (compress) the 3D data using, for example, the Geometry-based-Approach 3D compression technology discussed in MPEG.

[0036] The transmitting unit (104) transmits the transmission data formed in the formatting unit (103) to the receiving unit (105). The transmitting unit (104) transmits the transmission data to the receiving unit (105) after performing a series of processing of the data acquisition unit (101), the 3D model generation unit (102), and the formatting unit (103) offline. Additionally, the transmitting unit (104) may transmit the transmission data generated from the above-described series of processing to the receiving unit (105) in real time.

[0037] The receiving unit (105) receives transmission data transmitted from the transmitting unit (104) and supplies it to the rendering unit (106).

[0038] The rendering unit (106) performs rendering using transmission data received from the receiving unit (105). For example, the rendering unit (106) performs texture mapping by projecting the mesh of a 3D model from the viewpoint of a camera that draws the mesh and attaching a texture representing a color or shape. The drawing at this time can be arbitrarily set regardless of the camera position at the time of shooting and can be viewed from a free viewpoint.

[0039] The rendering unit (106) performs texture mapping, which attaches a texture representing the color, shape, or texture of the mesh according to the position of the mesh of the 3D model, for example. Texture mapping includes a method called View Dependent, which considers the user's viewing position, and a method called View Independent, which does not consider the user's viewing position. The View Dependent method has the advantage of realizing higher quality rendering than the View Independent method because it changes the texture attached to the 3D model according to the position of the viewing position. On the other hand, the View Independent method has the advantage of reducing processing load compared to the View Dependent method because it does not consider the position of the viewing position. In addition, the viewing position data is input from the display unit (107) to the rendering unit (106) by detecting the user's viewing area (Region of Interest). In addition, the rendering unit (106) may adopt billboard rendering, which renders the object so that the object maintains a vertical posture relative to the viewing position, for example. For example, when rendering multiple objects, the rendering unit (106) may render objects of low interest to viewers as billboards and render other objects using a different rendering method.

[0040] The rendering unit (106) supplies the rendering result data to the display unit (107).

[0041] The display unit (107) displays the result rendered by the rendering unit (106) on the display unit of the display device. The display device may be a 2D monitor or a 3D monitor, such as a head-mounted display, a spatial display, a mobile phone, a television, a PC (Personal Computer), etc.

[0042] Flow of System Processing

[0043] An example of the flow of system processing executed by the information processing system (100) is explained with reference to the flowchart of FIG. 2.

[0044] When processing begins, in step S101, the data acquisition unit (101) acquires image data for creating a 3D model of the subject.

[0045] In step S102, the 3D model generation unit (102) generates a 3D model having three-dimensional information of the subject based on the image data acquired in step S101.

[0046] In step S103, the formatting unit (103) encodes the shape and texture data of the 3D model generated in step S102 into a format suitable for transmission or accumulation.

[0047] In step S104, the transmitting unit (104) transmits the encoded data generated in step S103.

[0048] In step S105, the receiving unit (105) receives the data transmitted in step S104.

[0049] In step S106, the rendering unit (106) performs decoding processing to convert the data into shape and texture data required for display. Additionally, the rendering unit (106) performs rendering using the shape and texture data.

[0050] In step S107, the display unit (107) displays the rendered result.

[0051] When the processing of Step S107 is finished, the system processing is terminated.

[0052] By performing each processing as described above, the information processing system (100) can generate a 3D model of the subject and generate and display an image of the subject viewed from a free viewpoint. By doing so, the user, who is the viewer, can view the subject from a free viewpoint.

[0053] <Variation Example>

[0054] In the above description, the information processing system (100) has shown a series of flows from a data acquisition unit (101) that acquires an image, which is a material for creating content, to a display unit (107) that displays an image viewed by a user. However, this does not mean that all functional blocks are required for the implementation of the present invention, and the present invention can be implemented for each functional block or in combination of multiple functional blocks. For example, in FIG. 1, a transmitting unit (104) and a receiving unit (105) are provided to show a series of flows from the side creating content to the side viewing content through the distribution of content data, but the entire process from content creation to viewing may be carried out on the same information processing device (e.g., a personal computer). In that case, the formatting unit (103), the transmitting unit (104), and the receiving unit (105) may be omitted.

[0055] Additionally, when implementing the information processing system (100), the same implementer may implement all functions, or different implementers may implement each function block. For example, Operator A may implement the data acquisition unit (101), the 3D model creation unit (102), and the formatting unit (103) to create 3D content, Operator B may implement the transmission unit (104) (platform) to transmit 3D content, and Operator C may implement the reception unit (105), the rendering unit (106), and the display unit (107) to receive, render, and control the display of 3D content.

[0056] Additionally, each function block can be executed on the cloud. For example, the rendering unit (106) may be executed within the display device or on a server. In that case, there is an exchange of information between the display device and the server.

[0057] In FIG. 1, a data acquisition unit (101), a 3D model generation unit (102), a formatting unit (103), a transmission unit (104), a reception unit (105), a rendering unit (106), and a display unit (107) are integrated and described as an information processing system (100). However, the configuration of the information processing system (100) is not limited to this example, and it is sufficient to have at least a data acquisition unit (101). For example, among the configurations shown in FIG. 1, one or more of the 3D model generation unit (102) to the display unit (107) may be omitted. In addition, the information processing system (100) may have configurations (function blocks) other than those described above.

[0058] In addition, each of the above-described function blocks (data acquisition unit (101) to display unit (107)) is realized by any configuration. For example, each function block may be realized by one or more devices. In addition, multiple function blocks may be realized by one device.

[0059] Data Acquisition Department

[0060] FIG. 3 is a block diagram illustrating a main configuration example of the data acquisition unit (101) of FIG. 1. A data acquisition unit (101), which is an embodiment of an image processing system to which the present technology is applied, has an image illumination unit (121) and a transmission unit (122) as shown in FIG. 3.

[0061] The imaging and lighting unit (121) images a subject or illuminates a subject. The imaging and lighting unit (121) has an imaging unit (131-1) to an imaging unit (131-M) and an illumination unit (132-1) to an illumination unit (132-N) (M and N are integers greater than or equal to 2). When there is no need to distinguish between the imaging unit (131-1) to the imaging unit (131-M), it is referred to as the imaging unit (131). When there is no need to distinguish between the illumination unit (132-1) to the illumination unit (132-N), it is referred to as the illumination unit (132).

[0062] That is, the imaging and lighting unit (121) has a plurality of imaging units (131) and a plurality of lighting units (132). In addition, the number of imaging units (131) and lighting units (132) of the imaging and lighting unit (121) may be the same (i.e., M=N) or different.

[0063] The imaging unit (131) is composed of a single or multiple imaging devices and captures a subject to generate an image for creating a 3D model. That is, the imaging unit (131) generates an image used for extracting the silhouette or texture of the subject. The imaging unit (131) supplies data of the generated image to the transmitting unit (122).

[0064] The wavelength band of light received by the imaging device of the imaging unit (131) is arbitrary and may be visible light or non-visible light. For example, the imaging unit (131) may receive visible light (RGB light) to generate an image of visible light, or receive infrared light (IR (InfraRed) light) to generate an image of infrared light.

[0065] The lighting unit (132) is composed of a single or multiple lighting devices and illuminates a subject captured by the imaging unit (131). The wavelength band of light emitted by the lighting device of the lighting unit (132) is arbitrary and may be visible light or non-visible light. For example, the lighting unit (132) may illuminate the subject with visible light (RGB light) or illuminate the subject with infrared light (IR light).

[0066] The transmitting unit (122) transmits the captured image data supplied from the capturing unit (131) to the 3D model generating unit (102). At that time, the transmitting unit (122) may supply the captured image data to the 3D model generating unit (102) without encoding it, or it may encode the captured image data and supply it to the 3D model generating unit (102) as encoded data. Additionally, the transmitting unit (122) may perform any image processing on the captured image. For example, the transmitting unit (122) may extract a silhouette or texture from the captured image and supply the extracted silhouette or texture data to the 3D model generating unit (102).

[0067] Generally, if the brightness is too low (too dark), it becomes difficult to capture the subject. Therefore, by the lighting unit (132) illuminating the subject, the imaging unit (131) can capture the subject under sufficient brightness, thereby obtaining an image with sufficient brightness.

[0068] However, in the case of a data acquisition unit (101) that acquires an image for generating a 3D model, a plurality of imaging units (131) are arranged around the subject to reduce the angle of view. Therefore, there is a very high possibility that the lighting unit (132) will converge within the angle of view of the imaging unit (131). In other words, it was difficult to arrange the lighting unit (132) so that it does not converge within the angle of view of the imaging unit (131).

[0069] Since the lighting unit (132) (lighting device) is a high-intensity light source, when the lighting unit (132) converges within the angle of view, the light leaks into the dark area, and phenomena such as so-called flare, ghosting, and halation are likely to occur. When such phenomena occur, there was a concern that it would be difficult to extract the accurate silhouette of the subject from the captured image. There was also a concern that it would be difficult to extract the texture of the subject. Therefore, there was a concern that the accuracy of the 3D model generated from the captured image would be reduced.

[0070] Application of Polarizers

[0071] Therefore, a polarizer is provided in the imaging unit (131) (imaging device) and the lighting unit (132) (lighting device) to generate linear polarization from natural light (non-polarized) or circular polarization, and the lighting unit (132) illuminates the subject with polarization, and the imaging unit (131) receives the polarization to generate an image. In addition, the polarization direction of the polarization irradiated by the lighting unit (132) (i.e., the polarization direction of the polarizer in the lighting unit (132)) and the polarization direction of the polarization received by the imaging unit (131) (i.e., the polarization direction of the polarizer in the imaging unit (131)) are made different from each other.

[0072] In addition, in this specification, a light ray containing a vibration component mainly in a predetermined direction is referred to as polarization, and the main vibration direction of the polarization is referred to as the polarization direction (or polarization angle). In addition, a polarizer generates polarization in a predetermined polarization direction, and the polarization direction is also referred to as the polarization direction (or polarization angle) of the polarizer.

[0073] For example, an imaging processing system that generates a 3D model of an object using a plurality of images obtained by capturing an object comprises a plurality of polarizing illumination devices (e.g., illumination unit (132)) that are equipped with polarizers and irradiate the object with polarized light obtained by light emitted from a light-emitting unit passing through the polarizers at different locations, and a plurality of polarizing imaging devices (e.g., imaging unit (131)) that are equipped with polarizers and generate an image using polarized light obtained by light from the outside passing through the polarizers at different locations where the object and at least one polarizing illumination device are within the field of view, and the polarization direction of the polarizer of the polarizing imaging device is different from the polarization direction of the polarizer of the polarizing illumination device.

[0074] For example, at different locations, an image of an object is generated using polarization with a polarization direction different from the polarization irradiated from a polarization lighting device (e.g., lighting unit (132)) within the field of view, and a 3D model of the object is generated using a plurality of images obtained at different locations.

[0075] For example, the imaging processing system is provided with a plurality of polarizing illumination devices (e.g., illumination unit (132)) that are equipped with polarizers and irradiate an object with polarized light obtained by light emitted from a light-emitting unit passing through the polarizers at different locations, and a plurality of polarizing imaging devices (e.g., imaging unit (131)) that are equipped with polarizers and generate an image of the object using polarized light obtained by light from the outside passing through the polarizers at different locations where the object and at least one polarizing illumination device are within the field of view, and the polarization direction of the polarizer of the polarizing imaging device is different from the polarization direction of the polarizer of the polarizing illumination device.

[0076] If the polarization direction of the polarizer of the imaging unit (131) and the polarization direction of the polarizer of the illumination unit (132) are different from each other, the amount of direct light from the illumination unit (132) that passes through the polarizer of the imaging unit (131) and is incident on the sensor is reduced. Therefore, in the imaging image generated by the imaging unit (131), the luminance value of the portion of the illumination unit (132) that converges within the field of view can be reduced, so the occurrence of so-called flare, ghosting, halation, etc. can be suppressed. Consequently, since the extraction of silhouettes or textures from the imaging image can be performed more accurately, the 3D model generation unit (102) can generate a more accurate 3D model (the reduction in the accuracy of the 3D model can be suppressed).

[0077] In addition, the degree of reduction in the amount of direct light from the illumination unit (132) that passes through the polarizer of the imaging unit (131) and is incident on the sensor depends on the relationship (angle) between the polarization direction of the polarizer of the imaging unit (131) and the polarization direction of the polarizer of the illumination unit (132). Generally, the closer the angle between the two is to 90 degrees, the greater the reduction in the amount of light. That is, the closer the angle between the polarization direction of the polarizer of the imaging unit (131) and the polarization direction of the polarizer of the illumination unit (132) is to 90 degrees, the more strongly the occurrence of so-called flare, ghosting, halation, etc. can be suppressed.

[0078] Lighting Department

[0079] FIG. 4 is a block diagram illustrating a main configuration example of a lighting unit (132). As shown in FIG. 4, the lighting unit (132) has a polarizing filter (151) and a light-emitting unit (152).

[0080] The polarizing filter (151) is an example of a polarizer and generates polarization by transmitting light of a component vibrating in a predetermined direction. The light-emitting part (152) is a light source and emits light of a predetermined wavelength (non-polarized light) in a predetermined direction.

[0081] As illustrated in FIG. 4, a polarizing filter (151) is positioned in front of the light ray emission direction (irradiation direction) of the light emitting unit (152). The unpolarized light (161) emitted from the light emitting unit (152) is directed toward the polarizing filter (151). The polarizing filter (151) transmits the vibrational component of the unpolarized light (161) in a predetermined direction. That is, polarized light (162) having the predetermined direction as the polarization direction is generated by the polarizing filter (151). This polarized light (162) is emitted from the illumination unit (132). That is, the illumination unit (132) is a polarizing illumination device equipped with a polarizer and irradiating polarized light generated using light from a light source by the polarizer.

[0082] Since the lighting unit (132) is installed in a position and posture that illuminates an object that is the subject of the imaging unit (131), at least a portion of the polarized light (162) is irradiated onto the object. Then, at least a portion of the irradiated polarized light (162) is reflected from the object and becomes unpolarized light and directed toward the imaging unit (131). That is, by illuminating in this way, the brightness of the image can be increased.

[0083] Additionally, the wavelength band of the polarization (162) irradiated by the illumination unit (132) is arbitrary. For example, the polarization (162) may be visible light, non-visible light, or both. For example, the polarization (162) may be infrared light (IR light). Additionally, the illumination unit (132) may have a plurality of light-emitting units (152) (light sources) that emit light in different wavelength ranges, and the imaging illumination unit (121) may have a plurality of illumination units (132) that irradiate polarization (162) in different wavelength ranges.

[0084] Additionally, the polarization direction of the polarization filter (151) (i.e., the polarization direction of the polarization (162)) may be predetermined (fixed) or variable. For example, a polarization direction control mechanism (such as a movable ring) that controls the polarization direction of the polarization filter (151) may be provided, and the polarization direction of the polarization filter (151) may be variable by means of the polarization direction control mechanism.

[0085] <Image Unit>

[0086] FIG. 5 is a block diagram illustrating a main configuration example of an imaging unit (131). For example, as shown in FIG. 5A, the imaging unit (131) has a polarizing filter (171) and an image sensor (172).

[0087] A polarizing filter (171) is an example of a polarizer and generates polarization by transmitting light of a component vibrating in a predetermined direction. An image sensor (172) has a plurality of pixels and generates an image by photoelectric conversion of incident light at each pixel. The image sensor (172) supplies data of the generated image to a transmitter (122).

[0088] As illustrated in FIG. 5A, a polarizing filter (171) is positioned on the light incident side of an image sensor (172). Unpolarized light (181) incident on the imaging unit (131) is directed toward the polarizing filter (171). The polarizing filter (171) transmits vibrational components of the unpolarized light (181) in a predetermined direction. That is, polarized light (182) having the predetermined direction as the polarization direction is generated by the polarizing filter (171). This polarized light (182) is incident on the image sensor (172) and converted into photoelectric light. That is, the image sensor (172) generates an image corresponding to this polarized light (182). That is, the imaging unit (131) is a polarizing imaging device equipped with a polarizer and generating an image using the polarized light generated by the polarizer.

[0089] Additionally, when the illumination unit (132) is located within the field of view of the imaging unit (131), direct light from the illumination unit (132) may be incident on the imaging unit (131). That is, polarized light (162) irradiated from the illumination unit (132) may be directed toward the polarizing filter (171). Here, the polarization direction of the polarizing filter (171) is set to be different from the polarization direction of the polarizing filter (151). That is, the polarizing filter (171) and the polarizing filter (151) have different polarization directions. Therefore, at least a portion of the polarized light (162) is blocked by the polarizing filter (171). That is, the amount of polarized light (162) incident on the image sensor (172) is reduced.

[0090] That is, in the image generated by the imaging unit (131), the luminance value of the portion of the lighting unit (132) that converges within the field of view can be reduced, so the occurrence of so-called flare, ghosting, halation, etc. can be suppressed. Therefore, since the extraction of silhouettes or textures from the image can be performed more accurately, the 3D model generation unit (102) can generate a more accurate 3D model (the reduction in the accuracy of the 3D model can be suppressed).

[0091] Additionally, the wavelength band of light (i.e., the wavelength band of polarization (182)) that the image sensor (172) receives and converts is arbitrary. For example, the image sensor (172) may convert visible light, convert invisible light, or convert both. That is, the imaging unit (131) may generate data of an image of visible light, generate data of an image of invisible light, or generate images of both. For example, the image sensor (172) may convert infrared light (IR light). That is, the imaging unit (131) may generate an image of infrared light (IR light). Additionally, the imaging unit (131) may have a plurality of image sensors (172) that photoelectrically convert light rays of different wavelength regions, and the imaging illumination unit (121) may have a plurality of imaging units (131) that generate an image of light rays of different wavelength regions.

[0092] The image generated by the imaging unit (131) may be used to extract the silhouette of the object that is the subject. In other words, the imaging unit (131) may generate an image for extracting the silhouette of the object. By providing a polarizer (e.g., a polarizing filter (171)) to the imaging unit (131) that generates such an image, the extraction of the silhouette from the image can be performed more accurately.

[0093] Additionally, the image generated by the imaging unit (131) may be used to extract the texture of the object that is the subject. In other words, the imaging unit (131) may generate an image for extracting the texture of the object. By providing a polarizer (e.g., a polarizing filter (171)) to the imaging unit (131) that generates such an image, the extraction of the texture from the image can be performed more accurately.

[0094] Of course, the image generated by the imaging unit (131) may be used to extract both the silhouette and the texture of the object being the subject. In other words, the imaging unit (131) may generate an image to extract the silhouette and texture of the object. Additionally, the imaging unit (131) may generate an image to extract the silhouette of the object and an image to extract the texture of the object, respectively.

[0095] Additionally, the imaging lighting unit (121) may have an imaging unit (131) that generates an imaging image used to extract the silhouette of an object and an imaging unit (131) that generates an imaging image used to extract the texture of an object. In that case, a polarizer (e.g., a polarizing filter (171)) may be provided in the imaging unit (131) that generates an imaging image used to extract the silhouette of an object, in the imaging unit (131) that generates an imaging image used to extract the texture of an object, or in both imaging units (131).

[0096] Additionally, the polarization direction of the polarization filter (171) (i.e., the vibration direction of the polarization (182)) may be predetermined (fixed) or variable. For example, a polarization direction control mechanism (such as a movable ring) that controls the polarization direction of the polarization filter (171) may be provided, and the polarization direction of the polarization filter (171) may be variable by means of the polarization direction control mechanism.

[0097] Additionally, as illustrated in FIG. 5B, the imaging unit (131) may be configured by a polarization sensor (191). The polarization sensor (191) is an image sensor that generates an image by photoelectric conversion of polarization. The polarization sensor (191) has a plurality of pixels, and each pixel is provided with a polarizer that generates polarization from incident light, and a light receiving unit provided in each pixel receives the polarization generated by the polarizer and performs photoelectric conversion. That is, the polarization sensor (191) polarizes the incident unpolarized light (181) and performs photoelectric conversion to generate the image. Furthermore, the polarization direction of the polarizer provided in each pixel of the polarization sensor (191) is designed to be different from the polarization direction of the polarization filter (151). That is, the polarizer provided in each pixel of the polarization sensor (191) and the polarization filter (151) have different polarization directions. Therefore, since at least a portion of the polarization (162) is blocked by the polarizer, the amount of light (brightness in the image) of the polarization (162) that is converted into photoelectric is reduced.

[0098] Accordingly, just as with the polarizing filter (171), in the image generated by the imaging unit (131) (polarizing sensor (191)), the luminance value of the portion of the illumination unit (132) that converges within the field of view can be reduced, thereby suppressing the occurrence of so-called flare, ghosting, halation, etc. Consequently, since the extraction of silhouettes or textures from the image can be performed more accurately, the 3D model generation unit (102) can generate a more accurate 3D model (the reduction in the accuracy of the 3D model can be suppressed).

[0099] Additionally, the imaging unit (131) and the illumination unit (132) may be configured as a Time of Flight (ToF) sensor. That is, the illumination unit (132) may illuminate the subject, the imaging unit (131) may receive the reflected light, and the distance to the subject may be measured based on the timing of the light reception. In other words, the present technology can also be applied to optical distance sensors such as ToF sensors.

[0100] Imaging unit

[0101] The imaging unit (131) and the lighting unit (132) may be positioned in close proximity to each other. Additionally, the imaging unit (131) and the lighting unit (132) may be positioned such that the direction of light irradiation by the lighting unit (132) and the imaging direction of the imaging unit (131) (e.g., the direction of the center of the angle of view) are the same. In other words, each lighting unit (132) may be positioned near one of the imaging units (131) and may assume a position such that the direction of polarization irradiation is the same as the imaging direction of the imaging unit (131) in that vicinity. By doing so, the lighting unit (132) can illuminate the object, which is the subject, from the front as viewed from the imaging unit (131). Therefore, the imaging unit (131) can produce an image with less unnecessary shadows on the subject and sufficient brightness.

[0102] For example, an imaging and lighting unit may be formed with an imaging unit (131) and a lighting unit (132) placed near each other. FIG. 6 is a drawing illustrating an example of the imaging and lighting unit.

[0103] In the example of FIG. 6, the imaging illumination unit (210) has an RGB camera (211), an IR camera (212), and an IR light (213).

[0104] The RGB camera (211) is an imaging unit (131) that receives visible light and generates an image in the wavelength range of visible light. The IR camera (212) is an imaging unit (131) that receives infrared light and generates an image in the wavelength range of infrared light. The IR light (213) is an illumination unit (132) that emits infrared light.

[0105] For example, in outdoor or live venues, there is a high likelihood that visible light sources will change drastically. For instance, in live venues, spotlights or laser lights are considered to be irradiated onto the subject. Under such an environment, when imaging is performed using an imaging processing system as described above, the image in the visible light wavelength range is susceptible to the influence of such lighting, and optical phenomena such as so-called flare, ghosting, and halation are likely to occur. Consequently, there is a concern that it may be difficult to accurately extract the silhouette of the subject using such an image.

[0106] Thus, the imaging lighting unit (210) uses an IR camera (212) to generate an imaging image in the wavelength range of infrared light as an imaging image for extracting the silhouette of a subject. That is, the silhouette of a subject is extracted using the imaging image in the wavelength range of infrared light. Then, for imaging by the IR camera (212) (in order to secure sufficient brightness in the wavelength range of infrared light), an IR light (213) illuminates the subject using infrared light.

[0107] Additionally, since the IR camera (212) and the IR light (213) are installed in close proximity to each other and facing the same subject, the IR light (213) can illuminate the subject from the front as viewed from the IR camera (212). Therefore, the IR camera (212) can produce an image with less unnecessary shadowing on the subject and sufficient brightness. In other words, the IR camera (212) can produce an image that can extract a more accurate silhouette. In other words, by using the image produced by the IR camera (212), the silhouette of the subject can be extracted more accurately.

[0108] Additionally, the IR camera (212) has a polarizer as in the example of FIG. 5. Likewise, the IR light (213) has a polarizer as in the example of FIG. 4. And, the polarization direction of the polarizer of the IR camera (212) and the polarization direction of the polarizer of the IR light (213) are different from each other. Therefore, as described above, in the image captured by the IR camera (212), the occurrence of so-called flare, ghosting, halation, etc. caused by infrared light irradiated by the IR light (213) that converges within the field of view can be suppressed. Therefore, since the silhouette can be extracted more accurately from the image captured, the 3D model generation unit (102) can generate a more accurate 3D model (the reduction in the accuracy of the 3D model can be suppressed).

[0109] Since the RGB camera (211) can generate an image in the wavelength range of visible light, it can generate an image used to extract the texture of a subject. The RGB camera (211) and the IR camera (212) are installed in close proximity to each other, facing the same subject. That is, the angles of view of the RGB camera (211) and the IR camera (212) are the same or approximate. Therefore, using the image generated by the RGB camera (211), a texture corresponding to the silhouette of the subject extracted using the image generated by the IR camera (212) can be extracted.

[0110] <Layout Example>

[0111] An example of the arrangement of the imaging unit (131) and the lighting unit (132) is described using the imaging lighting unit (210) as a unit. A plurality of imaging lighting units (210) (i.e., the imaging unit (131) and the lighting unit (132)) may be installed around (surround) the object (231) which is the subject, as shown in FIG. 7. For example, each imaging lighting unit (210) may be arranged so that the object which is the subject is located in an area (plane or space) where the line connecting adjacent imaging lighting units (210) is the outer frame.

[0112] In that case, the object (231) and at least one lighting unit (132) may be arranged so that they converge within the field of view of each imaging unit (131). Additionally, other imaging units (131) may be arranged so that they converge within that field of view.

[0113] For example, as shown in FIG. 8, two imaging lighting units (210) (imaging lighting unit (210-1) and imaging lighting unit (210-2)) may be arranged to face each other. In the example of FIG. 8, the imaging lighting unit (210-1) and the imaging lighting unit (210-2) are installed facing the object (231) on opposite sides of the object (231) along a straight line (241) passing through the object (231). That is, the imaging direction and the illumination direction of the imaging lighting unit (210-1) and the imaging lighting unit (210-2) are opposite to each other.

[0114] By installing two imaging lighting units (210) (imaging lighting unit (210-1) and imaging lighting unit (210-2)) in this way, a wider area of ​​the object (231) can be imaged (the blind spot can be reduced).

[0115] In this arrangement, the IR light (213) converges within the field of view of the IR camera (212), but as described above, the incidence of direct light from the IR light (213) can be suppressed by using a polarizer, so the occurrence of so-called flare, ghosting, halation, etc. caused by the infrared light irradiated by the IR light (213) can be suppressed. Therefore, the silhouette of the object (231) can be extracted more accurately using the captured image generated by the IR camera (212).

[0116] Additionally, the number of imaging lighting units (210) installed is arbitrary. For example, eight imaging lighting units (210) may be installed. When multiple imaging lighting units (210) are installed in this manner, there may be cases where multiple other imaging units (131) or lighting units (132) converge within the field of view of the imaging unit (131).

[0117] That is, an imaging lighting unit (210) (imaging unit (131) and lighting unit (132)) may be installed so that a plurality of lighting units (132) converge within the field of view of the imaging unit (131). In that case, the polarization direction of each polarizer of the plurality of lighting units (132) may be made the same as each other. By doing so, the incidence of direct light from each lighting unit (132) converged within the field of view to the imaging unit (131) can be suppressed in the same way. That is, the occurrence of so-called flare, ghosting, halation, etc. can be further suppressed.

[0118] Additionally, an imaging lighting unit (210) may be installed so that another imaging unit (131) converges within the field of view of the imaging unit (131). Additionally, an imaging lighting unit (210) may be installed so that multiple other imaging units (131) converge within the field of view of the imaging unit (131).

[0119] Additionally, a plurality of polarizing illumination devices (e.g., illumination unit (132)) may include a first polarizing illumination device and a second polarizing illumination device, and a plurality of polarizing imaging devices (e.g., imaging unit (131)) may include a first polarizing imaging device located at a position where the object and the first polarizing illumination device are within the field of view, and a second polarizing imaging device located at a position where the object and the second polarizing illumination device are within the field of view, wherein the polarization direction of the polarizer of the first polarizing imaging device may be different from the polarization direction of the polarizer of the first polarizing illumination device, and the polarization direction of the polarizer of the second polarizing imaging device may be different from the polarization direction of the polarizer of the second polarizing illumination device. That is, there may be a plurality of imaging units (131) in which a single illumination unit (132) converges within the field of view.

[0120] In this case, the polarization directions of the polarizers of the plurality of illumination units (132) that converge within the field of view of different imaging units (131) may be the same or may not be the same. That is, the polarization direction of the polarizer of the second polarization imaging device may be different from the polarization direction of the polarizer of the second polarization illumination device.

[0121] That is, the polarization directions of the polarizers of the plurality of lighting units (132) do not have to be the same as each other. Likewise, the polarization directions of the polarizers of the plurality of imaging units (131) do not have to be the same as each other. For example, if the polarization direction of the polarizer of one of the imaging units (131) and the polarization direction of the polarizer of the lighting unit (132) that converges within the field of view of the imaging unit (131) are different from each other, the effect of the present embodiment is obtained.

[0122] For example, as shown in FIG. 9A, a plurality of imaging lighting units (210) (imaging unit (131) and lighting unit (132)) may be arranged in a circle centered on the object (231). In the example of FIG. 9A, eight imaging lighting units (210) are arranged on a circle (251) centered on the object (231). As shown in FIG. 9B, each imaging lighting unit (210) (imaging lighting unit (210-1) to imaging lighting unit (210-8)) is installed facing the object (231). More specifically, imaging lighting unit (210-1) and imaging lighting unit (210-5) are arranged to face each other on a straight line (252) passing through the object (231). An imaging lighting unit (210-2) and an imaging lighting unit (210-6) are positioned opposite each other on a straight line (253) passing through the object (231). An imaging lighting unit (210-3) and an imaging lighting unit (210-7) are positioned opposite each other on a straight line (254) passing through the object (231). An imaging lighting unit (210-4) and an imaging lighting unit (210-8) are positioned opposite each other on a straight line (255) passing through the object (231).

[0123] Even in such cases, by applying the present technology, the occurrence of so-called flare, ghosting, halation, etc. by the polarizer can be suppressed as described above.

[0124] For example, as shown in FIG. 10, a plurality of imaging and lighting units (210) (imaging unit (131) and lighting unit (132)) may be arranged in a circular shape with a vertical line (261) passing through an object (231) as the central axis. Even in such cases, by applying the present technology, the occurrence of so-called flare, ghosting, halation, etc., can be suppressed by a polarizer as described above.

[0125] For example, as shown in FIG. 11, a plurality of imaging and lighting units (210) (imaging unit (131) and lighting unit (132)) may be arranged in a spherical (or hemispherical) shape centered on an object (231). Even in such cases, by applying the present technology, the occurrence of so-called flare, ghosting, halation, etc., can be suppressed by a polarizer as described above.

[0126] Extraction of Silhouettes

[0127] For example, as shown in FIG. 12A, in the image (301) generated by the RGB camera (211), the IR light (213) of another image lighting unit (210) is projected (converged within the field of view) together with the object (311) which is the subject. In this case, since the image (301) is an image in the wavelength range of visible light, flares such as the ellipse (321) or ellipse (322) caused by direct light (infrared light) from the IR light (213) do not occur.

[0128] In contrast, the IR camera (212) generates an image (331) in the wavelength range of infrared light as shown in Fig. 12B. The angle of view of the RGB camera (211) and the IR camera (212) is approximately the same. Therefore, the IR light (213) of another imaging lighting unit (210) is projected onto this image (331) (converged within the angle of view). Consequently, when the present technology is not applied, flare (ellipse (321) or ellipse (322)) is generated in the image (331) by direct light (infrared light) from the IR light (213). Consequently, it is difficult to accurately extract the silhouette of the object (311).

[0129] By applying the present technology, the IR camera (212) can suppress direct light from the IR light (213) by means of a polarizer having a polarization direction different from the polarization direction of the IR light (213), thereby generating an image (331) as shown in Fig. 12C. That is, the occurrence of so-called flare, ghosting, halation, etc. can be suppressed. Therefore, since the silhouette can be extracted more accurately from the image (331), the 3D model generation unit (102) can generate a more accurate 3D model (the reduction in the accuracy of the 3D model can be suppressed).

[0130] <2. Second Embodiment>

[0131] Calibration

[0132] The polarization direction of the polarizers of the imaging unit (131) and the lighting unit (132) may be corrected (adjusted). As described above, the amount of light suppressed by the polarizer of the imaging unit (131) changes depending on the relative angle of the polarization direction of the polarizers of the imaging unit (131) and the lighting unit (132). That is, the degree of suppression of the occurrence of so-called flare, ghost, halation, etc. changes. Therefore, for example, depending on the location or position where the imaging unit (131) or the lighting unit (132) is installed, the polarization direction of each polarizer may be corrected so that the relative angle becomes an appropriate angle (so that the occurrence of so-called flare, ghost, halation, etc. can be further suppressed).

[0133] Data Acquisition Department

[0134] FIG. 13 is a block diagram illustrating a main configuration example of a data acquisition unit (101) in that case. As shown in FIG. 13, the data acquisition unit (101) has a calibration processing unit (401) and a display unit (402) in addition to the configuration of FIG. 3.

[0135] The calibration processing unit (401) is an example of a calibration device for correcting the polarization direction of a polarizing filter, and acquires an image generated by the imaging unit (131), and based on the image, derives a more suitable polarization direction (a polarization direction that can further suppress the occurrence of so-called flare, ghosting, halation, etc.). In addition, the calibration processing unit (401) generates a display image showing the derived polarization direction and supplies the display image to the display unit (402).

[0136] The display unit (402) displays a display image supplied from the calibration processing unit (401). By referring to the display image displayed on the display unit (402), the user identifies a polarization direction that can further suppress the occurrence of so-called flare, ghosting, halation, etc. The polarization direction of the polarizers of the imaging unit (131) and the lighting unit (132) is variable, and the imaging unit (131) and the lighting unit (132) have a polarization direction control mechanism (such as a movable ring) that controls the polarization direction of the polarizers. The user operates the polarization direction control mechanism to correct the polarization direction to a desired direction.

[0137] <Flow of Calibration Processing>

[0138] An example of the flow of calibration processing executed by such calibration processing unit (401) is explained with reference to the flowchart of FIG. 14.

[0139] When calibration processing is initiated, the calibration processing unit (401) acquires an image in step S201.

[0140] In step S202, the user sets the polarization direction (polarization angle) of the polarizers of the imaging unit (131) and the lighting unit (132) to a predetermined direction (angle) different from the previous one.

[0141] In step S203, the calibration processing unit (401) calculates the brightness value of the acquired image.

[0142] In step S204, the calibration processing unit (401) determines whether the polarization direction (polarization angle) has been set to all candidate directions (angles). That is, the calibration processing unit (401) determines whether an image has been captured in all candidate polarization directions (polarization angles) and whether a luminance value has been calculated.

[0143] If it is determined that there is an unprocessed direction (angle), the processing returns to step S202. That is, imaging is performed in a new polarization direction (polarization angle), and the luminance value of the captured image is calculated. If it is determined that processing has been performed for all candidate directions (angles) as described above, the processing proceeds to step S205.

[0144] In step S205, the calibration processing unit (401) determines the polarization direction (polarization angle) in which the luminance value is minimized among each polarization direction (polarization angle) for which the luminance value of the captured image is calculated, and generates a display image showing the polarization direction (polarization angle). The display unit (402) displays the display image.

[0145] Accordingly, the user can correct the polarization direction of the polarizers of the imaging unit (131) and the lighting unit (132) to a more appropriate direction based on the indication. Thus, the occurrence of so-called flare, ghosting, halation, etc. can be further suppressed.

[0146] Additionally, a polarization direction control unit (actuator) may be provided to update the polarization direction of the polarizers of the imaging unit (131) and the lighting unit (132) to the polarization direction derived by the calibration processing unit (401).

[0147] <3. Application Examples>

[0148] Additionally, a device for detecting the tilt of the camera by camera calibration may be further provided. The position of the calibrated camera is expressed in terms of rotation and translation with respect to a certain origin. Additionally, a device for changing the angle of a polarizing filter when camera rotation is detected may be provided.

[0149] In addition, as an imaging unit (131), when using a polarization sensor as in the example of B in FIG. 5, a sensor capable of capturing multiple polarization directions (e.g., four directions such as 0°, 90°, 180°, and 270°) may be used. For example, rotation information of an automatically controlled light source may be acquired, and the polarization sensor may select a pixel of the optimal polarization direction based on the opening information and generate an image corresponding to the polarization of that polarization direction.

[0150] In addition, a device for controlling the deflection angle of a light source according to the camera position may be provided. For example, when introducing an imaging unit with a changeable imaging position, such as a drone or crane camera, into a surrounding volumetric shooting environment, there is a risk that light may enter the opposite position due to the movement. Therefore, the polarization direction (polarization angle) of the light side may be controlled using information on the camera's position or rotation.

[0151] <4. Bookkeeping>

[0152] <Examples of Application of the Technology>

[0153] The technology described in this disclosure can be applied to various products or services.

[0154] (1. Content Creation)

[0155] For example, new video content may be produced by synthesizing the 3D model of the subject generated in the present embodiment with 3D data managed on a different server. Additionally, if background data acquired by an imaging device such as LiDAR exists, for example, by combining the 3D model of the subject generated in the present embodiment with the background data, content may be produced that makes it appear as if the subject is in the location represented by the background data. Furthermore, the video content may be 3D video content or 2D video content converted into 2D. Additionally, the 3D model of the subject generated in the present embodiment may be, for example, a 3D model generated by a 3D model generation unit or a 3D model reconstructed by a rendering unit.

[0156] (2. Experience in virtual space)

[0157] For example, a subject (e.g., an actor) created in this embodiment can be placed within a virtual space, which is a place where a user communicates as an avatar. In this case, the user can view the real-life subject in the virtual space as an avatar.

[0158] (3. Applications to Communication with Remote Locations)

[0159] For example, by transmitting the 3D model of the subject generated by the 3D model generation unit (102) to a remote location from the transmission unit (104), a user at the remote location can view the 3D model of the subject through a playback device at the remote location. For example, by transmitting the 3D model of the subject in real time, the subject and the user at the remote location can communicate in real time. For example, one can assume a case where the subject is a teacher and the user is a student, or a case where the subject is a doctor and the user is a patient.

[0160] (4. Others)

[0161] For example, free-viewpoint video of sports, etc., may be generated based on 3D models of multiple subjects generated in the present embodiment, and an individual may bet themselves, the 3D model generated in the present embodiment, on a betting platform. As such, the contents of the embodiments described in this specification can be applied to various technologies or services.

[0162] Computer

[0163] The aforementioned series of processes may be executed by hardware or by software. When the series of processes is executed by software, a program constituting the software is installed on a computer. Here, the computer includes a computer embedded in dedicated hardware, and, for example, a general-purpose personal computer, which is capable of executing various functions by installing various programs.

[0164] FIG. 15 is a block diagram illustrating an example of a computer hardware configuration that executes the aforementioned series of processes by a program.

[0165] In the computer (900) illustrated in FIG. 15, the CPU (Central Processing Unit) (901), ROM (Read Only Memory) (902), and RAM (Random Access Memory) (903) are connected to each other through a bus (904).

[0166] An input / output interface (910) is also connected to the bus (904). An input unit (911), an output unit (912), a memory unit (913), a communication unit (914), and a drive (915) are connected to the input / output interface (910).

[0167] The input unit (911) consists of, for example, a keyboard, mouse, microphone, touch panel, input terminal, etc. The output unit (912) consists of, for example, a display, speaker, output terminal, etc. The memory unit (913) consists of, for example, a hard disk, RAM disk, non-volatile memory, etc. The communication unit (914) consists of, for example, a network interface. The drive (915) drives removable media (921), such as a magnetic disk, optical disk, optical magnetic disk, or semiconductor memory.

[0168] In a computer configured as described above, the CPU (901) performs the aforementioned series of processes by loading a program stored, for example, in a memory unit (913) into RAM (903) via an input / output interface (910) and a bus (904) and executing it. In addition, data and other items necessary for the CPU (901) to execute various processes are also appropriately stored in RAM (903).

[0169] A program that the computer executes can be applied by writing it to a removable media (921), such as a package media. In that case, the program can be installed in a memory unit (913) through an input / output interface (910) by mounting the removable media (921) to a drive (915).

[0170] Additionally, this program may be provided via a wired or wireless transmission medium, such as a local area network, the internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit (914) and installed in the memory unit (913).

[0171] In addition, this program may be pre-installed in the ROM (902) or memory (913).

[0172] <Applicability of this technology>

[0173] In addition, although information processing systems and the like were described above as examples of the application of the present technology, the present technology can be applied to any configuration.

[0174] For example, the present technology can be applied to various electronic devices, such as transmitters or receivers (e.g., television receivers or mobile phones) in satellite broadcasting, wired broadcasting such as cable TV, distribution over the Internet, and distribution to terminals via cellular communication, or devices that record images on media such as optical discs, magnetic discs, and flash memory, or play images from such storage media (e.g., hard disk recorders or cameras).

[0175] In addition, the present technology may be implemented as a component of a device, such as a processor as a system LSI (Large Scale Integration), e.g., a video processor; a module using multiple processors, e.g., a video module; a unit using multiple modules, e.g., a video unit; or a set having additional functions added to the unit, e.g., a video set.

[0176] Furthermore, for example, this technology may also be applied to network systems composed of multiple devices. For example, this technology may be implemented as cloud computing in which multiple devices share and jointly process data through a network. For example, this technology may be implemented in a cloud service that provides video (video) services to any terminal, such as a computer, AV (Audio Visual) device, portable information processing terminal, or IoT (Internet of Things) device.

[0177] In addition, in this specification, the term "system" means a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are housed in the same housing. Accordingly, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are all systems.

[0178] <Fields and Uses Where This Technology Is Applicable>

[0179] Systems, devices, processing units, etc., to which this technology is applied can be used in any field, such as transportation, medical, security, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and natural monitoring. Furthermore, their applications are also arbitrary.

[0180] Others

[0181] Embodiments of the present technology are not limited to the embodiments described above, and various modifications are possible within the scope of not departing from the essence of the present technology.

[0182] For example, the configuration described as a single device (or processing unit) may be divided to form multiple devices (or processing units). Conversely, the configuration described as multiple devices (or processing units) above may be integrated to form a single device (or processing unit). Furthermore, configurations other than those described above may be added to the configuration of each device (or each processing unit). Additionally, if the configuration or operation of the system as a whole is substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0183] In addition, for example, the above-described program may be executed on any device. In that case, the device must have the necessary functions (function blocks, etc.) and be able to obtain the necessary information.

[0184] In addition, for example, each step of a flowchart may be executed by a single device or divided among multiple devices. Also, if a step includes multiple processes, the multiple processes may be executed by a single device or divided among multiple devices. In other words, multiple processes included in a single step may be executed as processes of multiple steps. Conversely, processes described as multiple steps may be integrated and executed as a single step.

[0185] Furthermore, for example, the processing of steps describing a program executed by a computer may be executed chronologically according to the order described in this specification, in parallel, or individually at necessary timings, such as when a call is made. That is, as long as no contradiction arises, the processing of each step may be executed in an order different from the order described above. Additionally, the processing of steps describing this program may be executed in parallel with the processing of other programs, or executed in combination with the processing of other programs.

[0186] Furthermore, for example, multiple technologies related to the present invention may be practiced independently as separate entities, provided that no contradiction arises. Of course, any multiple technologies may also be practiced in combination. For example, all or part of the present invention described in one embodiment may be practiced in combination with all or part of the present invention described in another embodiment. Additionally, any or part of any of the aforementioned technologies may be practiced in combination with other technologies not described above.

[0187] In addition, the present technology may also take the following configuration.

[0188] (1) An image processing system that generates a 3D model of an object using a plurality of images obtained by capturing an object,

[0189] A plurality of polarizing lighting devices equipped with polarizers, and irradiating the object with polarized light obtained by light emitted from a light-emitting part at different positions passing through the polarizers, and

[0190] A plurality of polarizing imaging devices, each equipped with a polarizer, wherein the object and at least one of the polarizing illumination devices are located at different positions within the field of view, and the imaging device generates the image using polarization obtained by light from the outside passing through the polarizer.

[0191] Equipped with,

[0192] The polarization direction of the polarizer of the polarizing imaging device is different from the polarization direction of the polarizer of the polarizing illumination device.

[0193] Image processing system.

[0194] (2) Within the field of view of the polarizing imaging device, among the plurality of polarizing imaging devices, another polarizing imaging device is located.

[0195] The image processing system described in (1).

[0196] (3) The other polarizing imaging device is opposite to the polarizing imaging device

[0197] The image processing system described in (2).

[0198] (4) The polarizing illumination device is located near the polarizing imaging device, and the direction of irradiation of the polarization is the same as the imaging direction of the polarizing imaging device.

[0199] The image processing system described in (3).

[0200] (5) A plurality of polarizing illumination devices are located within the field of view of the polarizing imaging device.

[0201] An image processing system described in any one of (1) to (4).

[0202] (6) A plurality of the above-mentioned polarizing illumination devices include a first polarizing illumination device and a second polarizing illumination device, and

[0203] A plurality of the above-mentioned polarizing imaging devices include a first polarizing imaging device positioned such that the object and the first polarizing illumination device are within the field of view, and a second polarizing imaging device positioned such that the object and the second polarizing illumination device are within the field of view.

[0204] The polarization direction of the polarizer of the first polarization imaging device is different from the polarization direction of the polarizer of the first polarization illumination device, and

[0205] The polarization direction of the polarizer of the second polarization imaging device is different from the polarization direction of the polarizer of the second polarization illumination device.

[0206] An image processing system described in any one of (1) to (5).

[0207] (7) The polarization direction of the polarizer of the first polarization illumination device is different from the polarization direction of the polarizer of the second polarization illumination device.

[0208] The image processing system described in (6).

[0209] (8) A plurality of the polarizing imaging devices and a plurality of the polarizing illumination devices are arranged to surround the object.

[0210] An image processing system described in any one of (1) to (7).

[0211] (9) A plurality of the polarizing imaging devices and a plurality of the polarizing illumination devices are arranged in a circular pattern centered on the object.

[0212] The image processing system described in (8).

[0213] (10) A plurality of the polarizing imaging devices and a plurality of the polarizing illumination devices are arranged in a circular shape with a vertical line passing through the position of the object as the central axis.

[0214] The image processing system described in (8).

[0215] (11) A plurality of the polarizing imaging devices and a plurality of the polarizing illumination devices are arranged in a spherical shape centered on the object.

[0216] The image processing system described in (8).

[0217] (12) The polarizing imaging device generates an image for extracting the silhouette of the object.

[0218] An image processing system described in any one of (1) to (11).

[0219] (13) Further comprising an imaging device for capturing the object and generating an image for extracting the texture of the object.

[0220] An image processing system described in any one of (1) to (12).

[0221] (14) The above polarizing light illuminating device irradiates the polarization of visible light, and

[0222] The above polarization imaging device generates the image using the polarization of visible light.

[0223] An image processing system described in any one of (1) to (13).

[0224] (15) The above polarizing light device irradiates the polarization of non-visible light, and

[0225] The above polarization imaging device generates the image using the polarization of non-visible light.

[0226] An image processing system described in any one of (1) to (13).

[0227] (16) The above polarizer is a polarizing filter

[0228] An image processing system described in any one of (1) to (15).

[0229] (17) The polarization direction of the above polarization filter is variable, and

[0230] The above polarization imaging device further comprises a polarization direction control mechanism for controlling the polarization direction of the polarization filter.

[0231] The image processing system described in (16).

[0232] (18) Further having a correction device for correcting the polarization direction of the above polarization filter

[0233] The image processing system described in (17).

[0234] (19) At different locations, a captured image of an object is generated using polarization with a polarization direction different from the polarization irradiated from a polarization lighting device within the field of view, and

[0235] Generating a 3D model of the object using multiple of the above-mentioned captured images obtained at different locations

[0236] 3D Model Creation Method

[0237] (20) A plurality of polarizing lighting devices, each having a polarizer, and irradiating an object with polarized light obtained by light emitted from a light-emitting unit at different positions passing through the polarizer, and

[0238] A plurality of polarizing imaging devices equipped with polarizers, wherein the object and at least one of the polarizing illumination devices are located at different positions within the field of view, and the polarizing imaging device generates an image of the object using polarization obtained by light from the outside passing through the polarizer.

[0239] Equipped with,

[0240] The polarization direction of the polarizer of the polarizing imaging device is different from the polarization direction of the polarizer of the polarizing illumination device.

[0241] Image processing system. Explanation of the symbols

[0242] 100: Information Processing System 101: Data Acquisition Unit 102: 3D Model Generation Section 103: Formatting section 104: Transmitter 105: Receiver 106: Rendering section 107: Display unit 121: Imaging Illumination Unit 122: Transmitter 131: Imaging unit 132: Lighting Section 151: Polarizing filter 152: Light-emitting part 171: Polarizing filter 172: Image sensor 191: Polarization sensor 210: Imaging light unit 211: RGB camera 212: IR Camera 213: IR Light 231: Object 401: Calibration processing unit 402: Display unit

Claims

Claim 1 An imaging processing system configured to generate a 3D model of an object using a plurality of captured images obtained by capturing an object, comprising a plurality of polarizing illumination devices and a plurality of polarizing imaging devices, wherein the plurality of polarizing illumination devices are provided at different locations, each individual polarizing illumination device includes an individual polarizer, and is configured to irradiate the object with polarized light obtained by light emitted from a light-emitting unit passing through the individual polarizer, wherein the plurality of polarizing imaging devices are provided at different locations, each individual polarizing imaging device includes an individual polarizer, and is configured to generate one or more of the plurality of captured images using polarized light obtained by receiving light from the outside with the individual polarizer, wherein the object and at least one of the individual polarizing illumination devices of the plurality of polarizing illumination devices are within the field of view of each individual polarizing imaging device, and the polarization direction of the individual polarizer of each individual polarizing imaging device is the individual of each individual polarizing illumination device within the field of view of each individual polarizing imaging device An image processing system with a polarization direction different from that of a polarizer. Claim 2 An imaging processing system according to claim 1, wherein, within the field of view of each individual polarizing imaging device, another polarizing imaging device among the plurality of polarizing imaging devices is located. Claim 3 In paragraph 2, the other polarizing imaging device is an imaging processing system facing each individual polarizing imaging device. Claim 4 In paragraph 3, each individual polarizing illumination device is located near one or more of the plurality of polarizing imaging devices, and the irradiation direction of the polarization emitted from each individual polarizing illumination device is the same as the imaging direction of one or more of the plurality of polarizing imaging devices, in an imaging processing system. Claim 5 An imaging processing system according to claim 1, wherein the plurality of polarizing illumination devices are located within the field of view of each individual polarizing imaging device. Claim 6 In claim 1, the plurality of polarizing illumination devices includes a first polarizing illumination device and a second polarizing illumination device, and the plurality of polarizing imaging devices includes a first polarizing imaging device located at a position where the object and the first polarizing illumination device are within the field of view, and a second polarizing imaging device located at a position where the object and the second polarizing illumination device are within the field of view, wherein the polarization direction of an individual polarizer of the first polarizing imaging device is different from the polarization direction of an individual polarizer of the first polarizing illumination device, and the polarization direction of an individual polarizer of the second polarizing imaging device is different from the polarization direction of an individual polarizer of the second polarizing illumination device, an imaging processing system. Claim 7 In paragraph 6, the polarization direction of the individual polarizer of the first polarization illumination device is different from the polarization direction of the individual polarizer of the second polarization illumination device, in an imaging processing system. Claim 8 An imaging processing system according to claim 1, wherein the plurality of polarizing imaging devices and the plurality of polarizing illumination devices are configured to be arranged to surround the object. Claim 9 An imaging processing system according to claim 8, wherein the plurality of polarizing imaging devices and the plurality of polarizing illumination devices are configured to be arranged in a circular shape centered on the object. Claim 10 An imaging processing system according to claim 8, wherein the plurality of polarizing imaging devices and the plurality of polarizing illumination devices are configured to be arranged in a circular shape with a vertical line passing through the position of the object as the central axis. Claim 11 An imaging processing system according to claim 8, wherein the plurality of polarizing imaging devices and the plurality of polarizing illumination devices are configured to be arranged in a spherical shape centered on the object. Claim 12 In claim 1, the plurality of polarizing imaging devices is configured to generate an imaging image for extracting the silhouette of the object, an imaging processing system. Claim 13 An imaging processing system according to claim 1, further comprising an imaging device configured to capture the object and generate an image for extracting the texture of the object. Claim 14 An imaging processing system according to claim 1, wherein the plurality of polarizing illumination devices irradiate the polarization of visible light, and the plurality of polarizing imaging devices generate each imaging image using the polarization of visible light. Claim 15 An imaging processing system according to claim 1, wherein the plurality of polarizing illumination devices irradiate the polarization of non-visible light, and the plurality of polarizing imaging devices generate each imaging image using the polarization of non-visible light. Claim 16 In paragraph 1, an imaging processing system in which each individual polarizer includes a polarization filter. Claim 17 In claim 16, the individual polarization direction of the polarization filter of each individual polarizer is variable, and each individual polarization imaging device further comprises a polarization direction control mechanism configured to control the polarization direction of the polarization filter, an imaging processing system. Claim 18 An image processing system according to claim 17, further comprising a correction device including a circuit portion configured to correct the polarization direction of the polarization filter. Claim 19 A method for generating a 3D model, comprising: a step of generating an image of an object using polarization having a polarization direction different from the polarization direction of polarization emitted from at least one polarization illumination device within the field of view of each individual polarization imaging device among a plurality of polarization imaging devices provided at different locations; and a step of generating a 3D model of the object using a plurality of images obtained from the plurality of polarization imaging devices provided at different locations, wherein the polarization direction of an individual polarizer of each individual polarization imaging device is different from the polarization direction of an individual polarizer of each individual polarization illumination device within the field of view of each individual polarization imaging device. Claim 20 An imaging processing system comprising a plurality of polarizing illumination devices and a plurality of polarizing imaging devices, wherein the plurality of polarizing illumination devices are provided at different locations, each individual polarizing illumination device includes an individual polarizer, and is configured to irradiate an object with polarized light obtained by light emitted from a light-emitting unit passing through the individual polarizer, wherein the plurality of polarizing imaging devices are provided at different locations, each individual polarizing imaging device includes an individual polarizer, and is configured to generate an image of the object using polarized light obtained by receiving light from the outside through the individual polarizer, wherein the object and at least one individual polarizing illumination device of the plurality of polarizing illumination devices are within the field of view of each individual polarizing imaging device, and the polarization direction of the individual polarizer of each individual polarizing imaging device is different from the polarization direction of the individual polarizer of each individual polarizing illumination device within the field of view of each individual polarizing imaging device.

Images