Image processing device, image processing method, and program

Abstract

To provide an image processing apparatus for appropriately correcting missing portions of three-dimensional shape data.SOLUTION: The image processing apparatus selects a plurality of elements corresponding to a surface of an object from among the plurality of elements included in three-dimensional shape data representing the shape of the object, determines whether or not each of the elements is visible from each of a plurality of viewpoints for each of the plurality of selected elements based on camera parameters, identifies the element to be processed from among the plurality of selected elements based on the number of viewpoints determined to be visible, and corrects the three-dimensional shape data based on the elements to be processed.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a technique for generating three-dimensional shape data. [Background technology]

[0002] One method involves generating silhouette images showing regions corresponding to objects from multiple captured images obtained by capturing images from multiple viewpoints, and then using each silhouette image to generate 3D shape data. However, in the generation of silhouette images as a preprocessing step, a defect may occur in the object region shown by the silhouette image due to an erroneous determination that a region of the object region in the captured image that has a color similar to the background color is mistakenly determined to be the background. In such cases, the 3D shape data generated based on a silhouette image that includes a defect in the object region will also have a through-hole-shaped defect (hereinafter referred to as a "defect") with a cross section corresponding to the shape of the defect.

[0003] Patent Document 1 discloses a technique for correcting such defects in three-dimensional shape data by performing correction through image processing. Specifically, the technique disclosed in Patent Document 1 corrects volume data by filling in defects in the three-dimensional shape data by performing filtering such as a median filter on slice images of each xyz plane of the three-dimensional shape data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-48627 Summary of the Invention [Problem to be solved by the invention]

[0005] The technology disclosed in Patent Document 1 applies filtering uniformly to the space when correcting three-dimensional shape data. As a result, filtering is also performed on parts of the three-dimensional shape data that are not missing. Therefore, the technology disclosed in Patent Document 1 adversely affects parts of the three-dimensional shape data that are not missing.

[0006] The present disclosure is made to solve such problems, and aims to provide an image processing device that can appropriately correct missing parts in three-dimensional shape data. [Means for solving the problem]

[0007] The image processing device according to the present disclosure includes an acquisition means for acquiring camera parameters for each of a plurality of viewpoints and three-dimensional shape data indicating the shape of an object generated based on the camera parameters and a plurality of captured images obtained by capturing images from each of the plurality of viewpoints; a selection means for selecting a plurality of elements corresponding to the surface of the object from a plurality of elements included in the three-dimensional shape data; and a selection means for selecting a plurality of elements that are visible from each of the plurality of viewpoints for each of the plurality of elements selected by the selection means based on the camera parameters. Number of viewpoints of identification do Viewpoint identification Means and Viewpoint identification By means identification The perspective of pieces Identify the element to process from among multiple selected elements based on the number of element The apparatus includes a specifying means and a correcting means for correcting the three-dimensional shape data based on the element to be processed. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to appropriately correct missing portions of three-dimensional shape data. [Brief explanation of the drawings]

[0009] [Figure 1]1 is a block diagram illustrating an example of a configuration of an image processing device according to a first embodiment. [Figure 2] 1 is a block diagram showing an example of a hardware configuration of an image processing device according to a first embodiment. [Figure 3] 4 is a flowchart showing an example of a processing flow of the image processing device according to the first embodiment. [Figure 4] 3A to 3C are schematic diagrams each showing a state of shape data that changes as a result of processing by the image processing device according to the first embodiment. [Figure 5] 10 is a flowchart showing an example of a processing flow of an image processing apparatus according to a second embodiment. [Figure 6] 10A and 10B are schematic diagrams each showing a state of shape data that changes as a result of processing by an image processing device according to a second embodiment. [Figure 7] FIG. 11 is an explanatory diagram for explaining processing by a target identifying unit according to the second embodiment. [Figure 8] 11 is a flowchart showing an example of a processing flow of an image processing apparatus according to the third embodiment. [Figure 9] 10A and 10B are schematic diagrams each showing a state of shape data that changes as a result of processing by an image processing device according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the scope of the present disclosure is not limited to these configurations.

[0011] [First embodiment] An image processing device 100 according to the first embodiment will be described with reference to Figures 1 to 4. The functional block configuration of the image processing device 100 according to the first embodiment will be described with reference to Figure 1. Figure 1 is a block diagram showing an example of the functional block configuration of the image processing device 100 according to the first embodiment. The image processing device 100 includes an information acquisition unit 101, a surface selection unit 102, a visibility determination unit 103, an object identification unit 104, and a shape correction unit 105.

[0012] The information acquisition unit 101 acquires camera parameters of each of a plurality of image capture devices and three-dimensional shape data indicating the shape of an object generated based on the camera parameters and a plurality of captured images obtained by capturing images using each of the plurality of image capture devices. Specifically, the information acquisition unit 101 acquires, as the three-dimensional shape data, three-dimensional shape data generated by a volume intersection method (also called a "shape from silhouette method") using silhouette images of the object appearing in each captured image. Furthermore, the information acquisition unit 101 acquires, as the camera parameters, camera parameters including information indicating the position, orientation, and viewing angle of the image capture device used to obtain the above-mentioned silhouette image.

[0013] Hereinafter, the information acquisition unit 101 will be described as acquiring volume data as three-dimensional shape data. Here, volume data refers to a data representation composed of a plurality of unit elements obtained by dividing the interior of a circumscribing rectangle (also referred to as a "bounding box") set in three-dimensional space into a plurality of unit areas in a grid pattern. Each of these unit elements is called a voxel, and each voxel is set with inclusion information (hereinafter referred to as a "voxel value") indicating whether or not the voxel exists inside the physical entity of an object in three-dimensional space (hereinafter simply referred to as an "object"). The voxel value may be binary, multi-valued, or continuous. In the present disclosure, the voxel value will be described as being binary as an example. Hereinafter, a voxel with a voxel value of 1 will be described as being partially or entirely inside the object, and the voxel will be referred to as an ON voxel. In the following description, a voxel whose voxel value is 0 is referred to as an OFF voxel, since it is assumed that the voxel does not exist entirely inside the object, that is, the voxel exists entirely outside the object.

[0014] The surface selection unit 102 selects, as a plurality of surface unit elements, a plurality of unit elements corresponding to the surface of an object from a plurality of unit elements included in the three-dimensional shape data acquired by the information acquisition unit 101. That is, the surface selection unit 102 selects, as a plurality of surface voxels, a plurality of voxels corresponding to the surface of an object from a plurality of voxels included in the volume data acquired by the information acquisition unit 101. Here, a surface voxel refers to either of two voxels, that is, an ON voxel where adjacent voxels exist inside the object, or an OFF voxel where adjacent voxels exist outside the object. Hereinafter, the surface selection unit 102 will be described as selecting the ON voxel of the two voxels as the surface voxel, as an example.

[0015] The visibility determination unit 103 determines whether each of the multiple surface unit elements is visible (hereinafter referred to as "visible") from a viewpoint corresponding to each of the multiple image capture devices, based on the camera parameters acquired by the information acquisition unit 101. That is, the visibility determination unit 103 determines whether each of the multiple surface voxels is visible from a viewpoint corresponding to each of the multiple image capture devices, based on the camera parameters. The visibility determination unit 103 generates information indicating the determination result (hereinafter referred to as "visibility information"). Here, "visible" refers to a state in which a voxel is within the angle of view from a certain viewpoint, and a line segment connecting the viewpoint and the voxel is not obstructed by other ON voxels. Specifically, for example, the visibility determination unit 103 acquires a depth image corresponding to the surface of the volume data from a certain viewpoint, projects the surface voxel to be determined onto the depth image, and associates the surface voxel with a pixel in the depth image. Furthermore, the visibility determination unit 103 determines whether or not a surface voxel is visible by comparing the distance between the viewpoint and the surface voxel with the depth value of the pixel associated with the surface voxel. Based on this determination, the visibility determination unit 103 identifies, for each surface voxel, the number of viewpoints from which the surface voxel is visible (hereinafter referred to as the "number of visible viewpoints") and generates visibility information indicating the number of visible viewpoints for each surface voxel.

[0016] The target identification unit 104 identifies a surface unit element to be processed from among the multiple surface unit elements based on the number of viewpoints determined to be visible by the visibility determination unit 103. Specifically, the target identification unit 104 identifies, from among the multiple surface unit elements, a surface unit element for which the number of viewpoints determined to be visible by the visibility determination unit 103 is equal to or less than a predetermined number as a surface unit element to be processed. That is, the target identification unit 104 identifies a voxel to be processed (hereinafter referred to as a "target voxel") from among the multiple surface voxels selected by the surface selection unit 102 based on the visibility information generated by the visibility determination unit 103. Specifically, for example, the target identification unit 104 identifies, from among the multiple surface voxels, a surface voxel for which the number of visible viewpoints indicated by the visibility information is equal to or less than a predetermined number as a target voxel. In the following description, it is assumed that the target identification unit 104 identifies, from among surface voxels that are ON voxels, a voxel for which the number of visible viewpoints is 0 as a target voxel.

[0017] The shape modification unit 105 modifies the three-dimensional shape data based on the surface unit elements of the processing target identified by the target identification unit 104. That is, the shape modification unit 105 modifies the shape of the volume data acquired by the information acquisition unit 101 based on the target voxel identified by the target identification unit 104, and acquires the modified volume data as modified volume data. Specifically, for example, the shape modification unit 105 modifies the shape of the volume data by converting OFF voxels around the target voxel to ON voxels.

[0018] The processing of each unit included in the image processing device 100 is performed by hardware such as an ASIC (Application Specific Integrated Circuit) built into the image processing device 100. The processing of each unit included in the image processing device 100 may also be performed by hardware such as an FPGA (Field Programmable Gate Array) built into the image processing device 100. Furthermore, the processing may also be performed by software using a CPU (Central Processor Unit) or a GPU (Graphic Processor Unit) and a memory.

[0019] The hardware configuration of the image processing device 100 when each unit included in the image processing device 100 operates as software will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the hardware configuration of the image processing device 100 according to the first embodiment. The image processing device 100 is configured by a computer, and the computer has a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, a display unit 205, an operation unit 206, a communication unit 207, and a bus 208, as shown as an example in Fig. 2.

[0020] The CPU 201 controls the computer using programs or data stored in the ROM 202 or RAM 203, causing the computer to function as each unit included in the image processing device 100 shown in FIG. 1 . The image processing device 100 may have one or more dedicated hardware components different from the CPU 201, and at least a portion of the processing performed by the CPU 201 may be executed by the dedicated hardware components. Examples of the dedicated hardware components include an ASIC, an FPGA, and a DSP (digital signal processor). The ROM 202 stores programs and the like that do not require modification. The RAM 203 temporarily stores programs or data supplied from the auxiliary storage device 204, or data and the like that are supplied from the outside via the communication unit 207. The auxiliary storage device 204 is formed, for example, by a hard disk drive or the like, and stores various data such as image data or audio data.

[0021] The display unit 205 is configured, for example, by a liquid crystal display or LED, and displays a GUI (Graphical User Interface) or the like for the user to operate or view the image processing device 100. The operation unit 206 is configured, for example, by a keyboard, mouse, touch panel, or the like, and receives operations by the user and inputs various instructions to the CPU 201. The CPU 201 also operates as a display control unit that controls the display unit 205 and an operation control unit that controls the operation unit 206. The communication unit 207 is used for communication with devices external to the image processing device 100. For example, when the image processing device 100 is connected to an external device via a wired connection, a communication cable is connected to the communication unit 207. When the image processing device 100 has a function for wireless communication with an external device, the communication unit 207 is equipped with an antenna. The bus 208 connects each unit included in the image processing device 100 to transmit information. In the first embodiment, the display unit 205 and the operation unit 206 are described as existing inside the image processing device 100, but at least one of the display unit 205 and the operation unit 206 may exist as a separate device outside the image processing device 100.

[0022] The operation of the image processing device 100 will be described with reference to Figures 3 and 4. Figure 3 is a flowchart showing an example of the processing flow of the image processing device 100 according to the first embodiment. Figure 4 is a schematic diagram showing the state of shape data that changes as a result of processing by the image processing device 100 according to the first embodiment. In the following description, the symbol "S" means step.

[0023] First, in S301, the information acquisition unit 101 acquires volume data 402a and camera parameters 402b. Here, the volume data 402a represents, as an example, volume data calculated by a volume intersection method using a silhouette image created based on captured images of an object 401a with an elliptical cross section captured from multiple viewpoints. For illustrative purposes, a region 401b, which is a part of the surface of the object 401a, represents a region erroneously determined as background in foreground / background separation of a captured image captured from viewpoint 401c, which is one of the multiple viewpoints. Due to the region erroneously determined as background, the silhouette image of the captured image captured from viewpoint 401c shows the region corresponding to region 401b as background. As a result, a through-hole-shaped defect having a cross section corresponding to the shape of the region in question is generated in the volume data 402a.

[0024] Next, in S302, the surface selection unit 102 selects multiple surface voxels 403 from the ON voxels of the volume data 402a acquired in S301. Specifically, for example, the surface selection unit 102 selects as surface voxels an ON voxel having at least one OFF voxel among six adjacent voxels on each of six sides of the ON voxel. Note that when an ON voxel is adjacent to a bounding box, the surface selection unit 102 selects surface voxels assuming that an OFF voxel exists on the periphery of the bounding box. Next, in S303, the visibility determination unit 103 determines, for each surface voxel 403 selected in S302, whether it is visible from each viewpoint at which the captured image was captured, using the camera parameters 402b acquired in S301. Based on this determination, the visibility determination unit 103 generates visibility information 404 a that indicates the number of visible viewpoints for each surface voxel 403 .

[0025] Next, in S304, the target identification unit 104 identifies surface voxels with a visible viewpoint count of 0 as target voxels based on the visibility information 404a generated in S303. Next, in S305, the shape modification unit 105 modifies the shape of the volume data 402a acquired in S301 based on the target voxels identified in S304 to acquire modified volume data 406. Specifically, for example, the shape modification unit 105 performs a predetermined expansion process, such as an expansion process by two voxels, on each target voxel to convert OFF voxels into ON voxels. The shape modification unit 105 acquires the volume data 405 after the expansion process as modified volume data 406 and outputs the acquired modified volume data 406 to the outside. After S305, the image processing device 100 ends the processing of the flowchart shown in FIG. 3.

[0026] As described above, according to the image processing apparatus 100 according to the first embodiment, only the OFF voxels around the target voxel can be converted to ON voxels, and therefore, missing portions of the volume data can be selectively corrected.

[0027] In the first embodiment, the volume data to be processed is described as being composed of voxels with binary voxel values. However, as described above, the voxel values ​​may be multi-valued or continuous. When the voxel values ​​are multi-valued or continuous, for example, the surface selection unit 102 selects surface voxels corresponding to the surface of the object by comparing the voxel values ​​with a predetermined threshold. Here, the threshold may be one that the surface selection unit 102 has stored in advance, or one that the information acquisition unit 101 has read and acquired from the ROM 202 or the like. Similarly, when the voxel values ​​are multi-valued or continuous, the visibility determination unit 103 determines whether the surface voxels are visible by, for example, comparing the voxel values ​​with a predetermined threshold to determine whether the voxels obstruct the above-mentioned line segments.

[0028] Furthermore, in the first embodiment, the volume data to be processed is described as being generated by the volume intersection method. However, the method is not limited to the volume intersection method as long as it is a method for generating volume data from captured images obtained by capturing images from multiple viewpoints. Specifically, for example, the volume data may be generated using multiple depth map images obtained by capturing images from multiple viewpoints using a depth camera. The volume data can be generated using the TSDF (Truncated Signed Distance Function) method, which is a method for representing three-dimensional shapes. Furthermore, for example, a point cloud may be acquired using the PMVS (Patch-based Multi View Stereo) method, which acquires feature amounts of each captured image obtained by capturing images from multiple viewpoints and creates a three-dimensional point cloud from the acquired feature amounts. Furthermore, the volume data may be generated by converting the acquired point cloud into volume data.

[0029] In the first embodiment, the information acquisition unit 101 is described as acquiring pre-generated volume data, but the information acquisition unit 101 may acquire captured images obtained by capturing images from multiple viewpoints instead of volume data. In this case, the information acquisition unit 101 acquires the volume data by generating the volume data based on the acquired captured images.

[0030] For example, the information acquisition unit 101 performs foreground / background separation on each captured image to generate a silhouette image corresponding to each captured image. Furthermore, the generated silhouette image is used to generate volume data by a volume intersection method, thereby acquiring the volume data. The information acquisition unit 101 may acquire two-dimensional images, such as depth images or RGB images, captured from multiple viewpoints, and generate volume data based on the acquired two-dimensional images to acquire the volume data. Specifically, for example, when the information acquisition unit 101 acquires a depth image, the information acquisition unit 101 can generate volume data by using the TSDF method. Furthermore, when the information acquisition unit 101 acquires an RGB image, the information acquisition unit 101 can generate volume data by using the PMVS method.

[0031] [Second embodiment] An image processing device 100 according to the second embodiment will be described with reference to FIGS. 5 to 7. The image processing device 100 according to the first embodiment identifies all surface voxels for which the number of visible viewpoints is 0 as target voxels and performs expansion processing on the identified target voxels. In contrast, the image processing device 100 according to the second embodiment (hereinafter simply referred to as "image processing device 100") adds processing for excluding surface voxels near a bounding box from the target voxels, among surface voxels for which the number of visible viewpoints is 0. Furthermore, the image processing device 100 adds processing for erosion of voxels that have been changed to ON voxels by the expansion processing. Furthermore, the image processing device 100 adds processing for generating a silhouette image based on the acquired corrected volume data and generating volume data using the generated silhouette image.

[0032] Like the image processing device 100 according to the first embodiment, the image processing device 100 includes an information acquisition unit 101, a surface selection unit 102, a visibility determination unit 103, an object identification unit 104, and a shape correction unit 105, examples of which are shown in FIG. 1. Note that the information acquisition unit 101, the surface selection unit 102, and the visibility determination unit 103 according to the second embodiment are similar to the information acquisition unit 101, the surface selection unit 102, and the visibility determination unit 103 according to the first embodiment, and therefore description thereof will be omitted. Hereinafter, the information acquisition unit 101, the surface selection unit 102, the visibility determination unit 103, the object identification unit 104, and the shape correction unit 105 according to the second embodiment will be simply referred to as the information acquisition unit 101, the surface selection unit 102, the visibility determination unit 103, the object identification unit 104, and the shape correction unit 105.

[0033] The target identification unit 104 identifies unit elements to be processed. Specifically, similar to the target identification unit 104 according to the first embodiment, the target identification unit 104 first identifies, as target voxels, surface voxels for which the number of visible viewpoints is 0 from among the surface voxels based on the visibility information 404a. Next, the target identification unit 104 excludes, from the identified surface unit elements to be processed, unit elements whose distance from a circumscribing rectangle preset in three-dimensional space is equal to or less than a predetermined threshold. Specifically, the target identification unit 104 excludes, from the identified target voxels, voxels around the bounding box of the volume data as non-target voxels (hereinafter referred to as "non-target voxels"). In other words, the target identification unit 104 identifies, as target voxels, surface voxels obtained by excluding the non-target voxels from among the surface voxels for which the number of visible viewpoints is 0.

[0034] The shape modification unit 105 modifies the 3D shape data based on the surface unit elements of the processing target identified by the target identification unit 104. That is, the shape modification unit 105 modifies the shape of the volume data acquired by the information acquisition unit 0101 based on the target voxels identified by the target identification unit 104. Specifically, the shape modification unit 105 first performs an expansion process to convert OFF voxels surrounding the target voxels into ON voxels, similar to the shape modification unit 105 according to the first embodiment. Next, the shape modification unit 105 performs an erosion process on the voxels converted to ON voxels by this conversion, thereby modifying the shape of the volume data. By performing this erosion process, when surface voxels that are not part of the volume data are locally identified as target voxels, the influence of the expansion process on the target voxels can be reduced. For example, the shape modification unit 105 acquires the volume data after the erosion process as corrected volume data after the correction.

[0035] Furthermore, the shape modification unit 105 may not only modify the 3D shape data based on the surface unit elements to be processed, but may also generate new 3D shape data based on the modified 3D shape data. That is, after modifying the volume data, the shape modification unit 105 may generate new volume data based on the modified volume data. Specifically, first, the shape modification unit 105 uses the modified volume data to generate silhouette images of an object represented by the modified volume data as viewed from each of multiple viewpoints. Here, each of the multiple viewpoints corresponds, for example, to the position at which each of the captured images described above was captured. The silhouette images generated by the shape modification unit 105 are silhouette images corresponding to each of the captured images described above, with missing portions corrected. Next, the shape modification unit 105 projects the generated silhouette images into the three-dimensional space of the volume data, and generates new volume data in which regions outside the object, i.e., regions outside the object (hereinafter referred to as "non-object regions"), are designated as OFF voxels. Through the above-described processing, the shape modification unit 105 can acquire newly generated volume data using the silhouette images with missing portions corrected. The shape correction unit 105 may acquire the newly generated volume data as corrected volume data.

[0036] The operation of the image processing device 100 will be described with reference to Figs. 5 to 7. Fig. 5 is a flowchart showing an example of a processing flow of the image processing device 100 according to the second embodiment. Fig. 6 is a schematic diagram showing the state of shape data that changes due to processing by the visibility determination unit 103, the object identification unit 104, and the shape correction unit 105 provided in the image processing device 100 according to the second embodiment. Fig. 7 is an explanatory diagram for explaining processing by the object identification unit 104 according to the second embodiment. In the following explanation, the symbol "S" means step. Furthermore, in Figs. 5 and 6, explanations of components that are assigned the same reference numerals as in Fig. 3 or 4 will be omitted.

[0037] First, the image processing device 100 executes the processes from S301 to S303. After S303, in S501, the target identification unit 104 identifies surface voxels having a visible viewpoint count of 0 as target voxels based on the visibility information 404a generated in S303. Next, in S502, the target identification unit 104 calculates the shortest distance from the bounding box of the volume data for each identified target voxel, and excludes voxels whose calculated distance is equal to or less than a predetermined threshold from the target voxels. This process makes it possible to determine non-target voxels 703, which are voxels not to be processed, from among the surface voxels 702 in the volume data 701 including the ground contact surface. Specifically, this process makes it possible to exclude voxels corresponding to regions of the volume data 701 that are missing parts and have low visibility, such as the soles of the feet, as non-target voxels 703 and exclude them from the target of the correction process by the shape correction unit 105.

[0038] Next, in S503, the shape modification unit 105 performs expansion and erosion processes based on the target voxels identified in S502 to modify the shape of the volume data 402a acquired in S301. Specifically, similar to the shape modification unit 105 according to the first embodiment, the shape modification unit 105 first performs expansion based on the target voxels identified in S502 to convert OFF voxels into ON voxels, thereby acquiring volume data 405 after expansion. Next, the shape modification unit 105 performs erosion 606 on the voxels in the acquired volume data 405 that have been converted to ON voxels by the expansion process, thereby acquiring volume data 607 after erosion. By performing the expansion and erosion processes in this manner, even if surface voxels that are not part of the volume data are identified as target voxels, the influence of the modification due to the expansion process can be reduced by the erosion process. The shape modification unit 105 then outputs the acquired volume data 607 to the outside as first modified volume data, for example.

[0039] Next, in S504, the shape correction unit 105 generates silhouette images 608 of the volume data 607 viewed from each of a plurality of viewpoints, using the camera parameters acquired in S301 and the volume data 607 after erosion processing acquired in S503. Next, in S505, the shape correction unit 105 generates new volume data 609 using the silhouette images 608 generated in S504. Through this processing, the shape correction unit 105 can acquire newly generated volume data using the silhouette images in which the missing portions have been corrected. The shape correction unit 105 outputs the newly generated volume data 609 to the outside, for example, as second corrected volume data.

[0040] As described above, the image processing apparatus 100 according to the second embodiment can selectively correct missing portions of volume data.

[0041] [Third embodiment] An image processing device 100 according to the third embodiment will be described with reference to Figures 8 and 9. The image processing device 100 according to the first embodiment selects ON voxels as surface voxels, identifies surface voxels with a visible viewpoint count of 0 as target voxels, and converts OFF voxels surrounding the target voxels into ON voxels. In contrast, the image processing device 100 according to the third embodiment (hereinafter simply referred to as "image processing device 100") selects OFF voxels as surface voxels, and identifies surface voxels with a visible viewpoint count of 0 or 1 as target voxels. Furthermore, the image processing device 100 converts target voxels that are OFF voxels into ON voxels.

[0042] Like the image processing device 100 according to the first embodiment, the image processing device 100 includes an information acquisition unit 101, a surface selection unit 102, a visibility determination unit 103, an object identification unit 104, and a shape correction unit 105, examples of which are shown in Fig. 1. Note that the information acquisition unit 101 and the visibility determination unit 103 according to the third embodiment are similar to the information acquisition unit 101 and the visibility determination unit 103 according to the first embodiment, and therefore a description thereof will be omitted. Hereinafter, the information acquisition unit 101, the surface selection unit 102, the object identification unit 104, and the shape correction unit 105 according to the third embodiment will be simply referred to as the information acquisition unit 101, the surface selection unit 102, the object identification unit 104, and the shape correction unit 105.

[0043] Similar to the surface selection unit 102 according to the first embodiment, the surface selection unit 102 selects a plurality of surface voxels from a plurality of voxels included in the volume data acquired by the information acquisition unit 101. Here, the surface voxels selected by the surface selection unit 102 are OFF voxels from two voxels where adjacent voxels are an ON voxel and an OFF voxel. Similar to the target identification unit 104 according to the first embodiment, the target identification unit 104 identifies, as target voxels, surface voxels whose number of visible viewpoints is equal to or less than a predetermined number based on visibility information. Specifically, the target identification unit 104 identifies, from among the surface voxels that are OFF voxels, voxels whose number of visible viewpoints is equal to or less than 1, i.e., whose number of visible viewpoints is 0 or 1, as target voxels.

[0044] The shape modification unit 105 modifies the shape of the volume data acquired by the information acquisition unit 0101, based on the target voxels identified by the target identification unit 104. Specifically, the shape modification unit 105 performs a process to modify the shape of the volume data by converting target voxels that are OFF voxels into ON voxels, and acquires a modified volume. In addition to converting the target voxels into ON voxels, the shape modification unit 105 may also perform a dilation process to convert OFF voxels surrounding the target voxels into ON voxels, and acquires a modified volume.

[0045] The operation of the image processing device 100 will be described with reference to Figures 8 and 9. Figure 8 is a flowchart showing an example of the processing flow of the image processing device 100 according to the third embodiment. Figure 9 is a schematic diagram showing the state of shape data that changes as a result of processing by the image processing device according to the third embodiment. In the following description, the symbol "S" means step. In Figures 8 and 9, the same reference numerals as those in Figures 3 and 4 will not be described.

[0046] First, the image processing device 100 executes the process of S301. After S301, in S802, the surface selection unit 102 selects a plurality of OFF voxels 901 corresponding to the surface of the object from among the OFF voxels of the volume data 402a acquired in S301. Next, in S803, the visibility determination unit 103 determines, for each surface voxel 901 selected in S802, whether or not the captured image is visible from each viewpoint from which it was captured, using the camera parameters 402b acquired in S301. Based on this determination, the visibility determination unit 103 generates visibility information 902 indicating the number of viewpoints from which each surface voxel 901 is visible.

[0047] Next, in S804, the object identification unit 104 identifies, as object voxels, surface voxels for which the number of visible viewpoints is 0 or 1, based on the visibility information 902 generated in S803. Next, in S805, the shape modification unit 105 converts the object voxels 903, which are OFF voxels, into ON voxels, based on the object voxels identified in S804. Through this conversion, the shape modification unit 105 modifies the shape of the volume data 402a acquired in S301, and outputs the modified volume data to the outside as a modified volume 904. After S805, the image processing device 100 ends the processing of the flowchart shown in FIG.

[0048] As described above, the image processing apparatus 100 according to the third embodiment can convert only target voxels that are OFF voxels, or only target voxels and their surrounding OFF voxels, to ON voxels, thereby selectively correcting missing portions of volume data.

[0049] [Other embodiments] The present disclosure can also be realized by supplying a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0050] It should be noted that within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted. [Explanation of symbols]

[0051] 100 Image processing device 101 Information acquisition department 102 Surface Selection Section 103 Visibility judgment section 104 Target Identification Department 105 Shape correction section

Claims

1. an acquisition means for acquiring camera parameters for each of a plurality of viewpoints and three-dimensional shape data representing a shape of an object, the three-dimensional shape data being generated based on the camera parameters and a plurality of captured images obtained by capturing images from each of the plurality of viewpoints; a selection means for selecting a plurality of elements corresponding to a surface of the object from among a plurality of elements included in the three-dimensional shape data; a viewpoint specifying means for specifying the number of viewpoints visible from each of the plurality of viewpoints for each of the plurality of elements selected by the selecting means based on the camera parameters; an element specifying means for specifying an element to be processed from the selected plurality of elements based on the number of viewpoints specified by the viewpoint specifying means; a correcting means for correcting the three-dimensional shape data based on the element to be processed; Having An image processing device comprising:

2. The element specifying means specifies, from among the selected plurality of elements, an element for which the number of viewpoints specified by the viewpoint specifying means is equal to or less than a predetermined number, as the element to be processed.

2. The image processing device according to claim 1, wherein:

3. The modifying means generates a plurality of silhouette images showing regions of the object corresponding to the plurality of viewpoints based on the modified three-dimensional shape data.

3. The image processing device according to claim 1, wherein:

4. the correcting means generates the plurality of silhouette images by projecting the corrected three-dimensional shape data onto each of the plurality of viewpoints.

4. The image processing device according to claim 3, wherein:

5. The correction means generates three-dimensional shape data by projecting each of the generated silhouette images into a three-dimensional space.

5. The image processing device according to claim 3, wherein:

6. The element specifying means excludes, from the elements to be processed that are specified by the element specifying means, elements whose distance from a circumscribing rectangle that is set in advance in a three-dimensional space is equal to or less than a predetermined threshold.

6. The image processing device according to claim 1, wherein:

7. The three-dimensional shape data is volume data in which each of the plurality of elements is a voxel.

7. The image processing device according to claim 1, wherein:

8. each of the selected elements is an ON voxel, which is a voxel having a voxel value of 1, corresponding to a surface of the object; The element specifying means specifies, from among the selected plurality of elements, an element for which the number of viewpoints specified by the viewpoint specifying means is 0, as the element to be processed.

8. The image processing device according to claim 7,

9. the selected elements are OFF voxels, which are voxels with a voxel value of 0, corresponding to a surface of the object; The element specifying means specifies, from among the selected plurality of elements, an element for which the number of viewpoints specified by the viewpoint specifying means is one or less, as the element to be processed.

8. The image processing device according to claim 7,

10. The modifying means modifies the three-dimensional shape data acquired by the acquiring means by converting the element to be processed into an ON voxel, which is a voxel with a voxel value of 1. The image processing device according to claim 9 ,

11. The modifying means modifies the three-dimensional shape data acquired by the acquiring means by converting elements that are OFF voxels, which are voxels with a voxel value of 0, among elements surrounding the element to be processed, into the ON voxels.

11. The image processing device according to claim 8 or 10,

12. The correction means converts the elements that are the OFF voxels into the ON voxels, and converts the elements to be processed that correspond to the surface of the object among the elements converted into the ON voxels from the ON voxels to the OFF voxels. The image processing device according to claim 11 ,

13. an acquisition step of acquiring camera parameters for each of a plurality of viewpoints and three-dimensional shape data representing a shape of an object, the three-dimensional shape data being generated based on the camera parameters and a plurality of captured images obtained by capturing images from each of the plurality of viewpoints; a selection step of selecting a plurality of elements corresponding to a surface of the object from among a plurality of elements included in the three-dimensional shape data; a viewpoint specifying step of specifying the number of viewpoints visible from each of the plurality of viewpoints for each of the plurality of elements selected in the selection step based on the camera parameters; an element specifying step of specifying an element to be processed from the selected plurality of elements based on the number of viewpoints specified by the viewpoint specifying step; a correction step of correcting the three-dimensional shape data based on the element to be processed; Having An image processing method comprising:

14. A program for causing a computer to operate as the device according to any one of claims 1 to 12.

Images