Image processing device, control method, and program

Abstract

To acquire a smooth surface shape from a binary volume.SOLUTION: An image processing apparatus of the present invention has: acquisition means that acquires a binary volume; first generation means that generates a multi-valued volume by performing filter processing on the binary volume acquired by the acquisition means; and second generation means that generates a surface shape of a three-dimensional shape model based on the multi-valued volume generated by the generation means.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus that generates a smooth surface shape from a volume.

Background Art

[0002] In video creation technology, there is volumetric video technology that reconstructs information in a three-dimensional space and visualizes it from an arbitrary viewpoint. In volumetric video technology, one method for generating a three-dimensional shape model of a subject is the volume intersection method. In the volume intersection method, the subject is photographed from multiple viewpoints, and a mask image (silhouette image) representing the silhouette of the subject is obtained from the photographed images by foreground / background separation processing, and the overlapping portions of the masked subject regions in each image are volumized. The volume thus obtained has binary voxel values representing the inside and outside of the subject.

[0003] Also, in order to make such a volume easier to render, a surface shape may be generated from the volume and converted into the form of a mesh model or a surface point cloud. Patent Document 1 describes a technique for generating a surface shape using a surface generation pattern by referring to voxel values for a volume composed of a voxel grid.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, Patent Document1 has a problem that a smooth surface shape cannot be calculated for a binary volume obtained by the volume intersection method or the like.

[0006] Therefore, one of the objectives of this invention is to obtain a smooth surface shape from a binary volume. [Means for solving the problem]

[0007] The image processing apparatus of the present invention includes an acquisition means for acquiring a binary volume, A complexity calculation means calculates the amount or frequency of fluctuation of neighboring voxel values ​​in the spatial or temporal direction as complexity for each voxel constituting the binary volume acquired by the acquisition means, The binary volume obtained by the acquisition means , with a smoothing intensity corresponding to the complexity. The system comprises a first generation means that generates a multi-level volume by performing a filtering process, and a second generation means that generates the surface shape of a 3D shape model based on the multi-level volume generated by the first generation means. Furthermore, when the first generation means performs a smoothing process as a filtering process, it reduces the smoothing intensity in the filtering process of the highly complex local region. . [Effects of the Invention]

[0008] According to the present invention, a smooth shape can be obtained from a binary volume. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing an example of the configuration of an image processing apparatus according to the first embodiment. [Figure 2] This is a block diagram showing an example of the hardware configuration of an image processing device according to the first embodiment. [Figure 3] This flowchart shows an example of the processing flow of the image processing apparatus according to the first embodiment. [Figure 4] This is a schematic diagram illustrating the changes in shape data due to the processing flow of the image processing apparatus according to the first embodiment. [Figure 5] This is a schematic diagram illustrating the changes in shape data that occur in the image processing apparatus according to the first embodiment, through a processing flow that includes a process for changing the resolution. [Figure 6] This is a block diagram showing an example of the detailed configuration of the multi-level volume generation unit according to the second embodiment. [Figure 7] This flowchart shows an example of a part of the processing flow of the image processing apparatus according to the second embodiment. [Figure 8]It is a schematic diagram showing the mode of shape data that changes by the processing of the processing flow of the image processing apparatus according to the second embodiment. [Figure 9] It is a flowchart showing a part of the processing flow of the image processing apparatus according to the third embodiment. [Figure 10] It is a schematic diagram showing the mode of shape data that changes by the processing of the processing flow of the image processing apparatus according to the third embodiment. [Figure 11] It is a schematic diagram showing the mode of shape data that changes by the processing flow of the image processing apparatus according to the first modification of the third embodiment. [Figure 12] It is a block diagram showing an example of the detailed configuration of the multi-value volume generation unit according to the fourth embodiment. [Figure 13] It is a flowchart showing an example of the processing flow of the image processing apparatus according to the fourth embodiment. [Figure 14] It is a schematic diagram showing the mode of shape data that changes by the processing of the processing flow of the image processing apparatus according to the fourth embodiment. [Figure 15] It is a diagram showing an example of the operation UI. [Embodiments for Carrying Out the Invention]

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the scope of the present disclosure is not limited only to those configurations.

[0011] [First Embodiment] Referring to FIGS. 1 and 2, the image processing apparatus 0100 according to the first embodiment will be described. First, referring to FIG. 1, the logical configuration of the image processing apparatus 0100 according to the first embodiment will be described. FIG. 1 is a block diagram showing an example of the logical configuration of the image processing apparatus 0100 according to the first embodiment. The image processing apparatus 0100 includes a binary volume generation unit 0101, a binary volume acquisition unit 0102, a multi-value volume generation unit 0103, and a surface shape generation unit 0104.

[0012] The binary volume generation unit 0101 takes multi-view captured images as input, generates a subject mask image by foreground-background separation processing, and generates a binary volume from the mask image using the viewing volume cross-section method.

[0013] The binary volume acquisition unit 0102 acquires the binary volume generated by the binary volume generation unit 0101.

[0014] Here, a volume is a data representation composed of multiple unit elements that divide a bounding box (also called a bounding rectangle) set in three-dimensional space into a grid. These unit elements are called voxels. Each voxel is set with data (hereinafter referred to as a "voxel value") indicating whether or not an object exists in three-dimensional space. In this embodiment, the voxel value is zero or a positive value, and can be binary, multi-valued (discrete), or continuous. In this embodiment, a volume with a binary value set as the voxel value is referred to as a binary volume, and a volume with a multi-valued or continuous value set as the voxel value is referred to as a multi-valued volume. The fineness of the unit elements that represent the bounding rectangle is referred to as the voxel resolution. That is, the voxel resolution increases as the size of the voxel decreases, and decreases as the size of the voxel increases.

[0015] In this embodiment, a voxel with a value of 1 in a voxel set with a binary value is partially or entirely located inside the object. Hereafter, a voxel with a value of 1 in a voxel set with a binary value will be referred to as an ON voxel. On the other hand, a voxel with a value of 0 (zero) in a voxel set with a binary value is not entirely located inside the object; that is, the entire voxel is located outside the object. Hereafter, a voxel with a value of 0 (zero) in a voxel set with a binary value will be referred to as an OFF voxel.

[0016] The multi-level volume generation unit 0103 calculates a weighted average value as a filtering process for the binary volume acquired by the binary volume acquisition unit 0102, and generates a multi-level volume corresponding to the shape of the binary volume.

[0017] The surface shape generation unit 0104 generates a mesh model as a surface shape based on the multi-level volume generated by the multi-level volume generation unit 0103. The surface shape generation unit 0104 outputs the generated mesh model to an external device.

[0018] The processing of each component of the image processing device 0100 is performed by hardware such as an ASIC (Application Specific Integrated Circuit) built into the image processing device 0100. Alternatively, the processing of each component of the image processing device 0100 may be performed by hardware such as an FPGA (Field Programmable Gate Array) built into the image processing device 0100. Furthermore, this processing may be performed by software using a CPU (Central Processor Unit) or GPU (Graphic Processor Unit) and memory.

[0019] Referring to Figure 2, the hardware configuration of the image processing device 0100 when each part of the image processing device 0100 operates as software will be described. Figure 2 is a block diagram showing an example of the hardware configuration of the image processing device 0100 according to the first embodiment. Here, the image processing device 0100 is composed of a computer. As shown as an example in Figure 2, the computer has a CPU 0201, ROM 0202, RAM 0203, auxiliary storage device 0204, display unit 0205, operation unit 0206, communication unit 0207, and bus 0208.

[0020] The CPU 0201 controls the computer using programs or data stored in ROM 0202 or RAM 0203, thereby causing the computer to function as the various components of the image processing device 0100 shown in Figure 1. The image processing device 0100 may have one or more dedicated hardware components separate from the CPU 0201, and at least a portion of the processing performed by the CPU 0201 may be executed by the dedicated hardware. Examples of dedicated hardware include ASICs, FPGAs, and DSPs (Digital Signal Processors). ROM 0202 stores programs and other data that do not require modification. RAM 0203 temporarily stores programs or data supplied from the auxiliary storage device 0204, or data supplied from external sources via the communication unit 0207. The auxiliary storage device 0204 is, for example, a hard disk drive and stores various types of data such as image data or audio data.

[0021] The display unit 0205 is composed of, for example, a liquid crystal display or LEDs, and displays a GUI (Graphical User Interface) for the user to operate or view the image processing device 0100. The operation unit 0206 is composed of, for example, a keyboard, mouse, or touch panel, and receives various instructions from the user and inputs them to the CPU 0201. The CPU 0201 also operates as a display control unit that controls the display unit 0205, and an operation control unit that controls the operation unit 0206. The communication unit 0207 is used for communication between the image processing device 0100 and external devices. For example, if the image processing device 0100 is connected to an external device by wire, a communication cable is connected to the communication unit 0207. If the image processing device 0100 has a function to communicate wirelessly with an external device, the communication unit 0207 is equipped with an antenna. The bus 0208 connects the various parts of the image processing device 0100 and transmits information. In the first embodiment, the display unit 0205 and the operation unit 0206 are described as being located inside the image processing device 0100, but at least one of the display unit 0205 and the operation unit 0206 may be located outside the image processing device 0100 as a separate device.

[0022] The operation of the image processing device 0100 will be described with reference to Figures 3 and 4. Figure 3 is a flowchart showing an example of the processing flow of the image processing device 0100 according to the first embodiment. Figure 4 is a schematic diagram illustrating the processing of a volume performed by the image processing device 0100 according to the first embodiment. In the following description, the symbol "S" represents a step.

[0023] In S0301, the binary volume generation unit 0101 acquires multi-viewpoint images. Multi-viewpoint images are multiple images taken from multiple viewpoints by multiple imaging devices (cameras). Each image is accompanied by information about the shooting conditions, including the camera's position at the time of shooting and information about the optical axis direction of the lens. Alternatively, if the shooting conditions are fixed regardless of the shooting and are recorded in ROM 0202, the binary volume generation unit 0101 reads that information from ROM 0202.

[0024] In S0302, the binary volume generation unit 0101 separates the foreground and background of the multi-view images acquired in S0301 and creates a mask image (silhouette image) of the object (subject). Here, the separation of the foreground and background is performed by identifying the subject and background, for example, by taking the difference with a previously captured background, or by identifying whether or not a specific color is the background color. Alternatively, the binary volume generation unit 0101 may separate the foreground and background using machine learning such as deep learning, utilizing a trained model for determining the foreground and background.

[0025] In S0303, the binary volume generation unit 0101 generates a binary volume from the mask image using the visual volume cross-eyed method. Specifically, first, the binary volume generation unit 0101 projects the mask image onto a predetermined voxel grid based on the mask image and information on the shooting conditions of each camera. Then, the binary volume generation unit 0101 generates a binary volume representing the shape of the object, assuming that the object exists in the overlapping area of ​​the object region of the mask image and the voxel grid.

[0026] In S0304, the binary volume acquisition unit 0102 acquires the binary volume 0401 generated in S0303. Figure 4(A) shows the binary volume. If the data format of the acquired binary volume is a compressed data format, the binary volume generation unit 0101 performs a decompression process (unzipping process) on the binary volume.

[0027] In Figures 4(A) to 4(E), the binary volume 0401 is represented as a two-dimensional grid of voxels. Each grid cell in Figures 4(A) to 4(E) is, in reality, a voxel (a three-dimensional volume). Figure 4(A) also shows the object shape 0401a before it is voxelized. In Figure 4(A), voxels with their centers inside the object shape 0401a are represented as ON voxels and are shown as black voxels. If a voxel center is outside the object shape 0401a, even if it overlaps with the object shape 0401a, it becomes an OFF voxel, resulting in an error of about half a voxel between the shapes of the binary volume 0401 and the object shape 0401a.

[0028] In step S0305, the multi-level volume generation unit 0103 first creates an 8-bit multi-level volume. Then, for each voxel value, the multi-level volume generation unit 0103 refers to the voxel at the same coordinate in the binary volume, assigning 0 to OFF voxels and 255 to ON voxels, thereby multi-leveling the data format. Next, the multi-level volume generation unit 0103 uses a weighted averaging filter 0402a on the 8-bit multi-level volume to generate a multi-level volume 0402b through the filtering process shown in Equation 1. A schematic diagram of the generated multi-level volume is shown in Figure 4(B). Here, Figure 4(C) shows an example of a schematic diagram of a multi-level volume when weighted averaging processing is performed. Through the above procedure, the multi-level volume generation unit 0103 generates an 8-bit multi-level volume with intermediate tones.

[0029]

number

[0030] Here, the voxel value V of the multivalued voxel at coordinate (i,j,k) m (x,y,z) is a binary volume V b It is obtained by convolution of (x,y,z) and filter F(x,y,z). X,Y,Z are the filtering ranges in the respective dimensional axes, and the weighted average filter 0402a is a 3x3 filter. Here, X,Y,Z are i-1≦X≦i+1, j-1≦Y≦j+1, and k-1≦Z≦k+1. In this embodiment, it is assumed that OFF voxels are set outside the display area of ​​the binary volume 0401.

[0031] Note that the filtering process is not limited to weighted average filtering, but may be other filtering methods. For example, Figure 4(D) shows an example of unweighted average filtering. When filtering is performed with this unweighted average filter 0402c, a multi-level volume 0402d is generated. In this embodiment, a multi-level volume with 8-bit integer values ​​is used as an example, but this does not limit the conditions. For example, a voxel value of 2 bits or more may be used in the multi-level volume, or a floating-point number that can be considered a continuous value, such as a Float type, may be used.

[0032] In S0306, the surface shape generation unit 0104 generates a smooth surface shape on the multi-level volume 0402 using the multi-level marching cube method. Figure 4(E) shows a mesh model 0403, which is an example of a smooth surface shape generated in this embodiment. Here, the marching cube method is a method that determines whether each voxel is inside or outside an object using a predetermined threshold, and generates a surface shape from the pattern of inside and outside. In this embodiment, the surface shape generation unit 0104 refers to the voxel values ​​in units of 2 × 2 × 2 eight voxels, and uses a predetermined threshold v THUsing this, the inside / outside determination of the object is performed for each voxel. Then, based on the inside / outside determination of the object, the surface shape generation unit 0104 determines the polygon to be generated from the inside / outside pattern of the voxels. Here, in two adjacent voxels, when the voxel values v0 and v1 of the two voxels satisfy v0 ≤ v TH < v1, vertices of the polygon are generated on the line connecting the centers of the two voxels. The vertex coordinates p of this polygon are calculated by Equation 2 using the voxel values v0 and v1 and the voxel center coordinates p0 and p1 of the two voxels respectively.

[0033]

Equation

[0034] Here, the density of the vertex coordinates p can be higher than the density of the voxel center coordinates. That is, at least a part of the density of the vertices of the surface shape determined by the surface shape generation unit 0104 is higher than the density of the voxel center coordinates. There is a part with a resolution higher than the voxel resolution in the resolution of the surface shape generated by the surface shape generation unit 0104. By such a vertex position determination method, the surface shape generation unit 0104 can generate a smooth surface shape.

[0035] As described above, using the voxel values of the multi-value volume, a smooth surface shape (mesh model) along the isosurface of the multi-value volume is generated. In FIG. (E), as an example, the mesh model 0403 schematically shows the surface shape when calculated with the threshold value v TH = 3.5 for the inside / outside determination of the marching cube. Here, the filter threshold value may use the setting previously held by the image processing apparatus 0100, or may be read and obtained by the surface shape generation unit 0104 from the ROM 0202 or the like. In this embodiment, as an example, a mesh model is created by the marching cube method, but a mesh model or a surface point cloud may be created by other methods and used as the surface shape.

[0036] In S0307, ​​the surface shape generation unit 0104 outputs the generated mesh model 0403 as a surface shape to an external device. If the external device is a display, the mesh model 0403 is displayed on the display. If the external device is a recording medium such as volatile memory or an HDD, it is output to the memory or storage medium as data in a predetermined file format. After S0307, ​​the image processing device 0100 completes the processing shown in the flowchart in Figure 3.

[0037] As described above, the image processing apparatus 0100 according to the first embodiment can generate a smooth surface shape by converting a binary volume into a multi-level volume through filtering and then calculating the surface shape.

[0038] In this embodiment, at S0305, the multi-level volume generation unit 0103 calculates the intermediate gradation of a multi-level volume with the same resolution as the binary volume by weighted averaging. However, it is also possible to generate a multi-level volume with a different resolution than the input binary volume. In this case, the multi-level volume generation unit 0103 calculates the intermediate gradation of the multi-level volume during the process of generating a multi-level volume with a different resolution. In other words, in this case, the multi-level volume generation unit 0103 performs a process to change the voxel resolution by filtering. An example of this case is shown in Figures 5(A) to (C).

[0039] Figure 5 is a schematic diagram illustrating the changes in shape data in the processing flow performed by the image processing device 0100. The multi-level volume generation unit 0103 creates the multi-level volume 0502 shown in Figure 5(B) by halving the resolution of the binary volume 0501 shown in Figure 5(A). The surface shape generation unit 0104 generates the mesh model 0503 shown in Figure 5(C) by applying the marching cube method to the multi-level volume 0502. Here, the sum of the voxel values ​​of the binary volume 0501 voxels contained within each multi-level voxel is defined as the voxel value of the multi-level volume. Note that while Figures 5(A) to (C) show an example of reducing the resolution, when increasing the resolution, the image processing device 0100 generates a multi-level volume with increased resolution through interpolation to create the surface shape. This allows the image processing device 0100 to generate a mesh model at any resolution, even when the resolution of the binary volume and the resolution of the mesh model are different. This allows the image processing device 0100 to process faster when the resolution of the mesh model is lower than that of the binary volume. Conversely, when the resolution is increased, the image processing device 0100 can generate a smoother surface shape.

[0040] [Second Embodiment] In the first embodiment, the image processing device 0100 generated a multi-level volume from a binary volume by spatially uniform filtering. However, for example, in the case of an object with a complex shape with uneven surfaces, ON voxels and OFF voxels are mixed in the binary volume, and there is a risk that the shape of the object may be distorted if spatially uniform smoothing is performed. Therefore, in the second embodiment, the complexity in a local region is calculated, and the filtering parameters are adaptively changed based on the complexity, thereby suppressing distortion of the shape of the generated object even for objects with complex shapes. The complexity will be described later.

[0041] The image processing apparatus 0100 according to the second embodiment will be described with reference to Figures 1 and 6. The image processing apparatus 0100 according to the second embodiment (hereinafter simply referred to as "image processing apparatus 0100") has the configuration of the block diagram shown in Figure 1. Here, as shown in Figure 6, in the image processing apparatus 0100, the multi-level volume generation unit 0103 is composed of a complexity calculation unit 0601 and a filter processing unit 0602. The complexity calculation unit 0601 calculates the spatial complexity from the binary volume acquired by the binary volume acquisition unit 0102. The filter processing unit 0602 performs filtering by adaptively changing parameters based on the complexity and generates a multi-level volume.

[0042] Since the binary volume generation unit 0101, binary volume acquisition unit 0102, and surface shape generation unit 0104 of the second embodiment are the same as those of the first embodiment, their description will be omitted.

[0043] The complexity calculation unit 0601 calculates the complexity based on the frequency of changes in voxel values ​​in the spatial direction of a local region in the binary volume acquired by the binary volume acquisition unit 0102. The specific method for calculating complexity will be described later.

[0044] The filter processing unit 0602 performs Gaussian filtering on the binary volume acquired by the binary volume acquisition unit 0102, adaptively changing parameters based on the complexity calculated by the complexity calculation unit 0601, to generate a multi-level volume. In this embodiment, the parameters are set to reduce the shape changes caused by filtering in regions with complex shapes. This suppresses the distortion of the shape of the generated object.

[0045] The operation of the image processing device 0100 will be described with reference to Figures 3, 7, and 8. A flowchart of the second embodiment is shown in Figure 3. Here, of the processes S0301 to S0307 in Figure 3, all except the process S0305 are the same as in the first embodiment, so a detailed explanation of the processes other than S0305 will be omitted. Figure 7 is a flowchart showing the detailed process of S0305 of the image processing device 0100 according to the second embodiment. Figures 8(A) to 8(D) are schematic diagrams showing the processing of a volume performed by the image processing device 0100 according to the second embodiment.

[0046] First, in the processing from S0301 to S0303, the binary volume generation unit 0101 generates a binary volume from the multi-view captured images. In S0304, the binary volume acquisition unit 0102 acquires the binary volume 0801 shown in Figure 8(A).

[0047] Next, in S0701, the complexity calculation unit 0601 counts the number of times each voxel switches between ON and OFF voxels for each axis, based on the binary volume 0801 acquired by the binary volume acquisition unit 0102.

[0048] Next, in S0702, the complexity calculation unit 0601 calculates the complexity from the total number of switching times in each axis direction for each voxel. Figure 8(B) shows complexity 0802, which is an example of the complexity calculated by the complexity calculation unit 0601. Here, the calculation process of complexity c by the complexity calculation unit 0601 in S0701 and S0702 is shown by the following formula 3.

[0049]

number

[0050] Here, the complexity c(i,j,k) at coordinate (i,j,k) is calculated as the sum of the absolute differences between adjacent voxels on each axis of the binary volume. X, Y, and Z are the complexity analysis ranges.

[0051] Here, as an example, we will explain a method for calculating complexity from the values ​​of two voxels in a cross shape around the voxel being analyzed, as shown in Figure 8(B) 0802a. As mentioned above, the complexity calculation unit 0601 uses the number of times the voxel switches between ON and OFF as the complexity. For example, let's calculate the complexity of voxel 0802b in Figure 8(A). First, the complexity calculation unit 0601 obtains the voxel value as shown in 0802c. In 0802c, the number of times the voxel switches between ON and OFF in the vertical direction is 1, and the number of times the voxel switches between ON and OFF in the horizontal direction is 4. In this case, the complexity calculation unit 0601 calculates the complexity of voxel 0802b as 5. In this embodiment, the number of times the binary voxel value switches was used as the complexity, but the power at a specific frequency or the frequency with high power may be used as the complexity based on frequency information obtained by analyzing the spatial frequency. Alternatively, the sum of the absolute differences between the voxel value of the voxel to be filtered and the voxel values ​​of neighboring voxels may be used as the complexity.

[0052] Next, in S0703, the filter processing unit 0602 adaptively changes the parameters of the Gaussian filter based on the complexity 0802, lowering the smoothing intensity in areas with high complexity and increasing the smoothing intensity in areas with low complexity, thereby performing smoothing. As a result, the filter processing unit 0602 generates a multi-level volume. Figure 8(C) shows a multi-level volume 0803, which is an example of a multi-level volume generated by the filter processing unit 0602. The three-dimensional Gaussian filter used in this embodiment is shown by equation 4.

[0053]

number

[0054] Here, G(x,y,z,σ) represents a three-dimensional Gaussian filter, x,y,z are coordinates, and σ of the filter represents the standard deviation. Here, the filter parameters are changed so that the standard deviation σ in Equation 4 changes based on complexity. For example, to adaptively change the standard deviation σ according to complexity, the standard deviation σ is defined using complexity c, as shown in Equation 5. σ = σ0(1-C) [Equation 5]

[0055] Here, σ0 is a predetermined standard deviation. The complexity c is normalized to a range of 0 to 1. The predetermined standard deviation σ0 may be a setting pre-stored by the image processing device 0100, or it may be obtained by the filter processing device 0602 by reading it from ROM 0202 or the like. By using the above filter, the distortion of the generated complex shape can be suppressed.

[0056] Subsequently, processing in S0306 generates a mesh model 0804 from the multi-level volume 0803 with a smooth surface shape and no breakdowns in complex shapes, as shown in Figure 8(D). Next, the surface shape 1005 is output by S0307, ​​but the processing in S0306 and S0307 is the same as in the first embodiment, so the explanation is omitted.

[0057] As described above, the image processing device 0100 according to the second embodiment makes it possible to acquire a mesh model while suppressing the breakdown of the complex shape of the binary volume.

[0058] [Third Embodiment] In the second embodiment, a multi-level volume was obtained from the binary volume of a single frame in the video by an adaptive spatial smoothing process according to the spatial complexity. However, if this process is performed over multiple frames, the variation in the shapes generated in each frame may result in an unnatural-looking video where the shape of the object changes irregularly. In other words, the method in the second embodiment does not take into account the temporal changes in surface shape. Because the temporal changes in surface shape are not taken into account, for example, the temporal shape changes of fast-moving objects in the video may become distorted. Therefore, in the third embodiment, the binary volumes of multiple frames are obtained, and a multi-level volume is obtained by an adaptive temporal smoothing process according to the temporal complexity. This generates a surface shape that changes smoothly over time while suppressing the distortion of the shape of fast-moving objects. An example is described below.

[0059] The image processing apparatus 0100 according to the third embodiment will be described with reference to Figures 1 and 6. The image processing apparatus 0100 according to the third embodiment (hereinafter simply referred to as "image processing apparatus 0100") has the same configuration as the image processing apparatus 0100 according to the second embodiment, so a detailed explanation will be omitted. The image processing apparatus 0100 according to the third embodiment (hereinafter simply referred to as "image processing apparatus 0100") performs the following processing. First, the binary volume generation unit 0101 acquires multi-view images of multiple frames and generates a binary volume for each frame. Next, the binary volume acquisition unit 0102 acquires the binary volumes of multiple frames, the complexity calculation unit 0601 calculates the complexity in the time direction, and the filter processing unit 0602 performs multi-level processing by filtering in the time direction based on the calculated complexity.

[0060] The binary volume generation unit 0101 acquires multi-view images of multiple frames, processes each frame in the same manner as the binary volume generation unit 0101 according to the first embodiment, and generates binary volumes of multiple frames.

[0061] The binary volume acquisition unit 0102 acquires binary volumes from multiple frames and performs decompression if the data format is compressed.

[0062] The complexity calculation unit 0601 calculates the amount of change in the time direction for each voxel of the binary volume of multiple frames acquired by the binary volume acquisition unit 0102, and uses this as the complexity.

[0063] The filter processing unit 0602 uses the amount of variation in voxel values ​​in the time direction in a local region obtained by the complexity calculation unit 0601 as the complexity, and performs Gaussian filtering by adaptively changing the smoothing intensity to obtain a multi-level volume from the binary volume of multiple frames.

[0064] The operation of the image processing device 0100 will be described with reference to Figures 3, 9, and 10. Figure 9 is a flowchart showing an example of the processing flow of the image processing device 0100 according to the third embodiment. Figure 10 is a schematic diagram illustrating the processing of a volume performed by the image processing device 0100 according to the third embodiment. In Figure 9, detailed explanations of components with the same reference numerals as in Figure 3 will be omitted.

[0065] In S0901, the binary volume generation unit 0101 acquires a multi-viewpoint image containing multiple frames.

[0066] From S0302 to S0303, the binary volume generation unit 0101 generates a binary volume for the multi-view captured image of each frame.

[0067] In S0902, the image processing device 0100 determines whether the binary volume generation process has been completed for all frames. If it has not been completed, it returns to S0302 and processes the frames for which processing has not been completed.

[0068] In S0903, the binary volume acquisition unit 0102 acquires binary volumes 1001 of multiple frames as shown in Figure 10(A). If the binary volumes 1001 of multiple frames are created from data sampled at unequal time intervals, the binary volume acquisition unit 0102 may perform a resampling process to equalize the intervals by interpolation.

[0069] In S0904, the complexity calculation unit 0601 calculates the absolute difference between the voxel value of each voxel in the processing frame 1002 that is the target of the filtering process of the binary volume 1001 and the voxel value of the voxel at the same position in a neighboring frame. Here, the processing frame is the frame that is the target of the complexity analysis process and the filtering process. Here, the neighboring frames to the processing frame are, for example, frames that are within a range of 3σ0 using a predetermined standard deviation σ0 of the Gaussian filtering process performed in S0702 from the processing frame.

[0070] In S0905, the complexity calculation unit 0601 calculates the complexity as the sum of the absolute differences with each frame calculated in S0904. In this embodiment, the complexity c obtained by the processing in S0904 and S0905 is calculated by formula 6. Figure 10(B) shows an example of a schematic diagram of the complexity 1003 calculated by formula 6.

[0071]

number

[0072] Here, complexity c(i,j,k,n) is the complexity of frame number n at coordinate (x,y,z)=(i,j,k), V b (i,j,k,n) is the voxel value of frame number n at coordinate (x,y,z)=(i,j,k) of the binary volume, and T is the range of neighboring frames. Here, as an example, the sum of the absolute differences with the processing frame is used as the complexity, but values ​​analyzed by other methods shown in the complexity calculation unit 0601 of the second embodiment may also be used as the complexity.

[0073] In S0906, the filter processing unit 0602 performs a time-direction Gaussian filter on each voxel of the processing frame 1002 of the binary volume 1001 to generate a multi-level volume 1004 as shown in Figure 10(C). Here, the time-direction Gaussian filter is performed by adaptively changing the standard deviation σ based on the complexity 1003 and performing a smoothing process. The adaptively changing standard deviation σ can be set using the complexity 1003, as in the second embodiment. Note that this does not limit the time-direction smoothing process to a Gaussian filter; smoothing can also be performed by other processes such as a low-pass filter.

[0074] Subsequently, the image processing device 0100 acquires the surface shape 1005 through the process in S0306 and outputs the surface shape 1005 shown in Figure 10(D) in S0307. However, since this process is the same as in the first embodiment, a detailed explanation is omitted.

[0075] As shown in Figure 10(D), the surface shape 1005 generated from the multi-level volume 1004 shows that deformation in a single frame, such as 1005a, also begins in the preceding and succeeding frames, as shown in 1005b. In this way, the multi-level volume is smoothed in the time direction, and a mesh model that changes smoothly over time, as shown in Figure 10(D), is generated from a binary volume, as shown in Figure 10(A).

[0076] As described above, according to the image processing device 0100 of the third embodiment, a mesh model that deforms smoothly in time can be obtained by performing a time-direction filter on a binary volume of multiple frames.

[0077] [Modification 1 of the third embodiment] In the third embodiment, the sum of the absolute differences between the processing frame and its neighboring frames in the time direction was used as the complexity. However, since the complexity increases even with small changes in voxel values, the smoothing intensity decreases as the complexity increases, which can impair smoothness. Therefore, in the modified example 1, the image processing device 0100 performs a low-pass filter in the spatial direction on the calculated complexity to prevent the complexity from increasing due to small changes in voxel values. As a result, the image processing device 0100 smooths areas where voxel values ​​change locally while suppressing smoothing areas where they change globally, thereby achieving both suppression of shape distortion in the generated surface shape and smooth shape changes.

[0078] In the modified example 1, as shown in the flowchart in Figure 9, at S0905, the complexity calculation unit 0601 performs an averaging process as a low-pass filter on the calculated complexity to achieve surface smoothing. Then, at S0906, the filter processing unit 0602 performs adaptive Gaussian filtering in the time direction using the low-pass processed complexity.

[0079] Referring to Figure 11, the image processing apparatus 0100 according to Modification 1 of the Third Embodiment will be described. Figure 11 is a schematic diagram that schematically represents the data format in the processing of the image processing apparatus 0100 according to Modification 1 of the Third Embodiment. Here, the processing other than that of the complexity calculation unit 0601 has already been described in the Third Embodiment, so its explanation will be omitted.

[0080] First, the complexity calculation unit 0601 calculates the temporal complexity of the binary volume acquired by the binary volume acquisition unit 0102. Figure 11(A) shows the calculated complexity 1003. Next, the complexity calculation unit 0601 performs spatial low-pass filtering on the complexity 1003 to obtain the low-pass complexity 1101 shown in Figure 11(B). The multi-level volume generation unit 0103 performs temporal filtering by adaptively changing the filter parameters based on the low-pass complexity 1101. Figure 11(C) shows the multi-level volume 1102 generated by the multi-level volume generation unit 0103. Subsequently, the image processing device 0100 performs the same processing as in the third embodiment and outputs the mesh model 1103 shown in Figure 11(D).

[0081] As a result, when voxel value fluctuations occur in a spatially localized area, as shown in 1003b of Figure 11(A), the complexity is reduced as shown in 1101b of Figure 11(B). Conversely, when voxel value fluctuations occur in a spatially wide area, as shown in 1003a of Figure 11(A), the conventional complexity can be maintained as shown in 1101a of Figure 11(B). Thus, in Modification 1, by utilizing complexity to suppress the breakdown of the general shape due to smoothing, and by performing strong temporal smoothing in areas where voxel value changes are localized, changes in minute spatial irregularities are reduced, and a smooth surface shape can be obtained.

[0082] [Fourth Embodiment] In the third embodiment and its modification 1, a mesh model whose shape changes smoothly over time was created by performing temporal smoothing on a binary volume of multiple frames. However, it is conceivable that a spatially smooth mesh model may not be obtained by temporal smoothing alone. Therefore, in the fourth embodiment, the image processing device 0100 first analyzes the spatial complexity of the binary volume of multiple frames for each frame and performs adaptive spatial smoothing to obtain a multi-level volume. Next, the image processing device 0100 analyzes the temporal complexity of the obtained multi-level volume and performs adaptive temporal smoothing to generate a spatiotemporalally smoothed multi-level volume. Next, a mesh model is generated from the spatiotemporally smoothed multi-level volume. As a result, the image processing device 0100 can obtain a mesh model in which the mesh has a smooth surface shape for each frame and whose surface shape changes smoothly as a video.

[0083] The image processing apparatus 0100 according to the fourth embodiment will be described with reference to Figures 1 and 12. The image processing apparatus 0100 according to the fourth embodiment (hereinafter simply referred to as "image processing apparatus 0100") first acquires binary volumes of multiple frames in the binary volume acquisition unit 0102 and calculates the spatial complexity in the spatial complexity calculation unit 1201. Then, the image processing apparatus 0100 converts the binary volumes into multi-level volumes by spatial filtering in the spatial filtering unit 1202. Furthermore, the image processing apparatus 0100 calculates the temporal complexity in the temporal complexity calculation unit 1203 and converts the binary volumes into multi-level volumes by temporal filtering in the temporal filtering unit 1204.

[0084] The image processing device 0100 includes a binary volume generation unit 0101, a binary volume acquisition unit 0102, a multi-level volume generation unit 0103, and a surface shape generation unit 0104, as shown in Figure 1. Here, the multi-level volume generation unit 0103 according to the fourth embodiment includes a spatial complexity calculation unit 1201, a spatial filter processing unit 1202, a temporal complexity calculation unit 1203, and a temporal filter processing unit 1204, as shown in Figure 12.

[0085] Note that the binary volume generation unit 0101, binary volume acquisition unit 0102, and surface shape generation unit 0104 according to the fourth embodiment are the same as those according to the third embodiment. Therefore, a detailed explanation of these will be omitted.

[0086] The spatial complexity calculation unit 1201 calculates the spatial complexity of the local region of the volume of each frame for the binary volumes of multiple frames acquired by the binary volume acquisition unit 0102, similar to the second embodiment.

[0087] The spatial filtering unit 1202 applies a Gaussian spatial filter to the binary volume by adaptively changing the filter parameters according to the spatial complexity, and obtains a multi-level volume from multiple frames.

[0088] The time complexity calculation unit 1203 calculates complexity for the multi-level volume calculated by the spatial filtering processing unit 1202 by finding the sum of the absolute differences between the voxel values ​​of each voxel in the processing frame and the voxel values ​​of other frames.

[0089] The time-filtering unit 1204 applies a Gaussian filter to the multi-level volume, adaptively changing the filter parameters according to the complexity in the time direction, and performs a Gaussian filter in the time direction to obtain a spatiotemporalally smoothed multi-level volume.

[0090] The operation of the image processing device 0100 will be described with reference to Figures 3, 13, and 14. The flowchart of the fourth embodiment is shown in Figure 3. Here, of the processes S0301 to S0307 in Figure 3, all except the process S0305 are the same as in the first embodiment, so a detailed explanation of the processes other than S0305 will be omitted. Figure 13 is a flowchart showing an example of a detailed process of S0305 in the flowchart shown in Figure 3 of the image processing device 0100 according to the fourth embodiment. Figure 14 is a schematic diagram that schematically represents the data format in the processing of the image processing device 0100 according to the fourth embodiment.

[0091] In S1301, the spatial complexity calculation unit 1201 performs a spatial complexity analysis on the volume of each frame of the binary volume 1401 acquired in S0304, similar to S0701 and S0702 of the second embodiment, and generates the spatial complexity 1402. Figure 14(A) shows a schematic diagram of the binary volume 1401 acquired in S0304, and Figure 14(B) shows a schematic diagram of the spatial complexity 1402 generated in S1301.

[0092] In S1302, the spatial filtering unit 1202 performs a spatial Gaussian filter on each frame of the binary volume 1401, similar to S0703 in the second embodiment, to generate the multi-level volume 1403 shown in Figure 14(C). Here, the spatial Gaussian filter is performed by adaptively changing the parameters based on the spatial complexity 1402.

[0093] In S1303, the time complexity calculation unit 1203 calculates the sum of the absolute differences between the voxel value of the processing frame and the voxel value of the neighboring frame for each voxel of the multilevel volume 1403 in the processing frame 1404. The calculated sum of absolute differences is the time complexity 1405. A schematic diagram of the time complexity 1405 is shown in Figure 14(D). Here, the complexity is calculated from the multilevel volume using equation 6. Note that equation 6 can also be calculated from a multilevel volume instead of a binary volume, so the complexity can be calculated using the same method as in the third embodiment. Note that, as shown in the second embodiment, the complexity may be calculated using the sum of the absolute differences between adjacent voxels, or frequency analysis may be performed and the power at a specific frequency or a frequency with strong power may be used as the complexity.

[0094] In S1304, the time filtering unit 1204 performs a time-direction Gaussian filter on the multi-level volume 1403 to generate a spatiotemporalally smoothed multi-level volume 1406 as shown in Figure 14(E). Here, the time-direction Gaussian filter is performed by adaptively changing the parameters based on the time complexity 1405. Note that the same filtering process as in the third embodiment can also be used for multi-level volumes.

[0095] In S0306, the surface shape generation unit 0104 obtains the mesh model 1407 shown in Figure 14(F) from the spatiotemporal smoothed multi-level volume 1406, and outputs this mesh model 1407 in S0307. Since the process is the same as in the first embodiment, a detailed explanation is omitted.

[0096] Through these processes, the image processing device 0100 can obtain a spatially and temporally smooth surface shape 1407 from a binary volume 1401 of multiple frames.

[0097] In the fourth embodiment, spatial processing was performed first, followed by temporal processing. However, the order of these processes may be reversed, with temporal processing being performed first, followed by spatial processing.

[0098] In addition, the spatial complexity calculation unit 1201 and the temporal complexity calculation unit 1203 may each perform spatial low-pass processing on the calculated complexity to convert it into low-pass complexity, similar to the complexity calculation unit 0601 in Modification 1 of the third embodiment. In that case, the spatial filter processing unit 1202 and the temporal filter processing unit 1204 adaptively change the filter processing parameters based on the low-pass complexity, similar to Modification 1 of the third embodiment.

[0099] Figure 15 shows an example of the operation UI displayed on the display unit 0205. The image display area 1501 displays the surface shape of the input volume, or a rendered image of the surface shape. Here, the image display area 1501 schematically displays a two-dimensional surface shape, but a three-dimensional surface shape may be displayed using shading or the like. Note that the operation UI shown in Figure 15 can be used in any of the first to fourth embodiments.

[0100] The input volume path setting area 1502 is the area where the file path of the binary volume acquired by the binary volume acquisition unit is set. When the read button in the input volume path setting area 1502 is pressed, the process S0304 in the processing flow shown in Figure 3 is executed. Note that this volume information may be recorded in ROM 0202 or the like, or it may be acquired externally by the communication unit 0207 via communication.

[0101] The display information setting area 1503 is an area for the user to specify (set) the type of surface shape to be displayed in the image display area 1501. The filter setting area 1504 is an area for the user to set whether or not to perform spatial and temporal filtering, the filter parameters, whether or not to use complexity, and whether or not to perform spatial smoothing of complexity, using radio buttons and sliders. When the settings in the filter setting area 1504 are changed, the S0305 process shown in Figure 3 is executed, and a surface shape generated with the filter settings set by the user is generated. Here, if the display of smooth surface shape in the display information setting area 1503 is turned on, the image display area 1501 displays the surface shape generated by the S0306 process shown in Figure 3 from the multi-level volume calculated according to the filter settings in the filter setting area 1504.

[0102] The play button 1505 is a UI button that executes a function to continuously display the surface shape shown in the image display area 1501 in frame number order. This allows the user to see the dynamic changes in the generated surface shape. The output button 1506 is a button for saving the smooth surface shape generated by the settings in the filter setting area 1504 to a file or memory, or outputting it to another application. When the output button 1506 is pressed, S0307 of the processing flow shown in Figure 3 is executed. In this way, users can change the settings for the smoothing process while referring to the surface shape displayed in the UI and manipulating the settings to see both the smoothed surface shape and the surface shape before smoothing. This makes it easy for users to create any desired smoothed surface shape.

[0103] [Other embodiments] In the first to fourth embodiments, the binary volume generation unit 0101 created a binary volume from multi-view images, and the surface shape generation unit 0104 generated a mesh model using the marching cube method. However, this is not limited to this input / output configuration, and other input / output configurations can be used.

[0104] The binary volume generation unit 0101 can accept data of other forms as input, not just multi-view images. As an example, the method for inputting a surface point cloud generated from a binary volume is shown. When the binary volume generation unit 0101 acquires a surface point cloud, it divides it using the voxel grid of the binary volume that is the source of the acquired surface point cloud, and voxels that contain the surface point cloud are designated as ON voxels. Next, the region surrounded by the ON voxels is also designated as an ON voxel to create a binary volume, which is then output to the binary volume acquisition unit. This is particularly effective when the binary volume is lightweight and stored, as storing data in surface point cloud format reduces the data size.

[0105] Furthermore, if the user directly inputs a binary volume in any of the first to fourth embodiments, the binary volume generation unit 0101 may not perform any processing, and the binary volume acquisition unit 0102 may directly acquire the binary volume from the user and begin processing.

[0106] The surface shape generation unit 0104 can generate not only mesh models but also other surface shape forms. As an example, a method for generating a surface point cloud is shown. First, the image processing device 0100 sets the voxel of interest to be processed and references adjacent voxels. Next, the image processing device 0100 determines pairs of voxels of interest and adjacent voxels where the voxel of interest value ≥ threshold > adjacent voxel value or the voxel of interest value ≤ threshold < adjacent voxel value. The image processing device 0100 assumes that the voxel value changes linearly between the voxel center coordinates of the determined voxel pair and generates vertices at the threshold position. By repeating the above process, a surface point cloud can be obtained.

[0107] Furthermore, the method for generating the mesh model is not limited to the marching cube method. Other methods such as the marching tetrahydro method or the dual contouring method may also be used to generate the mesh model.

[0108] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more of the functions.

[0109] Within the scope of this disclosure, it is possible to freely combine the embodiments, modify any component of each embodiment, or omit any component in each embodiment.

[0110] Furthermore, the disclosure of this embodiment includes the following configurations and methods.

[0111] (Composition 1) A means for obtaining a binary volume, A first generation means that generates a multi-level volume by performing a filtering process on a binary volume obtained by the acquisition means, A second generation means generates the surface shape of a 3D shape model based on the multi-level volume generated by the generation means, An image processing apparatus characterized by having (Configuration 2) The second generation means performs a spatial filtering process. The image processing apparatus according to configuration 1, characterized in that... (Composition 3) The acquisition means acquires binary volumes of multiple frames, The first generation means performs filtering in the time direction. The image processing apparatus according to configurations 1 and 2, characterized by the above. (Composition 4) The system further includes a third generation means that acquires multiple captured images taken from multiple viewpoints and generates a binary volume from the multiple captured images, The acquisition means acquires the binary volume generated by the third generation means. An image processing apparatus according to any one of configurations 1 to 3, characterized by the above. (Composition 5) The first generation means generates multi-valued voxels by performing a weighted average of each voxel in a binary volume with neighboring voxels as a filtering process. An image processing apparatus according to any one of configurations 1 to 4, characterized by the above. (Composition 6) The first generation means modifies the voxel resolution as a filtering process. An image processing apparatus according to any one of configurations 1 to 5, characterized by the above. (Composition 7) The second generation means determines the vertex positions with a resolution higher than the voxel resolution by referring to the voxel values ​​of the multi-level volume. An image processing apparatus according to any one of configurations 1 to 6, characterized by the above. (Composition 8) The second generation means performs an inside / outside determination for each voxel constituting the multi-level volume and generates a surface shape from the inside / outside patterns. The image processing apparatus according to configuration 7, characterized by the features described above. (Composition 9) The system includes a complexity calculation means that calculates the amount or frequency of variation of neighboring voxel values ​​in the spatial or temporal direction as complexity for each voxel constituting the binary volume acquired by the acquisition means, The first generation means locally modifies the filter intensity according to the complexity. An image processing apparatus according to any one of configurations 1 to 8, characterized by the above. (Composition 10) The complexity calculation means calculates the complexity based on frequency information in the spatial or temporal direction. The image processing apparatus according to configuration 9, characterized by the features described therein. (Composition 11) The first generation means reduces the smoothing intensity in the filtering of the highly complex local region when performing a smoothing process as a filtering process. An image processing apparatus according to any one of configurations 9 to 10, characterized by the above. (Composition 12) The complexity calculation means uses the amount or frequency of variation of the neighboring voxel values ​​as the complexity, obtained by spatially low-pass processing the variation amount or variation frequency. The first generation means performs filtering based on the complexity. An image processing apparatus according to any one of configurations 9 to 11, characterized by the above. (Method 13) The acquisition process for obtaining a binary volume, A first generation step involves generating a multi-level volume by applying a filter to the binary volume obtained in the acquisition step, A second generation step generates the surface shape of a 3D shape model based on the multi-level volume generated in the above generation step. A control method characterized by having the following features. (Program 14) The acquisition process for obtaining a binary volume, A first generation step involves generating a multi-level volume by applying a filter to the binary volume obtained in the acquisition step, A second generation step generates the surface shape of a 3D shape model based on the multi-level volume generated in the above generation step. A program to cause an image processing device to execute.

Claims

1. A means for obtaining a binary volume, A complexity calculation means calculates the amount or frequency of fluctuation of neighboring voxel values ​​in the spatial or temporal direction as complexity for each voxel constituting the binary volume acquired by the acquisition means, A first generation means generates a multi-level volume by filtering the binary volume obtained by the acquisition means with a smoothing intensity corresponding to the complexity, A second generation means generates the surface shape of a three-dimensional shape model based on the multi-level volume generated by the generation means, It has, The first generation means reduces the smoothing intensity in the filtering of the highly complex local region when performing a smoothing process as a filtering process. An image processing apparatus characterized by the following:

2. The second generation means performs a spatial filtering process. The image processing apparatus according to feature 1.

3. The acquisition means acquires binary volumes of multiple frames, The first generation means performs filtering in the time direction. The image processing apparatus according to feature 1.

4. The system further includes a third generation means that acquires multiple captured images taken from multiple viewpoints and generates a binary volume from the multiple captured images, The acquisition means acquires the binary volume generated by the third generation means. The image processing apparatus according to feature 1.

5. The first generation means generates multi-valued voxels by performing a weighted average of each voxel in a binary volume with neighboring voxels as a filtering process. The image processing apparatus according to feature 1.

6. The second generation means determines vertex positions by referring to the voxel values ​​of the multi-level volume in order to generate the surface shape of the three-dimensional shape model at a resolution higher than the voxel resolution. The image processing apparatus according to feature 1.

7. The second generation means performs an inside / outside determination for each voxel constituting the multi-level volume and generates a surface shape from the inside / outside patterns. The image processing apparatus according to claim 6.

8. The complexity calculation means calculates the complexity based on frequency information in the spatial or temporal direction. The image processing apparatus according to feature 1.

9. The complexity calculation means uses the amount or frequency of variation of the neighboring voxel values ​​as the complexity, obtained by spatially low-pass processing the variation amount or variation frequency. The first generation means performs filtering based on the complexity. The image processing apparatus according to feature 1.

10. The acquisition process for obtaining a binary volume, A complexity calculation step is performed to calculate the complexity of each voxel constituting the binary volume acquired in the acquisition step, by determining the amount or frequency of variation of neighboring voxel values ​​in the spatial or temporal direction. A first generation step involves generating a multi-level volume by filtering the binary volume obtained in the acquisition step with a smoothing intensity corresponding to the complexity, A second generation step generates the surface shape of a three-dimensional shape model based on the multi-level volume generated in the above generation step. It has, In the first generation step, when performing a smoothing process as a filtering process, the smoothing intensity is reduced in the filtering process of the highly complex local region. A control method characterized by the following:

11. The acquisition process for obtaining a binary volume, A complexity calculation step is performed to calculate the complexity of each voxel constituting the binary volume acquired in the acquisition step, by determining the amount or frequency of variation of neighboring voxel values ​​in the spatial or temporal direction. A first generation step that generates a multi-level volume by filtering the binary volume obtained in the acquisition step with a smoothing intensity corresponding to the complexity, wherein the first generation step reduces the smoothing intensity in the filtering of local regions with high complexity when performing the smoothing as a filtering process, A second generation step generates the surface shape of a three-dimensional shape model based on the multi-level volume generated in the above generation step. A program to cause an image processing device to execute.

Images