Image processing device, image processing method, program

Abstract

This provides image processing technology that reduces computational costs while suppressing visual inconsistencies in 3D data, including occlusion. [Solution] The image processing device 100 acquires 3D data having shape information and texture information from the imaging unit 106 and performs image processing. The resolution of the vertices constituting the shape information is smaller than the resolution of the texture information. The image processing unit 104 generates an additional shape to be added to the shape acquired in S101 (S102). The image processing unit 104 performs mapping between the shape information and the texture information (S103). In S103, the image processing unit 104 sets texture coordinates for the region of the additional shape generated in S102 and performs a process to map the acquired texture information.

Description

[Technical Field]

[0001] This invention relates to an image processing technique for three-dimensional data. [Background technology]

[0002] Imaging devices such as 3D cameras can acquire distance distribution information. By transforming this distance distribution information into a perspective projection, a point cloud can be obtained. Furthermore, by adding phase information between the vertices of the point cloud to generate polygon data, a 3D surface model can be created. By acquiring an image (color distribution information) simultaneously with the distance distribution information and generating 3D data with texture information, users can enjoy viewing from any virtual viewpoint.

[0003] Incidentally, in 3D data generated based on distance distribution information from a limited number of viewpoints, there may be areas that exist in the actual subject but cannot be restored as 3D data. For example, consider the creation of 3D data for a scene in which a hand is placed in front of a human body. In this case, information is lost as occlusion in a part of the human body that is obscured from the front by the hand. When rendering such 3D data from a viewpoint other than the front, the loss of information in the occluded area can be observed, potentially causing a sense of incongruity in the viewer.

[0004] Non-patent document 1 discloses an inpainting technique for reconstructing lost or degraded image data. For missing areas in the background caused by foreground occlusion, the shape and color information of the missing area are generated by deep learning based on image information of the background adjacent to the missing area, and then inpainting is performed. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Varun Jampani et al., "SLIDE: Single Image 3D Photography with Soft Layering and Depth-aware Inpainting", Proceedings of the IEEE International Conference on Computer Vision, 2021 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] In general, image data inpainting is computationally expensive. This effect is particularly pronounced when using high-resolution image information. In the prior art described in Non-Patent Document 1, the time required for inference may increase when the input image resolution is high. The objective of the present invention is to provide an image processing technique that reduces computational costs while suppressing the visual incongruity in 3D data including occlusion. [Means for solving the problem]

[0007] An image processing apparatus according to an embodiment of the present invention includes an acquisition means for acquiring three-dimensional data having shape information and texture information, a generation means for generating an additional shape to be added to the shape corresponding to the acquired shape information, and a matching means for matching the shape information with the texture information. The resolution of the vertices constituting the shape information is smaller than the resolution of the texture information, and the matching means matches the acquired texture information with the additional shape. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an image processing technique that suppresses the visual incongruity in 3D data including occlusion while further reducing computational costs. [Brief explanation of the drawing]

[0009] [Figure 1] It is a block diagram of an imaging device according to an embodiment. [Figure 2] It is a flowchart for explaining image processing according to an embodiment. [Figure 3] It is an explanatory diagram of an invisible region and a region with depth difference. [Figure 4] It is a conceptual diagram regarding an additional shape. [Figure 5] It is an explanatory diagram of a surface map. [Figure 6] It is a diagram for explaining the correspondence between a surface and a texture. [Figure 7] It is a diagram showing the positional relationship between an imaging means and a subject. [Figure 8] It is a diagram showing the correspondence between a distance map and a texture. [Figure 9] It is a schematic diagram showing surface maps and texture coordinate systems according to a comparative example and an embodiment. [Figure 10] It is a flowchart for explaining an association process in an embodiment. [Figure 11] It is a conceptual diagram regarding an additional shape in a first modification example. [Figure 12] It is a conceptual diagram for explaining the difference between a comparative example and a first modification example. [Figure 13] It is a diagram showing a reference region on a texture coordinate in a third modification example. [Figure 14] It is a flowchart for explaining image processing in a fourth modification example. [Figure 15] It is a diagram showing a modified region on a texture coordinate in a fourth modification example. [Figure 16] It is a diagram showing a reference region on a texture coordinate in a fifth modification example. [Figure 17] It is a schematic diagram showing surface maps and texture coordinate systems according to a comparative example, an embodiment, and a sixth modification example. [Figure 18] It is a diagram for explaining the correspondence between a distance map and a texture according to a seventh modification example. [Figure 19]This is a schematic diagram showing the surface map and texture coordinate system according to the seventh modified example. [Figure 20] This figure illustrates the correspondence between the distance map and texture in the eighth modified example. [Figure 21] This is a schematic diagram showing the surface map and texture coordinate system according to the eighth modified example. [Figure 22] This is a schematic diagram showing the surface map and texture coordinate system related to the comparative example. [Figure 23] This is a conceptual diagram showing a surface map relating to the ninth modified example. [Figure 24] This diagram shows the positional relationship between the imaging means and the subject in the 10th modified example. [Figure 25] This is a schematic diagram showing the surface map and texture coordinate system according to the Example and the 10th Modified Example. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The definitions of terms and preconditions used in relation to the image processing apparatus and image processing method according to the present invention will be explained below. • "3D data": Refers to data containing three-dimensional shape information of the object being processed, including point clouds and 3D surface models. It may also include color information in addition to shape information. • "Shape": Refers to vertices and the faces (surfaces) formed by the connections between vertices. • "Polygon": A face formed by connecting multiple vertices. A "surface" is a collection of polygons. • "Texture": Refers to color distribution information, or images, assigned to the shape of 3D data. • "Texture coordinates": Represents relative coordinates within a texture image, and is synonymous with, for example, UV coordinates.

[0011] Figure 1 is a block diagram showing an overview of the imaging device according to this embodiment. It shows an example of the configuration of an imaging device to which the image processing device 100 is applied. The imaging device comprises the image processing device 100 and the imaging unit 106.

[0012] The imaging unit 106 captures an image of the subject and outputs the acquired imaging information to the image processing device 100. For example, the imaging unit 106 may have a stereo camera or a time-of-flight camera configuration, or it may have a pupil-splitting image sensor with phase-difference detection. The imaging unit 106 is not limited to a specific configuration as long as it can acquire the information necessary for generating 3D data.

[0013] The image processing device 100 includes a control unit 101, a memory 102, a storage device 103, an image processing unit 104, and a data acquisition unit 105. The control unit 101 is equipped with a CPU (Central Processing Unit) and controls each block element of the image processing device 100. For example, the control unit 101 performs the process of acquiring imaging information from the imaging unit 106 via the data acquisition unit 105. The control unit 101 outputs the acquired imaging information to the image processing unit 104 and controls the storage device 103 to store the image processing results.

[0014] Memory 102 is used for storing temporary information, and a storage device such as volatile memory is used. The storage device 103 is a device capable of permanently storing information, and for example, an HDD (hard disk drive) is used. The storage device 103 stores imaging information acquired by the imaging unit 106, parameters used by the image processing unit 104, and so on.

[0015] The image processing unit 104 includes an additional shape generation unit 104a, a mapping processing unit 104b, and a texture correction unit 104c. Details of each unit will be described later. The image processing unit 104 applies predetermined image processing to the information acquired by the imaging unit 106 according to the image processing method shown below. The image processing unit 104 may be configured as an integrated circuit or as a functional module implemented by software.

[0016] The processing in this embodiment will be described with reference to Figure 2. Figure 2 is a flowchart illustrating the processing performed by the image processing unit 104 on the 3D data relating to the subject. Note that the process of generating 3D data relating to the subject based on the information acquired by the imaging unit 106 is well known, so a detailed explanation thereof will be omitted.

[0017] S101 corresponds to the step in which the data acquisition unit 105 acquires 3D data. In the 3D data acquisition step, processing is performed to acquire 3D data that has shape information and texture information. This step has the role of making the 3D data that will be processed in subsequent steps ready for handling. For example, the generated 3D data is imported and expanded into memory 102.

[0018] In this embodiment, it is assumed that the resolution of the vertices constituting the surface is smaller than the resolution of the texture. In other words, the total number of vertices constituting the surface is smaller than the number of pixels in the texture. Hereinafter, the texture at the end of process S101 will be referred to as the "acquired texture," and the surface at that time will be referred to as the "acquired surface." The region corresponding to the acquired surface will be referred to as the "acquired region." This acquired region corresponds to the shape region that is originally present and not an added shape. Of the regions included in the acquired 3D data, the region closer to the imaging means (imaging unit 106) will be referred to as the "first acquired region," and the region further away will be referred to as the "second acquired region."

[0019] The process proceeds from S101 to S102. S102 corresponds to the step in which the additional shape generation unit 104a generates additional shapes. In S102, additional shapes (vertices and surfaces) that do not exist in the surface acquired in S101 are generated. Details of the additional shapes will be described later.

[0020] The process proceeds from S102 to S103. S103 corresponds to the step in which the mapping processing unit 104b performs mapping between the surface and the texture. Generally, mapping between shape information and texture information of 3D data is performed by UV unwrapping. However, for shape information acquired from a single viewpoint, the image taken from that viewpoint can be used as a texture without UV unwrapping. The following explanation assumes that UV unwrapping is not performed, but UV unwrapping may be performed depending on the embodiment.

[0021] Refer to Figure 3 to explain invisible areas and areas with depth differences. Figure 3 is a schematic diagram to explain incomplete 3D data. Assume that the subject is positioned with a depth difference between the foreground and background sides relative to the imaging means (imaging unit 106). In this case, an invisible area of ​​the background subject is created due to occlusion by the foreground subject. In the case of 3D data consisting only of surfaces of the visible area (visible region) from the imaging means, the invisible area will be visible when the 3D data is rendered from a virtual viewpoint. No surface is generated for the invisible area of ​​the background subject, and a virtual viewpoint image with a hole in the 3D data is generated. Therefore, the user may feel a sense of incongruity when viewing the image.

[0022] Therefore, in this embodiment, a process is performed to generate a surface as an additional shape in areas other than the acquisition area (visible area) (Figure 2: S102). The area where the additional shape should be generated is the area near the area where there is a depth difference in the shape when the object related to the 3D data is viewed from the surface. Here, "surface" refers to the side of the polygon on which the texture is reflected. Generally, the texture is reflected on the surface where the normal of the polygon is positive, so viewing from the surface where the sign of the normal is positive can also be defined as "viewing from the surface". Furthermore, the "area with a depth difference" is the area where there is a depth difference between shapes, as shown in Figure 3. The area with a depth difference can be determined by the following method. A method for determining areas in 3D data where a surface is missing, like a hole. A method that sequentially focuses on each polygon constituting the surface, evaluates the Euclidean distance between the centroid coordinates of the polygon of interest and the centroid coordinates of the polygon adjacent to that polygon, and determines the region as one where the evaluated value is greater than or equal to a threshold.

[0023] An example of an added shape is explained with reference to Figure 4. In a region with a depth difference as viewed from the imaging means (imaging unit 106), the surface that connects the surface in the acquisition area on the far side and the surface in the acquisition area on the near side corresponds to the added shape. The depth difference corresponds to the difference in distance information or depth information in the line of sight direction or imaging direction. The added shape can be generated by manipulating the phase information so that the point cloud on the near side and the point cloud on the far side near the region with a depth difference are connected, and it is not always necessary to add a point cloud. In this way, a process is performed to generate an added shape that connects the vertices of the region with a depth difference (Figure 2: S102). As a result, even when rendering from a virtual viewpoint, the hole is not visible, so the sense of incongruity in viewing can be suppressed.

[0024] Figure 5 schematically illustrates a surface map. Although a surface is inherently three-dimensional information, in Figure 5 it is represented in a two-dimensional map format projected onto a projection plane using perspective projection. The surface map in Figure 5 represents the connection relationships of the point cloud on the projection plane with numerous line segments.

[0025] Refer to Figure 6 to explain the correspondence between surfaces and textures. Figure 6(A) shows the correspondence between the surface map and texture without texture coordinate correction. Figure 6(B) shows the correspondence between the surface map and texture with texture coordinate correction. In the texture coordinate system, the horizontal axis represents the U coordinate, and the vertical axis represents the V coordinate. Texture coordinate values ​​are relative coordinate values ​​normalized in the range of 0 to 1 on the texture coordinate system. Each vertex has one corresponding point in the texture coordinate system. Points in the surface map will be represented by uppercase alphabetic symbols, and points in the texture coordinate system will be represented by lowercase alphabetic symbols.

[0026] In FIG. 6(A), the vertices P, Q, and R of the triangle respectively correspond to the points p, q, and r in the texture coordinate system. When the three vertices P, Q, and R are in a connection relationship, the texture corresponding to the area of the triangle pqr in the texture coordinate system is assigned to the area of the triangle PQR. In the texture coordinate system, the range on the left side where the texture is dark gray is shown, and the range on the right side where the texture is light gray is shown. The area of the triangle pqr is within the range where the texture is dark gray. FIG. 6(B) shows the following modification examples. · Change the point p in the texture coordinate system corresponding to the vertex P to the point p

[0027] , . · Change the point q in the texture coordinate system corresponding to the vertex Q to the point q * . · Change the point r in the texture coordinate system corresponding to the vertex R to the point r * .

[0027] In the texture coordinate system, the area of the triangle pqr is within the range where the texture is dark gray, while the area of the triangle p * q * r * is within the range where the texture is light gray. That is, the texture associated with the triangle PQR in the surface map is changed from dark gray to light gray. The process by which the correspondence processing unit 104b changes the texture coordinate values in this way is referred to as "correction of texture coordinates" or simply "correction". Also, the magnitude of the difference vector between the texture coordinate values before and after correction is referred to as the "correction amount", and the direction of the difference vector is referred to as the "correction direction". The difference vector is the vector from the point p to the point p * heading to, the vector from the point q to the point q * heading to, and the vector from the point r to the point r * heading to, and the correction amount and correction direction are determined.

[0028] In step S103 of Figure 2, the process of setting texture coordinates is executed for the region of the added shape generated in S102. By setting the texture coordinates so that the acquired texture is assigned to the region of the added shape, it is possible to suppress the unnatural appearance of occlusion with lower computational cost. The reason why this effect can be achieved is explained below.

[0029] As mentioned above, there is prior art that suppresses unnaturalness by performing inpainting on textures in areas invisible to the imaging means (occlusion areas) based on information from nearby visible areas (acquired areas). Generally, since inpainting is performed by editing the image itself, there was a problem that the computational cost was high when the image resolution was high.

[0030] On the other hand, in this embodiment, the determination of the texture of the occlusion region is performed by UV assignment based on existing texture information, rather than editing the texture. When the resolution of the shape information is lower than the resolution of the texture, the amount of information that requires special assignment processing is reduced, making it possible to further reduce the computational cost.

[0031] In this embodiment, it is assumed that the resolution of the vertices constituting the surface is lower than the resolution of the image information. The 3D data format may include color information and 3D coordinate information for each vertex. When achieving high-resolution image representation, coordinate information is required along with color information, resulting in a large data size. Therefore, lowering the resolution of the shape information relative to the image information is effective, enabling higher-resolution image representation while reducing data size. Considering that for many subjects, the detail of the shape is coarser than the detail of the color information, the degradation of the 3D data quality due to lowering the resolution of the shape information than the resolution of the image information is limited.

[0032] Next, with reference to Figures 7 to 9, an overview of the process of mapping surfaces to textures (Figure 2: S103) will be described. Figure 7 shows the positional relationship between the imaging means (imaging unit 106) and the subject in the vicinity of a region with a difference in depth. The first subject (hereinafter referred to as "subject 1") is located in front of the imaging unit 106, and its texture is light gray. The second subject (hereinafter referred to as "subject 2") is located behind the imaging unit 106, and its texture is dark gray. The imaging unit 106 images subject 1 and subject 2, and 3D data is generated. The optical axis of the imaging unit 106 is shown by a dotted line passing through point o, and corresponds to the boundary line between subject 1 and subject 2. Shape information is acquired at intervals of points indicated by the numbers 1 to 6. Points indicated by 1 to 3 correspond to subject 2, and points indicated by 4 to 6 correspond to subject 1. The resolution of the vertices constituting the surface is smaller than the resolution of the image information. Therefore, the image information will be acquired at a finer resolution than the resolution corresponding to the interval between points 1 to 6.

[0033] In Figure 7, the optical axis direction of the imaging unit 106 is defined as the z-axis direction, the direction perpendicular to the plane of the paper as the y-axis direction, and the direction perpendicular to the z-axis and y-axis directions as the x-axis direction. In Figure 7, it is assumed that the imaging unit 106 acquires distance information with a resolution of 6 rows in the y-axis direction and 6 columns in the x-axis direction. Points indicated by 1 to 6 correspond to column numbers. Furthermore, the difference in texture coordinates between a given vertex and an adjacent vertex is assumed to be constant for all vertices. Here, the column number is denoted as c and the row number as r. The U coordinate of column number c and the V coordinate of row number r are respectively U c ,V r This is expressed as follows: the U coordinate of column number c+1 and the V coordinate of row number r+1 are respectively U c+1 ,V r+1 This is written as follows: For any c,r and constants ΔU,ΔV, "ΔU = U c+1 -U c ΔV=V r+1 -V r Let the following be true.

[0034] Figure 8 shows the distance map and texture acquired with the configuration shown in Figure 7. The vertical direction on the page corresponds to the y-axis, and the horizontal direction corresponds to the x-axis. Column numbers c=1 to 6 and point o are shown. The dotted line passing through point o represents the texture boundary. Since the boundary of the subject corresponds to the optical axis of the imaging unit 106, the color of the subject in the foreground (light gray) and the color of the subject in the background (dark gray) switch at the center of the texture in the x-axis direction. Column number c=3 corresponds to the background edge, and column number c=4 corresponds to the foreground edge.

[0035] On the other hand, shape information is acquired at the pitch indicated by points 1 to 6 in Figure 7. The edges of subject 1 (the set of vertices closest to the boundary) correspond to the 4th column, and the edges of subject 2 correspond to the 3rd column. Thus, due to the difference in resolution, a discrepancy occurs between the shape switching and the texture switching for each subject. This discrepancy between shape and texture can occur not only due to the difference in resolution explained here, but also due to various factors described later.

[0036] The virtual camera shown in Figure 7 schematically illustrates a situation where it is assumed that the subject is imaged from a virtual viewpoint. The virtual camera represents the assumed camera position without actually moving the imaging unit 106, and indicates its line of sight L. For example, if imaging is performed from the position of the virtual camera (virtual viewpoint) in the direction of line of sight L (between points 3 and 4), it is desirable that the texture (dark gray) of subject 2 be assigned. However, since information loss occurs in the area of ​​subject 2 that is occluded by subject 1 from the position of the imaging unit 106, the problem is how to compensate for the information loss.

[0037] Figure 9 is a schematic diagram showing a surface map and texture coordinate system consisting of 6 rows x 6 columns of vertices. Each of the columns 1 to 6 shown at the vertices in Figure 9 corresponds to each of the points 1 to 6 in Figure 7. Figure 9(A) represents Comparative Example 1, Figure 9(B) represents Comparative Example 2, and Figure 9(C) represents the Example. In each figure, the surface map is placed in the upper row and the texture coordinate system is placed in the lower row.

[0038] Comparative Example 1 in Figure 9(A) is an example in which 3D data is generated only from acquired information without generating additional shapes. In the texture coordinate system, the dotted circles represent the texture coordinate positions before correction in the mapping process (Figure 2: S103). The black dots represent the texture coordinate positions after correction in the mapping process. In Comparative Example 1, no additional shapes are generated, and no correction of the texture coordinates is performed. Here, of the vertices in the 6 columns, the vertices with column numbers c 1 to 3 belong to the acquisition area on the far side in Figure 3, and the vertices with column numbers c 4 to 6 belong to the acquisition area on the near side in Figure 3. In Comparative Example 1, no surface is generated between the vertex of column number 3 and the vertex of column number 4. As a result, when viewed from the virtual viewpoint in Figure 7 (see line of sight L), a hole is visible, resulting in an aesthetically unsettling appearance.

[0039] Comparative Example 2 in Figure 9(B) is an example in which an additional shape is generated and texture coordinate correction is not performed in the mapping process (Figure 2: S103). In the surface map, a surface is generated as an additional shape by connecting the surface in the acquisition area on the far side and the surface in the acquisition area on the near side as viewed from the imaging means (imaging unit 106). The vertex of column number 3 and the vertex of column number 4 are connected, and a surface is formed. In other words, the face containing both vertices of column number 3 and column number 4 is the additional shape. Texture coordinate correction is not performed in Comparative Example 2. Therefore, the surface of the additional shape is assigned a texture that is a mixture of the texture of subject 1 located on the near side and the texture of subject 2 located on the far side. As mentioned above (see Figure 7), while it is expected that the texture of subject 2 located on the far side will be visible in the area between points 3 and 4, the texture of subject 1 located on the near side is visible in that area. This may cause a sense of incongruity in viewing.

[0040] The embodiment in Figure 9(C) is an example in which an additional shape is generated and the texture coordinates are corrected in the mapping process (Figure 2: S103). It is desirable to perform the texture correction so that the texture of the second acquisition region is assigned to the additional shape. The U coordinate before correction at the vertex belonging to column number c is U c This is written as, and the corrected U coordinate is U* c This is written as follows. When c=4, "U * The U coordinate is set such that "4 ≤ U4 - ΔU / 2". The texture coordinate position before correction (dotted circle) is changed to the texture coordinate position after correction (black dot). This makes it possible to assign the texture of the subject located further back to the added shape. As a result, it is possible to easily reduce the sense of incongruity that occurs when viewing the occlusion area from a virtual viewpoint.

[0041] Next, referring to Figure 10, the mapping process (Figure 2: S103) will be explained. In S111, the process of setting the vertices to be corrected for texture coordinates is performed. For example, by calculating the gradient in the distance map, the region with a large depth difference can be determined as the region where the gradient value is greater than or equal to a threshold. The distance map is map information that has distance information as pixel values ​​and represents the distribution of depth information in the depth direction of the subject. For example, depth information can be calculated by known methods from detection information of image plane phase difference, image displacement information, defocus information, etc. In the determined region with a large depth difference, it is possible to set the vertices to be corrected by searching for the foreground edge (foreground vertex adjacent to the background distance value) within that region and its vicinity.

[0042] The process proceeds from S111 to S112, where the correction amount and direction of the texture coordinate values ​​are set. The correction amount for the texture coordinates can be a fixed or variable value. For example, ΔU / 2 is set as a fixed value. The correction direction is set to be perpendicular to the near edge found in S111 and pointing towards the far edge.

[0043] The process proceeds from S112 to S113, where the texture coordinate value correction process is performed. The texture coordinate value is updated by adding the correction value based on the correction amount and correction direction set in S112 to the pre-correction texture coordinate value at each vertex targeted for correction, as set in S111.

[0044] The example in Figure 10 is not the only possible embodiment. For example, in the process of determining areas with large depth differences in S111, areas where the Euclidean distance between the 3D coordinate values ​​of adjacent vertices is greater than or equal to a threshold can be determined. Furthermore, the area to be corrected in S111 may include not only the edge vertices on the near side, but also the edge vertices on the far side and their neighboring vertices. In addition, in the process of setting the correction amount in S112, the correction amount may be set to exceed a predetermined correction amount (fixed value) after detecting the positional misalignment between the texture and the shape. Furthermore, the correction amount may be set to change dynamically according to the depth difference of the shape.

[0045] In this embodiment, by setting the texture coordinates of the vertices constituting the added shape to be assigned to the texture of the acquired area, it is possible to further reduce computational costs while suppressing visual inconsistencies. In the embodiment shown in Figure 9(C), the texture is assigned continuously to areas where surfaces are continuous. Although not limited to this configuration, if some of the continuous surfaces refer to areas different from the others, the texture reflected in a patchwork-like manner may appear unnatural at the boundary. From the viewpoint of suppressing the appearance of unnatural texture boundaries, it is preferable to assign continuous texture areas to areas where surfaces are continuous.

[0046] [First variation] This modified example shows another example of an added shape. Figure 11 shows an example of an added shape that is continuous with the surface of the acquisition area on the far side as viewed from the imaging means (imaging unit 106), and discontinuous with the surface of the acquisition area on the near side as viewed from the imaging means. In this case, the addition of a point cloud and the formation of a surface are required in the additional shape generation process (Figure 2: S102). The addition of a point cloud can be done, for example, by extrapolation based on the information of the acquisition area on the far side. By generating an added shape that is continuous with the acquisition area on the far side, the hole will not be visible even when rendered from a virtual viewpoint, thus suppressing any sense of incongruity in viewing.

[0047] Referring to Figure 12, the details of the correspondence process (Figure 2: S103) in this modified example will be explained. Figure 12(A) shows a comparative example of surface maps near regions with depth differences, and Figure 12(B) shows this modified example.

[0048] In this modified example, since there may be two or more vertices and surfaces for a ray extended from the imaging means, a single surface map like the one in Figure 9 cannot represent all the shapes. Therefore, in Figures 12(A) and 12(B), the upper section shows a superimposed display of the surface maps in the order they are visible from the imaging means, while the lower section shows the acquisition area on the far side and the added shape. The acquired surface has six rows of vertices. Vertices with row numbers c 1 to 3 belong to the acquisition area on the far side in Figure 11, and vertices with row numbers c 4 to 6 belong to the acquisition area on the near side in Figure 11. In addition, as an added shape in this modified example, vertices with row numbers 7 and 8 are added near the vertex with row number 3, and these are connected to form a surface that is continuous with the acquisition area on the far side. In this case, the face consisting of vertices with row numbers 3, 7, and 8 is the added shape. From the perspective of the imaging device, the acquisition area on the near side (the surface consisting of vertices in rows 4 to 6) partially obscures the additional shape (the surface consisting of vertices in rows 3, 7, and 8). Therefore, the vertices in rows 7 and 8, and the surface consisting of them, are not shown in the superimposed display diagram above.

[0049] In the comparative example in Figure 12(A), columns 1-3 and 4-6 belong to the acquired region and are represented by applying a uniform texture of dark gray and light gray, respectively. On the other hand, the vertices in columns 7 and 8 are newly added vertices, and the color information that should be assigned to the region containing these vertices is not included in the acquired image. In other words, in the comparative example, there are no coordinate values ​​that should be set for U7 and U8, and in Figure 12(A), the texture information of the region of the added shape is represented in white.

[0050] In this modified example shown in Figure 12(B), the U coordinates of the acquisition region are assigned to the vertices that make up the added shape. Specifically, "U *7<U3 + ΔU / 2, U * 8The process of setting the U coordinate is executed so that it becomes <U3 + ΔU / 2>. Thus, by setting the texture coordinates of the vertices constituting the additional shape so that the texture of the acquisition area is assigned, it is possible to reduce the calculation cost while suppressing the sense of incongruity in appearance.

[0051] [Second Modified Example] The three-dimensional data acquired in the three-dimensional data acquisition step (Fig. 2: S101) is generated based on the imaging information by the imaging means. In this modified example, the imaging information is used in the determination process of the target area for generating the additional shape in the additional shape generation step (Fig. 2: S102). For example, the imaging unit 106 has a configuration of a stereo camera and can acquire image information with parallax or distance map information as the imaging information. In this modified example, information in the vicinity of the distance edge detected by the edge detection method with respect to the distance map information acquired by the imaging means can be used as auxiliary information for the target area for generating the additional shape. Based on the information acquired by the imaging means, it is possible to improve the detection accuracy of the target area (occlusion area) for which the additional shape is to be generated.

[0052] [Third Modified Example] The example is not limited to cases where each polygon within the added shape references a different area on the texture; as shown in this modified example, each polygon may reference the same area. Figure 13 is a diagram representing the reference area on the texture coordinate system. The horizontal axis represents the U coordinate, and the vertical axis represents the V coordinate. In the mapping process (Figure 2: S103), the texture coordinates can be set so that the polygons within the added shape reference a specific minute area (see Figure 13) in the texture coordinate system. Here, a minute area is defined as an area where the change in pixel values ​​(RGB values) within that area is sufficiently small. When referencing an area on the texture where the change in pixel values ​​is large, it is necessary to set the size of the minute area small enough so that the change in pixel values ​​within that area is sufficiently small. As an example, an area equivalent to about 2x2 pixels can be set as the minute area. On the other hand, when referencing an area on the texture where the change in pixel values ​​is small, the minute area can be made larger as long as the change in pixel values ​​within that area is sufficiently small. As an example, an area equivalent to about 50x50 pixels can be set as the minute area.

[0053] For example, consider a white area on a texture, and assume its RGB values ​​are (255,255,255). If a small area composed of three points within the white area is set as the reference area, and the texture coordinates are set so that all polygons within the added shape refer to this small area, the occlusion area in the 3D data will be filled with white.

[0054] Furthermore, for minute regions, image features on the background side of the subject may be detected and set as the region with the largest image feature quantity. The processing flow will be explained assuming a scene with a single subject in a monochrome background. First, the background region (background region) is identified by a subject region detection means based on deep learning. The average value of the pixel values ​​of the background region is obtained, and the relative coordinates (texture coordinates) of the region with the texture pixel value closest to the average value are calculated. Based on the statistical amount of the pixel values ​​of the texture background region, the texture coordinates are corrected so that the texture is assigned to the added shape.

[0055] In this modified version, a specific region on the texture coordinate system is always referenced. By correcting the texture coordinates of the added shape so that it always references a small region where the change in pixel value is below a threshold and the pixel change is sufficiently small, it is possible to achieve an effect similar to inpainting the occlusion region in a single color. For example, in a scene where it is clear that the occluded region is a single color, it is possible to effectively suppress the sense of incongruity near the point of perspective discontinuity.

[0056] [Fourth variation] This modified example describes a process for correcting the pixel values ​​of a small region in the texture coordinate system and adjusting the texture coordinates of the added shape so that the small region is referenced.

[0057] Referring to Figure 14, the image processing of this modified example will be explained. In this modified example, the steps in S121 and S122 are the same as the steps in S101 and S102 in Figure 2, so a detailed explanation of them will be omitted, and the differences will be explained. After S122, the process moves to S123. S123 corresponds to the texture correction step. The texture correction unit 104c performs a process to correct the pixel values ​​of a minute area on the texture. Although it is assumed that inpainting processing is not performed in order to reduce computational cost, the pixel values ​​of a minute area (for example, correction of a few pixels) are corrected. For corrections of this magnitude, the impact on the overall processing time is minor, and it is unlikely to increase computational cost.

[0058] In S123, the following methods can be used to set the pixel values ​​after correction by the texture correction unit 104c. • A method for setting predefined pixel values ​​such as black. • A method for setting pixel values ​​based on the statistical volume of pixel values ​​in the background area of ​​a texture, as explained in the third modification.

[0059] The area to be modified on the texture coordinate system (modification area) should preferably be a texture area that is not assigned to the acquired surface. Figure 15 shows the modification area on the texture coordinate system related to this modification. The horizontal axis represents the U coordinate, and the vertical axis represents the V coordinate. For example, if the texture of the surrounding area of ​​the image is not assigned, the surrounding area on the texture coordinate system is set as the modification area.

[0060] The next step, S124, corresponds to the process of mapping the surface to the texture. In S124, a process is performed to correct the texture coordinates of the region related to the added shape so that the texture modified in S123 is referenced.

[0061] This modification allows for an effect similar to inpainting occlusion areas with a single color. While this effect is similar to the third modification, this modification allows for assigning colors not included in the original texture as textures to the added shapes, thus reducing inconsistencies across a wider range of scenes.

[0062] [Fifth variation] This modified example describes processing based on statistical values ​​of the area of ​​the reference region in the texture coordinate system. These statistical values ​​include the mean, median, and mode. Texture information is assigned such that the statistical values ​​of the area of ​​the reference region in the texture coordinate system referenced by each polygon within the added shape are smaller than the statistical values ​​of the area of ​​the reference region in the texture coordinate system referenced by each polygon outside the added shape. This processing is performed by the mapping processing unit 104b.

[0063] Figure 16 is a diagram showing the reference area on the texture coordinate system related to this modification. The horizontal axis represents the U coordinate, and the vertical axis represents the V coordinate. It shows an example of assigning a texture in the same settings as the embodiment in Figure 9(C). In Figure 16, the area of ​​the area referenced by each polygon in the added shape is set to be smaller than the area of ​​the area referenced by each polygon in the acquired area. The acquired area consists of shapes and textures that are visible from the imaging means. On the other hand, the added shape consists of shapes and textures that are invisible from the imaging means. When viewing a subject related to 3D data from a viewpoint near the imaging means, the area visible is predominantly the acquired area, and the added shape is partial. Therefore, in most cases, the acquired area is of higher importance than the added shape in terms of information.

[0064] This method assumes that textures are assigned continuously to areas where surfaces are continuous. In this case, a correction is necessary to reduce the texture area that should be assigned to the acquired area, as a portion of the acquired texture is assigned to the added shape. This correction shifts the correspondence points between textures and shapes in the acquired information from their original positions, and if the amount of correction is large, it may cause an unnatural appearance. Therefore, in this modified example, in order to reduce the amount of this correction, the area of ​​the region that each polygon of the added shape references in the texture coordinate system is set to be relatively small.

[0065] [Sixth variation] This modified example describes a process in which the correction amount in the mapping between a surface and a texture is changed according to the depth difference. Figure 17 is a schematic diagram of a surface map and texture coordinate system consisting of 6 rows x 6 columns of vertices. In Figure 17, there is a depth difference between the 3rd and 4th columns, but the degree of this difference decreases from the 1st to the 4th row, and there is no depth difference in the 5th and 6th rows. Figure 17(A) is a comparative example, Figure 17(B) is an example, and Figure 17(C) is this modified example. In each figure, the surface map is placed in the upper row and the texture coordinate system is placed in the lower row. The display of dotted circles and black dots is as explained in Figure 9. Column numbers c=1~6 and row numbers r=1~6 are shown, and the coordinates of the vertices are denoted as (r,c).

[0066] The comparative example in Figure 17(A) is an example in which a surface connecting the third and fourth columns is generated as an additional shape, and no correction is performed in the texture coordinate system. The additional shape has a depth difference from (r,c)=(1,3) to (4,4). In the additional shape, the textures of the subject on the foreground and the subject on the background are mixed, which may cause a sense of incongruity in the viewer.

[0067] The embodiment in Figure 17(B) is an example in which a surface connecting the third and fourth columns is generated as an additional shape, and the texture coordinate system is corrected with a uniform correction amount. The texture coordinates corresponding to the vertices (r,c)=(1,4) to (4,4) are uniformly corrected in the negative direction of the U coordinate. As a result, in the additional shape with a depth difference from (r,c)=(1,3) to (4,4), the texture of the subject further back is always assigned, thus suppressing the visual incongruity of the occlusion area. However, in this embodiment, the correction is performed with a constant correction amount regardless of whether the depth difference is large or small. Therefore, a sudden shift in texture may occur at the boundary between the area with a depth difference and the area without a depth difference (fourth row to fifth row). Depending on the subject, this shift in texture may lead to a visual incongruity.

[0068] This modified example in Figure 17(C) generates a surface connecting the third and fourth columns as an additional shape and corrects the texture coordinate system with different correction amounts depending on the depth difference. The texture coordinates corresponding to the vertices (r,c)=(1,4)~(4,4) are corrected in the negative direction of the U coordinate according to the depth difference. The correction amount is large when the depth difference is relatively large, and small when the depth difference is relatively small. For example, the correction amount is set to be proportional to the depth difference. As a result, compared to the embodiment in Figure 17(B), the texture misalignment at the boundary between the area with a depth difference (or an area with a large depth difference) and the area without a depth difference (or an area with a small depth difference) becomes less noticeable.

[0069] In regions with relatively small depth differences (see Figure 17: (r,c)=(2,3)~(4,4)), the added shape may include the texture of the subject in the foreground. In this modified example, the degree of this is reduced compared to when texture coordinate correction is not performed. Furthermore, the smaller the depth difference, the smaller the area where occlusion is displayed when viewed from a virtual viewpoint, so even if the texture of the foreground is included in the added shape, it is less noticeable.

[0070] [7th variation] This modified version further includes a misalignment detection step that detects the amount of misalignment between the shape and texture of a region with a depth difference. The correction amount in the surface-texture correspondence step is set to be greater than or equal to the aforementioned misalignment amount.

[0071] Figure 18 shows an example where there is a discrepancy between the distance map and the texture in the data acquired with the configuration shown in Figure 7. The boundary of the distance map is shifted in the x-axis direction relative to the boundary of the texture. The difference from Figure 8 is that, for the texture boundary between the third and fourth columns, the far-side edge of the shape information corresponds to the fourth column, and the near-side edge corresponds to the fifth column. For example, when the distance map is acquired by stereo matching of the disparity image, it may not be possible to measure the distance correctly with respect to the disparity values ​​of the near and far edges. If the contrast of subject 2 is higher than that of subject 1 in Figure 7, the disparity values ​​of subject 2 may be assigned near the edges, resulting in a distance map that deviates from the actual boundary, as shown in Figure 18. When a discrepancy occurs between the texture and the shape, it is possible to reduce the discrepancy by appropriately setting the correction amount for the texture coordinate values.

[0072] Figure 19 is a schematic diagram of a surface map and texture coordinate system consisting of 6 rows x 6 columns of vertices. In the example in Figure 19, the added shape is a surface connecting the vertices of the 4th column and the vertices of the 5th column, and the vertices subject to texture coordinate value correction are the vertices of the 5th column and the vertices from the 3rd to the 5th columns. Figure 19(A) shows an embodiment, and Figure 19(B) shows this modified example. In each figure, the surface map is placed in the upper row and the texture coordinate system is placed in the lower row. The display of dotted circles and black dots is as explained in Figure 9.

[0073] The example in Figure 19(A) shows a case where the correction amount is ΔU / 2. The discrepancy between shape information and texture information in the settings of Figure 18 is greater than ΔU / 2. Therefore, the texture coordinate values ​​of the vertices in the 5th column remain the texture coordinate values ​​corresponding to subject 1 even after correction. As a result, it is not possible to assign the texture of the background (subject 2) to the added shape, which may cause an unnatural appearance.

[0074] In this modified example shown in Figure 19(B), the correction amounts are as follows: The texture coordinate correction amount for the vertices of the 5th column is 3ΔU / 2. The texture coordinate correction amount for the vertices of the fourth column is ΔU. The texture coordinate correction amount for the vertices of the third column is ΔU / 2.

[0075] According to this modification, even if there is a discrepancy between the texture and the shape, it is possible to assign the texture of the background to the added shape by setting a correction amount greater than the amount of the discrepancy. The detection of the amount of discrepancy can be achieved, for example, by comparing the edge detection results of the texture with the edge detection results of the distance map.

[0076] [8th variation] In this modified example, a process is performed to set a correction amount based on the defocus information of the imaging optical system regarding the texture when mapping texture to shape. Figure 20 shows an example where the texture is blurred in the data acquired with the configuration of Figure 7. The difference from Figure 8 is that the texture at the boundary between subject 1 and subject 2 has a blurred spread (denoted as δ). This blurring is mainly due to defocusing from the focus position (focused position) in the imaging optical system.

[0077] Figure 21 is a schematic diagram of a surface map and texture coordinate system consisting of 6 rows x 6 columns of vertices. In the example in Figure 21, the added shape is a surface connecting the vertices of the 3rd and 4th columns, and the vertices subject to texture coordinate value correction are the vertices of the 4th column, as well as the vertices of the 3rd and 4th columns. Figure 21(A) shows an embodiment, and Figure 21(B) shows this modified example. In each figure, the surface map is placed in the upper row, and the texture coordinate system is placed in the lower row. The display of dotted circles and black dots is as explained in Figure 9.

[0078] The example in Figure 21(A) shows a case where the correction amount is ΔU / 2. The texture coordinates corresponding to the vertices in the fourth column are the boundary between subject 1 and subject 2, but because the texture has spread due to blurring, the texture of subject 1 (the foreground side) remains in the added shape. Although the unnaturalness of occlusion can be suppressed, there is a possibility that a slight unnaturalness may remain in the viewing experience.

[0079] In this modified example shown in Figure 21(B), the correction amounts are as follows: The texture coordinate correction amount for the vertices of the 4th column is ΔU / 2 + δ / 2. The texture coordinate correction amount for the vertices of the third column is ΔU / 4 + δ / 4. In this modified example, even if the texture has blurred spread, it is possible to assign only the texture on the far side to the added shape by setting the correction amount to be greater than the amount of blur. The amount of blur can be estimated by calculating the amount of defocus based on information from the imaging optical system and the distance between the imaging means and the subject, for example.

[0080] [9th variation] This modified example describes the process performed by the mapping processing unit 104b to suppress the incongruity when the texture of the background object is assigned to the surface of the foreground object. Figure 22 is a schematic diagram of a surface map and texture coordinate system consisting of 6 rows x 6 columns of vertices. A comparative example is shown where the correction amount for the texture coordinates at the vertices of the 4th column is 3ΔU / 4 in the data acquired with the configuration of Figure 7. The texture of subject 2 (background object) is included between the corrected texture coordinates of the vertices of the 4th column and the texture coordinates of the vertices of the 5th column. Therefore, the texture of the background object is partially assigned to the surface of the foreground object, which may cause an incongruity in viewing.

[0081] Figure 23 is a conceptual diagram showing a surface map related to this modification. Figure 23(A) shows a first example related to this modification, and Figure 23(B) shows a second example related to this modification. In one embodiment of this modification, a shape modification step is provided in which the vertex coordinates are modified so that the area of ​​the polygons within the first acquisition region is reduced, thereby making it difficult to see the texture of the background subject that is assigned to the polygons on the foreground.

[0082] In the first example shown in Figure 23(A), the vertex coordinates of the fourth and fifth columns are modified so that the polygon formed by the vertices of the fourth and fifth columns within the first acquisition region becomes smaller. The spacing between the vertices of the third and fourth columns widens, and the spacing between the vertices of the fourth and fifth columns narrows, making the texture of subject 2 included in the first acquisition region harder to see. The modification of vertex coordinates can be achieved, for example, by smoothing the coordinate values ​​of the vertices of the fourth and fifth columns by a predetermined block size. In addition, the surface from the vertices of the first to the vertices of the fourth column and the surface from the vertices of the fifth to the vertices of the sixth column are translated so that they move closer to each other (not shown). This reduces the size of the polygon formed by the vertices of the fourth and fifth columns.

[0083] In the second example in Figure 23(B), the process includes adding vertices to the first acquisition region. The surface is reconstructed by adding vertices from the seventh column between the vertices of the fourth and fifth columns. By modifying the vertex coordinate values ​​so that the area of ​​the polygon formed by the vertices of the fourth and seventh columns is reduced, the texture of subject 2 included in the first acquisition region becomes less visible.

[0084] [10th variation] In this modified example, the process of evaluating at least one of color difference and shape continuity in setting the second acquisition region is described. Figure 24 is a diagram showing the positional relationship between the imaging means and the subject in this modified example. With the imaging means as the reference, the subject located closest to the viewer is designated as "Subject 1," the subject located in the middle is designated as "Subject 2," and the subject located furthest away is designated as "Subject 3." Furthermore, it is assumed that the 3D data acquired by imaging subjects 1 to 3 with the imaging means will be rendered from the viewpoint of a virtual camera. The distance measurement points are points 1 and 2 for subject 3, points 3 and 4 for subject 2, and points 5 and 6 for subject 1. As in the case of Figure 7, in Figure 24 as well, the texture resolution is assumed to be greater than the interval between distance measurement points (shape resolution).

[0085] Figure 25 is a schematic diagram of a surface map and texture coordinate system consisting of 6 rows x 6 columns of vertices. Figure 25(A) shows an example, and Figure 25(B) shows this modified example. In each figure, the surface map is placed in the upper row, and the texture coordinate system is placed in the lower row. The display of dotted circles and black dots is as explained in Figure 9.

[0086] In the embodiment shown in Figure 25(A), the first acquisition region corresponds to the surface formed by the vertices of the fifth and sixth columns. The second acquisition region corresponds to the surface formed by the vertices from the first to the fourth column. The additional shape corresponds to the surface formed by the surfaces of the fourth and fifth columns.

[0087] In the example above, the "second acquisition region" is only defined as a region adjacent to a region with a depth difference; there is no limitation on how far away from the region with a depth difference the region should be considered as the second acquisition region. In the configuration of Figure 24, when viewing each subject from the direction of the virtual camera's line of sight L, it is natural that the texture of subject 2 should be visible. That is, it is desirable that the texture of subject 2 be assigned to the additional shape consisting of the fourth and fifth columns.

[0088] In the embodiment shown in Figure 25(A), corrections are performed in the third, fourth, and fifth columns of the texture coordinate system. When a second acquisition region is set to have color differences and discontinuous depth differences in the textures, and corrections are performed so that the texture within the second acquisition region is assigned to the added shape, there is a possibility that the texture of subject 3, rather than subject 2, may be assigned to the added shape. If a different texture from subject 3 is assigned to the boundary between subject 1 and subject 2, it may cause a sense of incongruity in the viewer's perception.

[0089] In this modified example shown in Figure 25(B), the second acquisition region is set so that the color difference of the texture is small and there is no discontinuous depth difference. Specifically, the second acquisition region corresponds to the surface formed by the vertices from the third column to the vertices of the fourth column. By setting the second acquisition region in this way, the texture assigned to the additional shape located at the boundary between subject 1 and subject 2 can be made the texture of subject 2. This makes it possible to suppress any sense of incongruity in the viewing experience.

[0090] Regarding the second acquisition region, in order to set it so that the color difference of the texture is small and there are no discontinuous depth differences, it is necessary to evaluate the color difference and depth difference of the texture in the vicinity of the region with depth differences. The second acquisition region may be determined by evaluating only one of the color difference or depth difference of the texture, or the second acquisition region may be determined so that both the color difference and depth difference are below a threshold.

[0091] An example of the texture color difference evaluation process performed by the mapping processing unit 104b is described below. First, adjacent regions and neighboring regions are defined for regions with depth differences. Adjacent regions can be defined, for example, as regions where the pixel distance from a distance edge is within a predetermined number of pixels (e.g., 10 pixels or less). Distance edges are regions where the distance difference in the distance map is greater than a threshold. Neighboring regions can be defined, for example, as regions where the pixel distance from a distance edge is within a predetermined number of pixels (e.g., several tens of pixels) and which are not adjacent regions. Next, a process is performed to evaluate the average color of the texture within the adjacent region. Then, by evaluating the color difference between the average color of the texture within the adjacent region and the texture within the neighboring region, regions where the color difference is less than or equal to a threshold can be determined as the second acquisition region. Here, the color space and color difference formula are arbitrary, but as an example, the CIELab color space or the CIE1976 color difference formula can be used.

[0092] Furthermore, an example of the depth difference evaluation process performed by the correspondence processing unit 104b is described below. First, a neighboring region of the region with a depth difference is defined. The neighboring region can be defined, for example, as a region where the pixel distance from the distance edge is within a predetermined number of pixels (for example, several tens of pixels). Next, for the group of vertices within the neighboring region, the Euclidean distance to adjacent vertices is calculated, and the region where this distance value is less than or equal to a threshold can be determined as the second acquisition region. According to this embodiment, it is possible to further reduce computational costs while suppressing the visual incongruity associated with 3D data including occlusion.

[0093] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist.

[0094] Embodiments of this disclosure include the following configurations, methods, and programs. [Configuration 1] An acquisition means for acquiring 3D data having shape information and texture information, A generation means for generating an additional shape to be added to the shape corresponding to the acquired shape information, It has a matching means for matching the shape information with the texture information, The resolution of the vertices constituting the shape information is smaller than the resolution of the texture information, and the mapping means maps the acquired texture information to the added shape. An image processing apparatus characterized by the following: [Configuration 2] The three-dimensional data includes information about vertices and surfaces formed by the connections between vertices. The generation means determines the region for generating the additional shape using the imaging information of the imaging means acquired by the acquisition means. The image processing apparatus according to configuration 1, characterized in that... [Configuration 3] The region where the added shape is generated is near a region where there is a depth difference in the shape when the surface of the subject is viewed from the imaging means. When the region included in the 3D data is designated as the first acquisition region with respect to the imaging means, and the region on the near side is designated as the second acquisition region, the correspondence means performs a process of assigning the texture information of the second acquisition region to the region adjacent to the region with the depth difference. The image processing apparatus according to configuration 2, characterized in that... [Structure 4] The mapping means performs a correction to change the coordinate values ​​of the texture mapped to the surface and sets the texture coordinates for the region of the added shape. The image processing apparatus according to configuration 3, characterized in that [Composition 5] The aforementioned mapping means updates the texture coordinate values ​​by setting the vertices to be corrected for texture coordinates, setting the correction amount and correction direction of the texture coordinate values, and correcting the texture coordinate values ​​at each of the vertices to be corrected. The image processing apparatus according to configuration 4, characterized in that... [Composition 6] The generation means generates a surface as the additional shape that connects the surface in the first acquisition region and the surface in the second acquisition region. An image processing apparatus according to any one of configurations 3 to 5, characterized by the above. [Composition 7] The generation means generates a surface as the additional shape that is continuous with the surface in the second acquisition region but not continuous with the surface in the first acquisition region. An image processing apparatus according to any one of configurations 3 to 5, characterized by the above. [Structure 8] The mapping means assigns texture information corresponding to a specific reference region in the texture coordinate system to the added shape. An image processing apparatus according to any one of configurations 3 to 7, characterized by the above. [Composition 9] The matching means sets the reference region calculated from the statistical amount of pixel values ​​in the background region of the subject. The image processing apparatus according to configuration 8, characterized by the above. [Configuration 10] It includes a correction means for correcting the pixel values ​​of a small region in the texture coordinate system, The mapping means assigns the modified texture information from the modification means to the added shape. An image processing apparatus according to any one of configurations 3 to 7, characterized by the above. [Composition 11] The modification means sets the minute region that is not assigned to the first acquisition region and the second acquisition region as the region to be modified. The image processing apparatus according to configuration 10, characterized in that... [Composition 12] The aforementioned 3D data includes data of multiple polygons formed by connecting multiple vertices. The aforementioned mapping means assigns texture information such that the statistical value of the area of ​​the reference region in the texture coordinate system referenced by each polygon within the added shape is smaller than the statistical value of the area of ​​the reference region in the texture coordinate system referenced by each polygon outside the added shape. An image processing apparatus according to any one of configurations 1 to 11, characterized by the features described herein. [Composition 13] The matching means changes the correction amount according to the depth difference. The image processing apparatus according to configuration 5, characterized by the features described herein. [Composition 14] The aforementioned matching means sets a correction amount that is greater than or equal to the amount of discrepancy between the shape and texture of the region with the depth difference. The image processing apparatus according to configuration 5, characterized by the features described herein. [Composition 15] The matching means sets the correction amount based on the defocus information of the imaging optical system. The image processing apparatus according to configuration 5, characterized by the features described herein. [Composition 16] The aforementioned 3D data includes data of multiple polygons formed by connecting multiple vertices. The matching means performs shape modification to reduce the area of ​​the polygon within the first acquired region by correcting the vertex coordinates. The image processing apparatus according to configuration 5, characterized by the features described herein. [Composition 17] The aforementioned correspondence means adds vertices within the first acquisition region by the shape modification. The image processing apparatus according to configuration 16, characterized in that... [Composition 18] The matching means determines the second acquisition region by evaluating at least one of the color difference and the continuity of the shape. The image processing apparatus according to configuration 3, characterized in that [Method 1] An image processing method performed by an image processing device that performs image processing on 3D data having shape information and texture information, The acquisition process for acquiring the aforementioned 3D data, A generation step of generating an additional shape to be added to the shape corresponding to the acquired shape information, The system includes a matching step for matching the shape information with the texture information, The resolution of the vertices constituting the shape information is smaller than the resolution of the texture information, and in the mapping process, the acquired texture information is mapped to the added shape. An image processing method characterized by the following: [program] A program that causes a computer to execute each step described in Method 1. [Explanation of Symbols]

[0095] 100 Image Processing Devices 104 Image Processing Unit 105 Data Acquisition Unit

Claims

1. An acquisition means for acquiring 3D data having shape information and texture information, A generation means for generating an additional shape to be added to the shape corresponding to the acquired shape information, It has a matching means for matching the shape information with the texture information, The resolution of the vertices constituting the shape information is smaller than the resolution of the texture information, and the mapping means maps the acquired texture information to the added shape. An image processing apparatus characterized by the following:

2. The three-dimensional data includes information about vertices and surfaces formed by the connections between vertices. The generation means determines the region for generating the additional shape using the imaging information of the imaging means acquired by the acquisition means. The image processing apparatus according to feature 1.

3. The region where the added shape is generated is near a region where there is a depth difference in the shape when the surface of the subject is viewed from the imaging means. When the region included in the three-dimensional data is designated as the first acquisition region with respect to the imaging means, and the region on the near side is designated as the second acquisition region, the correspondence means performs a process of assigning the texture information of the second acquisition region to the region adjacent to the region with the depth difference. The image processing apparatus according to claim 2.

4. The mapping means performs a correction to change the coordinate values ​​of the texture mapped to the surface and sets the texture coordinates for the region of the added shape. The image processing apparatus according to claim 3.

5. The aforementioned mapping means updates the texture coordinate values ​​by setting the vertices to be corrected for texture coordinates, setting the correction amount and correction direction of the texture coordinate values, and correcting the texture coordinate values ​​at each of the vertices to be corrected. The image processing apparatus according to feature 4.

6. The generation means generates a surface as the additional shape that connects the surface in the first acquisition region and the surface in the second acquisition region. The image processing apparatus according to any one of claims 3 to 5.

7. The generation means generates a surface as the additional shape that is continuous with the surface in the second acquisition region but not continuous with the surface in the first acquisition region. The image processing apparatus according to any one of claims 3 to 5.

8. The mapping means assigns texture information corresponding to a specific reference region in the texture coordinate system to the added shape. The image processing apparatus according to claim 3.

9. The matching means sets the reference region calculated from the statistical amount of pixel values ​​in the background region of the subject. The image processing apparatus according to feature 8.

10. It includes a correction means for correcting the pixel values ​​of a small region in the texture coordinate system, The mapping means assigns the modified texture information from the modification means to the added shape. The image processing apparatus according to claim 3.

11. The modification means sets the minute region that is not assigned to the first acquisition region and the second acquisition region as the region to be modified. The image processing apparatus according to feature 10.

12. The aforementioned three-dimensional data includes data of multiple polygons formed by connecting multiple vertices, The aforementioned mapping means assigns texture information such that the statistical value of the area of ​​the reference region in the texture coordinate system referenced by each polygon within the added shape is smaller than the statistical value of the area of ​​the reference region in the texture coordinate system referenced by each polygon outside the added shape. The image processing apparatus according to feature 1.

13. The matching means changes the correction amount according to the depth difference. The image processing apparatus according to feature 5.

14. The aforementioned matching means sets a correction amount that is greater than or equal to the amount of discrepancy between the shape and texture of the region with the depth difference. The image processing apparatus according to feature 5.

15. The matching means sets the correction amount based on the defocus information of the imaging optical system. The image processing apparatus according to feature 5.

16. The aforementioned three-dimensional data includes data of multiple polygons formed by connecting multiple vertices, The aforementioned correspondence means performs shape modification to reduce the area of ​​the polygon within the first acquired region by correcting the vertex coordinates. The image processing apparatus according to feature 5.

17. The aforementioned correspondence means adds vertices within the first acquisition region by the shape modification. The image processing apparatus according to feature 16.

18. The matching means determines the second acquisition region by evaluating at least one of the color difference and the continuity of the shape. The image processing apparatus according to claim 3.

19. An image processing method performed by an image processing device that performs image processing on three-dimensional data having shape information and texture information, The acquisition process for acquiring the aforementioned three-dimensional data, A generation step of generating an additional shape to be added to the shape corresponding to the acquired shape information, The system includes a matching step for matching the shape information with the texture information, The resolution of the vertices constituting the shape information is smaller than the resolution of the texture information, and in the mapping process, the acquired texture information is mapped to the added shape. An image processing method characterized by the following:

20. A program that causes a computer to perform each of the steps described in claim 19.

Images