Image processing apparatus, image processing method, computer readable medium, and computer program product

Abstract

An image processing apparatus, image processing method, computer-readable medium, and computer program product for generating three-dimensional data of a subject based on a depth image are disclosed. The apparatus defines a boundary region between a subject region and a background region in an image to be segmented based on the depth image, and segments the image to be segmented into multiple sub-regions. The apparatus determines vertices for each sub-region including the subject region among the multiple sub-regions, and adds vertices to the boundary region. The apparatus then uses the determined vertices and the added vertices to generate three-dimensional data.

Description

Technical Field

[0001] This disclosure relates to image processing apparatus, image processing methods, computer-readable media, and computer program products. In particular, this disclosure relates to techniques for generating three-dimensional data. Background Technology

[0002] Traditionally, devices such as stereo cameras and time-of-flight cameras are known to be used to acquire distance distribution information (depth maps). Furthermore, point groups can be obtained by applying perspective projection transformations to the depth map. Additionally, a 3D surface model with surfaces can be generated by distributing topological information among the vertices of the point group and generating polygons.

[0003] The accuracy of 3D data can be improved by increasing the number of polygons; however, this in turn leads to an increase in data size. In Reference 1 (Renato Pajarola, “Overview of Quadtree-based Terrain Triangulation and Visualization”, UCI-ICS Technical Report No. 02-01, University of California, Irvine, January 2002), small polygons were used for regions with large shape variations, and large polygons were used for regions with small shape variations, thereby reducing the amount of data required for the same level of shape representation compared to using polygons of uniform size.

[0004] Depending on the location of the object from which 3D data is to be generated and the viewpoint from which the object's depth map is obtained, it may be impossible to successfully acquire distance information for regions on the object's surface. This raises the question of how to segment regions along the boundaries (edges) between areas where distance information has been acquired and those where it has not; however, Reference 1 does not address this issue.

[0005] Applying the method disclosed in Reference 1 to the edge portion resulted in the generation of unnecessary polygons and / or the absence of necessary polygons, and the generated 3D data caused the edge portion to appear unnatural. Summary of the Invention

[0006] This disclosure provides, in some embodiments thereof, an image processing apparatus and an image processing method for generating three-dimensional data in which the representation of the boundary between portions for which distance information has been obtained and portions for which distance information has not yet been obtained is improved.

[0007] According to an aspect of this disclosure, an image processing apparatus is provided that generates three-dimensional data of a subject based on a depth image, comprising: a setting component for setting a boundary region between the subject region and a background region in an image to be segmented based on the depth image; a segmentation component for segmenting the image to be segmented into a plurality of sub-regions; a determining component for determining vertices for each of the plurality of sub-regions including the subject region; an adjusting component for adding vertices to the boundary region; and a generating component for generating the three-dimensional data using the vertices determined by the determining component and the vertices added by the adjusting component.

[0008] According to another aspect of this disclosure, an image processing apparatus is provided that generates three-dimensional data of a subject based on a depth image, comprising: a setting component for setting a boundary region between the subject region and a background region in an image to be segmented based on the depth image; a segmentation component for segmenting the image to be segmented into a plurality of sub-regions; a determining component for determining vertices for each of the plurality of sub-regions that includes the subject region; and a generating component for generating the three-dimensional data using the vertices determined by the determining component, wherein the segmentation component segments the image to be segmented into the plurality of sub-regions in such a manner that no sub-region includes both the subject region and the background region.

[0009] According to another aspect of this disclosure, an image processing apparatus is provided that generates three-dimensional data of a subject based on a depth image, comprising: a setting component for setting a boundary region between the subject region and a background region in a subject region in an image to be segmented based on the depth image; a determining component for determining a plurality of vertices for the subject region; and a generating component for generating the three-dimensional data in such a manner that a subset of the determined vertices is used for the subject region outside the boundary region and all the determined vertices are used for the boundary region.

[0010] According to another aspect of this disclosure, an image processing method is provided for generating three-dimensional data of a subject based on a depth image, comprising: defining a boundary region between the subject region and a background region in a subject region of an image to be segmented based on the depth image; segmenting the image to be segmented into a plurality of sub-regions; determining vertices for each of the plurality of sub-regions including the subject region; adding vertices to the boundary region; and using the determined vertices and the added vertices to generate the three-dimensional data.

[0011] According to another aspect of this disclosure, an image processing method is provided for generating three-dimensional data of a subject based on a depth image, comprising: defining a boundary region between the subject region and a background region in a subject region of an image to be segmented based on the depth image; segmenting the image to be segmented into a plurality of sub-regions such that no sub-region includes both the subject region and the background region; determining vertices for each sub-region that includes the subject region among the plurality of sub-regions; and using the determined vertices to generate the three-dimensional data.

[0012] According to another aspect of this disclosure, a computer-readable medium is provided that stores a computer program executable by one or more processors, wherein the computer program, when executed by the one or more processors, causes the one or more processors to perform an image processing method according to this disclosure.

[0013] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program executable by one or more processors, wherein the computer program, when executed by the one or more processors, causes the one or more processors to perform an image processing method according to this disclosure.

[0014] The features of this disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings. The following description of the embodiments is given by way of example. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the specification, serve to explain the principles of the embodiments.

[0016] Figure 1 It is a graph used to describe the edges of three-dimensional data.

[0017] Figures 2A to 2C This is a diagram used to illustrate an example of reduced accuracy in the representation of edge regions.

[0018] Figure 3 This is a block diagram illustrating an example of the functional configuration of an image processing device according to an embodiment.

[0019] Figure 4 This is a flowchart related to the three-dimensional data generation process in the embodiments.

[0020] Figure 5 It is a flowchart related to the process of generating polygonal meshes for each grid unit region.

[0021] Figure 6A and Figure 6BIt is a graph relating to the grid unit region and the generated polygonal grid.

[0022] Figures 7A to 7D This is a schematic diagram illustrating the polygon mesh generation process in the first embodiment.

[0023] Figures 8A to 8E This is a diagram related to the vertex addition method in the second embodiment.

[0024] Figures 9A to 9D This is a schematic diagram illustrating the polygon mesh generation process in the second embodiment.

[0025] Figures 10A to 10F This is a diagram used to describe the problem to be solved in the third embodiment and its solution.

[0026] Figures 11A to 11F This is a diagram used to describe the problem to be solved in the fourth embodiment.

[0027] Figures 12A to 12C This is a schematic diagram illustrating the polygon mesh generation process in the fourth embodiment.

[0028] Figure 13 This is a schematic diagram of the polygonal mesh generated in the fifth embodiment.

[0029] Figure 14 This is a flowchart related to the polygon mesh generation process in the sixth embodiment.

[0030] Figures 15A to 15D This is a schematic diagram illustrating the polygon mesh generation process in the sixth embodiment.

[0031] Figure 16 This is a flowchart related to the polygon mesh generation process in the seventh embodiment. Detailed Implementation

[0032] In the following, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claims. Several features are described in the embodiments, but not all of them are required, and several such features can be appropriately combined. Furthermore, in the drawings, the same reference numerals are given the same or similar configuration, and redundant descriptions are omitted.

[0033] First Embodiment First, the terms used in the following description are defined as follows.

[0034] A "captured image" is an image obtained by photographing a subject. A captured image is formed by multiple pixels arranged in two dimensions, horizontally and vertically. Each pixel has a luminance component and a color component.

[0035] A "depth image" is information representing the distance distribution corresponding to a captured image. Here, the depth image represents the distance information of each pixel in the captured image; however, the resolution may differ between the depth image and the captured image. A depth image is also called a depth map or distance image.

[0036] The captured image and depth image can be pre-generated according to known methods, or they can be generated by the image processing device according to the embodiment. For example, by capturing an image using an image sensor compatible with the camera plane phase difference detection method, both the captured image and the depth image can be generated in a single image capture. Alternatively, the captured image can be generated by a camera device, and the depth image can be generated using a light detection and ranging (LiDAR) sensor, etc.

[0037] The "region to be segmented" is the image region included in the image used to generate 3D data and subjected to region segmentation processing. The image used to generate 3D data can be a captured image, a depth image, or an image based on a captured image or a depth image.

[0038] "Region segmentation processing" is a process used to divide a region into multiple sub-regions with predetermined shapes. The sub-regions have the same shape, but can be different sizes.

[0039] A "mesh unit region" is a partial region that is included within the region to be segmented and is the unit for generating polygonal meshes as 3D data. A "mesh unit region" is formed from one or more sub-regions.

[0040] Before describing the embodiments, the problems of the conventional art will be described in detail. Figure 1 The diagram schematically illustrates a depth image corresponding to a captured image of the subject for which 3D data is to be generated, and the subject represented by 3D data generated from the depth image.

[0041] exist Figure 1 In the example shown, only a depth image obtained from a single viewpoint is used to generate the 3D data; therefore, distance information for unphotographed portions such as the sides, back, and bottom of the subject is not obtained. Consequently, in the surface represented by the generated 3D data, edges are formed at the boundary portions of the subject area in the depth image.

[0042] For example, suppose that region segmentation is performed as disclosed in Reference 1, such that the parts of the subject region with large distance variations are segmented into smaller sub-regions, and the parts of the subject region with small distance variations are segmented into larger sub-regions.

[0043] In this situation, edge regions with small distance differences from the background are more likely to be segmented into relatively large sub-regions that include both the subject and the background. Therefore, if a polygonal mesh is generated such that each sub-region includes one vertex, the accuracy of representing the edge regions by the polygonal mesh will be reduced.

[0044] Figure 2A and Figure 2B Each region is shown as an 8×8 pixel area corresponding to the edge portion of the depth image. The horizontal numbers 1 to 8 are column numbers, and the vertical numbers 1 to 8 are row numbers. The background area is shown in dark gray, and the subject area is shown in light gray. Furthermore, the region is divided into square sub-regions using quadtree-based region segmentation. Additionally, a triangular polygonal mesh is generated to represent the subject area, such that for each sub-region, pixels near the top-left vertex are included as vertices.

[0045] exist Figure 2A and Figure 2B Of these two, it can be seen that the polygonal mesh exhibits low shape representation accuracy for the shape of the subject region included within the sub-region of interest, which is a relatively large sub-region encompassing both the subject and the background. Specifically, in Figure 2A In the example, the subject region within the sub-region of interest is represented as smaller than it actually is. This is because the vertices used for polygon mesh generation ( Only one pixel is set as a subject region of 6 pixels included in the sub-region of interest, thus there are subject regions that do not form polygons.

[0046] On the other hand, Figure 2B In the example, even parts that are not the subject area are represented as subject areas. This is because the polygon mesh generated from the vertices used to generate the polygon mesh includes the background area, as well as the sub-regions set as the focus and their adjacent sub-regions.

[0047] If this occurs across the entire edge portion, then as Figure 2C As illustrated schematically, in the subject area represented by the final generated polygonal mesh, the quality of the edge portions of the subject's shape will be low.

[0048] <Configuration of Image Processing Equipment> Figure 3 This is a block diagram illustrating an example of the functional configuration of an image processing device 100 according to an embodiment. The image processing device 100 may be, for example, any type of computer device capable of processing image data, such as a smartphone, smartwatch, tablet terminal, laptop computer, game console, or camera device, etc.

[0049] The control unit 101 includes one or more processors capable of executing programs. The control unit 101 implements the operation of the image processing device 100, described later, by loading one or more programs stored in the non-volatile memory 103 into the working memory to execute the programs. Note that at least some of the functions implemented by the control unit 101 executing the programs can be performed by one or more other hardware components.

[0050] The imaging unit 102 includes, for example, an image capturing optical system and an image sensor, which photoelectrically converts the subject image generated by the image capturing optical system into an image signal. The image capturing optical system includes a movable lens (such as a focusing lens) and an aperture driven by the control unit 101. The image sensor includes a plurality of pixels arranged in two dimensions, and converts the subject image into a group of pixel signals (analog image signals) by generating a charge corresponding to the amount of light incident on each pixel. The image sensor can be a CMOS image sensor or a CCD image sensor.

[0051] After noise reduction and A / D conversion, the analog image signal is output as a digital image signal (image data) from the camera unit 102. Note that the camera unit 102 may include other components, such as a focal plane shutter and a vibration-damping mechanism. The operation of the camera unit 102 is controlled by the control unit 101.

[0052] In this embodiment, each pixel arranged in the image sensor includes multiple photoelectric conversion regions and can generate a focus detection signal for performing automatic focus detection of the image plane phase difference detection method. Furthermore, since parallax image pairs can be generated based on image data read from the image sensor, depth images can be acquired in addition to capturing images. Note that if the image processing device 100 includes a distance information acquisition component such as a LiDAR sensor, a LiDAR sensor can be used to generate depth images.

[0053] Note that the image data used to generate the 3D data can be acquired, for example, from the recording medium 107 or an external device. Therefore, the camera unit 102 is not necessary for the image processing device 100.

[0054] The control unit 101 applies various types of image processing to the image data to generate image data corresponding to the settings and intended use. For example, data for recording generated depth images and captured images are stored in a data file with a settings-based format and recorded to the recording medium 107 or non-volatile memory 103.

[0055] The non-volatile memory 103 is electrically rewritable. The non-volatile memory 103 stores programs executable by the processor of the control unit 101, GUI data such as menu screens, and various settings and specific information of the image processing device 100. Furthermore, depth image and captured image data can also be stored in the non-volatile memory 103. Programs include an operating system (OS) and applications running on the OS (hereinafter referred to as "applications"). In this embodiment, the application used to generate 3D data is stored in the non-volatile memory 103.

[0056] The working memory 104 is, for example, RAM, and is used to temporarily store programs, data, etc. A portion of the working memory 104 is used as video memory for the display unit 106.

[0057] The operation unit 105 is a collective term for a group of input devices that allow the user of the image processing device 100 to input commands to the image processing device 100. For example, the operation unit 105 includes a power switch for indicating whether the image processing device 100 is turned on or off, and + (increase) and - (decrease) buttons for volume adjustment. In addition, the touch panel included in the display unit 106, which is a touch display, is also included in the operation unit 105.

[0058] The display unit 106 is a touch display and displays: image data acquired by the camera unit 102; image data reproduced from the non-volatile memory 103, recording medium 107, etc.; and GUIs such as screens provided by the OS. In the image processing device 100, application startup and operations on the GUI provided by the application are basically performed via touch operations on the display unit 106.

[0059] For example, the recording medium 107 can be at least one of a removable memory card and a built-in non-volatile memory. The recording medium 107 is the destination for recording image data obtained by capturing images using the imaging unit 102. Note that the destination for recording the image data can be the non-volatile memory 103 or an external storage device accessible by the image processing device 100.

[0060] The connection unit 108 is an interface for communicating with external devices and is capable of communication based on at least one wireless communication method. The connection unit 108 may include one or more wired communication interfaces, such as a USB interface and / or an HDMI (registered trademark) interface. The connection unit 108 includes components required for the supported communication methods. For example, components include antennas, connectors, modulation / demodulation circuitry, transmitting / receiving circuitry, etc.

[0061] In this embodiment, the connection unit 108 supports wireless LAN communication conforming to the IEEE 802.11x series standards (where x is a, b, g, n, ac, ax, etc.). Note that it is sufficient for the connection unit 108 to support infrastructure mode, and the connection unit 108 may optionally support self-organizing mode. The operation of the connection unit 108 is controlled by the control unit 101. Note that the connection unit 108 may support one or more other wireless communication methods. Furthermore, there are no particular restrictions on the protocol used for communication over an established wireless connection, and known protocols such as TCP / IP can be used.

[0062] Similar to the connection unit 108, the near-field wireless communication unit 109 is an interface for communication with external devices. The near-field wireless communication unit 109 supports wireless communication methods with a shorter communication distance than the connection unit 108. In this embodiment, the near-field wireless communication unit 109 supports wireless communication conforming to the Bluetooth Low Energy standard; however, the near-field wireless communication unit 109 may support other near-field wireless communication standards. The operation of the near-field wireless communication unit 109 is controlled by the control unit 101.

[0063] The image processing device 100 (control unit 101) is capable of communicating with external devices using one or more of the connection unit 108 and the near-field wireless communication unit 109. The external device that can communicate with the image processing device 100 is an external device that supports at least one of the communication standards supported by the connection unit 108 and the near-field wireless communication unit 109.

[0064] The public network connection unit 111 is a wireless communication interface for connecting to a cellular network. The image processing device 100 can connect to a cellular network conforming to 3GPP (registered trademark) standards (such as 3G, 4G, and / or 5G) via the public network connection unit 111 to make calls with landline and cellular phones and to perform data communication with external devices. During a call, the control unit 101 can use the microphone 112 as a voice input device and the speaker 113 as a voice output device. The public network connection unit 111 includes components required for the supported communication methods. For example, components include an antenna, modulation / demodulation circuitry, transmitting / receiving circuitry, etc. The antenna can be shared with the connection unit 108.

[0065] Next, based on Figure 4The flowchart shown describes the operation performed by the image processing device 100 to generate three-dimensional data (polygonal mesh data). Note, for example, that the operation described below can be performed by the control unit 101 executing an application for generating the three-dimensional data. Note that a portion of the operation performed by the control unit 101 may be performed by other components (e.g., hardware circuitry such as a GPU, NPU, or ASIC). Furthermore, the operation for generating the three-dimensional data is not limited to execution during the execution of a specific application and can be performed as part of other applications.

[0066] In the following text, it is assumed that the captured image of the subject for which three-dimensional data is to be generated, and the corresponding depth image, are pre-recorded in the recording medium 107; however, the generation of the captured image and the depth image, as well as the generation of the three-dimensional data, can be performed as a series of operations.

[0067] Furthermore, this embodiment is based on the following situation: distance information cannot be obtained for a portion of the subject from which 3D data is to be generated, and therefore, the surface represented by the 3D data does not form a closed surface and has edges. In the following description, the most typical example is the case of generating 3D data based on a depth image obtained from a single viewpoint; however, in the case of using depth images obtained from multiple viewpoints, the effect of this embodiment can also be achieved by applying similar processing to the edge portions.

[0068] <Boundary Area Setting Step S110> In the boundary region setting step S110, the control unit 101 sets the boundary region between the subject region and the background region in the depth image. First, the control unit 101 generates a mask image based on the depth image. The mask image is a binary image where the subject-region pixel value is 1 and the background-region pixel value is 0. The mask image can be generated using appropriate known methods for separating the subject region and the background region from each other. For example, locations in the depth image with a distance gradient greater than or equal to a threshold can be detected as the outer edge of the subject region, or the subject region can be detected from the captured image, and the remaining region can be set as the background region. Alternatively, the mask image can be generated from the captured image using a trained machine learning model.

[0069] The control unit 101 can define a region in the subject area that is no more than a threshold (e.g., several pixels) away from the background area as a boundary region. Note that the boundary region corresponds to the part that becomes an edge in the 3D data and the area nearby.

[0070] Figure 7AAn example of how to set a boundary region is shown. In this example, a region within the subject region with a chessboard distance (also known as a Chebyshev distance) of 1 from the background region is set as the boundary region. For example, control unit 101 stores in working memory 104 the position information of the outer edge of the subject region in the depth image and a threshold for setting the boundary region. Note that a mask image can be used as the position information of the outer edge of the subject region.

[0071] <Region segmentation preprocessing step S120> In the region segmentation preprocessing step S120, the control unit 101 performs the generation of the image to be segmented, the change of pixel values ​​in the image to be segmented, the setting of the segmentation threshold, the size adjustment of the image to be segmented, and the filling process.

[0072] For example, the control unit 101 can generate a differential image of the depth image as the image to be segmented. Thus, an image to be segmented with pixel values ​​corresponding to the magnitude of the distance gradient can be obtained.

[0073] In the image to be segmented, the pixel values ​​in the background region may or may not change. If the pixel values ​​in the background region do not change, during region segmentation, sub-regions that include either the subject or the background are more likely to be assigned to the regions with a large distance difference between the subject and the background. Conversely, sub-regions that include both the subject and the background are more likely to be assigned to the regions with a small distance difference between them.

[0074] When the pixel values ​​in the background region change, the control unit 101 can change the pixel values ​​in the background region to a value that does not affect the determination of whether to segment the region during the region segmentation process (e.g., a Not-a-N (NaN) value). If such a change is performed, there is a high probability that boundary portions in the subject region with significant shape changes will be segmented until the sub-region size becomes smaller. Furthermore, there is a high probability that sub-regions including both the subject and the background will be set as boundary portions in the subject region whose shapes do not change significantly because segmentation will stop before the sub-region size becomes smaller.

[0075] The segmentation threshold is a threshold used in the region segmentation step to determine whether to further segment the object into sub-regions. To efficiently represent the subject region using a small number of polygons, the following values ​​are desirable: regions with little shape change will not be segmented more than necessary, and regions with significant shape changes will be segmented repeatedly until the sub-region size becomes smaller.

[0076] For example, the segmentation threshold could be a threshold applied to a metric related to variations in pixel values ​​within a sub-region. For instance, a metric related to variations in pixel values ​​could be the maximum pixel value within the sub-region, the range of pixel values ​​within the sub-region (the difference between the maximum and minimum values), or the standard deviation of pixel values ​​within the sub-region.

[0077] For example, the segmentation threshold can be determined experimentally and pre-stored in non-volatile memory 103. Alternatively, the control unit 101 can dynamically set the segmentation threshold based on predetermined judgment criteria (such as the range of pixel values ​​included in the image to be segmented).

[0078] Resizing and padding are processes used to shape the image to be segmented into sub-regions of the maximum size that can be divided into (neither too much nor too little). Therefore, the control unit 101 applies resizing and padding corresponding to the region segmentation method and the shape and maximum size of the sub-regions to the image to be segmented. Note that the horizontal scaling factor and vertical scaling factor applied in the resizing process are equal. Furthermore, the pixels added in the padding process are set with a sufficiently large difference relative to the pixel values ​​in adjacent subject regions, and it is also ensured that there are no sub-regions that include both the regions added through padding and regions that existed before padding. The padded pixels may be set with NaN values.

[0079] The control unit 101 stores the image to be segmented after the above preprocessing in the working memory 104.

[0080] <Region segmentation step S130> In the region segmentation step S130, the control unit 101 performs region segmentation processing on the preprocessed image to be segmented. Region segmentation processing is a process used to divide the image to be segmented into sub-regions of predetermined shapes. The predetermined shape is typically a polygon (square, rectangle, or triangle, etc.), but can also be other shapes. Furthermore, the predetermined shape can also be a cuboid or a cube.

[0081] As methods for segmenting an image into sub-regions, there are top-down methods that first segment the image into the largest sub-region and then repeatedly segment it into smaller sizes based on conditions, and bottom-up methods that first segment the image into the smallest sub-region and then repeatedly integrate them into larger sizes based on conditions. Here, the top-down method is used as an example.

[0082] As a method for recursively partitioning a multidimensional space into subregions based on predetermined conditions, there are methods using tree structures such as kd-trees, quadtrees, or octrees. In kd-tree-based region partitioning, each node represents a K-dimensional space. By recursively performing the process of repeatedly partitioning the current node (K-dimensional space) into two K-dimensional spaces based on predetermined conditions and storing the obtained K-dimensional spaces as child nodes, the space is partitioned into subregions of different sizes.

[0083] On the other hand, in the quadtree-based method, the process of recursively dividing the two-dimensional space into four nodes (two-dimensional space) and storing the obtained nodes in the child nodes is performed. Methods using other tree structures are also applicable to this embodiment; however, the quadtree-based method will be described as an example.

[0084] First, the control unit 101 equally segments the image to be segmented in both the horizontal and vertical directions, dividing the image into four sub-regions of maximum size. Then, the control unit 101 determines whether to further segment each individual sub-region. For sub-regions determined not to be further segmented, segmentation stops. On the other hand, for sub-regions determined to require further segmentation, the control unit 101 segments the sub-region into four sub-regions and determines the necessity of further segmentation for each individual sub-region. The control unit 101 recursively performs this segmentation and determination process until segmentation stops for all sub-regions.

[0085] Control unit 101 uses the segmentation threshold determined in the preprocessing step S120 to determine the necessity of re-segmentation. For example, if an index related to the variation of pixel values ​​within the determined object sub-region is greater than or equal to the segmentation threshold, control unit 101 can determine that re-segmentation is necessary. The differential image of the depth image has large values ​​in regions with large distance variations and small values ​​in regions with small distance variations. Therefore, by using the segmentation threshold for determination, since re-segmentation is highly likely to occur in regions with large distance variations and stop in regions with small distance variations, region segmentation for accurately representing the shape of the subject region can be achieved while suppressing the number of polygons. Control unit 101 stores the location information of the sub-region in the working memory 104. Note that, for example, the location information of the sub-region may be information indicating the quadtree structure corresponding to the region segmentation result. Figure 7B Show Figure 7A The partial region segmentation results shown are shown.

[0086] Note that other conditions can be considered when determining whether to perform further subdivision. For example, the upper limit can be set as the ratio between the lengths of the edges (e.g., long or short edges) of adjacent sub-regions. This allows for the restriction of the local vertex layout pattern in the generation of the polygon mesh, which consists of vertices each corresponding to a different sub-region. Furthermore, this has the effect of increasing the minimum interior angles in the generated polygon mesh.

[0087] <Vertex Determination Step S140> In the vertex determination step S140, the control unit 101 determines the vertices of the polygonal mesh constituting the subject region based on the sub-regions generated in step S130.

[0088] Based on the region segmentation method described above, the parts of the subject region with large shape changes are divided into smaller sub-regions. Furthermore, to improve the representation accuracy of these parts, it is desirable to increase the vertex density in the polygonal mesh. Conversely, the parts of the subject region with small shape changes are divided into larger sub-regions. Moreover, even with a low vertex density in the polygonal mesh, these parts can still be accurately represented. Therefore, a polygonal mesh capable of efficiently representing the subject region can be generated by determining the vertices of each sub-region. Note that this does not preclude the possibility of determining multiple vertices within a single sub-region.

[0089] For example, control unit 101 can determine a predetermined number of polygonal mesh vertices for each sub-region. Specifically, for example, control unit 101 can determine the pixel closest to the top-left vertex as a polygonal mesh vertex. Note that if the sub-region is the size of a single pixel, control unit 101 determines the pixels constituting the sub-region as vertices. Control unit 101 stores the coordinates of the determined vertices in working memory 104. The coordinates can be a combination of the column and row numbers mentioned above. Note that the vertices determined here can be referred to as initial vertices because the vertices can be changed in the next step.

[0090] <Vertex Adjustment Step S150> In vertex adjustment step S150, control unit 101 adjusts the vertices determined in step S140. Specifically, control unit 101 adjusts the number of vertices such that the ratio of the number of vertices within the boundary region to the number of vertices outside the boundary region increases. Vertex "adjustment" includes adding, deleting, and moving vertices. Control unit 101 may perform one or more of the following: adding vertices within the boundary region; deleting vertices outside the boundary region; and moving vertices from outside the boundary region to within the boundary region.

[0091] Because the boundary region is set as the edge of the subject region, an improvement in the rendering accuracy of the outer edge of the subject region can be expected by adding vertices to the boundary region. In particular, since large sub-regions are highly likely to include both the subject and the background, the improvement in rendering accuracy achieved by adding vertices is significant for such sub-regions.

[0092] There are no particular limitations on the method used to add vertices. As a simple example, control unit 101 can add all pixels within the boundary region that have not yet been identified as vertices as vertices. Alternatively, control unit 101 can add all pixels within the boundary region that are horizontally closest to the subject region and have not yet been identified as vertices as vertices. Figure 7C In China, regarding Figure 7B The region segmentation results shown are from (Black dots) indicate the vertices determined in step S140. Furthermore, in Figure 7C In the process, for all pixels in the boundary region that have not yet been determined as vertices, by (Black star) indicates the vertices that have been added in step S150. The control unit 101 stores the adjusted vertex positions (image coordinates) in the working memory 104.

[0093] <Mesh generation step S160> In the mesh generation step S160, the control unit 101 generates a polygonal mesh using the vertices determined in steps S140 and S150. The polygonal mesh is generated using pixel values ​​in the depth image corresponding to the determined vertices. The control unit 101 can generate a polygonal mesh from multiple vertices using appropriate known methods. For example, the control unit 101 can generate a triangular polygonal mesh by performing constrained Delaunay triangulation. In this case, the control unit 101 can generate a polygonal mesh without constraining the formation of polygons in the background region. Figure 7D It shows from Figure 7C The example shown is a polygonal mesh generated from the vertices.

[0094] Furthermore, as an example of another method, the control unit 101 can generate polygonal meshes for individual mesh unit regions formed from multiple sub-regions. Reference will be made below. Figure 5 The flowchart and Figure 6A and Figure 6B To describe this method.

[0095] The mesh generation process is repeated on a per-mesh unit region basis. First, the mesh unit region will be described. For example, suppose... Figure 6AThe 2×2 pixel sub-region shown is the sub-region of interest. In this case, the candidate connection destinations for vertices in the sub-region of interest are the vertices of other sub-regions adjacent to the sub-region of interest, specifically those adjacent to the right, bottom, and lower right sides of the sub-region of interest.

[0096] The control unit 101 sets the region consisting of the sub-region of interest and adjacent sub-regions including vertices that serve as candidates for connection destinations as grid unit regions. Because the grid unit regions are determined in this way based on the sub-regions of interest, adjacent grid unit regions include the same sub-regions in their portions. By generating polygonal meshes for each of these grid unit regions, a polygonal mesh covering the entire subject region can ultimately be generated. When generating a polygonal mesh using grid unit regions, the control unit 101 sets the grid unit regions over the entire region to be segmented after completing vertex adjustments and executes... Figure 5 The operation is shown in the flowchart.

[0097] In step S161, the control unit 101 determines whether there is an unprocessed grid unit region, and if it determines that there is an unprocessed grid unit region, it executes step S162; otherwise, it terminates the polygon mesh generation.

[0098] In step S162, the control unit 101 retrieves the coordinates of the vertices included in the processed object mesh unit region from the working memory 104. Figure 6A In the example shown, the vertex coordinates are (1, 1), (3, 1), (3, 2), (1, 3), and (3, 3). In practice, the image coordinates of the image to be segmented are used to indicate the vertex positions.

[0099] In step S163, the control unit 101 connects the vertices to generate, for example, a triangular polygon mesh. Any suitable method can be applied as a method for forming triangles, and Delaunay triangulation can be used, for example.

[0100] In addition, for example, such as Figure 6B As shown, the control unit 101 can add vertices at the center of a grid unit region. ), and can be selected from the added vertices and the initially set vertices ( Generate triangles.

[0101] In step S164, the control unit 101 stores information related to the polygonal mesh generated for the mesh unit region of the processing object in the working memory 104. For example, the control unit 101 stores a list of vertices of each triangle as mesh information. The vertex information included in the list can be coordinates or vertex identification numbers. If identification numbers are used, the correspondence between identification numbers and image coordinates is stored separately in advance.

[0102] As described above, in the vertex adjustment step S150 of this embodiment, vertices used for generating the polygon mesh are added to the boundary region set in the subject region. Therefore, because the density of polygons representing the edge portions of the subject region increases, a polygon mesh that improves the representation accuracy of the edge portions can be generated.

[0103] Second Embodiment Next, the second embodiment will be described. Except for the operation of the control unit 101 in the vertex adjustment step S150, this embodiment is the same as the first embodiment. Therefore, the operation of the control unit 101 in the vertex adjustment step of this embodiment will be described below.

[0104] Figures 8A to 8E This diagram schematically illustrates the operation of the control unit 101 in the vertex adjustment step of this embodiment. Figures 8A to 8E The diagram shows the operation applied to only one grid unit region; however, in practice, the same operation is performed on each grid unit region that includes both the subject region and the background region.

[0105] Figure 8A An example grid unit region is shown, comprising a sub-region of interest, which is a 4×4 pixel square region that includes both the subject region and the background region in the image to be segmented. (See also: Regarding...) Figure 6A As described, a grid unit region consists of a sub-region of interest and sub-regions adjacent to its right, bottom, and bottom-right sides. Because the sub-regions adjacent to the sub-region of interest within the grid unit region do not include the subject region, no vertices for polygon mesh generation are set in these adjacent sub-regions. In this case, the initial vertices for polygon mesh generation set in the sub-region of interest ( Furthermore, no polygons are generated between sub-regions adjacent to the sub-region of interest within the grid unit area. Therefore, no polygons related to the subject region within the sub-region of interest are generated.

[0106] Figure 8B The shaded area in the image is an example of a boundary region, and Figure 8C The shaded areas are outlines formed by individual pixels at the outer edge of the grid unit region of the sub-region of interest. Furthermore, Figure 8D The shaded areas are the regions where the boundary regions and the outlines overlap. In this embodiment, the control unit 101 sets the regions where the outlines of the boundary regions and the mesh unit regions overlap as candidate regions for adding vertices used in polygon mesh generation.

[0107] Figure 8E The vertices used for generating polygon meshes are shown. and (White star)) is an example of each pixel included in the candidate region that has been added. This is achieved by adding the initial vertex (in the sub-region of interest) ) and the added vertices ( or Connecting these elements can generate triangles or polygons within the sub-regions of interest.

[0108] As described above, in this embodiment, the control unit 101 adds vertices for generating the polygon mesh to the area where the outlines of the boundary region and the mesh unit region overlap. Therefore, compared to adding vertices to individual pixels in the boundary region that have not yet had initial vertices set, the number of vertices can be reduced, thus reducing the amount of polygon mesh data.

[0109] When the boundary region within a sub-region of interest is substantially linear, the straight line obtained by linearly connecting the endpoints of the boundary region within the sub-region will substantially coincide with the original boundary region. In other words, the edges obtained by setting the points where the outline of the grid unit region intersects with the boundary region as vertices and connecting these vertices (assuming the boundary region is substantially linear) will substantially coincide with the boundary region. For this reason, by adding vertices to the points where the outline of the boundary region intersects with the outline of the grid unit region and connecting such vertices to form edges, it is possible to reduce the number of vertices while maintaining the fidelity of the subject region.

[0110] While vertices can be added across the entire candidate region, it's possible to add vertices only to the endpoints of the boundary regions within a single mesh unit. This discourages the addition of unnecessary vertices. Figure 8E The added vertices In setting the initial vertex and the vertices of the endpoints added to the boundary region On the connecting straight line; therefore, even without adding vertices. Mesh quality will not be affected. For example, polygon growth can be suppressed by adding vertices only at the endpoints of the boundary regions (pixels adjacent to the background region) within the candidate region's grid unit area. This corresponds to... Figure 8E Add vertices And without adding vertices The situation.

[0111] Based on Figures 9A to 9D A specific example of polygon mesh generation in this embodiment is described. Figure 9A and Figure 9B respectively with Figure 7A and Figure 7B The same applies. That is, the control unit 101 performs the determination from the setting of the boundary region to the initial vertex in the same manner as in the first embodiment. Figure 4 The processing of steps S110 to S140 in the process. The determined initial vertex. exist Figure 9C As shown in the image.

[0112] In vertex adjustment step S150, the control unit 101 adds vertices to the region (candidate region) where the outlines of the boundary region and the mesh unit region overlap. The added vertices... exist Figure 9C As shown in the image.

[0113] In the mesh generation step S160, the control unit 101 generates a polygon mesh using the initial vertices and the added vertices in the same manner as in the first embodiment. Figure 9D The generated polygonal mesh is shown.

[0114] In this embodiment, the same effect as in the first embodiment can be achieved. Furthermore, because the number of added vertices can be suppressed compared to the number added in the first embodiment, the amount of polygon mesh data can be reduced.

[0115] Third Embodiment Next, the third embodiment will be described. Except for the operation of the control unit 101 in the mesh generation step S160, this embodiment is the same as the first embodiment. Therefore, the operation of the control unit 101 in the mesh generation step of this embodiment will be described below.

[0116] Figures 10A to 10F This diagram schematically illustrates the operation of the control unit 101 in the mesh generation step of this embodiment. Figures 10A to 10F The diagram shows the operation applied to only one grid unit region; however, in practice, the same operation is performed on each grid unit region that includes both the subject region and the background region.

[0117] Figure 10A The diagram shows the state after vertex adjustment step S150 has been performed on a mesh unit region comprising two non-adjacent subject regions A and B. The processed object mesh unit region includes the initial vertices. and the added vertices .

[0118] In the mesh generation step S160 of the first and second embodiments, the control unit 101 uses all initial vertices and added vertices included in the mesh unit region to generate a polygon mesh. If the same processing as in the first and second embodiments is applied... Figure 10A The vertices shown will generate as follows: Figure 10B The polygonal grid shown. (As from...) Figure 10B It can be seen that if all vertices are used to generate a polygonal mesh when there are non-adjacent (or non-independent) subject regions A and B within a grid unit area, polygons will also be generated in the background area. This leads to a decrease in the accuracy of representing the subject region with a polygonal mesh.

[0119] Therefore, the mesh generation step in this embodiment is configured such that polygons are not generated between unconnected subject regions. For example, in the mesh generation step, the control unit 101 generates a polygon mesh under the constraint that vertices are not connected between non-adjacent subject regions within a mesh unit region. Thus, as Figure 10C As shown, a polygonal mesh was generated where vertices are not connected between non-adjacent subject regions.

[0120] Furthermore, instead of the mesh generation step, the region segmentation step or the mesh unit region creation step can be modified so that the mesh unit region does not include multiple subject regions. For example, with Figure 10D The corresponding grid unit region on the sub-region of the subject region A in the image is in Figure 10E As shown, and it can be seen, no grid unit area includes the subject area B. Therefore, as... Figure 10F As shown, a mesh can be generated in non-adjacent subject regions without them being connected by polygons.

[0121] According to this embodiment, the same effect as that in the first and second embodiments can be achieved. Furthermore, because the generation of polygons in the background area can be suppressed, the accuracy of the edge portions of the subject area represented by the polygon mesh can be improved.

[0122] Fourth embodiment Next, the fourth embodiment will be described. Except for the operation of the control unit 101 in the vertex adjustment step S150, this embodiment is the same as the second embodiment. Therefore, the operation of the control unit 101 in the vertex adjustment step of this embodiment will be described below.

[0123] In the second embodiment, vertices are added assuming that the boundary regions within a grid unit area are essentially straight lines. Figure 11AAn example of a grid unit region including a subject area and a background area is shown. The grid unit region is 5×5 pixels in size, and the sub-region of interest is 4×4 pixels in size. Note that the sub-region adjacent to the lower side of the sub-region of interest is also 4×4 pixels in size. Furthermore, Figure 11B and Figure 11C The outlines of the boundary region and the grid unit region are shown respectively. The boundary region is curved.

[0124] In the method described in the second embodiment, such as Figure 11D As shown, the control unit 101 determines candidate regions for vertex addition. Furthermore, in... Figure 11E The figure shows the determination of the initial vertex. and the added vertices Then, control unit 101 as follows Figure 11F The generated polygonal mesh is shown. Because the generated polygonal mesh includes a portion of the background area, the shape accuracy of the edges of the subject area is reduced.

[0125] This embodiment addresses the problem that may occur in the second embodiment. Specifically, in the region segmentation step S130, the control unit 101 evaluates the linearity of the boundary regions within the sub-region. Furthermore, compared to the possibility of re-segmenting sub-regions with low linearity, the control unit 101 increases the possibility of re-segmenting sub-regions with low linearity.

[0126] Specifically, in the region segmentation step S130, the control unit 101 performs an evaluation of the linearity of the boundary region for the processing object sub-region in the depth image, in addition to determining the change. Furthermore, if the logical OR (or OR) between the result of the change determination (which is 1 if equal to or exceeding the segmentation threshold) and the result of the boundary region linearity determination (which is 1 if the linearity is low) is 1, the control unit 101 determines that further segmentation is required.

[0127] For example, in determining the linearity of a boundary region, a regression line is obtained for the pixel coordinates of the boundary region within the sub-region. If the mean square error of the minimum distance between the regression line and each pixel within the boundary region is less than a threshold, the linearity is considered high; if the mean square error is greater than or equal to the threshold, the linearity is considered low. Alternatively, a straight line obtained by connecting the endpoints of the boundary region within the sub-region can be used as the regression line. Furthermore, the average value can be used instead of the mean square error.

[0128] Reference Figures 12A to 12C Describe the effects of this embodiment. Figure 12A It shows the relationship with Figure 11AThe same region, but it is in the following state: as a result of the low linearity of the region being judged as a boundary region, it makes... Figure 11A The region of interest within has been further divided into four. Due to this further segmentation, such as... Figure 12B As shown, the initial vertex is determined. and the added vertices Therefore, the generation of... Figure 12C The polygonal mesh shown. (Compared to...) Figure 11F Compared to the polygonal mesh shown, it can be seen that the rendering accuracy of the edge parts of the subject area has been improved.

[0129] According to this embodiment, even if the boundary regions included in the unit grid region do not have a linear shape, the same effect as the second embodiment can be achieved.

[0130] Fifth Embodiment Next, the fifth embodiment will be described. Except for the operation of the control unit 101 in the vertex adjustment step S150, this embodiment is the same as the first embodiment. Therefore, the operation of the control unit 101 in the vertex adjustment step of this embodiment will be described below.

[0131] In the above embodiment, in the vertex adjustment step S150, the ratio of the number of vertices within the boundary region to the number of vertices outside the boundary region is increased by adding vertices within the boundary region. In this embodiment, the number of vertices is adjusted by moving vertices outside the boundary region into the boundary region, thereby increasing the ratio of the number of vertices within the boundary region to the number of vertices outside the boundary region.

[0132] An example of a method for moving vertices will be described. Control unit 101 moves vertices located within a grid unit region that includes both the subject region and the background region, and outside the boundary region. The move destination is a location within the boundary region included in the same grid unit region. If multiple move destination candidates exist, control unit 101 can select a location such that the area of ​​the polygon including the moved vertex is maximized.

[0133] For example, consider the following grid unit area ( Figure 8A ):like Figure 7A As shown, the 4×4 pixel sub-region at the lower left of the subject area is the region of interest. Figure 8A The initial vertex determined in the sub-region of interest within the grid unit region shown is located at Figure 7A The initial vertex is located at coordinates (1, 5). This initial vertex satisfies the conditions for the vertex to be moved. Therefore, the control unit 101 moves the initial vertex to a position within the boundary region. The candidate move destination is... Figure 7C The added vertices are shown in the image. The five positions; in this case, the control unit 101 moves the initial vertex to coordinates (1, 7) to maximize the polygon area. The moved vertex and the generated polygon mesh are in Figure 13 As shown in the image.

[0134] According to this embodiment, the rendering accuracy of the edge portion of the subject area can be improved without increasing the number of vertices.

[0135] Sixth Embodiment Next, the sixth embodiment will be described. Except for the operation of the control unit 101 in the vertex adjustment step S150, this embodiment is the same as the first embodiment. Therefore, the operation of the control unit 101 in the vertex adjustment step of this embodiment will be described below.

[0136] The first to fifth embodiments improve the quality of polygonal meshes generated for sub-regions that include both the subject region and the background region. This embodiment improves polygonal mesh quality by suppressing the formation of sub-regions that include both the subject region and the background region.

[0137] Figure 14 This is a flowchart related to the polygon mesh generation operation performed by the control unit 101 in this embodiment. Figure 4 The same reference numerals as those in the accompanying drawings are provided for steps that perform the same operations as in the first embodiment, and their descriptions are omitted. In this embodiment, in the region segmentation preprocessing step S220 or the region segmentation step S230, processing corresponding to the vertex adjustment step S150 in the first embodiment is performed.

[0138] In the region segmentation preprocessing step S220, the control unit 101 performs the same processing as in the first embodiment, such as generating the image to be segmented, changing the pixel values ​​in the image to be segmented, setting the segmentation threshold, and adjusting the size and filling of the image to be segmented.

[0139] In this embodiment, the control unit 101 also performs processing that increases the likelihood of re-segmentation within the boundary region compared to the likelihood of re-segmentation outside the boundary region. For example, the control unit 101 replaces pixel values ​​within the boundary region of the image to be segmented, making it more likely that the re-segmentation judgment conditions are met.

[0140] For example, in the region segmentation step S230, if it is determined that re-segmentation should be performed if the maximum value of the pixel values ​​included in the processing object sub-region is greater than a threshold, the control unit 101 replaces the pixel values ​​in the boundary region with values ​​exceeding the threshold. This replacement can suppress the formation of sub-regions that include both the subject region and the background region, because in the region segmentation step S230, re-segmentation is repeated until the sub-regions in the boundary region are equal to the minimum size (a single pixel). The control unit 101 performs the subsequent vertex determination step and mesh generation step in the same manner as in the first embodiment.

[0141] Figures 15A to 15D This is a diagram used to describe this embodiment. The pixel positions of the subject area and the background area are... Figures 7A to 7D The pixel positions are the same.

[0142] In the boundary area setting step S110, the control unit 101 communicates with the first embodiment ( Figure 7A The boundary region is set in the same way. In the region segmentation preprocessing step S220, the control unit 101 replaces or changes the pixel values ​​in the boundary region as described above, so that the possibility of repeated re-segmentation in the boundary region increases compared with the possibility of repeated re-segmentation outside the boundary region.

[0143] Therefore, in the region segmentation step S230, as Figure 15B As shown, the boundary region is divided into sub-regions of the smallest possible size, and sub-regions that include both the subject region and the background region are eliminated. Therefore, in vertex determination step S140, as... Figure 15C As shown, the initial vertex It is set to all sub-regions within the boundary area, and no vertex adjustment step is required. Subsequently, the control unit 101 generates the mesh as follows in the mesh generation step S160: Figure 15D The polygonal grid shown.

[0144] In this embodiment, region segmentation is performed in the region segmentation preprocessing step S220 and the region segmentation step S230, such that many vertices are arranged within the boundary region. Therefore, a polygonal mesh with high-quality representation of the edge portions of the subject region can be generated. Furthermore, the increase of unnecessary vertices can be suppressed by dividing the subject region outside the boundary region into sub-regions of different sizes.

[0145] Variations The formation of sub-regions comprising both the subject region and the background region can be suppressed using methods other than replacing pixel values. For example, in the region segmentation preprocessing step S220, in addition to the differential image of the depth image, the control unit 101 also sets a mask image representing the subject region as the image to be segmented. Then, in the region segmentation step S230, in addition to judging changes within sub-regions in the depth image, the control unit 101 also performs a judgment on changes within sub-regions in the mask image. If the logical OR between the judgment result of changes in the depth image (1 if the result is greater than or equal to a threshold) and the judgment result of changes in the mask image (1 if the sub-region includes both the subject region and the background region) is 1, the control unit 101 can determine that re-segmentation should be performed. Therefore, sub-regions comprising both the subject region and the background region are always re-segmented, and thus the formation of sub-regions comprising both the subject region and the background region can be suppressed.

[0146] Alternatively, a configuration can be adopted such that, in the region segmentation step, the boundary region and the region outside the boundary region are segmented independently of each other, and the boundary region is unconditionally segmented into the smallest sub-region. Note that two or more of the first to sixth embodiments described above can be combined to implement this.

[0147] Seventh Embodiment Next, the seventh embodiment will be described. In the first to sixth embodiments, vertices within the boundary region are added after applying region segmentation. In this region segmentation, segmentation towards the largest sub-region is performed first, and then sub-regions with significant shape changes are repeatedly segmented. This embodiment relates to the case where region segmentation for segmenting into the smallest sub-region is applied from the beginning.

[0148] Figure 16 This is a flowchart related to the polygon mesh generation operation performed by the control unit 101 in this embodiment. Figure 4 The same reference numerals are used for steps that perform the same operations as in the first embodiment, and their descriptions are omitted.

[0149] In the region segmentation step S320, the control unit 101 performs region segmentation processing on the image to be segmented. In this embodiment, the control unit 101 segments the image to be segmented into sub-regions of the smallest possible size. Note that if the smallest size of a sub-region is 1 pixel, the region segmentation step is not required. Furthermore, if the smallest size of a sub-region is greater than 1 pixel, region segmentation can be performed after the aforementioned padding process.

[0150] In vertex determination step S330, the control unit 101 sets a vertex for each sub-region. For example, if the minimum size of the sub-region is 1 pixel, the control unit 101 sets the vertices to the individual pixels within the subject region of the image to be segmented.

[0151] In the mesh generation step S340, the control unit 101 generates a polygonal mesh using the vertices set in step S330. The polygonal mesh is generated using pixel values ​​in the depth image corresponding to the set vertices. The control unit 101 can generate a polygonal mesh from multiple vertices using any suitable known method.

[0152] In mesh adjustment step S350, control unit 101 performs mesh adjustment, including vertex deletion, on the subject region mesh outside the boundary region set in step S110. Note that control unit 101 is configured not to delete vertices within the boundary region during adjustment.

[0153] For mesh adjustments, methods such as Quadratic Error Metric (QEM) can be used. In QEM, edges that, even if removed, will not significantly alter the mesh shape are repeatedly deleted (by merging the two vertices at the endpoints of the edges). Therefore, in the subject region outside the boundary region, the number of polygons corresponding to areas where the shape does not change significantly can be reduced (this reduces the number of vertices). Thus, 3D data can be obtained where a subset of the already set vertices is used for the subject region outside the boundary region, and all the already set vertices are used for the boundary region.

[0154] Note that although vertices are deleted after mesh generation in this embodiment, the number of vertices generated after mesh generation can be reduced by integrating sub-regions corresponding to adjacent vertices in the subject region outside the boundary region, wherein the value variation between adjacent vertices is no greater than a threshold.

[0155] In this embodiment, after the image to be segmented is divided into the smallest possible sub-regions and vertices are set for each sub-region, vertices are deleted from the subject region outside the boundary region. This embodiment can also achieve the same effect as the first embodiment.

[0156] Other embodiments Embodiments of the present invention can also be implemented by providing software (including computer program products of computer programs) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU) or microprocessor unit (MPU) of the system or device) reads and executes the computer program.

[0157] While this disclosure has been described with reference to exemplary embodiments, it should be understood that this disclosure is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. An image processing apparatus for generating three-dimensional data of a subject based on a depth image, comprising: The setting component is used to set the boundary region between the subject region and the background region in the image to be segmented based on the depth image; A segmentation component is used to segment the image to be segmented into multiple sub-regions; A determining component is used to determine vertices for each of the plurality of sub-regions, including the subject region; Adjustment components are used to add vertices to the boundary region; as well as A generation component is used to generate the three-dimensional data using the vertices determined by the determining component and the vertices added by the adjusting component.

2. The image processing apparatus according to claim 1, wherein The adjustment component adds vertices to each pixel in the boundary region that has not yet had its vertices determined by the determining component.

3. The image processing apparatus according to claim 1, wherein, The adjustment component adds vertices to the area where the outline of the grid unit region overlaps with the boundary region, the grid unit region being formed from the sub-region of interest and adjacent sub-regions.

4. The image processing apparatus according to any one of claims 1 to 3, wherein The generating component generates the 3D data without connecting vertices between non-adjacent subject regions.

5. The image processing apparatus according to any one of claims 1 to 3, in, The segmentation component segments the image to be segmented into the plurality of sub-regions by dividing the image to be segmented into a predetermined number of sub-regions and then repeatedly dividing the sub-regions that meet the re-segmentation conditions into the predetermined number of sub-regions.

6. The image processing apparatus according to claim 1, in, The segmentation component divides the image to be segmented into the plurality of sub-regions by dividing the image to be segmented into a predetermined number of sub-regions and then repeatedly dividing the sub-regions that satisfy the re-segmentation conditions into the predetermined number of sub-regions. If the linearity of a boundary region within a sub-region is determined to be low, the segmentation component increases the probability of segmenting that sub-region, making the probability higher than if the linearity of a boundary region within the sub-region is determined to be not low.

7. The image processing apparatus according to claim 6, in, If the average or mean square error of the minimum distance between the regression line and the pixels in the boundary region is higher than or equal to a threshold, the segmentation component determines that the linearity is low, and the regression line is based on the pixel coordinates in the boundary region of the sub-region.

8. The image processing apparatus according to claim 1, in, The adjustment component adds vertices to the boundary region by moving vertices outside the boundary region into the boundary region.

9. The image processing apparatus according to claim 8, in, The adjustment component moves the vertices included in the sub-regions comprising the subject region and the background region to the boundary region.

10. The image processing apparatus according to claim 8 or 9, in, The three-dimensional data is a polygonal mesh, and The adjustment component determines the destination of the vertex's movement to maximize the area of ​​the polygon including the moved vertex.

11. The image processing apparatus according to any one of claims 1 to 3, in, The image to be segmented is a differential image of the depth image.

12. An image processing apparatus for generating three-dimensional data of a subject based on a depth image, comprising: The setting component is used to set the boundary region between the subject region and the background region in the image to be segmented based on the depth image; A segmentation component is used to segment the image to be segmented into multiple sub-regions; A determining component is used to determine vertices for each of the plurality of sub-regions, including the subject region; as well as A generation component is used to generate the three-dimensional data using the vertices determined by the determining component. The segmentation component divides the image to be segmented into the plurality of sub-regions in such a way that no sub-region includes both the subject region and the background region.

13. The image processing apparatus according to claim 12, in, The segmentation component divides the image to be segmented into multiple sub-regions by dividing the boundary region into the smallest possible sub-regions, such that no sub-region includes both the subject region and the background region.

14. The image processing apparatus according to claim 12 or 13, in, The segmentation component divides the image to be segmented into the plurality of sub-regions by dividing the image to be segmented into a predetermined number of sub-regions and then repeatedly dividing the sub-regions that satisfy the re-segmentation conditions into the predetermined number of sub-regions. The setting component changes the pixel values ​​within the boundary region in a manner that makes repeated segmentation within the boundary region more likely than outside the boundary region.

15. The image processing apparatus according to claim 14, in, The segmentation component further segments the sub-regions of the image to be segmented or the mask image representing the subject region that satisfy the re-segmentation conditions into the predetermined number of sub-regions.

16. The image processing apparatus according to claim 12 or 13, in, The image to be segmented is a differential image of the depth image.

17. An image processing apparatus for generating three-dimensional data of a subject based on a depth image, comprising: Setting component for setting a boundary region between the subject region and the background region in the subject region of the image to be segmented based on the depth image; A determining component is used to determine multiple vertices for the subject region; as well as A generation component is used to generate the 3D data in such a manner that a subset of the determined vertices are used for the subject region outside the boundary region and all the determined vertices are used for the boundary region.

18. The image processing apparatus according to claim 17, in, The generating component generates 3D data using all of the plurality of vertices, and then removes a subset of vertices for the subject region outside the boundary region.

19. The image processing apparatus according to claim 17, in, The generation component removes a subset of vertices from the subject region outside the boundary region, and then generates the 3D data.

20. An image processing method for generating three-dimensional data of a subject based on a depth image, comprising: Within the subject region of the image to be segmented based on the depth image, a boundary region is set between the subject region and the background region; The image to be segmented is divided into multiple sub-regions; Vertices are determined for each sub-region that includes the subject region among the plurality of sub-regions; Add vertices to the boundary region; as well as The three-dimensional data is generated using the determined vertices and the added vertices.

21. A computer-readable medium storing a computer program executable by one or more processors, wherein the computer program, when executed by the one or more processors, causes the one or more processors to perform the image processing method according to claim 20.

22. A computer program product comprising a computer program executable by one or more processors, wherein the computer program, when executed by the one or more processors, causes the one or more processors to perform the image processing method according to claim 20.

23. An image processing method for generating three-dimensional data of a subject based on a depth image, comprising: Within the subject region of the image to be segmented based on the depth image, a boundary region is set between the subject region and the background region; The image to be segmented is divided into multiple sub-regions in such a way that no sub-region includes both the subject region and the background region; Vertices are determined for each sub-region including the subject region among the plurality of sub-regions; and The determined vertices are used to generate the three-dimensional data.

24. A computer-readable medium storing a computer program executable by one or more processors, wherein the computer program, when executed by the one or more processors, causes the one or more processors to perform the image processing method according to claim 23.

25. A computer program product comprising a computer program executable by one or more processors, wherein the computer program, when executed by the one or more processors, causes the one or more processors to perform the image processing method according to claim 23.

Images