Image processing device, three-dimensional data generation program, and three-dimensional data generation method

Abstract

To reduce discomfort in viewing in three-dimensional data including occlusion.SOLUTION: An image processing apparatus includes: region dividing means of dividing coordinate information of a subject into a first region and a second region; and offset adding means of adding, to the coordinate information of the first region, an offset value that is a distance between a first representative coordinate representing a coordinate of the first region and a second representative coordinate representing a coordinate of the second region and that reduces a distance in an optical axis direction of distance distribution information acquiring means used for acquiring the coordinate information of the subject.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image processing device, a three-dimensional data generating program, and a three-dimensional data generating method. [Background technology]

[0002] Conventionally, technologies capable of acquiring distance distribution information, such as stereo cameras and time-of-flight (ToF) cameras, have been known. The distance distribution information can be converted into a point cloud by perspective projection transformation. Furthermore, the point cloud can be converted into a three-dimensional surface model having a surface by polygonization. Furthermore, by acquiring color distribution information simultaneously with the distance distribution information, three-dimensional data with texture information can be generated. Unlike two-dimensional images, three-dimensional data has the advantage that it can be viewed from any viewpoint.

[0003] However, when three-dimensional data is generated based on distance distribution information from a single viewpoint, it may contain areas that exist in the subject of the image but that the three-dimensional data itself cannot restore. For example, if a scene in which a person's hand is positioned in front of their body is converted into three-dimensional data, when the person is viewed from the front, part of the body is occluded by the hand, resulting in a loss of information. This condition is called occlusion. Furthermore, when such three-dimensional data is rendered from a viewpoint other than the front, the loss of the occluded area becomes visible, creating an unnatural feeling when viewed.

[0004] One example of a technique for solving such problems is the technique disclosed in Non-Patent Document 1. The technique disclosed in Non-Patent Document 1 applies an inpainting process to repair defects that occur in parts of the background due to occlusion of the foreground. This technique repairs defects by inferring shape information and texture information of the occluded area using an inpainting process based on deep learning, making it possible to make occlusion invisible even when viewing three-dimensional data from different viewpoints. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] J. Kopf, "One Shot 3D Photography", ACM Trans. Graph., Vol. 39, No. 4, Article 76., 2020 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology disclosed in Non-Patent Document 1 infers information about missing parts based on deep learning, which may result in information that is significantly different from the actual subject, potentially creating a stronger sense of incongruity. In addition, the use of deep learning generally requires high computational costs.

[0007] Therefore, an object of the present invention is to provide an image processing device, a three-dimensional data generation program, and a three-dimensional data generation method for reducing the sense of incongruity felt when viewing three-dimensional data that includes occlusion. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, the image processing device of the present invention is characterized by comprising an area dividing means for dividing coordinate information of a subject into a first area and a second area, and an offset adding means for adding an offset value to the coordinate information of the first area, which is the distance between a first representative coordinate representing the coordinate of the first area and a second representative coordinate representing the coordinate of the second area, and which reduces the distance in the optical axis direction of a distance distribution information acquiring means used to acquire the coordinate information of the subject. [Effects of the Invention]

[0009] According to the present invention, it is possible to reduce the sense of incongruity felt when viewing three-dimensional data that includes occlusion. [Brief explanation of the drawings]

[0010] [Figure 1] 10 is a flowchart illustrating an example of a three-dimensional data generation process according to the embodiment. [Figure 2] FIG. 10 is a diagram illustrating an example of region division according to the embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of region division according to the embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of region division according to the embodiment. [Figure 5] 10A and 10B are diagrams for explaining that the addition of an offset value according to the embodiment reduces the visibility of the occluded area. [Figure 6] 3A and 3B are diagrams illustrating examples of a first neighboring region and a second neighboring region according to an embodiment. [Figure 7] 10 is a flowchart illustrating an example of a three-dimensional data generation process according to a modified example of the embodiment. [Figure 8] 10 is a flowchart illustrating an example of a three-dimensional data generation process according to another modified example of the embodiment. [Figure 9] 10A and 10B are diagrams for explaining that the exposure of the occluded area is reduced by adding an offset value according to another modified example of the embodiment. [Figure 10] FIG. 1 is a diagram illustrating an example of a hardware configuration of an image processing apparatus according to an embodiment. [Figure 11] FIG. 2 is a diagram illustrating an example of an imaging unit according to an embodiment. [Figure 12] 5A and 5B are diagrams illustrating an example of the relationship between the defocus amount and the image shift amount of the imaging unit according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] First, definitions of terms used in the embodiments of the present invention will be explained.

[0012] The three-dimensional data is data including shape information that indicates a three-dimensional shape, such as a point cloud or a three-dimensional surface model. The three-dimensional data may also include color information that indicates a color in addition to the shape information.

[0013] When a point cloud is rendered, it is generally represented in a format in which point-like objects of finite size are placed at the positions of the points that make up the point cloud. When showing the rendering results of a point cloud in the following drawings, this format will be followed.

[0014] The coordinate information is distance distribution information, which will be described later, or the coordinate information is the three-dimensional coordinates of each point that constitutes the point cloud.

[0015] The Z axis extends from the origin of the distance distribution information acquisition means, such as a stereo camera or a ToF camera, and is parallel to the optical axis of the distance distribution information acquisition means. The X and Y axes are located in a plane perpendicular to the Z axis and are orthogonal to each other.

[0016] Distance distribution information is information that projects the distance to each point on the subject onto the XY plane. When distance distribution information is shown in the following drawings, the distance is expressed using a grayscale, with the closer the distance, the closer it is to white, and the farther the distance, the closer it is to black. Furthermore, one of the elements that make up distance distribution information is called a pixel.

[0017] The image and two-dimensional image refer to texture information indicating the texture of the subject and color distribution information indicating the color of the subject.

[0018] The occluded region is a region of the subject that is occluded by the occluding region, and therefore the coordinate information of the subject cannot be acquired by the distance distribution information acquisition means. The occluded region is missing in the three-dimensional data. The occluded region is a region that is occluded from the viewpoint of the distance distribution information acquisition means, making it impossible to acquire the coordinate information of the occluded region. Note that in the distance distribution information projected onto a two-dimensional plane, the occluded region and the occluded region coincide with each other.

[0019] The first and second regions are two-dimensional regions when the coordinate information is distance distribution information, and are three-dimensional regions when the coordinate information is three-dimensional coordinates of each point that constitutes the point cloud.

[0020] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, in which: Figure 1 is a flowchart showing an example of a three-dimensional data generation process according to an embodiment;

[0021] <Area division process step S101> Step S101 is executed by region dividing means included in the image processing device. In step S101, the region dividing means divides distance distribution information, which is an example of coordinate information of a subject, into a first region and a second region. The region dividing means defines the foreground including the occluded region as the first region, and the region other than the foreground including the occluded region as the second region.

[0022] 2A and 2B are diagrams illustrating an example of region division according to an embodiment. FIG. 2A shows an example of distance distribution information in a case where a circular subject is located in the foreground and a rectangular subject is located behind the circular subject in the background. FIG. 2B shows a region of the rectangular subject that is occluded by the circular subject. FIG. 2C shows an example of a first region and a second region in the case shown in FIGS. 2A and 2B. The region division means divides the coordinate information of the subject into a first region and a second region so that an occluding region that occludes an occluded region is included in the first region.

[0023] In the case shown in Fig. 2(A), the region dividing means recognizes the shaded region in Fig. 2(B), i.e., the region where the circular subject and the rectangular subject overlap, as the occluded region and the occluded region. In the case shown in Fig. 2(A), the region dividing means determines the region shown with diagonal hatching in Fig. 2(C), i.e., the region of the circular subject, as the first region. In the case shown in Fig. 2(A), the region dividing means determines the region shown with dotted hatching in Fig. 2(C), i.e., the region of the rectangular subject that does not overlap with the circular subject, as the second region.

[0024] FIG. 3 shows a case where there are multiple subjects. However, even if there is only one subject, one part of the subject may obscure another part of the subject. FIG. 3 is a diagram showing an example of region division according to an embodiment. FIG. 3(A) shows an example of distance distribution information when a person holds out the part from their right elbow to their right hand in front of their body. FIG. 3(B) shows the region of the person's body that is obscured by the part from their right elbow to their right hand. FIG. 3(C) shows an example of the first region and the second region in the case shown in FIG. 3(A) and FIG. 3(B).

[0025] In the case shown in Figure 3(A), the region dividing means recognizes the shaded region in Figure 3(B), i.e., the region where the part from the right elbow to the right hand overlaps with the body, as an occluded region and an occluded region. In the case shown in Figure 3(A), the region dividing means determines the region shown with diagonal hatching in Figure 3(C), i.e., the region from the right elbow to the right hand, as a first region. In the case shown in Figure 3(A), the region dividing means determines the region shown with dotted hatching in Figure 3(C), i.e., the region of the body that does not overlap with the part from the right elbow to the right hand, as a second region.

[0026] Fig. 4 is a diagram showing an example of region division according to the embodiment. Fig. 4(A) shows an example of distance distribution information when a person holds out the part from their left elbow to their left hand and the part from their right elbow to their right hand in front of their body. Fig. 4(B) shows the region of the person's body that is occluded by the part of their left hand and the part from their right elbow to their right hand. Fig. 4(C) shows an example of the first region and the second region in the case shown in Fig. 4(A) and Fig. 4(B).

[0027] In the case shown in FIG. 4(A), the region dividing means recognizes the shaded regions in FIG. 4(B), i.e., the regions where the body overlaps with the portion from the left elbow to the left hand or the portion from the right elbow to the right hand, as the occluded region and the occluded region. Next, in the case shown in FIG. 4(A), the region dividing means defines the two regions shown with diagonal hatching in FIG. 4(C), i.e., the region from the left elbow to the left hand and the region from the right elbow to the right hand, as sub-regions. The region dividing means then defines these two sub-regions as first regions. In addition, in the case shown in FIG. 4(A), the region dividing means defines the region shown with dotted hatching in FIG. 4(C), i.e., the region of the body that does not overlap with the portion from the left elbow to the left hand and the portion from the right elbow to the right hand, as second regions. In other words, in the case shown in FIG. 4(A), the region dividing means divides the coordinate information of the subject into a second region and a first region including multiple sub-regions that are not connected to each other.

[0028] The region division means performs the above-mentioned region division, for example, by segmentation based on deep learning. In particular, when the subject is a person, the body is often occluded by hands, arms, etc. In such cases, the region division means can appropriately perform region division by using a model that has been trained to assign different labels to regions of hands, arms, etc. and regions other than these regions.

[0029] <Offset Addition Process Step S102> Step S102 is executed by an offset adding means included in the image processing device. In step S102, the offset adding means adds an offset value to the distance distribution information of the first region. Here, the offset value is set so as to reduce the distance between a first representative coordinate representing the coordinates of the first region and a second representative coordinate representing the coordinates of the second region. Furthermore, this distance is the distance in the optical axis direction of the distance distribution information acquiring means used to acquire coordinate information of the subject.

[0030] Furthermore, the offset addition means calculates first representative coordinates by applying some rule to a set of coordinates of points belonging to the first region. Similarly, the offset addition means calculates second representative coordinates by applying some rule to a set of coordinates of points belonging to the second region. For example, the offset addition means may set the coordinates of the point closest to the center of gravity in the XY plane of the first region as the first representative coordinates, and the coordinates of the point closest to the center of gravity in the XY plane of the second region as the second representative coordinates. However, the offset addition means may calculate the first representative coordinates and the second representative coordinates using any method. Furthermore, at least one of the calculation of the first representative coordinates and the calculation of the second representative coordinates may be performed by an entity other than the offset addition means.

[0031] A perspective conflict edge is a boundary where the distance in the optical axis direction of the distance distribution information acquisition means changes discontinuously. A point cloud is an example of coordinate information of a subject, and when it includes a perspective conflict edge, the difference in distance in the optical axis direction of the distance distribution information acquisition means becomes large around the perspective conflict edge. When such a point cloud is viewed from a viewpoint that is not parallel to the optical axis direction, the part where the difference is large will be perceived as a gap.

[0032] 5A and 5B are diagrams illustrating how the addition of an offset value according to an embodiment reduces the appearance of an occluded area. FIG. 5A shows an example in which three-dimensional data based on a point cloud generated by converting the distance distribution information shown in FIG. 2A into a point cloud is viewed from a viewpoint not parallel to the optical axis direction, and no offset value is added. This gap is the appearance of an area in the background occluded by the foreground, i.e., an occluded area. When an occluded area appears, information is not present in an area where information actually exists or where information is expected to exist, which gives a significant sense of incongruity to a viewer of the point cloud.

[0033] FIG. 5(B) shows an example in which three-dimensional data based on a point cloud generated by converting the distance distribution information shown in FIG. 2(A) into a point cloud is viewed from a viewpoint not parallel to the optical axis direction, and an offset value is added. As described above, the offset adding means adds an offset value to the coordinate information of the first region so as to reduce the distance between the first representative coordinates and the second representative coordinates in the optical axis direction. As a result, the offset adding means reduces the difference in distance at the perspective boundary edge between the first region and the second region, as shown in FIG. 5(B). In other words, the offset adding means reduces the degree of perspective conflict between the first region and the second region at the perspective boundary edge, thereby reducing the gap. Therefore, the offset adding means can reduce the sense of incongruity that occurs when three-dimensional data is viewed from a viewpoint not parallel to the optical axis direction of the distance distribution information.

[0034] Note that the offset adding means adds an offset value to the coordinate information of the first region, which results in the generation of three-dimensional data that differs from the actual positional relationship of the subject when the distance distribution information was acquired. However, because the offset adding means does not change the shape of the subject in the first region or the shape information of the subject in the second region, it is possible to sufficiently minimize the sense of incongruity even when the positional relationship changes due to the addition of the offset value. Therefore, the image processing device can reduce the sense of incongruity felt when viewing three-dimensional data that includes occlusion.

[0035] As described above, FIG. 5 illustrates three-dimensional data generated based on a point cloud. When the three-dimensional data is surfaced, points on either side of the perspective conflict edge also form polygons, so the gaps shown in FIG. 5 are filled with polygons. Furthermore, when textured three-dimensional data is surfaced, the texture on the polygons formed by points on either side of the perspective conflict edge is a stretched version of the color information around the perspective conflict edge. In these two cases, the polygons formed by points on either side of the perspective conflict edge do not exist in the actual subject, and the larger the difference in distance around the perspective conflict edge, the larger these polygons become. Even in these two cases, the image processing device can reduce the sense of incongruity felt when viewing three-dimensional data containing occlusion.

[0036] The region dividing means may use the statistics of coordinates of a first neighboring region located in the first region near the boundary with the second region as the first representative coordinates. The region dividing means may use the statistics of coordinates of a second neighboring region located in the second region near the boundary with the first region as the second representative coordinates. As described above, in order to reduce the sense of incongruity when viewing three-dimensional data, it is necessary to reduce the degree of perspective conflict between the first region and the second region. Therefore, in order to reduce the degree of perspective conflict near the boundary between the first region and the second region, it is preferable that the region dividing means use the statistics of coordinates of the first neighboring region as the first representative coordinates and the statistics of coordinates of the second neighboring region as the second representative coordinates. The statistics may be, for example, the mean, median, or mode.

[0037] The first neighborhood region is a region encompassed by a pixel corresponding to the boundary between the first and second regions and a pixel a predetermined distance away from the first region. Similarly, the second neighborhood region is a region encompassed by a pixel corresponding to the boundary between the first and second regions and a pixel a predetermined distance away from the second region. FIG. 6 is a diagram illustrating examples of a first neighborhood region and a second neighborhood region according to an embodiment. In FIG. 6, the first neighborhood region is illustrated as a region filled with a dark color, and the second neighborhood region is illustrated as a region filled with a light color. Here, the predetermined length may be determined as, for example, 20 pixels using an index such as four neighbors from the pixel corresponding to the boundary between the first and second regions. Alternatively, if the diagonal length of the entire distance distribution information is L2 pixels, the predetermined length may be determined as 0.05 × L2 pixels from the pixel corresponding to the boundary between the first and second regions.

[0038] Furthermore, when the first region includes multiple sub-regions, as shown in Fig. 4, the offset adding means preferably adds a different offset value to each sub-region. This allows the image processing device to reduce gaps in the three-dimensional data, regardless of the degree of perspective conflict between each sub-region and the second region, by adding a different offset value to each sub-region, even if the degree of perspective conflict with the second region differs for each sub-region.

[0039] <Point Cloud Creation Step S103> Step S103 is executed by a point cloud generation means included in the image processing device. In step S103, the point cloud generation means converts the distance distribution information processed in step S102 into a point cloud by point cloud generation. Generally, distance distribution information acquired by distance measurement using distance distribution information acquisition means is perspectively projected. For this reason, the point cloud generation means can convert the distance distribution information into a point cloud, which is a set of coordinates of points in three-dimensional space, by back-projecting the distance distribution information based on the conditions applied when the distance distribution information was acquired. Of the X, Y, and Z coordinates of each point included in the point cloud, the Z coordinate is the distance itself included in the distance distribution information. The X coordinate of each point included in the point cloud is calculated using the following formula (1). Furthermore, the Y coordinate of each point included in the point cloud is calculated using the following formula (2). Formulas (1) and (2) are calculated using the coordinates (u, v) of a pixel based on the end points of the distance distribution information and the coordinates (u, v) of a pixel corresponding to the center of the distance distribution information. c ,v c ), the focal length f of the distance distribution information acquisition means, and the distance Z(u,v) at the coordinates (u,v).

[0040]

number

[0041]

number

[0042] <Surface processing> The image processing device may execute a surfacing step after the point clouding step of step S103 shown in Fig. 1. The surfacing step is executed by surfacing means included in the image processing device. In the surfacing step, the surfacing means generates three-dimensional data having a surface by connecting three or four adjacent points as polygons to form a mesh. Furthermore, if color distribution information of the same viewpoint as the distance distribution information exists, the surfacing means can generate three-dimensional data having color information by assigning the color distribution information as a texture to the surface.

[0043] [Variation 1] The above-described effects can also be obtained in inventions different from the inventions described with reference to Figures 1 to 6. Next, a first modification of the embodiment will be described with reference to Figure 7. Figure 7 is a flowchart showing an example of a three-dimensional data generation process according to the modification of the embodiment.

[0044] In step S201, the point cloud generation means converts the distance distribution information into a point cloud. Step S201 is the same process as step S103 described above.

[0045] In step S202, the region division means divides a point cloud, which is an example of coordinate information of the subject, into a first region and a second region. The region division means can perform region division, for example, by referring to the point cloud and applying segmentation based on deep learning. When the subject is a person, the region division means can appropriately perform region division by using a model that has been trained to assign different labels to regions such as hands and arms and regions other than these regions.

[0046] In step S203, the offset adding means adds an offset value to the coordinate information of the first region. Furthermore, in the first modification of the embodiment, the coordinate information of the subject is the coordinates of each point included in the point cloud. The offset value may be calculated based on the distance distribution information or the coordinates of each point included in the point cloud.

[0047] Furthermore, when the offset adding means calculates the offset value based on the distance distribution information, the offset adding means may calculate the offset value in the same manner as described above. The offset adding means uniformly adds the offset value to the Z coordinate, which is the coordinate parallel to the optical axis, of the point group belonging to the first region.

[0048] When calculating offset values based on the coordinates of each point included in the point cloud, the offset adding means may calculate offset values in each of the X, Y, and Z directions so as to reduce gaps in the perspective conflict edge. For example, for each point in a set U1 of points belonging to the first region that are close to the perspective conflict edge, the offset adding means may identify the nearest point in a set U2 of points belonging to the second region that are close to the perspective conflict edge, and evaluate the Euclidean distance to the nearest point. Next, the offset adding means may use the average value of the distance values for all points in set U1 as an index. The offset adding means may then calculate offset values in each of the X, Y, and Z directions so as to reduce the index by adding the offset value to the three-dimensional coordinates of each point constituting the point cloud in the first region.

[0049] The offset adding means may calculate the offset value using a method other than the above-described method. For example, the offset adding means may determine the above-described index using points randomly sampled from the points included in the set U1, rather than all points included in the set U1. Also, for example, the offset adding means may evaluate the difference in Z coordinates instead of Euclidean distance. When evaluating the difference in Z coordinates, the offset adding means sets the offset value in the X direction and the offset value in the Y direction to zero.

[0050] [Variation 2] In addition to the above-described processing, the image processing device can generate more suitable three-dimensional data with reduced discomfort by applying a filtering process to an area of the coordinate information that includes the first neighboring area and the second neighboring area. Next, a second modification of the embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of a three-dimensional data generation process according to another modification of the embodiment.

[0051] In step S301, the area dividing means divides the distance distribution information into a first area and a second area. Step S301 is the same as step S101 described above.

[0052] In step S302, the offset adding means adds an offset value to the distance distribution information of the first region. Step S302 is the same as step S102 described above.

[0053] Step S303 is executed by a filtering application means included in the image processing device. In step S303, the filtering application means applies filtering processing to an area of the coordinate information including the first neighboring area and the second neighboring area. Although the difference in distance in the Z direction around the perspective conflict edge is reduced by adding an offset value, it rarely becomes zero and often remains to some extent. However, by executing the processing of step S303, the filtering application means smooths the difference and can generate more suitable three-dimensional data with reduced unnaturalness. Furthermore, the filtering processing is, for example, processing that applies a smoothing filter or a median filter.

[0054] Fig. 9 is a diagram for explaining that the appearance of an occluded area is reduced by adding an offset value according to another modified example of the embodiment. Fig. 9 is a perspective view showing three-dimensional data generated by applying the process shown in Fig. 8 to the distance distribution information shown in Fig. 2(A). In the three-dimensional data shown in Fig. 9, the distance in the Z direction around the perspective conflict edge changes more continuously than in the three-dimensional data shown in Fig. 5(B), so gaps are hardly visible and the sense of incongruity is further reduced.

[0055] [Variation 3] The distance distribution information of the first region is Z1(u,v), the distance distribution information obtained by adding an offset value to Z1(u,v) is Z1'(u,v), and the distance distribution information used when calculating coordinates in a direction perpendicular to the optical axis direction in point grouping by the point grouping means is Z1"(u,v). In this case, it is preferable that the difference between Z1(u,v) and Z1'(u,v) is larger than the difference between Z1(u,v) and Z1"(u,v). The reason for this will be explained below.

[0056] As described above, in the point grouping process of step S103, the X coordinate and Y coordinate, which are coordinates in the direction perpendicular to the optical axis, are calculated by formula (1) and formula (2), respectively. When Z1(u,v) is substituted for Z(u,v) in formula (1), the left side is set to X1, when Z1'(u,v) is substituted for Z(u,v) in formula (1), the left side is set to X1', and when Z1"(u,v) is substituted for Z(u,v) in formula (1), the left side is set to X1". In this case, Z1 ≠ Z1', and therefore X1 ≠ X1'.

[0057] Z1(u,v) represents the distance distribution in the Z direction of the subject at the time the distance distribution information was acquired. Therefore, X1 represents the distance distribution in the X direction of the subject at the time the distance distribution information was acquired. On the other hand, Z1'(u,v) is obtained by adding an offset value to Z1(u,v) to correct the distance distribution of the subject. Therefore, X1' does not represent the distance distribution in the X direction of the subject at the time the distance distribution information was acquired, and considering equation (1), it is larger than X1. In other words, X1' is an enlarged version of the distance distribution in the X direction of the subject at the time the distance distribution information was acquired compared to X1, which may cause discomfort to viewers viewing the three-dimensional data.

[0058] From the above, it is preferable that the distance distribution information Z1"(u,v) used when calculating the X coordinate and Y coordinate of the point group in the first region is closer to Z1(u,v) than to Z1'(u,v) to which an offset value has been added. Furthermore, it is more preferable that Z1"(u,v) is equal to Z1(u,v).

[0059] Examples of indices for evaluating the magnitude relationship between the difference between Z1(u,v) and Z1′(u,v) and the difference between Z1(u,v) and Z1″(u,v) include the average error and the root mean square error. Furthermore, when the average error is the index, the condition that the difference between Z1(u,v) and Z1′(u,v) is greater than the difference between Z1(u,v) and Z1″(u,v) is expressed by the following equation (3).

[0060]

number

[0061] Next, an example of the hardware configuration of an image processing device will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of the hardware configuration of an image processing device according to an embodiment. The image processing device includes an imaging unit 401, a control unit 402, a memory 403, a storage device 404, and an image processing unit 405 shown in Fig. 10.

[0062] The imaging unit 401 functions as a means for acquiring distance distribution information and color distribution information of a subject. The control unit 402 is a control device such as a CPU (Central Processing Unit) and controls the operation of each block of the image processing device. For example, the control unit 402 transmits information such as distance distribution information and color distribution information acquired by the imaging unit 401 to an image processing unit 405, and stores the information processed by the image processing unit 405 in a storage device 404.

[0063] The memory 403 is, for example, a volatile memory, and is used to temporarily store information. The storage device 404 is, for example, a hard disk drive (HDD), and is used to permanently store information. The image processing unit 405 applies image processing such as the above-mentioned point cloud generation to information such as distance distribution information and color distribution information acquired by the imaging unit 401. The image processing unit 405 may be realized as an integrated circuit, or may be realized as a software functional module. The image processing unit 405 also functions as the above-mentioned point cloud generation means.

[0064] Next, an example of the configuration of the imaging unit 401 will be described with reference to FIG. 11 . The imaging unit 401 may have any configuration as long as it can function as a distance distribution information acquisition unit. Furthermore, it is preferable that the imaging unit 401 be configured to be able to acquire color distribution information simultaneously with distance distribution information. Examples of imaging units 401 having such a function include a stereo camera and a ToF camera. A stereo camera includes two imaging systems each having an optical system and an imaging element capable of acquiring RGB images. A ToF camera includes a ToF module capable of acquiring distance distribution information and an imaging system capable of acquiring RGB images. In the following description, an example of the configuration of the imaging unit 401 capable of simultaneously acquiring distance distribution information and color distribution information using an imaging surface phase difference ranging method will be given.

[0065] 11A and 11B are diagrams illustrating an example of an imaging unit according to an embodiment. Fig. 11A shows an imaging optical system 501 and an imaging element 502 included in the imaging unit 401. Fig. 11B shows an example of the configuration of the imaging element 502.

[0066] As shown in FIG. 11(A), the imaging optical system 501 collects light from a subject located on an object plane and captures an image by exposing an imaging element 502 located at a position that is approximately optically conjugate with the object plane.

[0067] 11(B), the image sensor 502 has a structure in which, for example, tens of millions of pixels 503, which are photoelectric conversion elements, are arranged in a grid pattern. The image sensor 502 also has a configuration in which color filters that transmit specific wavelengths such as red, green, and blue are arranged in a Bayer array in each pixel 503, thereby enabling the acquisition of color distribution information.

[0068] 11(B), each pixel 503 includes a microlens 504, a first photoelectric conversion unit 505, and a second photoelectric conversion unit 506, and acquires two different pieces of optical information from the photoelectric conversion units 505 and 506, which are arranged in the X direction. From the optical information acquired by each pixel 503, the imaging unit 401 acquires a first image configured as a luminance distribution of light received by the first photoelectric conversion unit 505, and a second image configured as a luminance distribution of light received by the second photoelectric conversion unit 506.

[0069] The incident surface of the image sensor 502 and the light receiving surfaces of the photoelectric conversion unit 505 and the photoelectric conversion unit 506 are in a nearly Fourier conjugate relationship via the microlens 504. Therefore, the light receiving surface of the photoelectric conversion unit and the exit pupil of the image pickup optical system 501 are in a nearly optically conjugate relationship. The position distribution at the exit pupil corresponds to the position distribution at the light receiving surfaces of the photoelectric conversion unit 505 and the photoelectric conversion unit 506. Therefore, by including different photoelectric conversion units 505 and 506, the image pickup unit 401 can separately receive light beams that have passed through different pupil regions. The first image and second image described above are luminance distribution information acquired from light beams that have passed through different pupil regions.

[0070] 12 is a diagram showing an example of the relationship between the defocus amount and the image shift amount of the image capturing unit according to the embodiment. As shown in the upper diagram of FIG. 12, in an ideal case, light rays formed on the imaging surface are incident on the same point on the image sensor 502 regardless of the position on the pupil through which they pass. However, as shown in the center or lower diagram of FIG. 12, when the light rays are defocused, the point at which they are incident on the image sensor 502 changes depending on the position on the pupil through which they pass. In other words, when the light rays are defocused, an image shift occurs according to the amount of defocus.

[0071] If the distance distribution information acquisition means can calculate the image shift amount, it can acquire distance distribution information by converting the image shift amount into a defocus amount. The distance distribution information acquisition means can calculate the image shift amount, for example, by performing stereo matching between the first image and the second image. The distance distribution information acquisition means can calculate the image shift amount by performing matching along the direction of the epipolar line for a patch of a small area in one image with the other image to identify the position of highest correlation. Furthermore, the distance distribution information acquisition means converts the image shift amount into a defocus amount by using optical parameters such as focal length based on the image shift amount.

[0072] The present invention also includes the following inventions that appropriately combine the above-mentioned features.

[0073] (Configuration 1) An image processing device comprising: an area dividing means for dividing coordinate information of a subject into a first area and a second area; and an offset adding means for adding an offset value to the coordinate information of the first area, the offset value being the distance between a first representative coordinate representing the coordinates of the first area and a second representative coordinate representing the coordinates of the second area, the offset value reducing the distance in the optical axis direction of a distance distribution information acquiring means used to acquire the coordinate information of the subject.

[0074] (Configuration 2) The image processing device according to configuration 1, wherein the area dividing means divides the coordinate information of the subject into the first area and the second area so that the first area includes an occluded area that occludes an occluded area for which coordinate information of the subject cannot be acquired by the distance distribution information acquisition means.

[0075] (Configuration 3) 3. The image processing device according to claim 1, wherein the region dividing means divides the coordinate information of the subject into the second region and the first region including a plurality of sub-regions that are not connected to each other, and the offset adding means adds a different offset value to each of the sub-regions.

[0076] (Configuration 4) 4. The image processing device according to any one of configurations 1 to 3, wherein the coordinate information of the subject is three-dimensional coordinates of each point that constitutes a point cloud.

[0077] (Configuration 5) 5. The image processing device according to any one of configurations 1 to 4, wherein the coordinate information of the subject is distance distribution information acquired by the distance distribution information acquisition means, and further comprising point cloud generation means for generating a point cloud from the distance distribution information.

[0078] (Configuration 6) The image processing device according to configuration 5, characterized in that a difference between the distance distribution information of the first region and the distance distribution information obtained by adding the offset value to the distance distribution information of the first region is larger than a difference between the distance distribution information of the first region and the distance distribution information used when calculating coordinates in a direction perpendicular to the optical axis direction in point cloud generation by the point cloud generation means.

[0079] (Configuration 7) The image processing device according to any one of configurations 1 to 6, wherein the first representative coordinates are statistics of coordinates of a first neighboring region of the first region that is located near the boundary with the second region, and the second representative coordinates are statistics of coordinates of a second neighboring region of the second region that is located near the boundary with the first region.

[0080] (Configuration 8) 8. The image processing device according to configuration 7, further comprising filtering application means for applying filtering to a region of the coordinate information that includes the first neighboring region and the second neighboring region.

[0081] (Configuration 9) a first representative coordinate representing the coordinates of the first region and a second representative coordinate representing the coordinates of the second region, and adding an offset value to the coordinate information of the first region, the offset value being the distance between a first representative coordinate representing the coordinates of the first region and a second representative coordinate representing the coordinates of the second region, the offset value reducing the distance in the optical axis direction of a distance distribution information acquisition means used to acquire the coordinate information of the subject.

[0082] (Configuration 10) A three-dimensional data generation method comprising: dividing coordinate information of a subject into a first region and a second region; and adding an offset value to the coordinate information of the first region, the offset value being the distance between a first representative coordinate representing the coordinates of the first region and a second representative coordinate representing the coordinates of the second region, the offset value reducing the distance in the optical axis direction of a distance distribution information acquisition means used to acquire the coordinate information of the subject.

[0083] <Other embodiments> Some of the elements constituting the present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a recording medium, and having one or more processors in the computer of the system or device read and execute the program. Also, some of the elements constituting the present invention can be realized by a circuit that realizes one or more functions, such as an ASIC (Application Specific Integrated Circuit).

[0084] The preferred embodiments of the present invention have been described above. However, the present invention is not limited to the above-described embodiments. In other words, the present invention includes embodiments in which various modifications have been made based on the spirit of the present invention, and these embodiments are not excluded from the scope of the present invention. [Explanation of symbols]

[0085] 401....imaging unit 402 Control unit 403...Memory 404‥‥Storage device 405...Image processing unit

Claims

1. an area dividing means for dividing coordinate information of a subject into a first area and a second area; an offset adding means for adding an offset value to the coordinate information of the first region, the offset value being a distance between a first representative coordinate representing the coordinates of the first region and a second representative coordinate representing the coordinates of the second region, the offset value reducing the distance in the optical axis direction of a distance distribution information acquiring means used to acquire coordinate information of the subject; An image processing device comprising:

2. the region dividing means divides the coordinate information of the subject into the first region and the second region so that the first region includes an occluded region that occludes an occluded region for which coordinate information of the subject cannot be acquired by the distance distribution information acquiring means; 2. The image processing device according to claim 1, wherein:

3. the region dividing means divides the coordinate information of the subject into the second region and the first region including a plurality of sub-regions that are not connected to one another; the offset adding means adds a different offset value to each of the sub-regions.

3. The image processing device according to claim 1, wherein the image processing device is a computer.

4. The coordinate information of the subject is the three-dimensional coordinates of each point constituting the point cloud.

3. The image processing device according to claim 1, wherein the image processing device is a computer.

5. the coordinate information of the subject is distance distribution information acquired by the distance distribution information acquisition means, further comprising a point cloud generation means for generating a point cloud of the distance distribution information; 3. The image processing device according to claim 1, wherein the image processing device is a computer.

6. a difference between the distance distribution information of the first region and the distance distribution information obtained by adding the offset value to the distance distribution information of the first region is larger than a difference between the distance distribution information of the first region and the distance distribution information used when calculating coordinates in a direction perpendicular to the optical axis direction in point grouping by the point grouping means; 6. The image processing device according to claim 5,

7. the first representative coordinates are statistics of coordinates of a first neighboring region located in the first region near the boundary with the second region, The second representative coordinates are statistics of coordinates of a second neighboring region of the second region that is located near the boundary with the first region.

3. The image processing device according to claim 1, wherein the image processing device is a computer.

8. The image processing device further includes a filtering application unit that applies a filtering process to a region of the coordinate information that includes the first neighboring region and the second neighboring region.

8. The image processing device according to claim 7,

9. Dividing the coordinate information of the subject into a first area and a second area; an offset value is added to the coordinate information of the first region, the offset value being a distance between a first representative coordinate representing the coordinates of the first region and a second representative coordinate representing the coordinates of the second region, the offset value reducing the distance in the optical axis direction of a distance distribution information acquisition means used to acquire the coordinate information of the subject; A three-dimensional data generation program comprising:

10. Dividing the coordinate information of the subject into a first area and a second area; an offset value is added to the coordinate information of the first region, the offset value being a distance between a first representative coordinate representing the coordinates of the first region and a second representative coordinate representing the coordinates of the second region, the offset value reducing the distance in the optical axis direction of a distance distribution information acquisition means used to acquire the coordinate information of the subject; Three-dimensional data generation method.

Images