Three-dimensional information processing device and three-dimensional information processing method
The three-dimensional information processing device addresses the challenge of capturing complete 3D shapes with fewer cameras by using image and distance information processing units to supplement data, achieving efficient and accurate 3D model generation, including hidden surfaces, with improved resource utilization and resolution maintenance.
Patent Information
- Application Number
- PCT/JP2024/014371
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-10
- Filing Date
- 2024-04-09
- Publication Date
- 2024-11-14
AI Technical Summary
Conventional three-dimensional information processing methods require multiple distance-measuring cameras to capture images from various viewpoints, making it difficult to obtain accurate 3D information, especially for dynamic shapes and the back surface of objects, due to resource-intensive data synthesis and limitations in real-time capture.
A three-dimensional information processing device that includes an image acquisition unit, distance information acquisition unit, boundary detection unit, thinning processing unit, and back surface complement processing unit to acquire, process, and supplement distance information, allowing for the generation of a complete 3D shape even when the back surface is not directly observable.
Enables the generation of accurate three-dimensional models with high reproducibility by reducing data processing load and enhancing the ability to capture dynamic shapes and hidden surfaces using fewer cameras, maintaining high image resolution without down-converting data.
Smart Images

Figure JP2024014371_14112024_PF_FP_ABST
Abstract
Description
Three-dimensional information processing device and three-dimensional information processing method
[0001] This application claims priority to Japanese Patent Application No. 2023-077668, filed May 10, 2023, the contents of which are incorporated herein by reference.
[0002] Conventionally, the three-dimensional shape of an object existing in the real world is acquired, and a three-dimensional model is modeled based on the acquired three-dimensional shape. In order to accurately acquire the three-dimensional shape of an object, there is a technique for acquiring three-dimensional information of the object from multiple viewpoints using multiple ranging cameras. The three-dimensional information acquired from the multiple ranging cameras is combined into one piece of three-dimensional information. By acquiring three-dimensional information of the object from multiple viewpoints using multiple ranging cameras, it is possible to acquire three-dimensional information with a higher degree of reproducibility than when three-dimensional information is acquired from one direction using a single ranging camera. For example, the technique described in Patent Document 1 can be exemplified as a technique for combining three-dimensional information acquired from multiple ranging cameras into one piece of three-dimensional information.
[0003] Japanese Unexamined Patent Publication No. 7-174538
[0004] However, according to the above-described conventional technology, the relative positions between images are calculated using multiple image data captured from multiple viewpoints, coordinate transformation parameters for the image data are obtained, and the three-dimensional information is stitched together based on the obtained coordinate transformation parameters to synthesize the images into a single piece of three-dimensional information. To obtain three-dimensional information with higher fidelity, it is necessary to use more ranging cameras to capture images of the subject from various viewpoints. When capturing images of a subject using multiple ranging cameras, resources are required to synthesize the three-dimensional information. This poses a problem, particularly when capturing the dynamic three-dimensional shape of the subject, making it difficult to capture the three-dimensional shape in real time. In view of this problem, it is conceivable to capture images of the subject using a small number of ranging cameras. For example, when attempting to capture three-dimensional information of a subject using a single ranging camera, it is not possible to capture three-dimensional information of the subject's back surface, making it difficult to generate a three-dimensional model based on the subject.
[0005] Therefore, the present invention has been made in consideration of such circumstances, and aims to provide a three-dimensional information processing device that is capable of generating the three-dimensional shape of a subject even when the three-dimensional shape of the back of the subject cannot be obtained.
[0006] [1] One aspect of this embodiment is a three-dimensional information processing device that includes an image acquisition unit that acquires an image of a subject, a distance information acquisition unit that acquires distance information to the subject, a boundary detection unit that detects the boundary between the subject and a background based on the acquired image, and a back surface completion processing unit that derives a function that indicates a change in a predetermined direction from the acquired distance information, and complements distance information on the back surface of the subject based on the derived function and points on the detected boundary.
[0007] [2] Furthermore, one aspect of this embodiment is a three-dimensional information processing device as described in [1] above, further comprising a thinning processing unit that extracts feature points from the acquired image and performs a thinning process to reduce the amount of data of the distance information by thinning out distance information other than the extracted feature points, and the back surface completion processing unit derives a function that passes through the three-dimensional coordinates of the feature points as the function, and complements the distance information on the back surface of the subject in the space thinned out by the thinning processing unit.
[0008] [3] Furthermore, one aspect of this embodiment is that in the three-dimensional information processing device described in [1] or [2] above, the back surface completion processing unit estimates distance information on the back surface of the subject so that it is within a range of maximum and minimum values of predetermined three-dimensional coordinates.
[0009] [4] Furthermore, one aspect of this embodiment is that in the three-dimensional information processing device described in any one of [1] to [3] above, the back surface completion processing unit further includes a back surface image information completion unit that complements image information on the back surface of the subject based on image information on the front surface of the subject.
[0010] [5] Furthermore, one aspect of this embodiment is a three-dimensional information processing method including an image acquisition process for acquiring an image of a subject, a distance information acquisition process for acquiring distance information to the subject, a boundary detection process for detecting a boundary between the subject and a background based on the acquired image, and a back surface completion process for deriving a function indicating a change in a predetermined direction from the acquired distance information, and completing distance information on the back surface of the subject based on the derived function and points on the detected boundary.
[0011] According to this embodiment, even if the three-dimensional shape of the rear surface of the subject cannot be acquired, the three-dimensional shape of the subject can be generated.
[0012] FIG. 1 is a functional configuration diagram showing an example of the functional configuration of a three-dimensional information generation system according to an embodiment; FIG. 1 is a functional configuration diagram showing an example of the functional configuration of a three-dimensional information processing device according to the embodiment; FIG. 2 is a functional configuration diagram showing an example of the functional configuration of a thinning processing unit according to the embodiment; FIG. 3 is a diagram for explaining feature point detection processing according to the embodiment; FIG. 4 is a diagram for explaining thinning processing according to the embodiment; FIG. 5 is a functional configuration diagram showing an example of the functional configuration of a back surface completion processing unit according to the embodiment; FIG. 6 is a diagram for explaining top of head completion processing according to the embodiment; FIG. 7 is a diagram for explaining temporal region completion processing according to the embodiment; FIG. 8 is a diagram showing an example of point cloud data after thinning processing and back surface completion processing according to the embodiment; FIG. 9 is a diagram for explaining ToF resolution up-conversion according to the embodiment; FIG. 10 is a diagram showing an example of point cloud data and mesh data according to the embodiment; FIG. 11 is a flowchart showing an example of a series of operations performed by the three-dimensional information processing device according to the embodiment; FIG. 12 is a block diagram showing an example of the internal configuration of the three-dimensional information processing device according to the embodiment;
[0013] A preferred embodiment of a three-dimensional information processing device according to the present invention will be described in detail below with reference to the accompanying drawings. Note that the embodiment described below is merely an example, and the embodiments to which the present invention is applicable are not limited to the following embodiments. Furthermore, "based on XX" in this application means "based on at least XX" and includes cases where the device is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is directly used, but also includes cases where the device is based on XX after calculation or processing. "XX" is any element (e.g., any information). Furthermore, in the following drawings, the scale and number of elements in each structure may differ from the scale and number of elements in the actual structure to make each configuration easier to understand.
[0014] 1 is a functional configuration diagram showing an example of the functional configuration of a three-dimensional information generation system according to an embodiment. An example of the functional configuration of the three-dimensional information generation system 1 will be described with reference to the diagram. In the following description, the orientation of each device in the three-dimensional information generation system 1, the positional relationships between each device, and the like will be described in terms of a three-dimensional Cartesian coordinate system of the x-axis, y-axis, and z-axis.
[0015] The three-dimensional information generation system 1 includes a three-dimensional information processing device 10 and an imaging device 20. By including the three-dimensional information processing device 10 and the imaging device 20, the three-dimensional information generation system 1 acquires three-dimensional information of a subject S and generates a three-dimensional model of the subject S by performing processing based on the acquired information. The imaging device 20 captures an image of the subject S from a point a distance D away from the subject S in the z-axis direction. A screen SCR such as a blue screen may be placed behind the subject S. Note that if the three-dimensional shape of the subject S can be easily separated from the background, the screen SCR is not required.
[0016] The imaging device 20 is a distance measuring camera capable of acquiring three-dimensional information of the subject S. The imaging device 20 acquires three-dimensional information of the subject S by two-dimensionally measuring the distance to the subject S corresponding to the captured image (or video). The three-dimensional information of the subject S acquired by the imaging device 20 may be, for example, a distance image having distance information at each coordinate in a two-dimensional coordinate system. The imaging device 20 may, for example, use a ToF (Time of Flight) method to two-dimensionally irradiate light onto the subject S and measure the distance based on the time it takes for the reflected light to be received. The imaging device 20 outputs image information IMG1 and distance information IMG2 to the three-dimensional information processing device 10 as the acquired three-dimensional information of the subject S.
[0017] Image information IMG1 includes image information (e.g., an RGB image) of subject S captured from a predetermined direction. Distance information IMG2 includes distance information corresponding to image information IMG1. Distance information IMG2 includes multiple pieces of distance information corresponding to coordinate information in the x-y plane. The coordinate information in the x-y plane contained in this distance information corresponds to the pixels contained in image information IMG1. While it is preferable for an image to have distance information for each pixel it contains, it is also possible for the image to have one piece of distance information for multiple pixels. In other words, the resolution of distance information IMG2 in the x-y plane may be lower than the resolution of image information IMG1.
[0018] In the following description, the surface of the subject S on which the imaging device 20 is present may be referred to as the front surface of the subject S, and the surface on which the screen SCR is present may be referred to as the back surface of the subject S. The front and back surfaces of the subject S are not identified by the shape of the subject S, but are identified by the positional relationship between the imaging device 20 and the subject S. Therefore, it can be said that the image information IMG1 includes image information of the front surface of the subject S, and the distance information IMG2 includes distance information of the front surface of the subject S.
[0019] The three-dimensional information processing apparatus 10 acquires image information IMG1 and distance information IMG2 from the imaging device 20. The three-dimensional information processing apparatus 10 generates a three-dimensional model having a three-dimensional shape of the subject S based on the acquired image information IMG1 and distance information IMG2. The three-dimensional model generated by the three-dimensional information processing apparatus 10 may be, for example, point cloud data or mesh data. Here, the three-dimensional information generation system 1 acquires information of the subject S from one direction using a single imaging device 20. Therefore, the three-dimensional information generation system 1 is unable to acquire sufficient information about the rear surface of the subject S. The three-dimensional information processing apparatus 10 complements the three-dimensional information about the rear surface of the subject S based on the information acquired from the imaging device 20 and generates a three-dimensional model. Note that this embodiment is not necessarily limited to the case where only one imaging device 20 is used, and multiple imaging devices 20 may be used.
[0020] FIG. 2 is a functional configuration diagram showing an example of the functional configuration of a three-dimensional information processing device according to this embodiment. An example of the functional configuration of the three-dimensional information processing device 10 will be described with reference to the diagram. The three-dimensional information processing device 10 includes an image acquisition unit 11, a distance information acquisition unit 12, a boundary detection unit 13, an array processing unit 14, a thinning processing unit 15, a back surface completion processing unit 16, a point cloud data generation unit 21, a meshing processing unit 17, a material generation unit 18, and an output unit 19. Each of these functional units is implemented using, for example, electronic circuits. Furthermore, each functional unit may internally include storage means such as a semiconductor memory or a magnetic hard disk drive, as necessary. Furthermore, each function may be implemented by a computer and software.
[0021] The image acquisition unit 11 acquires image information IMG1 obtained by capturing an image of the subject S from the imaging device 20. The image acquisition unit 11 outputs the acquired image information IMG1 to the boundary detection unit 13.
[0022] The distance information acquisition unit 12 acquires distance information IMG2 indicating the three-dimensional shape of the subject S from the imaging device 20. The distance information acquisition unit 12 outputs the acquired distance information IMG2 to the array processing unit 14. Note that the image information IMG1 acquired by the image acquisition unit 11 and the distance information IMG2 acquired by the distance information acquisition unit 12 are associated with each other using a predetermined method. The predetermined method may be a method based on time information, an identification number, or the like.
[0023] The boundary detection unit 13 acquires image information IMG1 from the image acquisition unit 11. The boundary detection unit 13 detects the boundary between the subject S and the background based on the acquired image information IMG1. The boundary between the subject S and the background is, for example, the outline of the person if the subject S is a person, and particularly the outline of the person's face if the subject S is the person's face. The outline of the person's face includes the top of the head, which is the boundary between the hair and the background. The boundary detection process performed by the boundary detection unit 13 may use a known object detection algorithm. The boundary detection unit 13 outputs information about the detected boundary to the array processing unit 14 as boundary detection information BDI.
[0024] The arraying processor 14 acquires boundary detection information BDI from the boundary detection unit 13 and acquires distance information IMG2 from the distance information acquisition unit 12. The arraying processor 14 extracts data inside the boundary portion identified by the boundary detection information BDI from the distance information IMG2 and arrays the extracted data. The arraying process deletes information from the distance information IMG2 in the background portion other than the subject S, i.e., information unrelated to the three-dimensional information of the subject S. The arraying processor 14 outputs the information obtained as a result of the arraying process to the thinning processor 15 as first array information SI1.
[0025] The thinning processor 15 acquires the first array information SI1 from the array processor 14. First, the thinning processor 15 extracts feature points of the subject S from the image information included in the acquired first array information SI1. Next, the thinning processor 15 reduces the amount of distance information data by thinning out distance information at coordinates other than those of the extracted feature points. In the following description, the processing performed by the thinning processor 15 may be referred to as thinning processing. The thinning processing will be described in detail with reference to FIGS. 3 to 5 .
[0026] 3 is a functional configuration diagram showing an example of the functional configuration of the thinning processing unit according to this embodiment. An example of the functional configuration of the thinning processing unit 15 will be described with reference to the same diagram. The thinning processing unit 15 includes a feature point detection unit 151 and a distance information extraction unit 152. The feature point detection unit 151 acquires first array information SI1 from the array processing unit 14. The first array information SI1 includes distance information for a portion of the subject S excluding the background portion from the distance information IMG2 acquired by the imaging device 20. The feature point detection unit 151 detects feature points of the subject S by analyzing image information for the portion of the subject S.
[0027] FIG. 4 is a diagram illustrating the feature point detection process according to this embodiment. The feature point detection process performed by the feature point detection unit 151 will be described with reference to FIG. In FIG. 4, when the subject S is a person, circles are placed around detected feature points. A feature point is a point used to identify the three-dimensional shape of the subject S; in other words, it may be a point at which the three-dimensional shape changes. When the subject S is a person's face, specifically, 486 feature points may be extracted. A known feature point detection algorithm may be used for the feature point detection process. Returning to FIG. 3 , the feature point detection unit 151 outputs information about the detected feature points to the distance information extraction unit 152 as feature point information FPI. The feature point information FPI includes three-dimensional coordinate information of the feature points.
[0028] The distance information extraction unit 152 acquires feature point information FPI from the feature point detection unit 151 and acquires first array information SI1 from the array processing unit 14. The distance information extraction unit 152 performs thinning processing of the point cloud data by extracting distance information of the feature point information FPI from the first array information SI1, i.e., by discarding information other than the feature point information FPI. If the subject S is a person's face, the feature point detection unit 151 detects feature points in the facial area. Here, the three-dimensional shape of the subject S may include parts other than the facial area, such as the neck. The distance information extraction unit 152 performs thinning processing only in the range where feature points are detected by the feature point detection unit 151, and does not perform thinning processing on other parts (such as the neck part other than the facial area).
[0029] Here, it is preferable that the feature point detection process performed by the feature point detection unit 151 detects feature points for all points of the subject S. "All points of the subject S" means all feature points within the contour of the subject S, that is, all points whose three-dimensional shape changes within the contour of the subject S. However, when a known feature point detection algorithm is used, it is possible to detect feature points for the facial part of the subject S, but it may not be possible to detect feature points for parts other than the face (for example, the top of the head, the temporal region, etc.). In such cases, it is preferable to extend the feature point detection process performed by the feature point detection unit 151.
[0030] 5 is a diagram for explaining the thinning process according to this embodiment. The extended feature point detection process and thinning process will be explained with reference to this diagram.
[0031] 5A shows a range AR1 in which feature points have been detected by feature point detection processing within distance information IMG2 of subject S, and a range AR2 that shows the outline of distance information IMG2 of subject S. While the distance information thinning process described above can reduce the amount of point cloud data within range AR1 within range AR2, it cannot reduce the amount of point cloud data within range AR2 outside range AR1. According to this embodiment, the thinning process performed within range AR1 is extended to range AR2, thereby performing overall thinning processing for the interior of subject S. Specifically, the overall thinning processing for the interior of subject S includes thinning processing at the top of the head and thinning processing at the sides of the head. The location where thinning processing is performed at the top of the head is shown as P1, and the location where thinning processing is performed at the sides of the head is shown as P2.
[0032] A plurality of arrows are shown within P1. The plurality of arrows shown within P1 are drawn at intervals between feature points present at the boundary between ranges AR1 and AR2. In the thinning process according to this embodiment, distance information on the arrows shown in the figure is retained within P1, and other distance information is thinned out. The distance information on the arrows is also thinned out at a predetermined interval. The predetermined interval may be an interval based on the interval between distance information within range AR1. The thinning process within P1 is performed in the vertical direction (y-axis direction) as shown in the figure.
[0033] Similarly, multiple arrows are also shown inside P2. The multiple arrows shown inside P2 are drawn at intervals between feature points present at the boundary between ranges AR1 and AR2. In the thinning process according to this embodiment, distance information on the arrows shown in the figure is retained inside P2, and other distance information is thinned out. The distance information on the arrows is also thinned out at a predetermined interval. The predetermined interval may be an interval based on the interval of distance information inside range AR1. The thinning process inside P2 is performed in the horizontal direction (x-axis direction) as shown in the figure.
[0034] 5B shows an example of distance information obtained by the extended thinning process described above. As shown in the figure, even for P1 and P2, it can be seen that a small amount of distance information exists after the thinning process. Returning to FIG. 3, the distance information extraction unit 152 outputs the distance information obtained as a result of the thinning process to the back surface completion processing unit 16 as second array information SI2.
[0035] Returning to FIG. 2 , the back surface completion processor 16 acquires image information IMG1 from the image acquisition unit 11 and acquires second array information SI2 from the thinning processor 15. The back surface completion processor 16 calculates a function representing the change in a predetermined direction of the point cloud from the acquired second array information SI2 (note that in the following description, this may also be referred to as "deriving a function"). This function is a function that passes through the three-dimensional coordinates of the feature points detected by the feature point detection unit 151. Specifically, in the case where the subject S is a person's face, this function represents the change in the y-z plane (see FIG. 5 ) of the person's face. The back surface completion processor 16 complements distance information on the back surface of the subject S based on the calculated function and the boundary points detected by the boundary detection unit 13. The back surface completion processor 16 may complement the obtained distance information (distance information on the back surface of the subject S) in the space thinned out by the thinning processor 15. In the following description, the processing performed by the back surface completion processor 16 may also be referred to as back surface completion processing. The back surface complement process will be described in detail with reference to FIGS.
[0036] 6 is a functional configuration diagram showing an example of the functional configuration of the back surface completion processing unit according to this embodiment. An example of the functional configuration of the back surface completion processing unit 16 will be described with reference to the same figure. The back surface completion processing unit 16 includes a parietal completion unit 161, a temporal completion unit 162, a back surface completion information generation unit 163, and a back surface image information completion unit 164.
[0037] The parietal complementing unit 161 includes a parietal function calculating unit 1611 and a parietal estimation unit 1612. The parietal function calculating unit 1611 acquires the second array information SI2 from the thinning processing unit 15. The parietal function calculating unit 1611 calculates a function at the parietal region. The function calculated by the parietal function calculating unit 1611 is a function that passes through the distance information of the subject S in the vertical direction. Specifically, when the subject S is a person's face, this function indicates a change in the three-dimensional shape of the person's face in the y-z plane (see FIG. 7), and is a function that passes through a point at the hairline on the forehead, a point at the parietal region (the boundary between the subject and the background), and a point at the back of the head in the y-z plane. This function may be, for example, a quadratic function. The parietal function calculating unit 1611 calculates multiple functions at predetermined intervals in the horizontal direction (x-axis direction). The predetermined intervals may be, for example, intervals that are large enough to adequately represent the shape of the subject S when a three-dimensional shape is generated. Hereinafter, the processing performed by the vertex completion unit 161 may be referred to as vertex completion processing.
[0038] FIG. 7 is a diagram for explaining the top of the head completion processing according to this embodiment. An example of a function obtained by the top of the head completion processing will be described with reference to the same figure. The same figure shows three-dimensional information of subject S viewed in the x-axis direction. Coordinates C1 (Z1, Y1) and coordinates C2 (Z2, Y2) are points that exist on the same y-z plane. In other words, the x-coordinates of coordinates C1 and C2 are the same. Furthermore, coordinates C1 and C2 are points on the distance information included in the second array information SI2. The top of the head function calculation unit 1611 calculates a first function FNC1 based on, for example, coordinates C1 and C2. The first function FNC1 may be calculated based on multiple points.
[0039] 6, the vertex function calculation unit 1611 outputs information about the calculated function as a first function FNC1 to the vertex estimation unit 1612. The first function FNC1 may include information about a plurality of functions.
[0040] The top of the head estimation unit 1612 acquires the second array information SI2 from the thinning processing unit 15, and acquires the first function FNC1 from the top of the head function calculation unit 1611. The top of the head estimation unit 1612 estimates distance information on the back of the subject S based on the first function FNC1 and the distance information included in the second array information SI2.
[0041] Turning now to FIG. 7 , a method for generating distance information on the back surface of subject S will be described. Here, the first function FNC1 is a function obtained based on coordinates C1 (Z1, Y1) and C2 (Z2, Y2) on the distance information included in the second array information SI2. Coordinates C3 (Z3, Y3) are illustrated as a point on this function. Coordinates C3 are, in other words, points on the back surface of subject S, and are information on points that cannot normally be obtained from the image capture device 20. In this way, the top of the head estimation unit 1612 estimates the three-dimensional shape of subject S, which cannot normally be obtained from the image capture device 20, based on the calculated function.
[0042] Depending on the function calculated by the parietal function calculation unit 1611, the distance information for the rear surface of the subject S may be an abnormal value. Here, if information about the identity of the subject S is known in advance, it is possible to determine whether the distance information for the rear surface of the subject S is an abnormal value, and if so, to correct it. For example, if it is known in advance that the subject S is a person's face, the range of coordinates that can realistically be taken as the rear surface of the person is limited to a predetermined range. Therefore, the coordinates of the rear surface of the subject S may be estimated so as to fall within a predetermined range of maximum and minimum values of three-dimensional coordinates. The range of maximum and minimum values of the three-dimensional coordinates may be acquired based on the class of the subject S obtained when object detection is performed by the boundary detection unit 13, for example. Returning to FIG. 6 , the parietal estimation unit 1612 outputs the estimated distance information for the rear surface of the subject S to the rear surface completion information generation unit 163 as first estimated information EI1.
[0043] The temporal complementing unit 162 includes a temporal function calculating unit 1621 and a temporal estimation unit 1622. The temporal function calculating unit 1621 acquires the second array information SI2 from the thinning processing unit 15. The temporal function calculating unit 1621 calculates a function for the temporal region. The function calculated by the temporal function calculating unit 1621 is a function that passes through the distance information of the subject S in the horizontal direction. Specifically, when the subject S is a person's face, this function indicates a change in the three-dimensional shape of the person's face in the x-z plane (see FIG. 8 ), and is a function that passes through a point on the x-z plane from the face to the hairline of the temporal region, a point on the temporal region (the boundary between the subject and the background), and a point on the occipital region. This function may be, for example, a quadratic function. The temporal function calculating unit 1621 calculates multiple functions at predetermined intervals in the vertical direction (y-axis direction). The predetermined intervals may be, for example, intervals that are sufficient to adequately represent the shape of the subject S when a three-dimensional shape is generated. Hereinafter, the processing performed by the temporal completion unit 162 may be referred to as temporal completion processing.
[0044] FIG. 8 is a diagram for explaining the temporal region completion processing according to this embodiment. An example of a function obtained by the temporal region completion processing will be described with reference to the same figure. The same figure shows three-dimensional information of subject S viewed in the y-axis direction. Coordinates C4 (Z1, X1) and C5 (Z2, X2) are points that exist on the same x-z plane. That is, the y coordinates of coordinates C4 and C5 are the same. Furthermore, coordinates C4 and C5 are points on the distance information included in the second array information SI2. The temporal function calculation unit 1621 calculates the second function FNC2 based on, for example, coordinates C4 and C5. The second function FNC2 may be calculated based on multiple points.
[0045] 6, the temporal function calculation unit 1621 outputs information about the calculated function as a second function FNC2 to the temporal estimation unit 1622. The second function FNC2 may include information about a plurality of functions.
[0046] The temporal estimation unit 1622 acquires the second array information SI2 from the thinning processing unit 15, and acquires the second function FNC2 from the temporal function calculation unit 1621. The temporal estimation unit 1622 estimates distance information on the back of the subject S based on the second function FNC2 and the distance information included in the second array information SI2.
[0047] Turning to FIG. 8 , a method for generating distance information on the rear surface of the subject S will be described. Here, the second function FNC2 is a function obtained based on coordinates C4 (Z1, X1) and C5 (Z2, X2) on the distance information included in the second array information SI2. Coordinates C6 (Z3, X3) are illustrated as a point on this function. Coordinates C6 are, in other words, points on the rear surface of the subject S, and are information on points that cannot normally be obtained from the imaging device 20. In this way, the temporal estimation unit 1622 estimates a three-dimensional shape that cannot normally be obtained from the imaging device 20 based on the calculated function.
[0048] Depending on the function calculated by the temporal function calculation unit 1621, the distance information for the rear surface of the subject S may be an abnormal value. If the identity of the subject S is known in advance, it is possible to determine whether the distance information for the rear surface of the subject S is an abnormal value, and if so, to correct it. For example, if it is known in advance that the subject S is a person's face, the range of coordinates that can realistically be taken as the rear surface of the person is limited to a predetermined range. Therefore, the coordinates of the rear surface of the subject S may be estimated so as to fall within a predetermined range of maximum and minimum values of the three-dimensional coordinates. The range of maximum and minimum values of the three-dimensional coordinates may be acquired based on the class of the subject S obtained when the boundary detection unit 13 performs object detection, etc. Returning to FIG. 6 , the temporal estimation unit 1622 outputs the estimated distance information for the rear surface of the subject S to the rear surface completion information generation unit 163 as second estimated information EI2.
[0049] The back surface completion information generation unit 163 acquires the second array information SI2 from the thinning processing unit 15, acquires the first estimated information EI1 from the vertex completion unit 161, and acquires the second estimated information EI2 from the temporal completion unit 162. Based on the acquired information, the back surface completion information generation unit 163 generates overall distance information including the front and back surfaces of the subject S. Here, the second array information SI2 includes distance information for the front surface of the subject S. Furthermore, the first estimated information EI1 and the second estimated information EI2 include estimated distance information for the back surface of the subject S. Therefore, based on this information, the back surface completion information generation unit 163 generates distance information including three-dimensional information about the front and back surfaces of the subject S. The back surface completion information generation unit 163 outputs the generated information to the back surface image information completion unit 164 as back surface completion information BCI.
[0050] FIG. 9 is a diagram showing an example of distance information after thinning processing and back surface completion processing according to this embodiment. Referring to the same figure, an example of back surface completion information BCI generated by the back surface completion information generation unit 163 is shown. As shown in the figure, the back surface completion information BCI generated by the back surface completion information generation unit 163 makes it possible to generate front and back surface distance information for the face portion of the person who is the subject S. Note that in the example shown, the distance information for the neck portion of the subject S has not been thinned. However, this embodiment is not limited to this example, and thinning processing may also be performed on the neck data, for example, using the method described with reference to FIG. 5.
[0051] Here, the back surface complement information generation unit 163 was able to generate distance information for the back surface of the subject S, but it is preferable to also perform complementation processing on the image of the back surface of the subject S. The three-dimensional information processing device 10, by including the back surface image information complement unit 164, also performs complementation processing on the image of the back surface of the subject S. Returning to FIG. 6 , the back surface image information complement unit 164 acquires image information IMG1 from the image acquisition unit 11 and acquires back surface complement information BCI from the back surface complement information generation unit 163. The back surface image information complement unit 164 complements the image information of the back surface of the subject S based on the acquired image information IMG1 of the front surface of the subject S and distance information of the back surface of the subject S. For example, the back surface image information complement unit 164 may extract color information of, for example, the hair portion based on the image information of the front surface of the subject S, and complement the image information of the back surface of the subject S using the extracted color information of the hair portion. After completing the image information of the rear surface of the subject S, the rear surface image information complementing unit 164 outputs data including the distance information and image information of the subject S to the point cloud data generating unit 21 as third array information SI3.
[0052] Currently, the resolution of image sensors is increasing, but the resolution of ToF sensors has not yet reached the same level as that of image sensors, and the ToF sensor resolution may be lower than that of the image sensor. In such cases, in order to use high-resolution image data without discarding it, it is preferable to upconvert the low-resolution ToF resolution to match the resolution of the image sensor. Hereinafter, the upconversion of ToF resolution will be described with reference to FIG. 10 .
[0053] FIG. 10 is a diagram for explaining ToF resolution upconversion according to this embodiment. Referring to the same figure, ToF resolution upconversion according to this embodiment will be described. FIG. 10(A) is a schematic diagram showing an image of distance data obtained by a low-resolution ToF sensor. FIG. 10(B) is a schematic diagram showing an image obtained by generating interpolation data using linear interpolation or the like and upconverting the original data shown in FIG. 10(A). Here, upconversion makes it possible to interpolate blank areas, but when meshing is actually performed using such information, the interpolated areas become surfaces, and there are cases where no difference is apparent between the presence and absence of interpolation processing. Therefore, according to this embodiment, rear ToF data is inserted into the space generated by upconversion. FIG. 10(C) is a schematic diagram showing an image obtained by inserting rear ToF data into the space generated by upconversion of FIG. 10(B). Using this method, it is possible to store not only rear data but also side data in the space generated by upconversion. Therefore, according to this embodiment, distance information (depth values) from multiple directions can be stored in a single distance image (depth data), thereby reducing the amount of data. Furthermore, by using this method, it is no longer necessary to down-convert the image information, and three-dimensional information can be generated while maintaining the high resolution of the image information.
[0054] Returning to FIG. 2 , the point cloud data generation unit 21 acquires third array information SI3 from the back surface completion processing unit 16. Here, the third array information SI3 is distance information of the subject S with information about the back surface completed by the back surface completion processing unit 16. Therefore, the point cloud data generation unit 21 generates point cloud data of the subject S with information about the back surface completed. That is, according to this embodiment, the arraying process, thinning process, and completion process are performed on the distance information (distance data or depth value) before the point cloud data is generated. Since processing using point cloud data generally imposes a high load, according to this embodiment, these processes are performed at the distance information stage before the point cloud data is generated. The point cloud data generation unit 21 outputs the generated point cloud data PCD to the meshing processing unit 17.
[0055] The meshing processing unit 17 acquires the point cloud data PCD from the point cloud data generating unit 21. The meshing processing unit 17 converts the point cloud data PCD into mesh data made up of a plurality of triangular faces.
[0056] FIG. 11 is a diagram showing an example of point cloud data and mesh data according to this embodiment. An example of the point cloud data and mesh data according to this embodiment will be described with reference to the same figure. FIG. 11(A) is an example of point cloud data. Point cloud data PCD generated by the point cloud data generation unit 21 is, for example, as shown in FIG. 11(A). FIG. 11(B) is an example of mesh data. The meshing processing unit 17 performs meshing processing based on the point cloud data PCD generated from the point cloud data PCD, thereby converting the point cloud data into mesh data as shown in FIG. 11(B). Note that a known algorithm may be used to convert the point cloud data into mesh data. Returning to FIG. 2, the meshing processing unit 17 outputs the converted mesh data to the material generation unit 18 as mesh information MSI.
[0057] The material generation unit 18 acquires mesh information MSI from the meshing processing unit 17. The material generation unit 18 generates three-dimensional information of the subject S based on image information IMG1 of the front of the subject S and the acquired mesh information MSI. Here, the mesh information MSI already includes three-dimensional information and image information, but according to this embodiment, the point cloud data is thinned out, resulting in a lack of image information and a low image resolution for the generated three-dimensional model. Therefore, the material generation unit 18 generates a three-dimensional model with high image resolution based on the image information IMG1 captured by the imaging device 20 and the mesh information MSI.
[0058] Specifically, first, the material generation unit 18 generates an object file (.obj file) from the point cloud data. A known algorithm may be used to generate the object file. Next, the material generation unit 18 maps the vertex coordinates of the object file to match the color image. Next, the material generation unit 18 generates material from the mapping result. The material generation unit 18 outputs the generated material to the output unit 19 as material information MTI.
[0059] The output unit 19 acquires the material information MTI from the material generation unit 18. The output unit 19 outputs the acquired material information MTI to an information processing device (not shown) or the like.
[0060] 12 is a flowchart showing an example of a series of operations performed by the three-dimensional information processing apparatus according to this embodiment. The flow of a series of three-dimensional information processing steps performed by the three-dimensional information processing apparatus 10 will be described with reference to this drawing.
[0061] First, the image acquisition unit 11 acquires image information IMG1 from the imaging device 20, and the distance information acquisition unit 12 acquires distance information IMG2 from the imaging device 20 (step S11). Next, the boundary detection unit 13 performs object detection processing based on the acquired image information IMG1. By performing object detection processing, the boundary detection unit 13 detects the boundary between the subject S and the background (step S12). Next, the array processing unit 14 extracts distance information from the area where the object is detected, thereby performing arraying (step S13). Next, the thinning processing unit 15 performs thinning processing on the extracted distance information (step S14). Next, the back surface completion processing unit 16 complements distance information on the back surface of the subject S to generate distance information for the entire subject S (step S15). Furthermore, the back surface completion processing unit 16 also performs complementing processing on image information of the back surface (step S16). Next, the point cloud data generation unit 21 performs point cloud data generation processing based on the complemented distance information and image information of the back surface. The meshing processing unit 17 performs meshing processing based on the generated point cloud data (step S17). Finally, the material generation unit 18 performs material generation processing based on the mesh data and the image information IMG1 of the subject S (step S18).
[0062] FIG. 13 is a block diagram showing an example of the internal configuration of a three-dimensional information processing apparatus 10 according to this embodiment. At least some of the functions of the three-dimensional information processing apparatus 10 can be implemented using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be implemented using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902 or the like. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices. The input / output devices 904 and 905 are input / output devices. Input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from and to RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906.
[0063] [Summary of the embodiment] According to the embodiment described above, the three-dimensional information processing device 10 includes the image acquisition unit 11 to acquire image information IMG1 obtained by capturing an image of the subject S, the distance information acquisition unit 12 to acquire distance information IMG2 indicating the three-dimensional shape of the subject S, the boundary detection unit 13 to detect the boundary between the subject S and the background based on the acquired image information IMG1, and the back surface completion processing unit 16 to calculate a function indicating a change in distance information in a predetermined direction from the acquired distance information IMG2, and to complement the distance information of the back surface of the subject S based on the calculated function and points on the detected boundary. That is, according to the present embodiment, the three-dimensional information processing device 10 can generate the three-dimensional shape of the back surface of the subject S, which would not normally be obtainable from the imaging device 20.
[0064] Furthermore, according to the above-described embodiment, the thinning processing unit 15 is further provided, which extracts feature points from the acquired image information IMG1 and performs a thinning process to reduce the amount of distance information data by thinning out distance information other than the extracted feature points. Furthermore, the back surface interpolation processing unit 16 calculates a function passing through the three-dimensional coordinates of the feature points and interpolates distance information for the back surface of the subject S into the space thinned out by the thinning processing unit 15. Here, the resolution of the distance image acquired by the ToF sensor may be lower than that of the image information IMG1. That is, according to this embodiment, even if the image is upconverted to match the resolution of the image information IMG1, the back surface distance information can be stored without reducing the resolution of the distance image. Therefore, according to this embodiment, the amount of data can be reduced.
[0065] Furthermore, according to the above-described embodiment, the back surface completion processing unit 16 estimates distance information for the back surface of the subject S so that it falls within a predetermined range of maximum and minimum values of three-dimensional coordinates. Here, when the three-dimensional information processing device 10 estimates distance information for the back surface based on a calculated function, it may erroneously estimate a shape that is different from the actual shape. However, according to this embodiment, since the maximum and minimum values are determined, it is possible to prevent the estimation of a shape that is different from the actual shape. Note that the maximum and minimum values may be determined according to the class of the subject S for which object detection has been performed, etc.
[0066] Furthermore, according to the above-described embodiment, the back surface completion processing unit 16 further includes a back surface image information completion unit 164, thereby completing image information on the back surface of the subject S based on image information on the front surface of the subject S. Therefore, according to this embodiment, not only the three-dimensional shape of the back surface of the subject S but also the image information on the back surface of the subject S can be completed.
[0067] Furthermore, according to the above-described embodiment, by further including a meshing processing unit 17, the point cloud data of the subject S, the information about the back surface of which has been complemented by the back surface complementing processing unit 16, is converted into mesh data composed of a plurality of triangular faces, and by further including a material generating unit 18, three-dimensional information of the subject S is generated based on the image information IMG1 of the front surface of the subject S and the mesh data. The three-dimensional information generated in this manner is based on the image information IMG1 captured by the imaging device 20, and therefore has high image resolution. Therefore, according to this embodiment, a three-dimensional model of the subject S with a high degree of reproduction can be generated.
[0068] In the above-described embodiment, the case where the subject S is a person's face has been described. However, the present embodiment is not limited to this example, and can also be applied to cases where the subject S is something other than a person's face. Other examples of the subject S include animals such as dogs and cats. Furthermore, the present embodiment can also be applied even when the subject S is something other than an animal. The case where the subject S is something other than an animal may be, for example, a car, a bicycle, a building, etc.
[0069] In the above-described embodiment, an example has been described in which three-dimensional information about one subject S is generated using information acquired by the imaging device 20 through a single imaging session, i.e., one piece of image information and distance information. However, this embodiment is not limited to this example, and three-dimensional information about multiple subjects S may be generated from information acquired by the imaging device 20 through a single imaging session. In this case, it is possible to generate three-dimensional information about various objects by detecting multiple objects through object detection, detecting classes of the detected objects, and using different complementary parameters (calculation formulas) for each detected class. The parameters for each class may be compiled into a database and stored in a predetermined server device, etc.
[0070] In addition, all or part of the functions of each unit of each device in the above-mentioned embodiments may be realized by recording a program for realizing these functions on a computer-readable recording medium, and reading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.
[0071] Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage units such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over networks like the Internet or communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients in such cases. Furthermore, the above-mentioned programs may be programs that realize some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system.
[0072] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention. In addition, the above-described embodiments may be combined as appropriate.
[0073] According to the present invention, even if the three-dimensional shape of the back surface of the subject cannot be acquired, the three-dimensional shape of the subject can be generated.
[0074] 1...3D information generation system, 10...3D information processing device, 20...imaging device, S...subject, SCR...screen, IMG...image information, PCD...point cloud data, 11...image acquisition unit, 12...distance information acquisition unit, 13...boundary detection unit, 14...array processing unit, 15...thinning processing unit, 16...back surface completion processing unit, 17...mesh processing unit, 18...material generation unit, 19...output unit, 151...feature point detection unit, 152...distance information extraction unit, 161...vertex completion unit, 162...temporal completion unit, 1 63...back surface completion information generation unit, 164...back surface image information completion unit, 1611...vertex function calculation unit, 1612...vertex estimation unit, 1621...temporal function calculation unit, 1622...temporal estimation unit, BDI...boundary detection information, SI1...first array information, SI2...second array information, SI3...third array information, MSI...mesh information, MTI...material information, FPI...feature point information, FNC1...first function, FNC2...second function, EI1...first estimated information, EI2...second estimated information, BCI...back surface completion information
Claims
1. A three-dimensional information processing device comprising: an image acquisition unit that acquires an image of a subject; a distance information acquisition unit that acquires distance information to the subject; a boundary detection unit that detects a boundary between the subject and a background based on the acquired image; and a back surface completion processing unit that derives a function that indicates a change in a specified direction from the acquired distance information, and complements distance information on the back surface of the subject based on the derived function and points on the detected boundary.
2. A three-dimensional information processing device as described in claim 1, further comprising a thinning processing unit that extracts feature points from the acquired image and performs a thinning process to reduce the amount of data of the distance information by thinning out distance information other than the extracted feature points, wherein the back surface completion processing unit derives a function that passes through the three-dimensional coordinates of the feature points as the function, and complements distance information on the back surface of the subject into the space thinned out by the thinning processing unit.
3. The three-dimensional information processing device according to claim 1 or 2, wherein the back surface complement processing section estimates distance information on the back surface of the subject so as to be within a range of maximum and minimum values of predetermined three-dimensional coordinates.
4. The three-dimensional information processing device according to claim 1 or 2, wherein the back surface completion processing section further comprises a back surface image information completion section that completes image information of a back surface of the subject based on image information of a front surface of the subject.
5. A three-dimensional information processing method including: an image acquisition step of acquiring an image of a subject; a distance information acquisition step of acquiring distance information to the subject; a boundary detection step of detecting a boundary between the subject and a background based on the acquired image; and a back surface completion processing step of deriving a function indicating a change in a specified direction from the acquired distance information, and completing distance information on the back surface of the subject based on the derived function and points on the detected boundary.
Citation Information
Patent Citations
Image processing system, imaging device, control method, program and recording medium
JP2020181400A
Method and system for reconstructing scene color and depth information - Patents.com
JP2021535466A
