Three-dimensional information generating device and three-dimensional information generating method

The three-dimensional information generation device uses a single camera to combine image and distance information for efficient three-dimensional shape acquisition, addressing the resource-intensive challenges of conventional methods by synthesizing composite images and distance images for accurate and real-time data capture.

WO2024232205A1PCT designated stage expired Publication Date: 2024-11-14JVC KENWOOD CORP
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/014382
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-10
Filing Date
2024-04-09
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Conventional methods for obtaining accurate three-dimensional information of objects require multiple cameras and significant hardware resources, making it difficult to acquire dynamic three-dimensional shapes efficiently, especially when large-scale equipment and multiple viewpoints are needed.

Method used

A three-dimensional information generation device and method that uses a single camera to capture image and distance information, combining it with complementary pixel information to generate point cloud data, allowing for the synthesis of composite images and distance images to create a three-dimensional model with reduced processing requirements.

Benefits of technology

Enables the easy acquisition of three-dimensional shapes using fewer cameras, reducing hardware needs and enabling real-time data capture of moving subjects with lighter processing, while maintaining high accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024014382_14112024_PF_FP_ABST
    Figure JP2024014382_14112024_PF_FP_ABST
Patent Text Reader

Abstract

This three-dimensional information generating device comprises: a first acquiring unit that acquires first image information including image information obtained by imaging a subject and distance information corresponding to the image information; a second acquiring unit that acquires second image information including at least pixel information; an image combining unit that generates a combined image by combining the image information included in the acquired first image information and information included in the second image information; and a point cloud data generating unit that generates point cloud data on the basis of the generated combined image and the distance information included in the first image information.
Need to check novelty before this filing date? Find Prior Art

Description

Three-dimensional information generating device and three-dimensional information generating method

[0001] The present invention relates to a three-dimensional information generating device and a three-dimensional information generating method.This application claims priority to Japanese Patent Application No. 2023-077667, filed on May 10, 2023, the contents of which are incorporated herein by reference.

[0002] Conventionally, the three-dimensional shape of an object existing in the real world is acquired, and a three-dimensional model is modeled based on the acquired three-dimensional shape. In order to accurately acquire the three-dimensional shape of an object, there is a technique for acquiring three-dimensional information of the object from multiple viewpoints using multiple ranging cameras. The three-dimensional information acquired from the multiple ranging cameras is combined into one piece of three-dimensional information. By acquiring three-dimensional information of the object from multiple viewpoints using multiple ranging cameras, it is possible to acquire three-dimensional information with a higher degree of reproducibility than when three-dimensional information is acquired from one direction using a single ranging camera. For example, the technique described in Patent Document 1 can be exemplified as a technique for combining three-dimensional information acquired from multiple ranging cameras into one piece of three-dimensional information.

[0003] Japanese Unexamined Patent Publication No. 7-174538

[0004] According to the conventional techniques described above, the relative positions between images are calculated using multiple image data captured from multiple viewpoints, coordinate transformation parameters for the image data are determined, and three-dimensional information is stitched together based on the determined coordinate transformation parameters to synthesize the images into a single piece of three-dimensional information. However, obtaining three-dimensional information with a higher degree of fidelity requires the use of more cameras, and the greater the number of cameras, the more hardware resources and time are required for the synthesis process. In particular, when capturing the dynamic three-dimensional shape of a subject, it is difficult to obtain the three-dimensional shape. Furthermore, the conventional techniques described above require multiple cameras and large-scale facilities, such as studio equipment.

[0005] The present invention has been made in consideration of this situation, and aims to provide a three-dimensional information generation device and a three-dimensional information generation method that can easily acquire the three-dimensional shape of a subject using a small number of cameras.

[0006] [1] One aspect of this embodiment is a three-dimensional information generation device comprising: a first acquisition unit that acquires first image information including image information of an image of a subject and distance information corresponding to the image information; a second acquisition unit that acquires second image information including at least pixel information; an image synthesis unit that generates a synthetic image by synthesizing image information included in the acquired first image information and information included in the acquired second image information; and a point cloud data generation unit that generates point cloud data based on the generated synthetic image and the distance information included in the first image information.

[0007] [2] Furthermore, one aspect of this embodiment is that in the three-dimensional information generating device described in [1] above, the image synthesis unit generates, as the synthesized image, an image that alternately has pixel information for columns or rows in image information contained in the acquired first image information and pixel information contained in the second image information.

[0008] [3] Furthermore, one aspect of this embodiment is a three-dimensional information generating device as described in [1] or [2] above, wherein the second image information includes image information of the subject captured from a direction different from the direction in which the subject was captured in the image information included in the first image information, and distance information corresponding to the image information, and further includes a distance information synthesis unit that generates a synthetic distance image by synthesizing the distance information included in the acquired first image information and the distance information included in the second image information.

[0009] [4] Furthermore, one aspect of this embodiment is that, in the three-dimensional information generating device described in [3] above, the device further includes an installation information acquisition unit that acquires information regarding the distance between a point where a first imaging device that captures the first image information is installed and a point where a second imaging device that captures the second image information is installed, and the distance information synthesis unit generates, as the synthetic distance image, a distance image that alternately has column or row distance information in the distance information included in the acquired first image information and distance information obtained by subtracting column or row distance information in the distance information included in the second image information from the distance acquired by the installation information acquisition unit.

[0010] [5] Furthermore, one aspect of this embodiment is a three-dimensional information generation method executed by a three-dimensional information generation device, the three-dimensional information generation method including: a first acquisition step of acquiring first image information including image information of a subject and distance information corresponding to the image information; a second acquisition step of acquiring second image information including at least pixel information; an image synthesis step of generating a synthetic image by synthesizing the image information included in the acquired first image information and the information included in the acquired second image information; and a point cloud data generation step of generating point cloud data based on the generated synthetic image and the distance information included in the first image information.

[0011] According to this embodiment, the three-dimensional shape of the subject can be easily acquired using a small number of cameras.

[0012] FIG. 1 is a functional configuration diagram showing an example of the functional configuration of a three-dimensional information generation system according to the first embodiment. FIG. 2 is a functional configuration diagram showing an example of the functional configuration of a three-dimensional information generation device according to the first embodiment. FIG. 3 is a diagram for explaining the synthesis of image information according to the first embodiment. FIG. 4 is a flowchart showing an example of a series of operations of the three-dimensional information generation device according to the first embodiment. FIG. 5 is a block diagram showing an example of the internal configuration of the three-dimensional information generation device according to the first embodiment. FIG. 6 is a functional configuration diagram showing an example of the functional configuration of a three-dimensional information acquisition system according to the second embodiment. FIG. 7 is a functional configuration diagram showing an example of the functional configuration of a three-dimensional information generation device according to the second embodiment. FIG. 8 is a diagram for explaining the synthesis of image information according to the second embodiment. FIG. 9 is a diagram for explaining the offset of distance information according to the second embodiment. FIG. 10 is a diagram for explaining the synthesis of distance images according to the second embodiment. FIG. 11 is a flowchart showing an example of a series of operations of the three-dimensional information generation device according to the second embodiment.

[0013] A preferred embodiment of a three-dimensional information generating device and a three-dimensional information generating method according to the present invention will be described in detail below with reference to the accompanying drawings. Note that the embodiment described below is merely an example, and the embodiments to which the present invention is applicable are not limited to the following embodiments. Furthermore, "based on XX" in this application means "based on at least XX" and includes cases where the information is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is directly used, but also includes cases where the information is based on XX after calculation or processing. "XX" is an arbitrary element (e.g., arbitrary information). Furthermore, in the following drawings, the scale and number of elements in each structure may differ from the scale and number of elements in the actual structure to make each configuration easier to understand.

[0014] First Embodiment First, a three-dimensional information generation system 1 according to a first embodiment will be described with reference to FIGS. 1 to 5. FIG.

[0015] 1 is a functional configuration diagram showing an example of the functional configuration of a three-dimensional information generation system according to a first embodiment. An example of the functional configuration of the three-dimensional information generation system 1 will be described with reference to the diagram. In the following description, the orientation of each device in the three-dimensional information generation system 1, the positional relationships between each device, and the like will be described in terms of a three-dimensional Cartesian coordinate system of the x-axis, y-axis, and z-axis.

[0016] The three-dimensional information generation system 1 includes a three-dimensional information generation device 10 and an imaging device 20. By including the three-dimensional information generation device 10 and the imaging device 20, the three-dimensional information generation system 1 acquires three-dimensional information of a subject S and generates a three-dimensional model. The imaging device 20 captures an image of the subject S from a point a distance D away from the subject S in the z-axis direction. A screen SCR such as a blue screen may be placed behind the subject S. Note that if the three-dimensional shape of the subject S can be easily separated from the background, the screen SCR is not required.

[0017] The imaging device 20 is a distance measuring camera capable of acquiring three-dimensional information of the subject S. The imaging device 20 acquires the three-dimensional information of the subject S by two-dimensionally measuring the distance to the subject S corresponding to the captured image (or video). The three-dimensional information of the subject S acquired by the imaging device 20 may be, for example, three-dimensional point cloud data. The imaging device 20 may, for example, use a ToF (Time of Flight) method to two-dimensionally irradiate light onto the subject S and measure the distance based on the time it takes to receive the reflected light. The imaging device 20 outputs the acquired three-dimensional information of the subject S to the three-dimensional information generation device 10 as first image information IMG1.

[0018] The first image information IMG1 includes image information (e.g., an RGB image) of the subject S captured from a predetermined direction, and distance information corresponding to the image information. The distance information included in the first image information IMG1 includes multiple pieces of distance information corresponding to coordinate information in the x-y plane. The coordinate information in the x-y plane contained in the distance information corresponds to pixels contained in the image. Hereinafter, multiple pieces of distance information corresponding to coordinate information in the x-y plane may be referred to as a distance image. Note that while it is preferable for an image to have distance information for each pixel, it may also have one piece of distance information for multiple pixels. In other words, the resolution of the distance information in the x-y plane may be lower than the resolution of the image information.

[0019] In the following description, the surface of the subject S on which the imaging device 20 is present may be referred to as the front surface of the subject S, and the surface on which the screen SCR is present may be referred to as the back surface of the subject S. The front surface and back surface of the subject S are not specified by the shape of the subject S, but are specified by the positional relationship between the imaging device 20 and the subject S. Therefore, it can also be said that the first image information IMG1 includes image information of the front surface of the subject S and distance information corresponding to the image information.

[0020] The three-dimensional information generation device 10 acquires first image information IMG1 from the imaging device 20. The three-dimensional information generation device 10 generates a three-dimensional model having a three-dimensional shape of the subject S based on the acquired first image information IMG1. The three-dimensional model generated by the three-dimensional information generation device 10 may be, for example, point cloud data. Here, the three-dimensional information generation system 1 acquires information of the subject S from one direction using one imaging device 20. Therefore, the three-dimensional information generation system 1 is unable to acquire sufficient information about the rear surface of the subject S. The three-dimensional information generation device 10 complements the three-dimensional information about the rear surface of the subject S based on the information acquired from the imaging device 20 and generates a three-dimensional model. Note that this embodiment is not necessarily limited to the use of only one imaging device 20; multiple imaging devices 20 may be used.

[0021] FIG. 2 is a functional configuration diagram showing an example of the functional configuration of a three-dimensional information generation device according to the first embodiment. An example of the functional configuration of the three-dimensional information generation device 10 will be described with reference to the diagram. The three-dimensional information generation device 10 includes a first acquisition unit 11, a second acquisition unit 12, an image synthesis unit 13, a point cloud data generation unit 14, and an output unit 15. Each of these functional units is implemented using, for example, electronic circuits. Furthermore, each functional unit may include internal storage means such as a semiconductor memory or a magnetic hard disk drive, as necessary. Furthermore, each function may be implemented by a computer and software.

[0022] The first acquisition unit 11 acquires first image information IMG1 from the imaging device 20. The first image information IMG1 includes at least image information of an image of the subject S and distance information corresponding to the image information. The first acquisition unit 11 outputs the image information included in the acquired first image information IMG1 to the image synthesis unit 13 as first image information IMG1-1. The first acquisition unit 11 also outputs the distance information included in the acquired first image information IMG1 to the point cloud data generation unit 14 as first image information IMG1-2.

[0023] The second acquisition unit 12 acquires second image information IMG2 from the storage device 25. The second image information IMG2 includes pixel information for completing the rear face of the subject S. Preferably, the second image information IMG2 may also include image information for completing the rear face of the subject S. For example, if the subject S is a person's face and the front of the person is captured by the imaging device 20, the pixel information or image information for completing the rear face of the subject S may be color information of the rear face of the subject S or image information of the rear face of the subject S. The color information of the rear face of the subject S may be, for example, hair color information. In this case, the second image information IMG2 is required to include information on at least one color that specifies the hair color. In other words, the second image information IMG2 is required to include at least one piece of pixel information. The image information for completing the rear face of the subject S may also be an image pattern having at least image information of hair.

[0024] The second image information IMG2 does not need to be stored in advance in the storage device 25. For example, the second image information IMG2 may be acquired by extracting pixel information (color information) of part of the image information included in the first image information IMG1 acquired by the first acquisition unit 11. The second image information IMG2 may also be acquired by extracting a pattern of part of the image information included in the first image information IMG1 acquired by the first acquisition unit 11. The second image information IMG2 may also be acquired by performing image analysis on the first image information IMG1 acquired by the first acquisition unit 11, thereby acquiring an estimated color of the back surface of the subject S as the second image information IMG2.

[0025] The image synthesis unit 13 acquires the first image information IMG1-1 from the first acquisition unit 11, and acquires the second image information IMG2 from the second acquisition unit 12. The image synthesis unit 13 generates a synthesized image by synthesizing the image information included in the acquired first image information IMG1-1 with the pixel information or image information included in the second image information IMG2. An example of a method of image synthesis performed by the image synthesis unit 13 will be described in detail with reference to FIG. 3.

[0026] FIG. 3 is a diagram for explaining the composition of image information according to the first embodiment. An example of the image composition method performed by the image composition unit 13 will be described in detail with reference to the diagram. FIG. 3(A) is an example of image information included in the first image information IMG1-1. As shown in the diagram, image information captured from the front of subject S is shown. In the example shown in the diagram, subject S is a person's face. FIG. 3(B) is an example of pixel information included in the second image information IMG2. As shown in the diagram, an image having pixels of the color behind subject S is shown. The pixels of the color behind subject S indicate the color of subject S's hair. These pixels may be estimated from subject S. Furthermore, the image shown in FIG. 3(B) may be a pattern image (i.e., a hair pattern) representing an image of the back of subject S.

[0027] 3C shows an example of a composite image obtained as a result of image synthesis processing performed by the image synthesis unit 13. The image synthesis unit 13 first performs a cropping process on the portion of the subject S based on the image information included in the first image information IMG1-1. After the cropping process, the image synthesis unit 13 generates a composite image of the portion of the subject S, which alternates between the image information included in the first image information IMG1-1 and the pixel information included in the second image information IMG2. The image synthesis unit 13 may alternately synthesize the image information included in the first image information IMG1-1 and the pixel information included in the second image information IMG2 in the column direction or in the row direction. In the example shown in the figure, the image synthesis unit 13 alternately synthesizes the image information included in the first image information IMG1-1 and the pixel information included in the second image information IMG2 in the column direction. That is, the image synthesis unit 13 generates a synthesized image that alternates between pixel information for columns or rows in the image information included in the first image information IMG1 and pixel information included in the second image information IMG2.

[0028] The image synthesis unit 13 is not limited to the example of alternately synthesizing every column or every row, but may also synthesize every multiple columns or every multiple rows. Furthermore, the image synthesis unit 13 is not limited to the example of alternately synthesizing every column or every row, but may also synthesize every other direction. Furthermore, the image synthesis unit 13 is not limited to the example of alternately synthesizing every predetermined direction, but may also synthesize every pixel, every multiple pixels, or every pixel block including a plurality of pixels, for example.

[0029] Returning to FIG. 2, the image synthesis unit 13 outputs information about the generated synthetic image to the point cloud data generation unit 14 as a synthetic image CIMG.

[0030] The point cloud data generation unit 14 acquires first image information IMG1-2 from the first acquisition unit 11 and acquires a composite image CIMG from the image synthesis unit 13. The point cloud data generation unit 14 generates point cloud data based on the acquired information. Here, the first image information IMG1-2 acquired from the first acquisition unit 11 includes distance information on the front side of the subject S, and the composite image CIMG acquired from the image synthesis unit 13 includes image information on the front and rear sides. The point cloud data generation unit 14 generates point cloud data on the front side based on the distance information on the front side of the subject S and pixel information on the front side of the composite image CIMG. The point cloud data generation unit 14 also interpolates the distance data on the rear side of the subject S, and generates point cloud data on the rear side based on the interpolated distance information and pixel information on the rear side of the subject S in the composite image CIMG. The point cloud data generation unit 14 combines the point cloud data for the front surface and the point cloud data for the rear surface to generate point cloud data of the subject S. The point cloud data generation unit 14 outputs the generated point cloud data to the output unit 15 as point cloud data PCD.

[0031] Here, the interpolation process of the distance information of the rear surface of the subject S performed by the point cloud data generation unit 14 may be performed according to the type of object by object detection of the subject S. Furthermore, in the case of a subject in which the shape of the rear surface is not important, the distance information of the rear surface may be interpolated as a constant value so that the rear surface becomes flat.

[0032] The output unit 15 acquires the point cloud data PCD from the point cloud data generation unit 14, and outputs the acquired point cloud data PCD as a three-dimensional model to an information processing device, a storage device, a display device, or the like (not shown).

[0033] 4 is a flowchart showing an example of a series of operations of the three-dimensional information generating apparatus according to the first embodiment. The series of operations of the three-dimensional information generating process performed by the three-dimensional information generating apparatus 10 will be described with reference to the same figure.

[0034] First, the first acquisition unit 11 acquires first image information IMG1 from the imaging device 20, which includes image information and distance information of the front of the subject S (step S11). Next, the second acquisition unit 12 acquires second image information IMG2 from the storage device 25 (step S12). The image synthesis unit 13 generates a synthesized image CIMG by synthesizing image information based on the acquired first image information IMG1 and second image information IMG2 (step S13). The point cloud data generation unit 14 generates point cloud data based on the generated synthesized image CIMG and the distance image included in the first image information IMG1 (step S14).

[0035] FIG. 5 is a block diagram showing an example of the internal configuration of the 3D information generation apparatus 10 according to the first embodiment. At least some of the functions of the 3D information generation apparatus 10 can be implemented using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be implemented using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902 or the like. In accordance with these instructions, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. RAM stands for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices. The input / output devices 904 and 905 are also input / output devices. Input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from and to RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906.

[0036] Summary of First Embodiment According to the embodiment described above, the three-dimensional information generation device 10 includes a first acquisition unit 11 to acquire first image information IMG1 including image information of a subject and distance information corresponding to the image information, a second acquisition unit 12 to acquire second image information IMG2 including information on at least one pixel, an image synthesis unit 13 to generate a synthetic image CIMG by synthesizing the image information included in the acquired first image information IMG1 with the information included in the second image information IMG2, and a point cloud data generation unit 14 to generate point cloud data PCD based on the generated synthetic image CIMG and the distance information included in the first image information IMG1. In other words, according to this embodiment, the three-dimensional shape of a subject S can be easily acquired using a single camera.

[0037] Furthermore, according to this embodiment, unlike conventional technology in which point cloud data for the front and rear of the subject S are separately created and then combined, point cloud data PCD is generated by combining image information containing information on the front and rear of the subject S with distance information. Combining point cloud data requires a large amount of resources. According to this embodiment, the point cloud data PCD for the subject S can be generated with less processing than conventional technology by combining pre-combined image data and distance data, without the need to combine point cloud data. Furthermore, according to this embodiment, since the point cloud data PCD for the subject S can be generated with less processing, it is possible to acquire the three-dimensional shape of the subject S that changes in real time, even when capturing the movement of the subject S in three dimensions.

[0038] Furthermore, according to the above-described embodiment, the image composition unit 13 generates a composite image CIMG that alternates between pixel information for columns or rows in the image information included in the acquired first image information IMG1 and pixel information included in the second image information IMG2. This image is used to generate point cloud data PCD. According to this embodiment, point cloud data PCD representing the three-dimensional shape of the subject S can be easily generated based on the composite image CIMG.

[0039] Second Embodiment Next, a three-dimensional information generation system 1A according to a second embodiment will be described with reference to FIGS.

[0040] 6 is a functional configuration diagram showing an example of the functional configuration of a three-dimensional information acquisition system according to the second embodiment. An example of the functional configuration of a three-dimensional information generation system 1A will be described with reference to the same figure.

[0041] The three-dimensional information generation system 1A includes a three-dimensional information generation device 10A and a plurality of imaging devices 20. In the following description, an example in which the three-dimensional information generation system 1A includes two imaging devices 20 will be described. However, the present embodiment is not limited to this example, and the system may include three or more imaging devices 20. Note that in the example shown in the figure, a first imaging device 20-1 and a second imaging device 20-2 are specifically described as an example of the plurality of imaging devices 20. The three-dimensional information generation system 1A differs from the three-dimensional information generation system 1 in that the system includes two or more imaging devices 20. The three-dimensional information generation system 1A also differs from the three-dimensional information generation system 1 in that the system includes a three-dimensional information generation device 10A instead of the three-dimensional information generation device 10.

[0042] The three-dimensional information generation system 1A includes a three-dimensional information generation device 10A and a plurality of image capturing devices 20, thereby observing three-dimensional information of a subject S from multiple viewpoints and generating three-dimensional information of the subject S. Specifically, the three-dimensional information generation system 1A includes two image capturing devices 20 disposed in positions facing each other with the subject S in between, thereby acquiring image information and distance information for each of the front and back surfaces of the subject S.

[0043] The first imaging device 20-1 images the subject S from a point that is a first distance D1 away from the subject S in the z-axis direction. The second imaging device 20-2 images the subject S from a point that is opposite the first imaging device 20-1 and is a second distance D2 away from the subject S in the z-axis direction. The opposite position is preferably a position where the optical axis of the first imaging device 20-1 and the optical axis of the second imaging device 20-2 coincide with each other. However, this embodiment is not limited to this example, and it is sufficient that the optical axis of the first imaging device 20-1 and the optical axis of the second imaging device 20-2 substantially coincide with each other. Here, the range in which the optical axes substantially coincide with each other may be a range in which the optical axes are substantially parallel to each other and are acceptable as an error when generating a three-dimensional shape.

[0044] In consideration of the calculation described below, it is preferable that the first distance D1 and the second distance D2 are the same. However, this embodiment is not limited to this example, and the first distance D1 and the second distance D2 may be different distances. The three-dimensional information generation device 10A is assumed to store information regarding the first distance D1 and the second distance D2 in advance.

[0045] In the following description, the surface of the subject S on which the first imaging device 20-1 is present may be referred to as the front surface of the subject S, and the surface on which the second imaging device 20-2 is present may be referred to as the back surface of the subject S. The front and back surfaces of the subject S are not identified by the shape of the subject S, but are identified by the positional relationship between the subject S and the first imaging device 20-1 and second imaging device 20-2. Therefore, it can be said that the first image information IMG1 includes image information of the front surface of the subject S and distance information corresponding to that image information, and the second image information IMG2 includes image information of the back surface of the subject S and distance information corresponding to that image information.

[0046] The first imaging device 20-1 outputs three-dimensional information of the front side of the subject S to the three-dimensional information generation device 10A as first image information IMG1. The second imaging device 20-2 outputs three-dimensional information of the back side of the subject S to the three-dimensional information generation device 10A as second image information IMG2. The three-dimensional information generation device 10A may estimate the first distance D1 and the second distance D2 based on the images captured by the first imaging device 20-1 and the second imaging device 20-2, respectively.

[0047] The three-dimensional information generation device 10A acquires first image information IMG1 from the first imaging device 20-1 and second image information IMG2 from the second imaging device 20-2. The three-dimensional information generation device 10A generates a three-dimensional model having a three-dimensional shape of the subject S based on the acquired first image information IMG1 and second image information IMG2. The three-dimensional model generated by the three-dimensional information generation device 10A may be, for example, point cloud data. Here, the three-dimensional information generation system 1A acquires information of the subject S from two different directions using two imaging devices 20. Therefore, the three-dimensional information generation system 1A can acquire information on the front and rear of the subject S. The three-dimensional information generation device 10A generates a three-dimensional model based on the acquired information on the front and rear of the subject S. Note that in this embodiment, when three or more imaging devices 20 are used, the three-dimensional shapes of the side, top, and bottom surfaces of the subject S may also be acquired.

[0048] FIG. 7 is a functional configuration diagram showing an example of the functional configuration of a three-dimensional information generation device according to the second embodiment. An example of the functional configuration of a three-dimensional information generation device 10A will be described with reference to the same diagram. The three-dimensional information generation device 10A differs from the three-dimensional information generation device 10 in that it includes a second acquisition unit 12A instead of the second acquisition unit 12, an image synthesis unit 13A instead of the image synthesis unit 13, and a point cloud data generation unit 14A instead of the point cloud data generation unit 14. The three-dimensional information generation device 10A also differs from the three-dimensional information generation device 10 in that it further includes a distance information offset unit 16, an installation information acquisition unit 17, and a distance information synthesis unit 18. In the description of the three-dimensional information generation device 10A, components similar to those of the three-dimensional information generation device 10 are denoted by the same reference numerals, and description thereof may be omitted.

[0049] The first acquisition unit 11 acquires first image information IMG1 from the first imaging device 20-1. The first image information IMG1 includes at least image information obtained by capturing an image of the subject S from the front and distance information corresponding to the image information. The first acquisition unit 11 outputs the image information included in the acquired first image information IMG1 to the image synthesis unit 13A as first image information IMG1-1. The first acquisition unit 11 also outputs the distance information included in the acquired first image information IMG1 to the distance information synthesis unit 18 as first image information IMG1-2.

[0050] The second acquisition unit 12A acquires second image information IMG2 from the second imaging device 20-2. The second image information IMG2 includes at least image information obtained by capturing an image of the subject S from behind and distance information corresponding to the image information. The second acquisition unit 12A outputs the image information included in the acquired second image information IMG2 to the image synthesis unit 13A as second image information IMG2-1. The second acquisition unit 12A also outputs the distance information included in the acquired second image information IMG2 to the distance information offset unit 16 as second image information IMG2-2.

[0051] The image synthesis unit 13A acquires the first image information IMG1-1 from the first acquisition unit 11 and acquires the second image information IMG2-1 from the second acquisition unit 12A. The image synthesis unit 13A generates a synthesized image by synthesizing the image information contained in the acquired first image information IMG1-1 and the image information contained in the second image information IMG2-1. An example of a method of image synthesis performed by the image synthesis unit 13A will be described in detail with reference to FIG. 8.

[0052] FIG. 8 is a diagram for explaining the composition of image information according to the second embodiment. An example of a method of image composition performed by the image composition unit 13A will be described in detail with reference to the diagram. FIG. 8(A) is an example of image information included in the first image information IMG1-1. As shown in the diagram, image information obtained by capturing an image of a subject S from the front is shown. In the example shown in the diagram, the subject S is the face of a person. FIG. 8(B) is an example of image information included in the second image information IMG2-1. As shown in the diagram, image information obtained by capturing an image of a subject S from the back is shown.

[0053] Here, the image synthesis unit 13A first performs left-right flipping of the image information contained in the second image information IMG2-1. FIG. 8C shows an example of the second image information IMG2-1 after left-right flipping has been performed by the image synthesis unit 13A. In the image synthesis process, the image synthesis unit 13A can easily perform image synthesis by first flipping the rear image left-right. Note that left-right flipping is not essential in this embodiment; it is also possible to directly perform image synthesis using the image before left-right flipping by flipping the pixels extracted from the image before left-right flipping. In this case, FIG. 8B can be considered an auxiliary diagram for simplifying the explanation.

[0054] 8(D) is an example of a composite image obtained as a result of image synthesis processing performed by the image synthesis unit 13A. The image synthesis unit 13A first performs a trimming process on the front portion of the subject S based on the image information included in the first image information IMG1-1. The image synthesis unit 13A then performs a trimming process on the back portion of the subject S based on the image information included in the second image information IMG2-1. The trimming process on the back portion may be performed on a similar portion based on the result of the trimming process on the front portion.

[0055] After the trimming process, the image synthesis unit 13A generates a composite image of the portion of the subject S, which alternates between the image information included in the first image information IMG1-1 and the image information included in the second image information IMG2-1. The image synthesis unit 13A may alternately synthesize the image information included in the first image information IMG1-1 and the image information included in the second image information IMG2-1 in the column direction or in the row direction. In the example shown in the figure, the image synthesis unit 13A alternately synthesizes the image information included in the first image information IMG1-1 and the image information included in the second image information IMG2-1 in the column direction. In other words, the image synthesis unit 13A generates a composite image of the portion of the subject S, which alternates between pixel information for the columns or rows of the image information included in the first image information IMG1 and the image information included in the second image information IMG2.

[0056] The image synthesis unit 13A is not limited to the example of alternately synthesizing every column or every row, but may also synthesize every multiple columns or every multiple rows. Furthermore, the image synthesis unit 13A is not limited to the example of alternately synthesizing every column or every row, but may also synthesize every other direction. Furthermore, the image synthesis unit 13A is not limited to the example of alternately synthesizing every predetermined direction, but may also synthesize every pixel, every multiple pixels, or every pixel block including a plurality of pixels, for example.

[0057] Returning to FIG. 7, the image composition unit 13A outputs information about the generated composite image to the point cloud data generation unit 14A as a composite image CIMG.

[0058] The installation information acquisition unit 17 acquires information about the distance between the location where the first imaging device 20-1 that captures the first image information IMG1 is installed and the location where the second imaging device 20-2 that captures the second image information IMG2 is installed. This distance is the sum of the first distance D1 and the second distance D2. The installation information acquisition unit 17 may acquire the first distance D1 and the second distance D2 separately and independently, or may acquire them as a single piece of distance information.

[0059] Distance information offset unit 16 acquires second image information IMG2-2 from second acquisition unit 12A. Distance information offset unit 16 also acquires information about the distance between the location where first imaging device 20-1 capturing first image information IMG1 is installed and the location where second imaging device 20-2 capturing second image information IMG2 is installed, i.e., information about the sum of first distance D1 and second distance D2, from installation information acquisition unit 17. Distance information offset unit 16 converts the distance information acquired by second imaging device 20-2, i.e., distance information from second imaging device 20-2 to subject S, into distance information from the position of first imaging device 20-1 to subject S. Hereinafter, the conversion process of distance information performed by distance information offset unit 16 may be referred to as offset processing.

[0060] 9 is a diagram illustrating offsetting of distance information according to the second embodiment. The offset processing performed by the distance information offset unit 16 will be described with reference to the diagram. The diagram illustrates the positional relationship between the first imaging device 20-1, the second imaging device 20-2, and the subject S. The distance from the first imaging device 20-1 to the center position of the subject S is a first distance D1, and the distance from the second imaging device 20-2 to the center position of the subject S is a second distance D2.

[0061] Here, the distance from the second imaging device 20-2 to the subject S is distance D21. The distance information offset unit 16 converts distance D21 into distance D22, assuming that the image was captured from a fictitious second imaging device 20-2' that exists on the same z-axis as the first imaging device 20-1. The conversion formula from distance D21 to distance D22 is given by the following formula (1).

[0062] Distance D22 = (second distance D2 - distance D21) + first distance D1... (1)

[0063] Distance information offset unit 16 performs offset processing based on equation (1) for each coordinate on the zy plane. Returning to Figure 7, distance information offset unit 16 outputs information obtained as a result of performing the offset processing to distance information synthesis unit 18 as second image information IMG2-3.

[0064] The distance information synthesis unit 18 acquires the first image information IMG1-2 from the first acquisition unit 11, and acquires the second image information IMG2-3 from the distance information offset unit 16. The distance information synthesis unit 18 generates a synthesized distance image CDI by synthesizing the distance information contained in the acquired first image information IMG1-2 and the distance information contained in the second image information IMG2-3.

[0065] FIG. 10 is a diagram illustrating the synthesis of distance images according to the second embodiment. An example of a method for synthesizing distance information performed by the distance information synthesis unit 18 will be described in detail with reference to the diagram. FIG. 10(A) is an example of distance information included in first image information IMG1-2. As shown in the diagram, information about a distance image captured from the front of subject S is shown. In the example shown in the diagram, subject S is the face of a person. That is, first image information IMG1-2 includes information about a distance image captured in front of subject S, which is the face of a person. FIG. 10(B) is an example of distance information included in second image information IMG2-3. As shown in the diagram, information about a distance image captured from behind subject S is shown. In the example shown in the diagram, subject S is the face of a person, so second image information IMG2-3 includes information about a distance image captured behind subject S, which is the face of a person.

[0066] Here, the distance information synthesis unit 18 first performs left-right flipping of the distance image included in the second image information IMG2-3. FIG. 10C shows an example of the second image information IMG2-3 after left-right flipping has been performed by the distance information synthesis unit 18. In the distance information synthesis process, the distance information synthesis unit 18 first flips the rear distance image left-right, thereby easily performing the distance image synthesis process. Note that left-right flipping is not essential in this embodiment; it is also possible to directly perform the distance image synthesis process using the distance image before left-right flipping by flipping the distance information extracted from the distance image before left-right flipping. In this case, FIG. 8B can be considered an auxiliary diagram for simplifying the explanation.

[0067] FIG. 10D shows an example of a composite distance image obtained as a result of distance information synthesis processing performed by the distance information synthesis unit 18. The distance information synthesis unit 18 first trims the front portion of the subject S based on the distance image included in the first image information IMG1-2. The distance information synthesis unit 18 then trims the back portion of the subject S based on the distance image included in the second image information IMG2-3. The trimming of the back portion may be performed by trimming a similar portion based on the results of the trimming of the front portion. After the trimming processing, the distance information synthesis unit 18 generates a composite distance image that alternates between the distance information included in the first image information IMG1-2 and the distance information included in the second image information IMG2-3. The distance information synthesis unit 18 may alternately combine the distance information included in the first image information IMG1-2 and the distance information included in the second image information IMG2-3 in the column direction or the row direction. In the illustrated example, the distance information composition unit 18 alternately combines the distance information included in the first image information IMG1-2 and the distance information included in the second image information IMG2-3 in the column direction. That is, the image composition unit 13A generates a composite image that alternates between distance information for columns or rows in the distance information included in the first image information IMG1 and distance information included in the second image information IMG2. Here, it is preferable that the direction of composition performed by the distance information composition unit 18 (i.e., alternate composition in the row direction, etc.) be the same as the direction of composition performed by the image composition unit 13A.

[0068] Returning to FIG. 7, the distance information synthesis unit 18 outputs information about the synthesized distance image to the point cloud data generation unit 14A as a synthesized distance image CDI.

[0069] The point cloud data generation unit 14A acquires the composite image CIMG from the image synthesis unit 13A and acquires the composite distance image CDI from the distance information synthesis unit 18. The point cloud data generation unit 14A generates point cloud data based on the acquired information. The composite image CIMG acquired from the image synthesis unit 13A includes image information on the front and back sides, and the composite distance image CDI acquired from the distance information synthesis unit 18 includes distance information on the front and back sides. Therefore, the point cloud data generation unit 14A can generate point cloud data having the entire three-dimensional shape of the subject S based on the composite image CIMG and the composite distance image CDI. The point cloud data generation unit 14A outputs the generated point cloud data to the output unit 15 as point cloud data PCD.

[0070] 11 is a block diagram showing an example of a series of operations of the 3D information generation device according to the second embodiment. With reference to this diagram, the series of operations of the 3D information generation process performed by the 3D information generation device 10A will be described.

[0071] First, the first acquisition unit 11 acquires, from the first imaging device 20-1, first image information IMG1, which includes image information and distance information of the front side of the subject S (step S21). Next, the second acquisition unit 12A acquires, from the second imaging device 20-2, second image information IMG2, which includes image information and distance information of the back side of the subject S (step S22). The image synthesis unit 13A generates a synthesized image CIMG by synthesizing the image information based on the acquired first image information IMG1 and second image information IMG2 (step S23). The distance information synthesis unit 18 generates a synthesized distance image CDI by synthesizing the distance information based on the acquired first image information IMG1 and second image information IMG2 (step S24). The point cloud data generation unit 14A generates point cloud data based on the generated synthesized image CIMG and synthesized distance image CDI (step S25).

[0072] Summary of the Second Embodiment According to the embodiment described above, the second image information IMG2 acquired by the second acquisition unit 12A includes image information of the subject S captured from a different direction (e.g., from the rear) than the image information included in the first image information IMG1, and distance information corresponding to the image information. Furthermore, the 3D information generation device 10A further includes a distance information synthesis unit 18, which synthesizes the distance information included in the acquired first image information IMG1 and the distance information included in the second image information IMG2 to generate a composite distance image CDI. Furthermore, the point cloud data generation unit 14A generates point cloud data PCD based on the generated composite image CIMG and composite distance image CDI. In other words, according to this embodiment, the three-dimensional shape of the rear surface of the subject S can also be acquired. Therefore, according to this embodiment, the three-dimensional shape of the subject S can be acquired with high accuracy.

[0073] Furthermore, according to this embodiment, unlike conventional technology in which point cloud data for the front and back of the subject S are separately created and then the point cloud data are combined, point cloud data PCD is generated by combining image information having information on the front and back of the subject S with distance information. Therefore, according to this embodiment, point cloud data PCD for the subject S can be generated with light processing. Furthermore, according to this embodiment, because point cloud data PCD for the subject S can be generated with light processing, even in cases where it is desired to obtain the movement of the subject S in three dimensions, the three-dimensional shape of the subject S that changes in real time can be obtained.

[0074] Furthermore, according to the above-described embodiment, the installation information acquisition unit 17 is provided to acquire information regarding the distance between the location where the first imaging device 20-1 that captures the first image information IMG1 is installed and the location where the second imaging device 20-2 that captures the second image information IMG2 is installed. Furthermore, the distance information synthesis unit 18 generates a synthetic distance image CDI, which alternately contains distance information in the column or row direction in the distance information included in the acquired first image information IMG1 and distance information obtained by subtracting the distance information in the column or row direction in the distance information included in the second image information IMG2 from the distance acquired by the installation information acquisition unit 17. That is, according to this embodiment, distance information acquired from the back of the subject S is converted into distance information from the front of the subject S, and a synthetic distance image CDI is generated that contains distance information for the front and back of the subject S as viewed from the front of the subject S. The three-dimensional information generation device 10A generates point cloud data based on a composite image CIMG having image information on the front and back surfaces of the subject S, and a composite distance image CDI having distance information on the front and back surfaces of the subject S. Therefore, according to this embodiment, point cloud data can be generated easily with light processing.

[0075] Furthermore, according to the above-described embodiment, the image synthesis unit 13A generates a synthesized image CIMG by synthesizing the image information contained in the acquired first image information IMG1 with image information obtained by left-right flipping the image information contained in the second image information IMG2, and the distance information synthesis unit 18 generates a synthesized distance image CDI by synthesizing the distance information contained in the acquired first image information IMG1 with image information obtained by left-right flipping the distance information contained in the second image information IMG2. Therefore, according to this embodiment, the image information for the front and back sides can be easily synthesized with the synthesized distance information.

[0076] Furthermore, according to the above-described embodiment, the three-dimensional information generation system 1A is equipped with a first imaging device 20-1 to capture first image information IMG1 including image information of the subject S captured from a first direction and distance information corresponding to the image information, and is equipped with a second imaging device 20-2 to capture second image information IMG2 including image information of the subject S captured from a second direction and distance information corresponding to the image information, and is equipped with a three-dimensional information generation device 10A to generate three-dimensional information of the subject S based on the first image information IMG1 and the second image information IMG2. The 3D information generation device 10A also includes a first acquisition unit 11 to acquire first image information IMG1, a second acquisition unit 12A to acquire second image information IMG2, an image synthesis unit 13A to generate a composite image CIMG based on the image information contained in the acquired first image information IMG1 and the image information contained in the second image information IMG2, a distance information synthesis unit 18 to generate a composite distance image CDI based on the distance information contained in the acquired first image information IMG1 and the distance information contained in the second image information IMG2, and a point cloud data generation unit 14A to generate point cloud data PCD based on the generated composite image CIMG and composite distance image CDI. The first and second image capture devices 20-1 and 20-2 are installed in positions facing each other. That is, according to this embodiment, the three-dimensional shape of the subject S is acquired using the two opposing image capture devices 20. Therefore, according to this embodiment, it is possible to perform accurate measurement of three-dimensional shapes in real time using a small number of imaging devices 20.

[0077] In addition, all or part of the functions of each unit of each device in the above-mentioned embodiments may be realized by recording a program for realizing these functions on a computer-readable recording medium, and reading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.

[0078] Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage units such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over networks like the Internet or communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients in such cases. Furthermore, the above-mentioned programs may be programs that realize some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system.

[0079] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention. In addition, the above-described embodiments may be combined as appropriate.

[0080] According to the present invention, the three-dimensional shape of a subject can be easily acquired using a small number of cameras.

[0081] 1, 1A... Three-dimensional information generation system, 10, 10A... Three-dimensional information generation device, 20... Imaging device, S... Subject, SCR... Screen, IMG1... First image information, 11... First acquisition unit, 12... Second acquisition unit, 13... Image synthesis unit, 14... Point cloud data generation unit, 15... Output unit, 25... Storage device, 20... Imaging device, CIMG... Synthetic image, PCD... Point cloud data, IMG2... Second image information, 16... Distance information offset unit, 17... Installation information acquisition unit, 18... Distance information synthesis unit, CDI... Synthetic distance image, DII... Installation information

Claims

1. A three-dimensional information generation device comprising: a first acquisition unit that acquires first image information including image information of a subject and distance information corresponding to the image information; a second acquisition unit that acquires second image information including at least pixel information; an image synthesis unit that generates a synthetic image by synthesizing image information included in the acquired first image information and information included in the acquired second image information; and a point cloud data generation unit that generates point cloud data based on the generated synthetic image and the distance information included in the first image information.

2. The three-dimensional information generating device according to claim 1, wherein the image synthesis unit generates, as the synthetic image, an image that alternates between pixel information for columns or rows in image information contained in the acquired first image information and pixel information contained in the second image information.

3. The three-dimensional information generating device of claim 1 or claim 2, further comprising a distance information synthesis unit that generates a synthetic distance image by synthesizing the distance information included in the acquired first image information and the distance information included in the second image information, the second image information including image information in which the subject is captured from a direction different from the direction in which the subject was captured in the image information included in the first image information, and distance information corresponding to the image information.

4. A three-dimensional information generating device as described in claim 3, further comprising an installation information acquisition unit that acquires information regarding the distance between a point where a first imaging device that captures the first image information is installed and a point where a second imaging device that captures the second image information is installed, and the distance information synthesis unit generates, as the synthetic distance image, a distance image that alternately has distance information in the column or row direction in the distance information included in the acquired first image information and distance information obtained by subtracting the distance information in the column or row direction in the distance information included in the second image information from the distance acquired by the installation information acquisition unit.

5. A three-dimensional information generating method executed by a three-dimensional information generating device, comprising: a first acquisition step of acquiring first image information including image information of a subject and distance information corresponding to the image information; a second acquisition step of acquiring second image information including at least pixel information; an image synthesis step of generating a synthetic image by synthesizing image information included in the acquired first image information and information included in the acquired second image information; and a point cloud data generation step of generating point cloud data based on the generated synthetic image and the distance information included in the first image information.

Citation Information

Patent Citations

  • Communication terminal device, communication method, and software program

    WO2021220494A1