Three-dimensional image generation system, image processing method for three-dimensional image generation, and composite image generation system

Abstract

This invention provides a three-dimensional image generation system, an image processing method for three-dimensional image generation, and a composite image generation system that generate three-dimensional images at higher speed and with higher quality when synthesizing two-dimensional images containing depth information captured by multiple depth cameras. [Solution] The three-dimensional image generation system comprises at least a plurality of depth cameras 10 and a three-dimensional image generation unit 20 that generates a three-dimensional image based on two-dimensional images having depth information captured by each depth camera 10. When the three-dimensional image generation unit 20 generates a mesh using a point cloud for a two-dimensional image captured by at least one of the plurality of depth cameras 10, it does not generate a mesh for faces that include edges where the distance between adjacent points exceeds a predetermined threshold.

Description

Technical Field

[0001] The present invention relates to a three-dimensional video generation system, an image processing method for three-dimensional video generation, and a composite video generation system for generating a three-dimensional video by synthesizing two-dimensional videos having depth information respectively captured by a plurality of depth cameras.

Background Art

[0002] As shown in the following prior art documents, a three-dimensional video generated based on a plurality of two-dimensional videos respectively captured by a plurality of cameras is also called a "volumetric video".

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In generating this three-dimensional video (volumetric video), as a camera for capturing each two-dimensional video, a depth camera capable of acquiring depth information may be used. Two-dimensional images containing depth information obtained by a depth camera (sometimes called "depth map and color information data") have the characteristic that there is no overlap in the camera's projection direction for each point cloud. Therefore, when the distance between adjacent points is large, there is a large difference in distance in the camera's projection direction. When a mesh is generated that includes the line segments connecting these points as edges, a three-dimensional image is sometimes generated that includes areas that look stretched and unattractive.

[0006] Therefore, the present invention aims to provide a means for generating three-dimensional images at higher speed and with higher quality. [Means for solving the problem]

[0007] The present invention, made to solve the above problems, comprises at least a plurality of depth cameras and a three-dimensional image generation unit that generates a three-dimensional image based on two-dimensional images having depth information captured by each depth camera, wherein the three-dimensional image generation unit, when generating a mesh using a point cloud for a two-dimensional image captured by at least one of the plurality of depth cameras, does not generate a mesh for faces that include edges where the distance between adjacent points exceeds a predetermined threshold. Furthermore, the present invention relates to an image processing method performed when generating a three-dimensional image based on a two-dimensional image having depth information captured by a depth camera using an information processing device, characterized in that when generating a mesh using a point cloud for the two-dimensional image captured by the depth camera, the mesh generation of a face that includes an edge where the distance between adjacent points exceeds a predetermined threshold is not performed. Furthermore, the present invention relates to a composite image generation system for displaying a composite image formed by combining multiple images to a user, comprising at least: a real image acquisition unit that acquires real images, which are images taken in approximately the user's line of sight, in a manner that includes depth information; a measurement unit that acquires measurement information including the user's position and orientation; a three-dimensional image management unit that manages three-dimensional images consisting of multiple two-dimensional images taken by multiple depth cameras arranged at the shooting site; a synthesis unit that generates a composite image by synthesizing a processed image, which is obtained by removing an arbitrary portion based on the depth information from the real image, and the three-dimensional image based on the measurement information; and a display unit that can be attached to the user's head and displays the composite image to the user, wherein the three-dimensional image is characterized in that, when generating a mesh using a point cloud on a two-dimensional image taken by at least one of the multiple depth cameras, mesh generation of faces including edges where the distance between adjacent points exceeds a predetermined threshold is not performed. [Effects of the Invention]

[0008] According to the present invention, it is possible to generate three-dimensional images at higher speed and with higher quality. [Brief explanation of the drawing]

[0009] [Figure 1] A schematic plan view showing an example of the construction of a three-dimensional image generation system according to Example 1. [Figure 2] An illustrative diagram of mesh generation using point clouds. [Figure 3] Three-dimensional image before the stretched surface removal process. [Figure 4] Three-dimensional image after the stretched surface removal process. [Figure 5] A three-dimensional image where the deleted area has been filled in by the other depth camera. [Figure 6] A schematic diagram showing the configuration of the composite video generation system according to Example 2. [Figure 7] A schematic diagram showing the image of the processed video generation process. [Figure 8]Schematic diagram showing the generation image of the synthetic image.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

Embodiment

[0011] <1>Overall Configuration (Fig. 1) The three-dimensional video generation system according to this embodiment is configured to include at least a plurality of depth cameras 10 and a three-dimensional video generation unit 20. Each component can be realized by arbitrarily combining hardware and software. In addition, a configuration in which each part is an individual device, a configuration in which a plurality of each part is incorporated into one device and integrated, etc. can be adopted. In the present invention, the number of each component, the location of each component, the number of videos used for generating the three-dimensional video, etc. are not limited. Hereinafter, details of each component will be described.

[0012] <2>Depth Camera (Fig. 1) The depth camera 10 is a device installed at the shooting site to acquire a two-dimensional video having depth information. In the present invention, the "two-dimensional video having depth information" acquired by the depth camera 10 is data obtained by combining information indicating the distance from the depth camera 10 to each pixel of the captured image by the depth camera 10 (which may also be called a "depth map") and color information data. In the present invention, the type and number of depth cameras 10 are not particularly limited.

[0013] <3>Three-Dimensional Video Generation Unit (Fig. 1) The three-dimensional video generation unit 20 has a function of generating a three-dimensional video 21 by combining two-dimensional videos having depth information captured by each depth camera 10. The three-dimensional image generation unit 20 can be realized by software or the like installed in an information processing apparatus such as a PC. In the present invention, a known method may be used for the basic processing method for generating the three-dimensional image 21 from a plurality of two-dimensional images, and a detailed description thereof will be omitted. Further, in the present invention, in the three-dimensional image generation unit 20, an extension plane deletion process is performed on a two-dimensional image captured by at least one of the plurality of depth cameras 10. Hereinafter, the details of the extension plane deletion process will be described.

[0014] <4> Extension plane deletion process (Figs. 2 to 4) The extension plane deletion process refers to a process in which generation of a mesh that meets a predetermined condition is deliberately not performed during mesh generation processing executed when performing a three-dimensionalization process using a point cloud of a two-dimensional image having depth information.

[0015] <4.1> Image diagram of the extension plane deletion process (Fig. 2) Fig. 2 shows an image diagram of the extension plane deletion process. As shown in Fig. 2, in the two-dimensional image having depth information captured by the depth camera 10, each point cloud is set so as not to overlap in the projection direction of the depth camera 10 (the Z-axis direction in Fig. 2). Therefore, as shown in Fig. 2, at the boundary between the wall B and the object C composed of a sphere, the distance between adjacent points in each point cloud is separated, and a large distance difference occurs in the projection direction of the depth camera 10. Therefore, in the three-dimensional image generation unit 20, the distance of a line segment connecting adjacent points in the point cloud is obtained, and for a line segment (line segment D1 to be deleted) whose line segment distance exceeds a predetermined threshold, a mesh of a surface (in the present invention, the surface shape is not particularly limited, and a triangle, a quadrilateral, etc. are conceivable) including this line segment D1 as a side is not generated, and for a surface composed only of the sides of a line segment (line segment D2 not to be deleted) whose line segment distance does not exceed a predetermined threshold, normal mesh generation processing is performed. This "predetermined threshold" may be set to an arbitrary value by the user of the three-dimensional image generation system according to the present invention.

[0016] <4.2> Illustrated diagram of a three-dimensional image (Figures 3 and 4) Next, a three-dimensional image generated by the three-dimensional image generation system according to the present invention will be described. In this invention, the display destination of the three-dimensional image is not particularly limited and can be any known display device, such as a display built into or attached to an information processing device, or an HMD (head-mounted display) worn by the user. Figures 3 and 4 are illustrative diagrams of three-dimensional images showing the presence or absence of the stretched surface removal process. Note that both Figures 3 and 4 are three-dimensional images generated using only two-dimensional images from a single depth camera 10, intended as reference to facilitate understanding of the stretched surface removal process. They show the three-dimensional image 21 being viewed from a direction different from the shooting direction of the depth camera 10 (a side view). Figure 3 is a three-dimensional image without the stretched surface removal process. The area between the worker object C and the back wall B is also rendered as if the image has been stretched, resulting in an unattractive area (degraded area E). On the other hand, in Figure 4, the degraded region E in Figure 3 has become a blank region (deleted region F) due to the stretched surface deletion process performed by the three-dimensional image generation unit 20.

[0017] <4.3>Illustration of the completed three-dimensional image (Figure 5) Figure 5 is an illustrative diagram of a three-dimensional image 21 generated by combining a two-dimensional image from one depth camera 10 (shown in Figure 4) with a two-dimensional image from another depth camera 10. In the three-dimensional image 21 shown in Figure 4, the area F is deleted because mesh generation was not performed by one of the depth cameras 10. In contrast, in the three-dimensional image 21 shown in Figure 5, the area F deleted in Figure 4 is interpolated with a mesh generated by the other depth camera 10, which is capturing images from a different direction. At this time, since the degraded region E is removed from each depth camera before the three-dimensional images obtained from each depth camera are combined, the degraded region E from one depth camera 10 and the mesh generated by the other depth camera 10 will not be displayed in overlapping view, thus enabling the acquisition of a more precise and high-quality three-dimensional image 21.

[0018] <5> summary According to the three-dimensional image generation system of the present invention, at least one of the effects described below can be obtained. (1) In the three-dimensional image generation unit 20, when generating a three-dimensional image based on the two-dimensional image acquired from the depth camera 10, mesh generation is not performed in areas with large distance differences in the projection direction of the depth camera 10, and these areas are not drawn on the three-dimensional image 21. This saves processing resources for mesh generation in areas where the information is not particularly necessary (degraded region E), resulting in the image simply appearing stretched, thus contributing to a reduction in image processing time. This is especially useful for improving the speed when generating the three-dimensional image 21 in real time. (2) When generating a three-dimensional image 21 using multiple depth cameras 10, areas where mesh generation was not performed by one depth camera 10 (deleted areas F) are filled in with meshes generated by the other depth camera 10 that captured images from a different direction. Furthermore, on the final three-dimensional image 21, the degraded areas E from one depth camera 10 and the meshes from the other depth camera do not overlap, thus enabling the creation of a high-quality three-dimensional image 21. [Examples]

[0019] [Composite Video Generation System]

[0020] The three-dimensional image 21 generated by the three-dimensional image generation system according to the present invention can also be used as source image in a system (synthetic image generation system) that displays a composite image to the user, which is formed by combining real-world images of the user's surroundings with source images, as described below.

[0021] <1> Overall structure (Figure 6) The composite image generation system shown in Figure 6 comprises at least a real-world image acquisition unit 100, a measurement unit 200, a three-dimensional image management unit 300, a synthesis unit 400, and a display unit 500. Each component can be implemented by arbitrarily combining hardware and software, and various configurations can be adopted, such as each component being a separate device, or multiple components being integrated into a single device. Furthermore, in this invention, there are no particular limitations on the number of parts, the location of each part, or the number of images used for synthesis. The details of each part are explained below.

[0022] <2> Real-world image acquisition unit (Figure 6) The real-world image acquisition unit 100 has the function of acquiring real-world images 110 in the approximate direction of the user's line of sight, in a manner that includes depth information. This "Real-World Image 110" is a video recording of the real world, including the approximate direction of the user's line of sight; in other words, it is a real-time video showing the space around the user. The real-world image acquisition unit 100 can use a camera built into a head-mounted display (hereinafter also referred to as "HMD") worn by the user, a camera attached externally to the HMD, or any other camera that can be worn by the user. Cameras that can be used as the real-world image acquisition unit 100 include so-called depth cameras or depth sensors, which have a built-in depth sensor. In this invention, the real-world image acquisition unit 100 only needs to be a camera capable of capturing images in at least the approximate direction of the user's line of sight, and does not exclude cameras capable of capturing images around the user's entire surroundings.

[0023] <3> Measurement unit (Figure 6) The measurement unit 200 has the function of acquiring user measurement information 210. This "measurement information 210" includes at least information that allows for the recognition of the user's orientation and position. The synthesis unit 400, described later, generates a synthesized image 420 based on this measurement information 210. The measurement unit 200 can use a group of sensors such as an angle sensor, an acceleration sensor, and a gyroscope, as well as a motion capture device, a GPS device, and the like. Furthermore, the measurement unit 200 includes all types, such as those installed in the living space where the user resides, those built into the HMD worn by the user, those separately attached to the user's body, and those held by the user.

[0024] <4> Three-dimensional image management unit (Figure 6) The three-dimensional image management unit 300 has the function of managing at least one three-dimensional image 310. This "three-dimensional image 310" includes a stereoscopic image consisting of still images or videos (including slow-motion videos and time-lapse videos) that include a 3D model generated based on multiple two-dimensional images obtained by photographing a subject (including anything such as a person, object, or structure) with multiple cameras. For example, the three-dimensional image 21 generated by the three-dimensional image generation system according to the present invention, as described in Example 1, can be used.

[0025] In the present invention, "management" of the three-dimensional video 310 by the three-dimensional video management unit 300 includes the act of generating a three-dimensional video 310 based on a plurality of two-dimensional videos received by the three-dimensional video management unit 300, as well as the act of receiving and saving three-dimensional videos 310 that are generated in real time (streaming video) or have already been generated by other functional units. Therefore, the three-dimensional video management unit 300 can be configured to have the functions of the three-dimensional video generation system according to the present invention, or to be configured to handle three-dimensional videos generated by the three-dimensional video generation system according to the present invention. In other words, the three-dimensional video 310 managed by the three-dimensional video management unit 300 may be a video generated by the three-dimensional video management unit 300 itself, which is equipped with the functions of the three-dimensional video generation system according to the present invention, or it may be a video generated by a three-dimensional video generation system according to the present invention, which is provided separately from the three-dimensional video management unit 300.

[0026] <5> Composite section (Figure 6) The synthesis unit 400 has the function of generating an image (processed image) in which at least an arbitrary portion based on depth information has been removed from the real image 110 (depth-based removal processing), and the function of synthesizing the processed image and the three-dimensional image 410 acquired and / or stored by the three-dimensional image management unit 300 based on the measurement information 210 acquired by the measurement unit 200. The synthesis unit 400 can be configured with hardware such as a CPU, memory, and GPU controlled by software. In the present invention, the synthesis unit 400 may further combine and synthesize other source images (such as VR images, CG images, and 360-degree images) in addition to the three-dimensional image 310 and the processed image.

[0027] <5.1> Example of removal process (Figure 6) In the present invention, the removal process performed on the real image 110 includes at least a depth-based removal process, and also includes a chroma-based removal process as necessary. "Depth-based removal processing" is a process that removes elements from a real-world image 110 containing depth information based on the depth information values. For example, it involves removing parts of the image that have depth information exceeding a predetermined distance. "Chroma removal processing" is a process that performs removal based on the values ​​of color space information on the image after depth removal processing has been performed on the real image 110. More specifically, it is a process that removes the parts corresponding to the homogeneous color of background materials placed in the user's living space that are visible within the field of view of the real image 110, and makes it possible to composite other images onto the removed parts. These processes can be found in detail in the patent documents conceived by the present applicant (Japanese Patent Publication No. 6717486, Japanese Patent Publication No. 6991494, Japanese Patent Publication No. 7157271, etc.).

[0028] <5.2> Image of processed video generation (Figure 7) Figure 7 shows an image illustrating the results of applying each removal process to the real-world image 110. The real-world image 110 shown in Figure 7(a) is an image captured by a real-world image acquisition unit 100 installed on an HMD worn by the user, and shows the user's hands and the background in the direction of the user's line of sight. By applying a depth-based removal process to this real-world image 110, for example, by removing areas with depth information of 50 cm or more, a processed image 410a in which only the user's hands are extracted can be generated, as shown in Figure 7(b). If the user's hands are sufficiently extracted in the processed image 410a after this depth-based decompression process, this processed image 410a can be used as the final processed image 410 for synthesis. For example, if you want to extract them in more detail, you can further apply chroma-based decompression to the processed image 410a to remove the background color area remaining around the contour lines of the user's hands, generating the processed image 410b shown in Figure 7(c), and this processed image 410b can be used as the final processed image 410.

[0029] <6> Display unit (Figure 6) The display unit 500 is a device equipped with the function of displaying the composite image generated by the synthesis unit 400 to the user. The display unit 500 can use a user-worn HMD, goggles, or other portable devices such as smartphones that can be housed in a housing to function as goggles.

[0030] <7> Image of the generation of a composite image (Figure 8) Figure 8 shows an example of how the composite image is generated. Figure 8(a) shows a three-dimensional image 310 from a three-dimensional image management unit 300 equipped with the functions of the three-dimensional image generation system according to the present invention, in which the entire shooting site and workers are rendered in three dimensions. Figure 8(b) shows the processed image 410 after the real image 110 acquired by the real image acquisition unit 100 has been processed by the synthesis unit 400 to remove at least arbitrary parts based on depth information (depth-based removal processing). Only the hands of the user wearing the HMD and the rod held by both hands are visible. Figure 8(c) is a composite image 420 created by combining the images shown in Figures 7(a) and 7(b) using the composite unit 400 based on user D's measurement information. In this image, the user's hands and the stick are visible in front of the workers and the work site.

[0031] <8> summary The composite image generation system according to the present invention can achieve, for example, the following effects. (1) Because the composite image includes a three-dimensional image generated from multiple two-dimensional images taken by multiple cameras, it is possible to provide users with a more realistic composite image experience. (2) In a composite image, even if there are areas in a particular field of view where the user cannot see them because they are obstructed by 3D models of people or objects included in the three-dimensional image, the user can see those areas by moving or changing the direction of their head to avoid the obstructing 3D models. (3) By moving so that the user overlaps with the 3D model in the three-dimensional image, the user can experience a view as if they were one with the 3D model. (4) Synthesized video can be used for on-site instruction, training materials for workers (distributing and saving footage of skilled workers at work), recording and managing work details, maintenance work such as visual inspections of equipment at work sites, and as reference material for tracing the cause of malfunctions by looking back at past incidents. (5) By using real-time generated streaming video for the three-dimensional image, the user can see real-time video of the shooting site, which is the source of the three-dimensional image. For example, when a construction site is used as the shooting site, and a skilled technician who is a user of the system according to this embodiment is giving instructions to assembly workers at the construction site, the skilled technician can display a composite image, which is generated from streaming video of the construction site captured in real time, on a display unit worn by the skilled technician. This allows the skilled technician, even from a distance, to perceive the construction site in three dimensions as if they were actually there, and to give instructions to the assembly workers. [Explanation of Symbols]

[0032] 10: Depth Camera 20: 3D image generation section 21: Three-dimensional images A: Filming location B:Wall C: Object D1: Line segment to be deleted D2: Line segments that will not be deleted E: Deterioration area F :Delete area 100: Real-world image acquisition unit 110: Real footage 200: Measurement Unit 210: Measurement Information 300: Three-Dimensional Video Management Department 310: Three-dimensional images 400: Synthesis part 410: Processed video 420: Composite video 500:Display section

Claims

1. Multiple depth cameras, A three-dimensional image generation unit generates a three-dimensional image based on two-dimensional images containing depth information captured by each depth camera. It must have at least the following: The three-dimensional image generation unit is The method is characterized by not generating a mesh using a point cloud for a two-dimensional image captured by at least one of the aforementioned multiple depth cameras, and not generating a mesh for faces that include edges where the distance between adjacent points exceeds a predetermined threshold. Three-dimensional image generation system.

2. An image processing method performed when generating a three-dimensional image based on a two-dimensional image containing depth information captured by a depth camera using an information processing device, The method for generating a mesh using a point cloud from the two-dimensional image captured by the depth camera is characterized by not generating a mesh for faces that include edges where the distance between adjacent points exceeds a predetermined threshold. Image processing method for generating three-dimensional images.

3. A composite video generation system for displaying a composite video, which is made by combining multiple videos, to a user, A real-world image acquisition unit acquires real-world images, which are images taken in approximately the direction of the user's line of sight, in a manner that incorporates depth information. A measurement unit that acquires measurement information including the user's position and orientation, The 3D video management unit manages the 3D video composed of multiple 2D images captured by multiple depth cameras placed at the filming location, A synthesis unit generates a composite image by combining the processed image, which is obtained by removing arbitrary parts based on the depth information from the real image, with the three-dimensional image, based on the measurement information. A display unit that can be attached to the user's head and displays the synthesized image to the user, It must have at least the following: The aforementioned three-dimensional image is The method is characterized in that, when generating a mesh using a point cloud for a two-dimensional image captured by at least one of the aforementioned multiple depth cameras, mesh generation is not performed for faces that include edges where the distance between adjacent points exceeds a predetermined threshold. A system for generating composite images.

Images