360-degree video projection mapping method, device therefor, and projection system comprising same

Abstract

Disclosed are a 360-degree video projection mapping method using a 360-degree spherical camera for matching and registering a 360-degree spherical camera image and a projector, a device therefor, and a projection system comprising same. A 360-degree video projection system comprises: at least one projector that projects an image onto a stereoscopic screen; at least one 360-degree spherical camera, positioned within a display space including the stereoscopic screen, for photographing an image projected onto the stereoscopic screen; and a projection mapping device that calculates a set of pixel correspondence points between a 360-degree image acquired from the 360-degree spherical camera and the at least one projector using a pre-arranged structured light pattern, repeats the step of calculating the set of pixel correspondence points for all projectors to generate a set of pixel correspondence points for each projector, and generates a 2D mesh optimized for geometric correction of each projector using the set of pixel correspondence points for each projector. Accordingly, an immersive display environment can be efficiently constructing by presenting a novel approach of utilizing a low-cost 360-degree spherical camera as an automatic correction device.

Description

360-degree video projection mapping method and device thereof and projection system including the same

[0001] The present invention relates to a 360-degree video projection mapping method and a device therefor and a projection system including the same, and more particularly, to a 360-degree video projection mapping method and a device therefor using a 360-degree spherical camera for matching and registering a 360-degree spherical camera image and a projector, and a projection system including the same.

[0002] Most conventional video was produced as two-dimensional images, and accordingly, conventional video projector systems were designed to project a single image onto a single screen. However, as the types of images played through projector systems have diversified in recent years, the video playback environment has also diversified.

[0003] Immersive display environments, such as dome theaters and ScreenX theaters, envelop the viewer in a immersive experience. These environments utilize multiple projectors, connected to form a single screen. To achieve a clean, high-quality multi-projection experience, the projectors' projected images must be geometrically calibrated.

[0004] However, this manual calibration is labor-intensive and time-consuming. Furthermore, automatic calibration requires the use of cameras. The wraparound nature of immersive display environments necessitates a large number of cameras, making camera installation complex, expensive, and demanding. Furthermore, when playing 360-degree videos, additional calibration is required to accurately represent the perspective.

[0005] [Patent Document]

[0006] (Patent Document 0001) Korean Patent Registration No. 10-1391380 (May 7, 2014) (Title of Invention: Image Correction System and Method for Multi-Face Screening)

[0007] (Patent Document 0002) Korean Patent Publication No. 10-2016-0031869 (March 23, 2016) (Title of invention: Stereoscopic image projection system using a single projector)

[0008] (Patent Document 0003) Korean Patent Publication No. 10-2016-0031966 (March 23, 2016) (Title of invention: Multi-projection system and projector correction method thereof)

[0009] Accordingly, the technical task of the present invention is to provide a 360-degree video projection mapping method for matching and registering a 360-degree spherical camera image and a projector by performing optimization for each projector through a pre-arranged structured light pattern in order to efficiently construct an immersive display environment.

[0010] Another object of the present invention is to provide a 360-degree video projection mapping device for performing the above-described 360-degree video projection mapping method.

[0011] Another object of the present invention is to provide a 360-degree video projection system including the above-described 360-degree video projection mapping device.

[0012] In order to achieve the above object of the present invention, a 360-degree video projection mapping method according to an embodiment includes the steps of: acquiring a 360-degree image obtained from at least one 360-degree spherical camera arranged in a display space; calculating a set of pixel correspondence points between the acquired 360-degree image and at least one projector using a pre-arranged structured light pattern; generating a set of pixel correspondence points for each projector by repeating the step of calculating the set of pixel correspondence points for all projectors; and generating a two-dimensional mesh optimized for geometric correction of each projector using the set of pixel correspondence points for each projector.

[0013] In one embodiment of the present invention, the 360-degree video projection mapping method may further include a step of removing outliers and centering corresponding points in a 360-degree image space for each projector for the generated set of corresponding points for each projector.

[0014] In one embodiment of the present invention, the 360-degree video projection mapping method may further include a step of generating an image shape correction map using the optimized two-dimensional mesh.

[0015] In one embodiment of the present invention, the 360-degree video projection mapping method may further include a step of generating a color correction map by performing brightness correction in an overlapping area where areas projected by multiple projectors overlap.

[0016] In one embodiment of the present invention, the 360-degree video projection mapping method may further include a step of playing back in real time by utilizing the generated color correction map.

[0017] In one embodiment of the present invention, the pixel corresponding points can be obtained by utilizing a Gray code pattern.

[0018] In order to achieve another object of the present invention, a 360-degree video projection mapping device according to an embodiment includes: a 360-degree spherical camera for acquiring a 360-degree image acquired from at least one 360-degree spherical camera disposed within a display space; a pixel correspondence point set calculation unit for calculating a pixel correspondence point set between the acquired 360-degree image and at least one projector using a pre-arranged structured light pattern; a correspondence point set generation unit for generating a pixel correspondence point set for each projector by repeating the step of calculating the pixel correspondence point set for all projectors; and a 2D mesh optimization unit for generating a 2D mesh optimized for geometric correction of each projector using the pixel correspondence point set for each projector.

[0019] In one embodiment of the present invention, the 360-degree video projection mapping device may further include a correspondence point outlier removal unit that removes correspondence point outliers and centers the corresponding point sets generated for each projector in a 360-degree image space.

[0020] In one embodiment of the present invention, the corresponding point outlier removal unit can remove corresponding points belonging to a small segment whose camera pixel coordinates are less than 100 Х 100 pixels.

[0021] In one embodiment of the present invention, the corresponding point outlier removal unit may calculate a 3D bounding box using spherical coordinates of a unit sphere for a set of camera pixels corresponding to the same projector pixel, and remove all corresponding points of the set when the diagonal angle of the bounding box is greater than 3 degrees.

[0022] In one embodiment of the present invention, the corresponding point outlier removal unit removes the camera pixel point of the j-th corresponding point. If the distance between the mean and the mean position is greater than the standard deviation, (Here, The corresponding point can be removed by removing the projector pixel point of the j-th corresponding point.

[0023] In one embodiment of the present invention, the 360-degree video projection mapping device may further include a shape correction map generation unit that generates an image shape correction map using the optimized two-dimensional mesh.

[0024] In one embodiment of the present invention, the 360-degree video projection mapping device may further include a color correction map generation unit that generates a color correction map by performing brightness correction in an overlapping area where areas projected by multiple projectors overlap.

[0025] In one embodiment of the present invention, the pixel corresponding points can be obtained by utilizing a Gray code pattern.

[0026] In order to achieve another object of the present invention, a 360-degree video projection system according to an embodiment includes: at least one projector for projecting an image onto a stereoscopic screen; at least one 360-degree spherical camera disposed within a display space including the stereoscopic screen and photographing an image projected onto the stereoscopic screen; and a projection mapping device for calculating a set of pixel correspondence points between a 360-degree image acquired from the 360-degree spherical camera and the at least one projector using a pre-arranged structured light pattern, repeating the step of calculating the set of pixel correspondence points for all projectors to generate a set of pixel correspondence points for each projector, and generating a two-dimensional mesh optimized for geometric correction of each projector using the set of pixel correspondence points for each projector.

[0027] According to this 360-degree video projection mapping method and device thereof and projection system including the same, a new approach is provided that utilizes a low-cost 360-degree spherical camera as an automatic correction device, projects a pre-prepared structured light pattern onto a stereoscopic screen, photographs the imaged structured light pattern using a 360-degree spherical camera, compares the projected structured light pattern with the photographed structured light pattern, and performs optimization for each projector, thereby matching and registering the 360-degree spherical camera image and the projector, thereby efficiently constructing an immersive display environment.

[0028] FIG. 1 is a block diagram illustrating a 360-degree video projection system according to one embodiment of the present invention.

[0029] Figures 2a to 2d are images for explaining the operation of the 360-degree video projection system shown in Figure 1.

[0030] FIG. 3 is a block diagram illustrating the 360-degree video projection mapping device illustrated in FIG. 1.

[0031] FIG. 4 is a flowchart illustrating a 360-degree video projection mapping method according to one embodiment of the present invention.

[0032] Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those skilled in the art can easily implement them. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, irrelevant parts have been omitted for clarity of description, and similar reference numerals have been used throughout the specification to indicate similar elements.

[0033] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element in between. Furthermore, when a part is said to "include" a component, this should be understood to mean that, unless specifically stated to the contrary, it may include other components rather than excluding them, and does not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0034] The terms “step of ~” or “step of ~” as used throughout the specification of the present invention do not mean “step for ~.”

[0035] In this specification, the term "~unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be realized by using two or more pieces of hardware, and two or more units may be realized by one piece of hardware. Meanwhile, the "~unit" is not limited to software or hardware, and the "~unit" may be configured to be in an addressable storage medium or may be configured to reproduce one or more processors. Accordingly, as an example, the "~unit" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and "~units" may be combined into a smaller number of components and "~units," or further separated into additional components and "~units." Additionally, components and '~parts' may be implemented to regenerate one or more CPUs within a device or secure multimedia card.

[0036] In this specification, some of the operations or functions described as being performed by a terminal, apparatus, or device may instead be performed by a server connected to the terminal, apparatus, or device. Similarly, some of the operations or functions described as being performed by a server may also be performed by a terminal, apparatus, or device connected to the server.

[0037] In this specification, some of the operations or functions described as mapping or matching images and projectors may be interpreted to mean mapping or matching a stereoscopic screen on which images are projected and a projector that projects the images.

[0038] In this specification, viewers can enjoy 360-degree video with a strong sense of immersion in an immersive display environment (e.g., a dome theater). Building a successful immersive environment requires two sophisticated steps. First, multiple projectors constituting the peripheral display must be closely registered on the surface to create a single seamless screen. Second, the 360-degree video must be mapped to the projection area, taking into account the display surface geometry and the sweet spot (i.e., the reference viewing position), to ensure the viewer perceives the correct perspective.

[0039] To effectively address these two issues, the present invention proposes a novel 360-degree video projection system that utilizes a consumer-grade spherical camera as a calibration device. The 360-degree video projection system first establishes correspondence points between the 360-degree spherical camera and the projector using a structured light pattern, and then filters out outliers using heuristic criteria. Next, the 360-degree video projection mapping device solves the geometric registration of the projector by formulating a simple 2D grid mesh parameterization with correspondence point constraints, assuming the camera is positioned at the sweet spot.

[0040] Then, the present invention will be described in detail with reference to the attached drawings.

[0041] FIG. 1 is a block diagram illustrating a 360-degree video projection system according to one embodiment of the present invention. FIGS. 2A to 2D are images for explaining the operation of the 360-degree video projection system illustrated in FIG. 1. In particular, FIG. 2A is an image illustrating a gray code pattern, FIG. 2B is an image illustrating initial corresponding points, FIG. 2C is an image illustrating outlier removal and center alignment, and FIG. 2D is an image illustrating grid mesh optimization.

[0042] Referring to FIGS. 1 and 2D, a 360-degree video projection system (100) according to an embodiment of the present invention includes at least one projector (110), at least one 360-degree spherical camera (120), and a 360-degree video projection mapping device (130), and before projecting an image frame onto a stereoscopic screen, projects a predetermined structured light pattern onto the stereoscopic screen, photographs the projected structured light pattern with a consumer 360-degree spherical camera (120), and compares the projected structured light pattern with the photographed structured light pattern, thereby modeling geometrical characteristics and color characteristics of the stereoscopic screen to correct distortion of the projected image frame.

[0043] The projector (110) projects an image for projector matching onto a stereoscopic screen. For example, the projector (110) may project an image having a grid pattern onto the stereoscopic screen. In the present embodiment, one or more projectors (110) may be arranged. When multiple projectors (110) are arranged, the same projectors (110) may have the same resolution or different resolutions. In the present embodiment, the image for projector matching may include a structured light pattern. The structured light may have a specific type of pattern, such as a black-and-white binary pattern, a black-and-white striped boundary pattern, or a sinusoidal striped pattern (fringe pattern).

[0044] A 360-degree spherical camera (120) is placed within a display space including a stereoscopic screen to capture an image (i.e., a structured light pattern) projected onto the stereoscopic screen and provide the captured image to a 360-degree video projection mapping device (130). In the present embodiment, the 360-degree spherical camera (120) can capture omnidirectional light of the surrounding environment, so it can be used to create 360-degree panoramic images and videos for virtual reality purposes. The 360-degree spherical camera (120) can be composed of two or more image sensors and lenses. In the present embodiment, the 360-degree spherical camera (120) can be positioned at a sweet spot. Here, the sweet spot is a reference viewing position where a viewer can experience maximum immersion. It is usually located at the center of the space, but can be moved according to the creator's intention.

[0045] A 360-degree video projection mapping device (130) calculates a set of pixel correspondence points between a 360-degree image acquired from a 360-degree spherical camera (120) and a projector (110) using a structured light pattern, repeats the step of calculating the set of pixel correspondence points for all projectors (110), generates a set of pixel correspondence points for each projector, and generates a two-dimensional mesh optimized for geometric correction of each projector (110) using the set of pixel correspondence points for each projector.

[0046] Specifically, the 360-degree video projection mapping device (130) can first set a correspondence point between each projector (110) and a 360-degree spherical camera (120). In the case of the 360-degree spherical camera (120), a conformal image having a pixel resolution of 5,376 Х 2,688 can be generated. The x-position and y-position of each pixel can be interpreted as the longitude and latitude of the sphere, respectively.

[0047] The 360-degree video projection mapping device (130) can reliably generate numerous accurate corresponding points using a structured light pattern. In the present embodiment, a complementary gray code pattern (illustrated in FIG. 2A) is used, but is not limited thereto and other structured light patterns may be used.

[0048] More specifically, the i-th projector among the plurality of projectors (110) A series of gray code patterns are projected, which are then captured by a 360-degree spherical camera (120) positioned at the sweet spot (as shown in FIG. 2a). A 360-degree video projection mapping device (130) decodes the captured images to generate a set of camera-to-projector pixel correspondence points. can be created. Here, and represent the projector and camera pixel points of the j-th corresponding point, respectively.

[0049] FIG. 3 is a block diagram for explaining the 360-degree video projection mapping device (130) illustrated in FIG. 1.

[0050] Referring to FIGS. 1 to 3, a 360-degree video projection mapping device (130) according to the present invention may include a pixel correspondence point set calculation unit (131), a correspondence point set generation unit (132), and a two-dimensional mesh optimization unit (133). In the present embodiment, the 360-degree video projection mapping device (130) is described as being composed of a pixel correspondence point set calculation unit (131), a correspondence point set generation unit (132), and a two-dimensional mesh optimization unit (133), but this is only logically distinguished for the convenience of explanation and is not distinguished in terms of hardware. That is, since each component corresponds to a logical component for realizing the technical idea of ​​the present invention, even if each component is configured to be integrated or separated, if the function performed by the logical configuration of the present invention can be realized, it should be interpreted as being within the scope of the present invention, and it should be understood that any component that performs the same or similar function should be interpreted as being within the scope of the present invention regardless of whether the name thereof is consistent.

[0051] The pixel correspondence point set calculation unit (131) calculates a pixel correspondence point set between a 360-degree image acquired from a 360-degree spherical camera (120) placed at a location within the display space and at least one projector.

[0052] The correspondence point set generation unit (132) generates a set of pixel correspondence points for each projector for all projectors (110).

[0053] The 2D mesh optimization unit (133) uses the projector-specific pixel correspondence point set generated by the correspondence point set generation unit (132) to generate an optimized 2D mesh for geometric correction of each projector (110) (as shown in FIG. 2d).

[0054] In this embodiment, computing a mesh of an appropriate shape for the image can be considered as mesh parameterization, which is mainly used to map textures to 3D surfaces and can be solved using a mesh optimization scheme. Similarly, assuming that a 360-degree spherical camera (120) is positioned at a sweet spot, the mapping problem can be expressed as a triangular grid mesh within the camera image space (i.e., spherical coordinates). can be formulated as an optimization of . More specifically, the two-dimensional mesh of the i-th projector in this embodiment is an m Х n triangular grid mesh. can be expressed as . Here, 'V' is a 2D vertex (or apex) represents a set of , and 'k' represents the index of the grid vertex. can be initially arranged at regular intervals in the projector image space. 'F' represents a set of triangular faces.

[0055] Below, the process of generating an optimized two-dimensional mesh by the two-dimensional mesh optimization unit (133) is described.

[0056] First, a set of camera-to-projector pixel correspondence points The corresponding camera coordinate set of the 2D mesh optimizer (133) can constrain the mesh during optimization. The 360-degree image generated by the 360-degree spherical camera (120) is a conformal image, showing a distorted area near the pole and discontinuity of the image boundary. Therefore, the acquired pixel correspondence points may be in such a problematic area, and the 2D mesh has a problem in that it cannot properly handle such problematic pixel correspondence points. Therefore, to prevent this, all camera pixel points are rotated to spherical coordinates as shown in FIG. 2c before optimizing the 2D mesh. Align the center of the

[0057] All camera pixel points After transforming the fields into three-dimensional rectangular coordinates on the unit sphere, the mean position is calculated. Then, using Rodrigues' rotation formula, the rotation matrix is ​​generated using two vectors pointing from the origin to the mean position and the frontal center. can induce. is originally used in the post-processing step to calculate the pixel positions in spherical coordinates.

[0058] In this embodiment, the camera pixel point of the j-th corresponding point and surrounding vertices The energy function measuring the difference between them is defined as follows: Equation (1).

[0059] [Formula 1]

[0060]

[0061] Here, is the projector pixel point of the j-th corresponding point in the projector image space. is a set of indices for vertices that contain faces containing , is the projector pixel point of the j-th corresponding point to the surrounding triangle vertex. is the center of gravity weight. Equation (1) is the camera pixel point of the j-th corresponding point in the camera image space. Surrounding vertices to match A linear combination of can be implemented.

[0062] To smooth the distribution of vertices and the deviation of each face, which is not constrained by Equation (1), we can add an energy function, such as Equation (2), based on the similarity transformation term used to measure local shape distortion.

[0063] [Formula 2]

[0064]

[0065]

[0066] Here, represents the vertices that make up the surface.

[0067] In this embodiment, local coordinates 'u' and 'v' can be calculated from the initial mesh. Equation (2) can induce each triangle face to undergo a similarity transformation.

[0068] The final optimization can be expressed as a linear least squares problem by combining equations (1) and (2), as shown in equation (3).

[0069] [Formula 3]

[0070]

[0071] Solving the sparse linear system of Equation (3) yields the i-th projector The distorted projector mesh of is generated (as shown in Fig. 2d). Grid mesh optimization can be applied sequentially for each projector.

[0072] In this embodiment, it is desirable that the optimization by the 2D mesh optimization unit (133) maps the projector mesh with a sufficient number of corresponding points. This is because if the corresponding points are not obtained evenly in the projector space, the mesh obtained by optimization often becomes unsmooth. The similarity transformation term in Equation (2) can only allow rotation and uniform scaling. Therefore, it is recommended that the shape of the free surface composed of unrestricted vertices be similar to the initial shape (i.e., a regular rectangle). As a solution to this, the raw projector mesh before optimization is Pre-warping the warped projector mesh with corresponding points as in the following formula (4) Creates.

[0073] [Formula 4]

[0074]

[0075] Here, denotes a 3Х3 homography matrix that can be transformed into perspective. Also, 'w' is a normalization number used to transform homogeneous coordinates into original coordinates. Find the n closest corresponding points within the projector image space from the location of . It was set to 5% of the total number of response points.

[0076] The direct linear transformation method is from these corresponding points. can be estimated. Finally, the surrounding vertices is a 3Х3 homography matrix and can be obtained by multiplying the homogeneous coordinates of . In optimization, the local coordinates of Equation (2) are computed from the pre-warped mesh. These spatially diverse perspective transformations can provide reasonable approximations.

[0077] Referring again to FIG. 3, the 360-degree video projection mapping device (130) according to the present invention may further include a correspondence point outlier removal unit (134) that removes and centers correspondence point outliers in the 360-degree image space for each projector for the generated set of pixel correspondence points for each projector before generating an optimized 2D mesh in the 2D mesh optimization unit (133).

[0078] In this embodiment, the following three heuristic criteria can be employed to remove outliers by the corresponding point outlier removal unit (134).

[0079] First, since the projector pixels are arranged continuously and the horizontal and vertical projection angles are greater than 10 degrees, the corresponding points whose camera pixel coordinates fall within a small segment less than a preset number of pixels (e.g., less than (100 Х 100 pixels)) are removed. The set of camera-to-projector pixel correspondence points To segment the image, a flood-fill algorithm is applied to divide the image into regions by connecting the corresponding points. Here, the flood-fill algorithm is an algorithm that expands the region centered on a specified reference value in a two-dimensional or more array. When the difference between the value of the reference pixel and the adjacent pixels is below a specified threshold, the region is expanded, and the expanded pixel is set as the reference pixel again, and this is repeated until the region does not expand any further. Therefore, in order to perform the flood-fill algorithm, a starting reference pixel must be set.

[0080] Second, although the projected area of ​​a single projector pixel can be observed from multiple camera pixels, the pixels must be close to each other. For a set of camera pixels corresponding to the same projector pixel, a 3D bounding box can be calculated using the spherical coordinates of the unit sphere. The 3D bounding box should be small because the light emitted from the projector pixel is projected near the same surface point. Therefore, if the diagonal angle of the 3D bounding box is greater than a preset angle (e.g., 3 degrees), all corresponding points in the set can be removed. For an inlier set, the many-to-one correspondence points are converted to one-to-one correspondence points by taking the average position of the camera pixels.

[0081] Third, the adjacent pixels of the projector must be projected onto adjacent surface points. The projector pixel point of the j-th corresponding point A set of camera-to-projector pixel correspondences at the location The ten closest corresponding points within the projector image space can be determined. The average position and standard deviation within the camera image space are calculated from the found corresponding points. The camera pixel point of the j-th corresponding point If the distance between the mean and the mean position is greater than the standard deviation, You can remove the corresponding point.

[0082] Meanwhile, the 360-degree video projection mapping device (130) according to the present invention may further include a shape correction map generation unit (135) that generates an image shape correction map that can be expressed in various forms, such as a displacement map, using the optimized two-dimensional mesh.

[0083] The shape correction map generation unit (135) is a raw projector mesh Wow, a twisted projector mesh Using the i-th projector, which can be used to sample the input 360-degree image and generate a geometrically aligned projection image. It is possible to generate an image shape correction map, which is a displacement map for .

[0084] Specifically, the shape correction map generation unit (135) generates the raw projector mesh , twisted projector mesh , rotation matrix A per-pixel displacement map is generated that transforms the pixel in the i-th projector space into a common global spherical image space. The displacement map is used to warp the input video frames for geometric alignment while the 360-degree video is projected in real time.

[0085] Accordingly, even if the stereoscopic screen is not completely flat, the image shape can be corrected so that geometric distortion due to the curvature of the stereoscopic screen does not occur in the projected image frame.

[0086] Meanwhile, when the areas projected by multiple projectors overlap, artifacts may occur that appear brighter than non-overlapping areas. Artifacts can seriously disrupt the sense of immersion. To eliminate such artifacts, the 360-degree video projection mapping device (130) according to the present invention may further include a color correction map generation unit (136) that performs brightness correction in the overlapping areas where the areas projected by multiple projectors overlap, thereby generating a color correction map that can be expressed in various forms, such as an alpha mask.

[0087] Specifically, the brightness compensation described above may employ an edge blending method that generates an alpha mask for each projector to achieve smooth transitions in overlapping areas. The alpha mask, which defines transparency levels (e.g., translucency, opacity, etc.), stores intensity weights per pixel to attenuate the brightness of edges. Gamma correction may be applied to the intensity weights to account for projector nonlinearity.

[0088] Accordingly, even if the stereoscopic screen has its own color, the image frame projected onto the stereoscopic screen can be color-corrected so that the color is not distorted from the actual image frame.

[0089] In addition, the 360-degree video projection mapping device (130) according to the present invention may further include a real-time playback unit (137) that renders or plays back in real time by utilizing the generated image shape correction map and color correction map. For example, the real-time playback unit (137) applies an alpha mask together with a displacement map within a fragment shader to render a projection image of the projector and plays back the rendered image in real time.

[0090] FIG. 4 is a flowchart illustrating a 360-degree video projection mapping method according to one embodiment of the present invention.

[0091] Referring to FIGS. 1 to 4, a 360-degree image is acquired from at least one 360-degree spherical camera (120) positioned at an appropriate position within a display space, and a set of pixel correspondence points between the acquired 360-degree image and at least one projector is calculated using a structured light pattern captured from the 360-degree image (step S110). The calculation of the pixel correspondence point set can be performed by the pixel correspondence point set calculation unit (131) described in FIG. 3. In the present embodiment, the projector (110) can project a structured light pattern to obtain pixel correspondence points. For example, a gray code pattern can be used as the structured light pattern, but other patterns can also be used as the structured light pattern for obtaining pixel correspondence points.

[0092] Next, the step of calculating the pixel correspondence point set at least twice for all projectors (110) is repeated to generate a pixel correspondence point set for each projector (step S120). The generation of the pixel correspondence point set described above can be performed by the correspondence point set generation unit (132) described in FIG. 3.

[0093] Next, outliers in the corresponding point are filtered out and center-aligned in the 360-degree image space for each projector (step S130). Here, the filtering and center-alignment operations for outliers in the corresponding points may be omitted. The above-described outlier filtering and center-alignment can be performed by the outlier removal unit (134) described in FIG. 3.

[0094] Next, a two-dimensional mesh optimized for geometric correction of each projector is generated using a set of pixel correspondence points for each projector (step S140). The above two-dimensional mesh optimization can be performed by the two-dimensional mesh optimization unit described in FIG. 3.

[0095] Next, an image shape correction map is generated using the optimized two-dimensional mesh (step S150). The image shape correction map described above can be expressed in various forms, such as a displacement map. Here, the displacement map is a modeling method formed based on the brightness and darkness of the image to apply curves that are difficult to model individually. While a normal map is an optical illusion technique, a displacement map is a technique that generates an actual mesh according to the brightness of a bitmap. The generation of the image shape correction map described above can be performed by the shape correction map generation unit (135) described in FIG. 3. Accordingly, even if the stereoscopic screen is not completely flat, the image shape can be corrected so that geometric distortion due to the curvature of the stereoscopic screen does not occur in the projected image frame.

[0096] Next, brightness correction is performed in the overlapping area to generate a color correction map (step S160). The generation of the above-mentioned color correction map can be performed by the color correction map generation unit (136) described in FIG. 3. Accordingly, even if the stereoscopic screen has its own color, the image frame projected onto the stereoscopic screen can be color corrected so that the color is not distorted in the actual image frame.

[0097] Next, rendering or real-time playback operations are performed using the generated image shape correction map and color correction map (step S170). The above-described rendering or real-time playback operations can be performed by the real-time playback unit (137) described in FIG. 3.

[0098] As described above, according to the present invention, a novel system, a 360-degree video projection system, which utilizes a consumer 360-degree spherical camera as a calibration device, first establishes correspondence points between the 360-degree spherical camera and the projector using a structured light pattern, then filters out outliers using a heuristic criterion, and formulates a simple 2D grid mesh parameterization with correspondence point constraints assuming that the camera is located in the sweet spot to solve the geometric registration of the projector. By presenting a novel approach that utilizes a low-cost 360-degree spherical camera as an automatic calibration device, an immersive display environment can be efficiently built. In particular, since only one low-cost 360-degree spherical camera is used, the cost is significantly reduced, and the installation and use are simple. In addition, 360-degree images can be accurately expressed in the viewpoint without additional calibration.

[0099] The above-described operations are exemplary, and embodiments may vary from the above-described operations.

[0100] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, a central processing unit (CPU), a graphics processing unit (GPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, an application specific integrated circuit (ASICS), or any other device capable of executing instructions and responding to them.

[0101] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands may include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0102] Although the above has been described with reference to embodiments, it will be understood by those skilled in the art that various modifications and changes can be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below.

[0103] [Explanation of symbols]

[0104] 100: 360-degree video projection system

[0105] 110: Projector 120: 360-degree spherical camera

[0106] 130: 360-degree video projection mapping device 131: Pixel correspondence point set calculation unit

[0107] 132: Corresponding point set generation section 133: 2D mesh optimization section

[0108] 134: Outlier removal section for corresponding points 135: Shape correction map generation section

[0109] 136: Color correction map generation unit 137: Real-time playback unit

Claims

1. A step of acquiring a 360-degree image by capturing a structured light pattern projected by at least one projector from at least one 360-degree spherical camera placed within a display space; A step of calculating a set of pixel correspondence points between the acquired 360-degree image and at least one projector using the above structured light pattern; A step of generating a set of pixel correspondence points for each projector by repeating the step of calculating the set of pixel correspondence points for all projectors; and A 360-degree video projection mapping method, comprising the step of generating an optimized two-dimensional mesh for geometric correction of each projector using the set of pixel correspondence points for each projector.

2. In paragraph 1, A 360-degree video projection mapping method, characterized in that it further includes a step of removing outliers and centering corresponding points in a 360-degree image space for each projector for the set of pixel corresponding points generated for each projector.

3. In paragraph 1, A 360-degree video projection mapping method, characterized in that it further includes a step of generating an image shape correction map using the above-mentioned optimized two-dimensional mesh.

4. In paragraph 3, A 360-degree video projection mapping method, further comprising the step of generating a color correction map by performing brightness correction in an overlapping area where areas projected by multiple projectors overlap.

5. In paragraph 4, A 360-degree video projection mapping method, characterized in that it further includes a step of playing back in real time by utilizing the generated image shape correction map and color correction map.

6. In paragraph 1, A 360-degree video projection mapping method, characterized in that the above-mentioned structured light pattern is a gray code pattern.

7. A pixel correspondence point set calculation unit that obtains a 360-degree image by capturing a structured light pattern projected by at least one projector from at least one 360-degree spherical camera placed within a display space, and calculates a set of pixel correspondence points between the obtained 360-degree image and at least one projector using the structured light pattern; A correspondence point set generation unit that generates a set of pixel correspondence points for each projector by repeating the step of calculating the set of pixel correspondence points for all projectors; and A 360-degree video projection mapping device comprising a two-dimensional mesh optimization unit that generates a two-dimensional mesh optimized for geometric correction of each projector using a set of pixel correspondence points for each projector.

8. In paragraph 7, A 360-degree video projection mapping device further comprising a correspondence point outlier removal unit for removing and centering correspondence point outliers in a 360-degree image space for each projector for the above-mentioned set of pixel correspondence points generated for each projector.

9. In the 8th paragraph, the corresponding point outlier removal unit, A 360-degree video projection mapping device characterized in that the camera pixel coordinates remove corresponding points belonging to a small segment less than a preset pixel.

10. In the 8th paragraph, the corresponding point outlier removal unit, A 360-degree video projection mapping device characterized in that, for a set of camera pixels corresponding to the same projector pixel, a 3D bounding box is calculated using spherical coordinates of a unit sphere, and all corresponding points belonging to the set are removed if the diagonal angle of the 3D bounding box is greater than a preset angle.

11. In the 8th paragraph, the corresponding point outlier removal unit, Camera pixel point of the j-th corresponding point If the distance between the mean and the mean position is greater than the standard deviation, (Here, A 360-degree video projection mapping device characterized in that the corresponding point (the projector pixel point of the j-th corresponding point) is removed.

12. In paragraph 7, A 360-degree video projection mapping device further comprising a shape correction map generation unit that generates an image shape correction map using the above-mentioned optimized two-dimensional mesh.

13. In paragraph 7, A 360-degree video projection mapping device further comprising a color correction map generation unit that generates a color correction map by performing brightness correction in an overlapping area where areas projected by multiple projectors overlap.

14. In paragraph 7, A 360-degree video projection mapping device characterized in that the above structured light pattern is a gray code pattern.

15. At least one projector projecting a structured light pattern onto a stereoscopic screen; At least one 360-degree spherical camera positioned within a display space including the stereoscopic screen and photographing a structured light pattern projected onto the stereoscopic screen; and A 360-degree video projection system characterized by including a projection mapping device that calculates a set of pixel correspondence points between the acquired 360-degree image and the at least one projector using the above-mentioned structured light pattern, repeats the step of calculating the set of pixel correspondence points for all projectors to generate a set of pixel correspondence points for each projector, and generates a two-dimensional mesh optimized for geometric correction of each projector using the set of pixel correspondence points for each projector.

Images