Apparatus and method for generating 360-degree panoramic images

Abstract

The present invention provides an apparatus and method for achieving consistency in 360-degree panoramic images. [Solution] The 360-degree panoramic image generation device 100 includes: a depth data processing unit that processes depth data of multiple images acquired from a single camera stream via a camera, utilizes color images to process resolution-related sampling, and constructs a depth map based on a deep learning model; a panoramic processing unit that splatters the images into a panoramic image space based on the depth map, derives GCC parameters related to the correction of the color values ​​of the splatted images and residual pose parameters related to the amount of pose change, splatters the images to which the residual pose parameters have been applied back into the panoramic image space, stitches them together, performs local color correction on overlapping areas, and performs seam blending to generate a panoramic image.

Description

[Technical Field]

[0001] [Cross-reference with related applications] This application claims priority under Korean Patent Application No. 10-2025-0019373 dated 14 February 2025, and all content disclosed in the said Korean Patent Application is incorporated herein by reference.

[0002] The present invention relates to a device and method for generating 360-degree panoramic images, and more specifically, to a technique for achieving consistency in 360-degree panoramic images through splatting of three-dimensional images using depth and optimization of camera pose. [Background technology]

[0003] Generally, to align images, a method is used in which feature point matching is performed on images obtained from two or more cameras, using epipolar lines as a reference, and then geometric alignment is applied.

[0004] However, conventional image matching methods that are not based on deep learning have the problem of relying on feature point matching and energy optimization (energy minimization) at every matching stage. Feature point matching methods are not as robust as deep learning-based feature map extraction methods, and failure in feature point matching can lead to the failure of the entire algorithm.

[0005] Furthermore, conventional image matching methods require estimating local homography for each image, estimating the optimal epipolar lines, and optimizing the energy function in calculations for 2D transformation selection. This significantly increases the computational complexity of the algorithm, making it difficult to match high-resolution panoramic images in real time.

[0006] Korean Published Patent No. 10-2023-0043668, presented as prior art, proposes a configuration that increases the success rate of image matching even under adverse conditions such as image brightness, contrast, color changes, low image quality, and limited overlapping areas by extracting features using a deep learning network.

[0007] However, this conventional technology has the problem of requiring multiple cameras because it processes images obtained simultaneously from multiple cameras.

[0008] Furthermore, conventional techniques have the limitation that distortion problems occur in the generated panoramic images because they achieve only visual consistency by distorting the image in a two-dimensional domain. [Overview of the project] [Problems that the invention aims to solve]

[0009] The present invention aims to provide a 360-degree panoramic image generation apparatus and method that achieves the consistency of a 360-degree panoramic image through splatting of a 3D image using depth and optimization of the camera pose.

[0010] The present invention aims to generate a panoramic image with physical accuracy that achieves the integrity of a 360-degree panoramic image while simultaneously enabling the measurement of spatial dimensions or the realization of experiences such as augmented reality (AR).

[0011] The present invention aims to provide a 360-degree panoramic image generation apparatus and method that generates 360-degree panoramic images through shooting with a mobile phone, which is a general user terminal, without the need for special equipment such as a 360-degree camera.

[0012] The present invention aims to achieve the realization of a natural image in the panoramic region by deleting some of the internal components and then inpainting them, in the implementation of an application program for generating 360-degree panoramic images. [Means for solving the problem]

[0013] According to one embodiment of the present invention, a depth data processing unit performs sampling related to resolution using color images on depth data of multiple images acquired from a single camera stream via a camera, and constructs a depth map based on a deep learning model; a virtual panoramic viewpoint is determined considering the different camera poses of the images, the images are splatted into a panoramic image space based on the respective depth maps of the images, global color correction (GCC) parameters related to the correction of the color values ​​of the splatted images, and residual pose parameters related to the amount of pose change are derived, and an error correction processing unit calculates at least one loss function for the splatted images to correct the GCC parameters and residual pose parameters; the corrected GCC parameters and corrected residual pose parameters are applied to the image again, the image is splatted into the panoramic image space again, the splatted image is stitched together, local color correction is performed on overlapping regions in the stitched image, and seam blending is performed. It may include a panoramic processing unit that generates a panoramic image by performing blending.

[0014] The error correction processing unit back-projects each pixel of the image into a 3D point based on the depth map, then converts the 3D point from the image coordinate system to the world coordinate system, from the world coordinate system to the panoramic coordinate system, projects it into the panoramic image space to calculate coordinates, maps the pixel to a panoramic pixel, and can splat the image into the panoramic image space based on the calculated pixel mapping.

[0015] The error correction processing unit can calculate at least one loss function among photometric consistency loss, gradient consistency loss, depth consistency loss, and feature anchoring loss related to the GCC parameters for the splatted image, and calculate at least one loss function among rotation regularizer loss, translation regularizer loss, and GCC regularizer loss related to the remaining pose parameters.

[0016] The error correction processing unit calculates a total loss based on the sum of the at least one loss function, sets weights for each of the at least one loss function so that the calculated total loss is minimized, and can correct the GCC parameters and the remaining pose parameters.

[0017] The error correction processing unit calculates the average color value for all pixels where the splatted images overlap on the panoramic image space, and then calculates the photometric consistency loss based on the sum of losses with individual color values. After deriving the gradient image for the image and splatting the gradient image, it calculates the average gradient value for all pixels that overlap with each other, and then calculates the gradient consistency loss based on the sum of losses with individual gradient values. After calculating the average depth value using the reciprocal of the distance calculated when projecting the back-projected 3D points onto the panoramic image space, it calculates the depth consistency loss based on the sum of losses with individual values, and can calculate the feature anchoring loss based on feature extraction and matching.

[0018] The error correction processing unit can calculate the rotation normalization loss based on the sum of losses between the rotation matrix of each residual pose of the image and the identity matrix, calculate the translation normalization loss based on the sum of losses between the translation vector of each residual pose of the image and the zero vector, and calculate the GCC normalization loss based on the sum of losses between each color correction matrix of the image and the truncated identity matrix.

[0019] The panoramic processing unit derives pixel values for the splatted image, measures the depth of 3D points with respect to the derived pixel values, and based on the measured depth, selects the image used for the generated panoramic image from the splatted images, and can perform the local color correction that corrects the color difference for overlapping regions in the selected image.

[0020] The panoramic processing unit can perform seam blending, which corrects color differences in the boundary portions within the generated panoramic image, by correcting the color corrected after the local color correction using residual color related to the gradient of the image.

[0021] According to one embodiment of the present invention, the application processing unit may further include a unit that selects at least one object from the panoramic image, deletes the selected at least one object to process it into an augmented reality (AR) plane, and restores the processed augmented reality plane to the panoramic image.

[0022] The aforementioned application processing unit can orthographically project the texture of the panoramic image onto each plane, and then perform inpainting on the portion perceived as the foreground region to process it into the augmented reality plane.

[0023] According to one embodiment of the present invention, the depth data processing unit performs sampling related to resolution using color images on depth data of multiple images acquired from a single camera stream via a camera, and constructs a depth map based on a deep learning model. The error correction processing unit determines a virtual panoramic viewpoint considering the different camera poses of the images, splatters the images into a panoramic image space based on the respective depth maps of the images, derives global color correction (GCC) parameters related to the correction of the color values ​​of the splatted images and residual pose parameters related to the amount of pose change, calculates at least one loss function for the splatted images, and corrects the GCC parameters and residual pose parameters. The panoramic processing unit splatters the image to which the corrected GCC parameters and corrected residual pose parameters have been applied back into the panoramic image space, stitches the re-splatted image, performs local color correction on overlapping regions in the stitched image, and performs seam blending. This may include a step of generating a panoramic image by performing blending.

[0024] The steps of determining a virtual panoramic viewpoint considering the different camera poses of the images, splatting the images into the panoramic image space based on the depth map of each image, deriving global color correction (GCC) parameters related to the correction of the color values ​​of the splatted image and residual pose parameters related to the amount of pose change, calculating at least one loss function for the splatted image to correct the GCC parameters and residual pose parameters, back projecting each pixel of the image to a 3D point based on the depth map, converting the 3D point from the image coordinate system to the world coordinate system and from the world coordinate system to the panoramic coordinate system, projecting it into the panoramic image space to calculate coordinates and mapping the pixel to a panoramic pixel, splatting the image into the panoramic image space based on the calculated pixel mapping, and applying photometric consistency loss and gradient consistency loss to the splatted image in relation to the GCC parameters. The process may include the steps of: calculating at least one loss function from among loss, depth consistency loss, and feature anchoring loss, and calculating at least one loss function from among rotation regularizer loss, translation regularizer loss, and GCC regularizer loss in relation to the residual pose parameter; and calculating a total loss based on the sum of the at least one loss function, and correcting the GCC parameter and the residual pose parameter by setting weights for each of the at least one loss function so that the calculated total loss is minimized.

[0025] For the splatted image, at least one loss function is calculated in relation to the GCC parameters, consisting of photometric consistency loss, gradient consistency loss, depth consistency loss, and feature anchoring loss. In relation to the residual pose parameters, rotation regularizer loss, translation regularizer loss, and GCC regularizer loss are calculated. The steps of calculating at least one loss function (loss) may include: calculating an average color value for all pixels where the splatted image overlaps with each other in the panoramic image space, and then calculating the color consistency loss based on the sum of the losses with the individual color values; deriving a gradient image for the image and splatting the gradient image, calculating an average gradient value for all pixels where they overlap, and then calculating the gradient consistency loss based on the sum of the losses with the individual gradient values; calculating an average depth value using the reciprocal of the distance calculated when the back-projected 3D points are projected onto the panoramic image space, and then calculating the depth consistency loss based on the sum of the losses with the individual values; and calculating the feature anchoring loss based on feature extraction and matching.

[0026] The steps of calculating at least one loss function from among photometric consistency loss, gradient consistency loss, depth consistency loss, and feature anchoring loss in relation to the GCC parameters for the splatted image, and at least one loss function from among rotation regularizer loss, translation regularizer loss, and GCC regularizer loss in relation to the residual pose parameters, may include the steps of calculating the rotation regularizer loss based on the sum of losses between the rotation matrix and identity matrix of each residual pose of the image, the translation regularizer loss based on the sum of losses between the translation vector and zero vector of each residual pose of the image, and the GCC regularizer loss based on the sum of losses between the color correction matrix and truncated identity matrix of each image.

[0027] The steps of generating a panoramic image by splattering the image to which the corrected GCC parameters and the corrected residual pose parameters have been applied back into the panoramic image space, stitching the splatted image back together, performing local color correction on overlapping areas in the stitched image, and performing seam blending, may include the steps of: deriving pixel values ​​for the splatted image, measuring the depth of 3D points relative to the derived pixel values, selecting an image from the splatted image to be used in the generated panoramic image based on the measured depth, performing local color correction to correct color differences in overlapping areas in the selected image, and performing seam blending to correct color differences in boundary areas within the generated panoramic image by correcting the color corrected after local color correction using residual colors related to the gradient of the image.

[0028] According to one embodiment of the present invention, a method for generating a 360-degree panoramic image may further include the steps of selecting at least one object from the panoramic image in an applied processing unit, deleting the selected at least one object to process it into an augmented reality (AR) plane, and restoring the processed augmented reality plane to the panoramic image. [Effects of the Invention]

[0029] The present invention provides a 360-degree panoramic image generation apparatus and method that achieves the consistency of a 360-degree panoramic image through splatting of a 3D image using depth and optimization of the camera pose.

[0030] This invention can generate physically accurate panoramic images that can measure the size of a space or embody experiences such as augmented reality (AR) by achieving consistency in 360-degree panoramic images.

[0031] The present invention provides a 360-degree panoramic image generation apparatus and method that generates a 360-degree panoramic image through shooting with a mobile phone, which is a general user terminal, without the need for special equipment such as a 360-degree camera.

[0032] In the implementation of an application program for generating 360-degree panoramic images, the present invention can achieve the realization of a natural image in the panoramic area by deleting some of the internal components and then inpainting them. [Brief explanation of the drawing]

[0033] [Figure 1] This figure illustrates a 360-degree panoramic image generation device according to one embodiment of the present invention. [Figure 2] This figure illustrates a screen illustrating the generation of a 360-degree panoramic image according to one embodiment of the present invention. [Figure 3] This figure illustrates image splatting for generating a 360-degree panoramic image according to one embodiment of the present invention. [Figure 4] This figure illustrates camera pose optimization for generating a 360-degree panoramic image according to one embodiment of the present invention. [Figure 5A] This figure illustrates the configuration of a frame selection-based preprocessing for generating a 360-degree panoramic image according to one embodiment of the present invention. [Figure 5B] This figure illustrates a frame selection-based local color correction configuration for generating a 360-degree panoramic image according to one embodiment of the present invention. [Figure 6]This figure illustrates the configuration of seam blending for generating a 360-degree panoramic image according to one embodiment of the present invention. [Figure 7] This figure illustrates the configuration of panoramic inpainting related to the application of 360-degree panoramic images according to one embodiment of the present invention. [Figure 8] This figure illustrates a method for generating a 360-degree panoramic image according to one embodiment of the present invention. [Modes for carrying out the invention]

[0034] The specific structural or functional descriptions of embodiments of the concept of the present invention disclosed herein are merely illustrative for the purpose of illustrating embodiments of the concept of the present invention, and embodiments of the concept of the present invention can be implemented in various forms and are not limited to those described herein.

[0035] Embodiments of the concept of the present invention can be modified in various ways and may take on various forms; such embodiments are illustrated in the drawings and described in detail herein. However, this is not intended to limit embodiments of the concept of the present invention to any particular disclosure, but rather to include modifications, equivalents, or substitutions that fall within the spirit and technical scope of the present invention.

[0036] Terms such as "first" or "second" can be used to describe various components, but the components should not be limited by such terms. The terms are used solely for the purpose of distinguishing one component from another; for example, without departing from the scope of rights under the concept of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0037] When it is mentioned that one component is "linked" or "connected" to another, it must be understood that this may mean that it is directly linked or connected to the other component, but that there may also be other components between those components. Conversely, when it is mentioned that one component is "directly linked" or "directly connected" to another, it must be understood that there are no other components between those components. Expressions describing the relationship between components, such as "between," "immediately between," or "directly adjacent to," must be interpreted in the same way.

[0038] The terms used herein are used solely to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “includes” or “having” are intended to specify the existence of a described feature, figure, stage, operation, component, part, or combination thereof, and should not be understood to preemptively exclude the existence or possibility of adding one or more other features, figures, stages, operations, components, parts, or combinations thereof.

[0039] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as those generally understood by a person of ordinary skill in the art to which this invention pertains. Terms as defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as ideal or overly formal unless explicitly defined herein.

[0040] The following examples will be described in detail with reference to the attached drawings. However, the scope of the patent application is not limited or restricted by these examples. The same reference numerals shown in each drawing indicate the same component.

[0041] Figure 1 is a diagram illustrating a 360-degree panoramic image generation device 100 according to one embodiment of the present invention.

[0042] Figure 1 illustrates the components of a 360-degree panoramic image generation device 100 according to one embodiment of the present invention.

[0043] Referring to Figure 1, a 360-degree panoramic image generation device 100 according to one embodiment of the present invention includes a depth data processing unit 110, an error correction processing unit 120, and a panoramic processing unit 130.

[0044] A 360-degree panoramic image generation device 100 according to one embodiment of the present invention can generate a 360-degree panoramic image (picture) through shooting with a mobile phone, without the need for special equipment such as a 360-degree camera.

[0045] For example, the 360-degree panoramic image generation device 100 achieves consistency through 3D image splatting using depth and optimization of camera pose, and can use the results to generate a panoramic image with physical accuracy for measuring the size of space or to embody an experience such as augmented reality.

[0046] For example, depth is based on depth data acquired via a LiDAR sensor included in a mobile camera.

[0047] According to one embodiment of the present invention, the depth data processing unit 110 can process resolution-related sampling using color images for depth data of multiple images acquired from a single camera stream via a camera, and construct a depth map based on a deep learning model.

[0048] For example, the depth data processing unit 110 adjusts the color image and resolution by performing an upsampling technique that utilizes the color image as guidance.

[0049] The depth data processing unit 110 utilizes a deep learning network as the backbone of the deep learning model, creates training data considering the characteristics of data obtained from mobile devices, and performs model training.

[0050] For example, the depth data processing unit 110 can apply a commercial MDE (monocular depth estimation) method to various types of color images to obtain a corresponding depth map.

[0051] The depth data processing unit 110 uses the inaccurate depth estimated by the MDE method to account for the inaccuracies inherent in the LiDAR depth data.

[0052] The depth data processing unit 110 downsamples each depth map obtained using the MDE method by 8 times using the nearest method to obtain a low-resolution depth map.

[0053] The depth data processing unit 110 can also construct a dataset using pairs of high-resolution and low-resolution depth maps for training.

[0054] An error correction processing unit 120 according to one embodiment of the present invention can determine a virtual panoramic viewpoint considering the different camera poses of the images, splatter the images into the panoramic image space based on the depth map of each image, derive GCC (global color correction) parameters related to the correction of the color values ​​of the splatted images and residual pose parameters related to the amount of pose change, and calculate at least one loss function for the splatted images to correct the GCC parameters and residual pose parameters.

[0055] For example, the error correction processing unit 120 back-projects each pixel of the image onto a 3D point based on the depth map, converts the 3D point from the image coordinate system to the world coordinate system, and from the world coordinate system to the panoramic coordinate system, projects it onto the panoramic image space to calculate the coordinates, calculates the pixels using panoramic pixel mapping, and splatters the image onto the panoramic image space based on the calculated pixel mapping.

[0056] According to one embodiment of the present invention, the error correction processing unit 120 can calculate at least one loss function from among photometric consistency loss, gradient consistency loss, depth consistency loss, and feature anchoring loss in relation to the GCC parameters for the splatted image, and can calculate at least one loss function from among rotation regularizer loss, translation regularizer loss, and GCC regularizer loss in relation to the residual pose parameters.

[0057] For example, the error correction processing unit 120 can calculate the total loss based on the sum of at least one loss function, and correct the GCC parameters and the residual pause parameters by setting weights for each of the at least one loss function so that the calculated total loss is minimized.

[0058] An error correction processing unit 120 according to one embodiment of the present invention can calculate an average color value for all pixels where images splatted on a panoramic image space overlap each other, then calculate a color consistency loss based on the sum of the losses with individual color values, derive a gradient image for the image and splat the gradient image, then calculate an average gradient value for all pixels where they overlap each other, then calculate a gradient consistency loss based on the sum of the losses with individual gradient values, calculate an average depth value using the reciprocal of the distance calculated when back-projected 3D points onto the panoramic image space, then calculate a depth consistency loss based on the sum of the losses with individual values, and calculate a feature anchoring loss based on feature extraction and matching.

[0059] For example, the error correction processing unit 120 can calculate a rotation normalization loss based on the sum of losses between the rotation matrix and identity matrix of each residual pose of the image, calculate a translation normalization loss based on the sum of losses between the translation vector and zero vector of each residual pose of the image, and calculate a GCC normalization loss based on the sum of losses between the color correction matrix and truncated identity matrix of each color correction matrix of the image.

[0060] For example, a configuration that optimizes GCC and camera pose using a loss function based on the error correction processing unit 120 will be explained in detail with reference to Figure 4.

[0061] A panoramic processing unit 130 according to one embodiment of the present invention can generate a panoramic image by splattering an image to which corrected GCC parameters and corrected residual pose parameters have been applied into the panoramic image space, stitching the splatted image again, performing local color correction on overlapping areas in the stitched image, and performing seam blending.

[0062] Local color correction based on the panoramic processing unit 130 is explained in detail using Figures 5A and 5B, and seam blending is explained in detail using Figure 6.

[0063] For example, the panorama processing unit 130 can derive pixel values ​​from the splatted image, measure the distance between three-dimensional points relative to the derived pixel values, select an image from the splatted image to be used for the generated panoramic image based on the measured distance, and perform local color correction to correct the color difference in overlapping areas of the selected image.

[0064] A panoramic processing unit 130 according to one embodiment of the present invention can perform seam blending to correct color differences in the boundary areas within the generated panoramic image by correcting the corrected color after local color correction using residual color related to the gradient of the image.

[0065] A 360-degree panoramic image generation apparatus 100 according to one embodiment of the present invention further includes an application processing unit 140.

[0066] According to one embodiment of the present invention, the application processing unit 140 can select at least one object from a panoramic image, delete the selected at least one object to process it into an augmented reality (AR) plane, and restore the processed augmented reality plane to the panoramic image.

[0067] The applied processing unit 140 can process the panoramic image's texture into the augmented reality plane by orthographic projection onto each plane and then inpainting the portion perceived as the foreground region. Here, the inpainting based on the applied processing unit 140 will be further explained with reference to Figure 7.

[0068] Therefore, the present invention can provide a 360-degree panoramic image generation apparatus and method that achieves the consistency of a 360-degree panoramic image through splatting of a 3D image using depth and optimization of the camera pose.

[0069] Figure 2 illustrates a screen illustrating the generation of a 360-degree panoramic image according to one embodiment of the present invention.

[0070] Figure 2 illustrates a screen for generating a 360-degree panoramic image according to one embodiment of the present invention.

[0071] Referring to Figure 2, images 200 and 220 are consecutive screens, and images 210 and 230 are consecutive screens.

[0072] Image 200 splatters the screen captured by the mobile camera onto a panoramic spatial image as Image 201.

[0073] Image 210 is created by splattering a screen captured by a mobile camera onto a panoramic spatial image as Image 211.

[0074] Image 220 is created by splattering a screen captured by a mobile camera onto a panoramic spatial image as Image 221.

[0075] Image 230 is created by splattering a screen captured by a mobile camera onto a panoramic spatial image as Image 231.

[0076] Comparing image 201 and image 221, we can see that the captured images were cumulatively added together.

[0077] Comparing image 211 and image 231, we can confirm that the captured images were accumulated and added together.

[0078] If the camera moves beyond a certain angle from the previous image and meets the sharpness requirements, an image is automatically captured.

[0079] Capture images so that there is at least a 1 / 3 overlap between adjacent images.

[0080] Images 201, 211, 221, and 231 may be preview images.

[0081] Images 201, 211, 221, and 231 show only one side of the space, but in reality, it is possible to capture the entire 360 ​​degrees horizontally.

[0082] For example, if a user wants to generate a 360-degree panoramic image, they would position the camera close to their body and take numerous images. In this case, the user would slowly move in a zigzag pattern, capturing images without gaps, within a range of 120 to 180 degrees vertically and 180 to 360 degrees horizontally.

[0083] In this case, the less the camera's physical position moves, the higher the quality of the panoramic image that can be generated.

[0084] The system simultaneously records the pose of the captured image and the camera's internal parameters, providing on-screen visual feedback to enable the user to consistently capture images while recognizing areas already captured during the shooting process.

[0085] Once shooting is complete, the data is transmitted to a server and fed into a panorama generation solution, where the processed panoramic image can be provided to the user.

[0086] Figure 3 illustrates image splatting for generating a 360-degree panoramic image according to one embodiment of the present invention.

[0087] Figure 3 illustrates the results of image splatting for generating a 360-degree panoramic image according to one embodiment of the present invention.

[0088] Referring to Figure 3, Image 300 shows the captured image, and Image 310 shows the panoramic spatial image.

[0089] In Image 310, the result of splattering the previously captured Image 300 onto the panoramic space is shown as Preview Image 311.

[0090] For Image 300 to be splatted onto the panoramic spatial image, errors exist in the camera pose for each image, so correction for these errors is required.

[0091] To splatter images captured onto a panoramic spatial image, a gradient-descent style optimization algorithm must be applied.

[0092] As a preliminary step, a virtual panoramic viewpoint can be determined that best combines N images with different poses.

[0093] After constructing a 3D line using the camera position and viewing direction of each image, a point in space can be calculated where the sum of squared Euclidean distances to all lines is minimized, and this point can be determined as the position of the virtual panoramic camera.

[0094] The panoramic camera's axis direction is set so that the down direction aligns with the direction of gravity, and since it will cover the entire 360 ​​degrees horizontally anyway, the looking direction can be set arbitrarily.

[0095] The splatting method of the present invention applies depth-based image splatting, but splatters each input image into a panoramic image space based on a depth map.

[0096] The splatting method of this invention calculates image pixels using panoramic pixel mapping.

[0097] Each pixel in the image is back-projected onto a 3D point based on the depth map.

[0098] For example, it can be summarized as shown in equation 1 below.

[0099]

number

[0100] In Equation 1, X is a three-dimensional point of the back projection and represents the image reference coordinate system, K represents the camera-specific (intrinsic) matrix, u and v represent the image coordinates, and d represents the depth value.

[0101] The 3D points are sequentially transformed from the image coordinate system to the world coordinate system and then to the panoramic coordinate system.

[0102]

number

[0103] In equation 2, X pano This shows the panoramic reference coordinate system of 3D points, T w2p This shows the transformation to the world coordinate system and the panoramic coordinate system, T f2w This shows the transformation from the frame coordinate system to the world coordinate system.

[0104] By projecting it onto a panoramic image space, the coordinates can be calculated using the following formula 3.

[0105]

number

[0106] In equation 3, W pano The 'width' indicates the panorama width, and the '%' can indicate the mode operation.

[0107] By projecting it onto a panoramic image space, the coordinates can be calculated using the following formula 4.

[0108]

number

[0109] In equation 4, Hpano This indicates the height of the panorama.

[0110] Apply an arbitrarily set panorama size based on equations 3 and 4; for example, use 1024 and 512, and then use 8192 and 4096 in subsequent stitching.

[0111] The image is splatted into a panoramic image space based on the mapping of pixel-by-pixel image coordinates (u,v).

[0112] In this case, since the coordinates in the panoramic image space are decimal (real) coordinates, the values ​​to be splatted are weighted using a method (scheme) similar to bilinear interpolation for the 2x2 pixel region surrounding those coordinates, and then accumulated.

[0113] Figure 4 illustrates camera pose optimization for generating a 360-degree panoramic image according to one embodiment of the present invention.

[0114] Figure 4 illustrates the results of optimizing the camera pose for generating a 360-degree panoramic image according to one embodiment of the present invention.

[0115] Referring to Figure 4, the results of sequentially applying camera pose optimization are shown through images 400, 401, and 402.

[0116] Images 400, 401, and 402 have GCC (Global Color Correction) applied.

[0117] By applying GCC to adjust the RGB values ​​for each image, the result of the application can be defined as shown in equation 5 below.

[0118]

number

[0119] In Equation 5, [r’, g’, b’] represents the corrected color, M GCC represents the color correction transformation, and [r, g, b] represents the unique color.

[0120] Here, the color correction transformation is a unique 3×4 matrix for each image, in a form where color crosstalk and color offset are connected horizontally.

[0121] The residual pose representations related to Image 400, Image 401, and Image 402 can parameterize the amount of change from the pose and can be summarized as shown in Equation 6 below.

[0122]

Equation

[0123] In Equation 6, T refined can represent the corrected pose, T orig can represent the original AR pose, and T residual can represent the residual pose.

[0124] T refined is the rigid body transformation of the SE group, and the SE group is [[R, t]; [0, 0, 0, 1]]. Here, R is the rotation matrix and t is the transformation vector, and these can each be represented by three variables.

[0125] The error correction processing unit according to an embodiment of the present invention calculates a loss function and performs optimization to minimize the loss.

[0126] In this case, the variables to be optimized are the residual poses of all images and the GCC parameters.

[0127] The sum of the loss functions can be summarized as shown in Equation 7.

[0128]

number

[0129] In equation 7, L total This shows the overall loss function, L color This indicates a loss of optical color consistency, L gradient This can show gradient consistency loss, L depth This can show the depth consistency loss, L anchor This can show the feature anchoring loss, L rot.reg This can show the rotational normalization loss, L trans.reg This can show the moving normalization loss, L gcc.reg This can be used to show the GCC normalized loss.

[0130] Color consistency loss is calculated by first determining the average color value for all pixels where splatted color images in panoramic space overlap, and then summing the L1 losses of the average color value with the individual color values.

[0131] It uses GCC-applied color values ​​and guides overlapping areas to ensure they are aligned with each other.

[0132] The reason for using the difference between the mean and individual values ​​as the loss is that when there are only two overlapping images, it is sufficient to simply compare them directly. However, when there are two or more overlapping images, comparing them all in pairs would involve too many cases.

[0133] Gradient consistency loss is similar to color consistency loss, but instead of a color image, a gradient image of the color image is acquired beforehand, and the same method is used to splatter and calculate the sum of the L1 losses of the average value and individual values.

[0134] It further enhances the consistency of strong textures and, due to its robust properties against scale and offset differences caused by automatic exposure, is highly effective in the early stages of optimization when color correction is insufficient.

[0135] Depth consistency loss does not directly splat the depth image, but rather uses the reciprocal calculated when projecting the back-projected 3D points from the splatting process onto the panoramic image space.

[0136] After calculating the average of the depth values ​​obtained in this way, the sum of the L1 losses with the individual values ​​is calculated.

[0137] When using inverse depth, errors at nearby points can be corrected more effectively than errors at distant points, which is advantageous for this alignment method that utilizes a perspective image space and helps to obtain good alignment results.

[0138] Feature anchor consistency loss is a loss function that utilizes feature extraction and matching techniques commonly used in the vision field.

[0139] Feature extraction involves extracting feature points from each of the N input images.

[0140] A fast and highly accurate method like XFeat[2] is preferred, but it doesn't matter if you use other methods.

[0141] Feature matching returns a list of matching feature points for all image-image combinations (= N × (N-1) / 2 possibilities).

[0142] When using methodologies such as LightGlue[3] (or the even lighter LighterGlue), reliable matching results can be obtained quickly.

[0143] To perform this for all image combinations, N 2 As the complexity increases proportionally, it can be made more efficient by only performing the process on combinations where the difference in image poses is within a certain range.

[0144] When using only three local consistency losses, problems can arise where distant corresponding points cannot be aligned initially. An anchoring effect prevents this.

[0145] The rotation normalization loss represents the sum of the L2 losses between the rotation matrix and identity matrix of the residual pose for each image, and it penalizes (imposes a penalty on) the corrected pose rotating too far from the original pose.

[0146] The transformation normalization loss represents the sum of the L2 losses between the translation vector of the residual pose of each image and a zero vector of equal magnitude.

[0147] A penalty will be imposed if the corrected pose moves far away from the original pose.

[0148] The GCC normalized loss is derived as the sum of the losses between the color correction matrix and the omission matrix for each image.

[0149] The law punishes those who perform color correction that significantly alters the color and brightness of the input image.

[0150] Performance can be adjusted by setting different weights for each loss, and an example of weight application can be summarized in equation 8 below.

[0151]

number

[0152] Equation 8 illustrates the application of weights, which can be applied using Equation 7.

[0153] For computational efficiency, the input image is downsampled to 1 / 4 size and used for splatting, making use of the reduced resolution of the panoramic image.

[0154] Optimization iterations allow us to visualize the average of color / gradient / depth images splatted into a panoramic image space.

[0155] Image 400, which corresponds to the initial optimization stage (iter: 40), shows that the splatted image is not well aligned due to an inaccurate camera pose, resulting in a blurred average image. However, as the iteration progresses to images 401 and 402 (140 → 280), alignment improves, and the average image becomes sharper.

[0156] Furthermore, it can be observed that the brightness and tone of the image change uniformly due to GCC.

[0157] This invention does not simply reduce stitching errors by distorting the image, but achieves consistency through splatting of 3D images using depth and optimization of camera pose.

[0158] This allows the present invention to generate panoramas that are geometrically accurate (and therefore enable XR applications) and have minimal stitching errors without additional image distortion.

[0159] Figures 5A and 5B illustrate the configuration of frame selection-based preprocessing and local color correction for generating a 360-degree panoramic image according to one embodiment of the present invention.

[0160] Figure 5A illustrates a configuration of frame selection-based preprocessing for generating a 360-degree panoramic image according to one embodiment of the present invention.

[0161] Referring to Figure 5A, the corrected camera pose and GCC are applied, and the input image is splatted into the panoramic image space once again.

[0162] When splatting an additional N images, the minimum and maximum inverse depth (w) for each panoramic pixel. min ,w max Record ).

[0163] Unlike previous steps that used downsampled images, this step generates the final panoramic result, so the original input image is used, and the resolution of the panorama to be splatted is also set to high resolution to preserve as much high-frequency detail as possible.

[0164] Images 500 and 510 may have multiple splatting images contributing to each panoramic pixel in the best frame selection.

[0165] Select each image from the selection and obtain its pixel value.

[0166] At this point, the panoramic processing unit back-projects the minimum and maximum inverse depths onto three-dimensional points.

[0167] Two 3D points are projected onto a candidate image, and the distance between the two points is measured in each image.

[0168] The image with the shortest distance is selected as the optimal image.

[0169] This distance represents the length of each line on the image corresponding to a given depth uncertainty.

[0170] This length indicates how many pixels in the image can be involved with that panoramic pixel.

[0171] Therefore, selecting an image with a shorter distance and taking the pixel value from that image can be seen as having a higher expected value that the pixel value is true.

[0172] Image 510 illustrates the panoramic color image obtained through the initial stitching process, as well as the optimal image index used.

[0173] As shown in Image 510, even though GCC was applied beforehand, there are significant differences in brightness and color tone between the stitched images. However, these differences will be resolved through subsequent additional corrections.

[0174] Figure 5B illustrates a configuration of frame selection-based local color correction for generating a 360-degree panoramic image according to one embodiment of the present invention.

[0175] Referring to Figure 5B, we can see a comparison of Image 520 and Image 530 regarding the correction of color differences between splatting images that have been stitched together through the preprocessing steps described in Figure 5A.

[0176] Comparing section 521 of image 520 with section 531 of image 530, we can see that each patch image has been regionally color-corrected to provide a more uniformly altered screen.

[0177] Only a portion of the images were selected through a preprocessing step that selected the best frame from each splatter image.

[0178] The image obtained by dilating each of these areas by just one pixel is called a patch image.

[0179] As a result of the expansion, adjacent patch images will have an overlap of about 2 pixels.

[0180] The following optimization is performed to regionally correct the color of each patch image in order to minimize the difference in overlapping regions between patch images.

[0181] Here, the method used to represent regional color correction is polynomial fitting.

[0182] For each patch image, the amount of color change (dr, dg, db) based on pixel coordinates is modeled using a polynomial function, where the variables are the coefficients of the polynomial function.

[0183] A notable characteristic is that, due to the nature of the panoramic images we are currently working with, simply constructing a monomial using only u and v coordinates makes it impossible to represent wrap-around in the horizontal direction.

[0184] Therefore, in this invention, u and v are first converted to θ and φ of the panoramic projection model, and then the three values ​​(sin(θ), cos(θ), φ) are used. However, sin(θ) and cos(θ) are constructed as unary terms so that they are not used simultaneously, thereby allowing curved parts to be handled naturally and can be illustrated as follows. Example) 1, sin(θ), cos(θ), φ, sin(θ) 2 ,cos(θ) 2 ,φ 2 ,sin(θ)φ,cos(θ)φ,sin(θ) 3 ,cos(θ) 3 ,φ 3 ,sin(θ) 2 φ, cos(θ) 2 φ, sin(θ)φ 2 ,cos(θ)φ 2 ,…

[0185] Therefore, if we call the multinomial coefficient vector of a single patch image C, and the coordinates of the pixel to be corrected are (u,v)→(θ,φ), the changed color can be summarized by equation 9 as follows.

[0186]

number

[0187] In equation 9, C can represent a multinomial coefficient vector, and M can represent a unary vector derived from unary θ and φ, and the optimization variables can be a set of multinomial coefficients for each patch image.

[0188] The loss function related to this can be summarized as shown in equation 10 below.

[0189]

number

[0190] In equation 10, L overlapThis shows the sum of the corrected L2 loss between colors measured in the overlapping region between patch images, and it serves to reduce the difference between adjacent images.

[0191] L reg This can represent the sum of the L1 losses between the color change vector and a zero vector of equal magnitude, which penalizes excessive changes from the original color.

[0192] λ reg This indicates the weight, 10 -3 This can be illustrated by using the multinomial order 3.

[0193] The LCC process is completed when the polynomial coefficients optimized for each patch image, utilizing only the overlapping region, are applied to all the remaining regions.

[0194] Considering that the resolution of images currently in general use is 8K, it is possible to effectively optimize by utilizing polynomial fitting while simultaneously using only the overlapping regions between patch images.

[0195] Thanks to the smooth characteristics of the low-order polynomial function, the effect of local color correction can be naturally achieved in the remaining areas as well.

[0196] Figure 6 illustrates the configuration of seam blending for generating a 360-degree panoramic image according to one embodiment of the present invention.

[0197] Figure 6 illustrates the application results of a seam blending configuration for generating a 360-degree panoramic image according to one embodiment of the present invention.

[0198] Referring to Figure 6, Images 600 and 610 are cropped versions of the same panoramic scene, showing only the left portion. By comparing Image 600 with Image 610, it can be seen that Image 610 is well corrected.

[0199] Image 600 shows a noticeable seam in the vertical direction, but this seam is not visible in Image 610.

[0200] Seam blending is an additional correction process that addresses the fact that boundaries between patch images may still be noticeable after LCC (Low-Cooling Color Correction). It involves pre-extracting the gradients of each patch image during the pre-processing stage and guiding the gradient calculated with the corrected colors to resemble the original, and a loss function may be applied.

[0201] For example, this could be a correction that prevents the color change from becoming too large, calculated by adding the loss to the pre-extracted color gradient values ​​and the current gradient value, and adding the loss to the amount of color change.

[0202] Figure 7 illustrates the configuration of panoramic inpainting related to the application of a 360-degree panoramic image according to one embodiment of the present invention.

[0203] Figure 7 illustrates the results of an implementation based on a panoramic inpainting configuration related to the application of a 360-degree panoramic image according to one embodiment of the present invention.

[0204] Referring to Figure 7, a panoramic inpainting can be realized from image 700 to image 710.

[0205] The conversion from Image 700 to Image 710 is a result of providing a function like "furniture removal" when realizing augmented reality on a panorama application, which involves removing specific objects in inpainting to exclude them.

[0206] When transforming Image 700 into Image 710, room layout data is used to represent structures such as walls and floors in a mesh or plane-like manner. This data is obtained either via the room plan API during AR shooting or by processing the AR plane.

[0207] The panoramic image texture can be realized by orthogonally projecting it onto each plane, then inpainting the parts detected as foreground regions, and finally bringing them back into the panoramic image space.

[0208] Therefore, in the embodiment of an application program for generating 360-degree panoramic images, the present invention can achieve the realization of a natural image in the panoramic region by deleting some of the internal components and then inpainting them.

[0209] Figure 8 illustrates a method for generating a 360-degree panoramic image according to one embodiment of the present invention.

[0210] Figure 8 illustrates a procedure for generating a 360-degree panoramic image according to one embodiment of the present invention, which achieves consistency of the 360-degree panoramic image through 3D image splatting using depth and optimization of camera pose.

[0211] Referring to Figure 8, in step S801, a method for generating a 360-degree panoramic image according to one embodiment of the present invention processes depth data of multiple images acquired from a single camera stream via a camera.

[0212] In other words, a method for generating a 360-degree panoramic image according to one embodiment of the present invention involves processing depth data of multiple images acquired from a single camera stream via a camera, utilizing color images to perform resolution-related sampling, and constructing a depth map based on a deep learning model.

[0213] In step S802, the method for generating a 360-degree panoramic image according to one embodiment of the present invention calculates a loss function for the GCC parameters and residual pose parameters to optimize pose correction and global color correction.

[0214] In other words, a method for generating a 360-degree panoramic image according to one embodiment of the present invention determines a virtual panoramic viewpoint considering the different camera poses of the images, splatters the images into the panoramic image space based on the depth map of each image, derives GCC (global color correction) parameters related to the correction of the color values ​​of the splatted image and residual pose parameters related to the amount of pose change, and calculates at least one loss function for the splatted image to correct the GCC parameters and residual pose parameters.

[0215] In step S803, the method for generating a 360-degree panoramic image according to one embodiment of the present invention applies corrected poses and GCC parameters to the panoramic image space, and generates a panoramic image by stitching the splatted images, performing local color correction, and seam blending.

[0216] In other words, a method for generating a 360-degree panoramic image according to one embodiment of the present invention involves splatting an image to which corrected GCC parameters and the corrected residual pose parameters have been applied into the panoramic image space, stitching the splatted image again, performing local color correction on overlapping areas in the stitched image, and performing seam blending to generate a panoramic image.

[0217] Therefore, the present invention provides a 360-degree panoramic image generation apparatus and method that generates a 360-degree panoramic image through shooting with a mobile phone, which is a general user terminal, without the need for special equipment such as a 360-degree camera.

[0218] The devices described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, controller, ALU (arithmetic logic unit), digital signal processor, microcomputer, FPA (field programmable array), PLU (programmable logic unit), microprocessor, or any other device capable of executing and responding to instructions. The processing device can run an operating system (OS) and one or more software applications run on the operating system. The processing device can also access, store, manipulate, process, and generate data in response to software execution. For convenience of understanding, the processing device may have been described as being used as one, but a person with ordinary skill in the art will see that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Furthermore, other processing configurations, such as parallel processors, are also possible.

[0219] Software may include computer programs, code, instructions, or a combination of one or more of these, which can configure a processing unit to operate as desired, or independently or collectively, instruct the processing unit. Software and / or data may be permanently or temporarily embodied in a certain type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, in order to be interpreted by a processing unit or to provide instructions or data to a processing unit. Software may be distributed on a networked computer system and stored or executed in a distributed manner. Software and data may be stored on a recording medium readable by one or more computers.

[0220] As described above, although the embodiments have been explained with limited drawings, various modifications and variations can be made from the above description by a person with ordinary skill in the art. For example, the described technique may be performed in a different order than described, and / or the components of the described system, structure, apparatus, circuit, etc. may be combined or assembled in a different manner than described, or substituted or replaced by other components or equivalents, and satisfactory results may be achieved.

[0221] Therefore, other embodiments, other embodiments and equivalents of the claims also fall within the scope of the attached claims.

Claims

1. A depth data processing unit that uses color images to process resolution-related sampling on depth data of multiple images acquired from a single camera stream via a camera, and constructs a depth map based on a deep learning model, An error correction processing unit determines a virtual panoramic viewpoint considering the different camera poses of the aforementioned images, splatters the images into the panoramic image space based on the depth map of each of the aforementioned images, derives a global color correction (GCC) parameter related to the correction of the color values ​​of the splatted image and a residual pose parameter related to the amount of pose change, calculates at least one loss function for the splatted image, and corrects the GCC parameter and the residual pose parameter. A 360-degree panoramic image generating apparatus, comprising: a panoramic processing unit that splatters the image to which the corrected GCC parameters and the corrected residual pose parameters have been applied back into the panoramic image space; stitches the splatted image back together; performs local color correction on overlapping areas in the stitched image; and performs seam blending to generate a panoramic image.

2. The error correction processing unit is characterized by back-projecting each pixel of the image onto a three-dimensional point based on the depth map, converting the three-dimensional point from the image coordinate system to the world coordinate system and from the world coordinate system to the panoramic coordinate system, projecting it onto the panoramic image space to calculate coordinates and mapping the pixels to panoramic pixels, and splatting the image onto the panoramic image space based on the calculated pixel mapping, as described in claim 1, for a 360-degree panoramic image generation apparatus.

3. The 360-degree panoramic image generating apparatus according to claim 1, characterized in that the error correction processing unit calculates at least one loss function from among photometric consistency loss, gradient consistency loss, depth consistency loss, and feature anchoring loss with respect to the splatted image in relation to the GCC parameters, and calculates at least one loss function from among rotation regularizer loss, translation regularizer loss, and GCC regularizer loss in relation to the residual pose parameters.

4. The 360-degree panoramic image generating apparatus according to claim 3, characterized in that the error correction processing unit calculates a total loss based on the sum of the at least one loss function, and corrects the GCC parameter and the residual pose parameter by setting weights for each of the at least one loss function so that the calculated total loss is minimized.

5. The error correction processing unit is characterized in that it calculates an average color value for all pixels where the splatted images overlap on the panoramic image space, then calculates the optical color consistency loss based on the sum of the losses with the individual color values, derives a gradient image for the image, splatts the gradient image, then calculates an average gradient value for all pixels where they overlap, then calculates the gradient consistency loss based on the sum of the losses with the individual gradient values, calculates an average depth value using the reciprocal of the distance calculated when the back-projected 3D points are projected onto the panoramic image space, then calculates the depth consistency loss based on the sum of the losses with the individual values, and calculates the feature anchoring loss based on feature extraction and matching, as described in claim 3.

6. The 360-degree panoramic image generating apparatus according to claim 3, characterized in that the error correction processing unit calculates the rotation normalization loss based on the sum of losses between the rotation matrix and identity matrix of each residual pose of the image, calculates the translation normalization loss based on the sum of losses between the translation vector and zero vector of each residual pose of the image, and calculates the GCC normalization loss based on the sum of losses between the color correction matrix and truncated identity matrix of each image.

7. The 360-degree panoramic image generating apparatus according to claim 1, characterized in that the panoramic processing unit derives pixel values ​​from the splatted image, measures the depth of three-dimensional points with respect to the derived pixel values, selects an image from the splatted image to be used for the generated panoramic image based on the measured depth, and performs local color correction to correct the color difference for overlapping areas in the selected image.

8. The 360-degree panoramic image generating apparatus according to claim 1, characterized in that the panoramic processing unit performs seam blending to correct color differences in boundary portions within the generated panoramic image by correcting the color corrected after the local color correction using residual color related to the gradient of the image.

9. The 360-degree panoramic image generating apparatus according to claim 1, further comprising an application processing unit that selects at least one object from the panoramic image, deletes the selected at least one object to process it into an augmented reality (AR) plane, and restores the processed augmented reality plane to the panoramic image.

10. The 360-degree panoramic image generating apparatus according to claim 9, characterized in that the applied processing unit orthographically projects the texture of the panoramic image onto each plane, and then performs inpainting on the portion perceived as the foreground region on each plane to process it into the augmented reality plane.

11. In the depth data processing unit, the steps include: processing depth data from multiple images acquired from a single camera stream via a camera, using color images to perform resolution-related sampling, and constructing a depth map based on a deep learning model; In the error correction processing unit, a virtual panoramic viewpoint is determined considering the different camera poses of the images, the images are splatted into the panoramic image space based on the depth map of each image, a global color correction (GCC) parameter related to the correction of the color values ​​of the splatted image and a residual pose parameter related to the amount of pose change are derived, and at least one loss function for the splatted image is calculated to correct the GCC parameter and the residual pose parameter. A method for generating a 360-degree panoramic image, characterized by comprising the steps of: in a panoramic processing unit, splatting the image to which the corrected GCC parameters and the corrected residual pose parameters have been applied again into the panoramic image space; stitching the splatted image again; performing local color correction on overlapping areas in the stitched image; and performing seam blending to generate a panoramic image.

12. The steps of determining a virtual panoramic viewpoint considering the different camera poses of the images, splatting the images into the panoramic image space based on the depth map of each image, deriving a global color correction (GCC) parameter related to correcting the color values ​​of the splatted image and a residual pose parameter related to the amount of pose change, and calculating at least one loss function for the splatted image to correct the GCC parameter and the residual pose parameter are as follows: The steps include: back-projecting each pixel of the image onto a three-dimensional point based on the depth map; converting the three-dimensional point from the image coordinate system to the world coordinate system and from the world coordinate system to the panoramic coordinate system; projecting it onto the panoramic image space to calculate the coordinates and mapping the pixels to panoramic pixels; and splatting the image onto the panoramic image space based on the calculated pixel mapping. The steps include: calculating at least one loss function from among photometric consistency loss, gradient consistency loss, depth consistency loss, and feature anchoring loss in relation to the GCC parameters for the splatted image; and calculating at least one loss function from among rotation regularizer loss, translation regularizer loss, and GCC regularizer loss in relation to the residual pose parameters; A method for generating a 360-degree panoramic image according to claim 11, comprising the steps of: calculating a total loss based on the sum of the at least one loss function; and correcting the GCC parameter and the residual pose parameter by setting weights for each of the at least one loss function so that the calculated total loss is minimized.

13. The steps of calculating at least one loss function among photometric consistency loss, gradient consistency loss, depth consistency loss, and feature anchoring loss in relation to the GCC parameters for the splatted image, and at least one loss function among rotation regularizer loss, translation regularizer loss, and GCC regularizer loss in relation to the residual pose parameters, are as follows: The steps include: calculating the average color value for all pixels where the splatted images overlap each other in the panoramic image space, and then calculating the color consistency loss based on the sum of the losses with the individual color values; The steps include: deriving a gradient image for the aforementioned image, splatting the gradient image, calculating an average gradient value for all overlapping pixels, and then calculating the gradient consistency loss based on the sum of the losses with the individual gradient values; The steps include: calculating the average depth value using the reciprocal of the distance calculated when projecting the back-projected 3D points onto the panoramic image space, and then calculating the depth consistency loss based on the sum of the losses with the individual values; A method for generating a 360-degree panoramic image according to claim 12, characterized by comprising the step of calculating the feature anchoring loss based on feature extraction and matching.

14. The steps of calculating at least one loss function among photometric consistency loss, gradient consistency loss, depth consistency loss, and feature anchoring loss in relation to the GCC parameters for the splatted image, and at least one loss function among rotation regularizer loss, translation regularizer loss, and GCC regularizer loss in relation to the residual pose parameters, are as follows: The steps include: calculating the rotation normalization loss based on the sum of the losses between the rotation matrix and identity matrix of each residual pose of the image; The steps include: calculating the moving normalization loss based on the sum of the losses between the transformation vector and the zero vector of each residual pose of the image; A method for generating a 360-degree panoramic image according to claim 12, characterized by comprising the step of calculating the GCC normalized loss based on the sum of the losses between each color correction matrix of the image and a truncated identity matrix.

15. The steps of generating a panoramic image are as follows: splatting the image to which the corrected GCC parameters and the corrected residual pose parameters have been applied again into the panoramic image space; stitching the splatted image again; performing local color correction on overlapping areas in the stitched image; and performing seam blending. The steps include: deriving pixel values ​​from the splatted image; measuring the depth of three-dimensional points relative to the derived pixel values; selecting an image from the splatted image to be used in the generated panoramic image based on the measured depth; and performing local color correction to correct for color differences in overlapping areas in the selected image. A method for generating a 360-degree panoramic image according to claim 11, characterized by comprising the step of performing seam blending, which corrects the color after local color correction using residual color related to the gradient of the image, thereby correcting the color difference in the boundary portion within the generated panoramic image.

16. A method for generating a 360-degree panoramic image according to claim 11, further comprising the steps of: in an applied processing unit, selecting at least one object from the panoramic image; deleting the selected at least one object to process it into an augmented reality (AR) plane; and restoring the processed augmented reality plane to the panoramic image.

Images