A 360-degree panoramic image upright adjustment method based on deep learning

Abstract

The application belongs to the technical field of panoramic image upright adjustment, and discloses a 360-degree panoramic image upright adjustment method based on deep learning, which comprises the following steps: S101, expanding the boundary of the panoramic image; S102, converting one-dimensional tilt features into a spherical coordinate mapping matrix, and outputting a three-dimensional coordinate mapping matrix; S103, obtaining the three-dimensional coordinate mapping matrix after up-sampling; and S104, mapping the input non-upright panoramic image and the three-dimensional coordinate mapping matrix after up-sampling to obtain an upright panoramic image; the application inputs a non-upright image through an end-to-end neural network and directly outputs an adjusted panoramic upright image, so as to solve the problem that the existing technology is limited by two-dimensional projection, cannot converge at the edge, and causes cracks in the upright-adjusted panoramic image.

Description

Technical Field

[0001] This invention relates to the field of panoramic image vertical adjustment technology, specifically a method for vertical adjustment of 360-degree panoramic images based on deep learning. Background Technology

[0002] Today, commercially available 360-degree panoramic cameras are commonplace. Unlike the limited field of view of ordinary cameras, these cameras can capture the surrounding 360-degree scene from all angles, ultimately forming a 360-degree panoramic image. Figure 1 The image shown is a scene captured with the camera upright. Although projecting three-dimensional space onto a two-dimensional plane will distort the top and bottom of the image, it is generally easy to understand the scene. However, in real-world scenarios, it is difficult for users to ensure that the camera is perfectly upright. Therefore, a tilted camera will produce a non-upright 360-degree panoramic image, i.e. Figure 2 As shown, it can be seen that non-upright images are more difficult to understand than upright images.

[0003] Traditional panoramic image uprightness correction methods first estimate the tilt angle, then compensate for the estimated tilt by resampling the non-upright image to generate an upright image. This calculation involves a spherical mathematical model, is complex in principle, time-consuming, and suffers from some accuracy loss during resampling, resulting in jagged edges in the generated image. In recent years, a better approach has been to utilize deep learning networks to treat panoramic image uprightness correction as a pure end-to-end two-dimensional image task. However, existing methods often neglect its three-dimensional spatial continuity, such as... Figure 1 As shown, the left and right ends of the image should be a continuous scene in the three-dimensional scene, but the two-dimensional image cuts it off, resulting in poor image quality and cracks. Summary of the Invention

[0004] This invention aims to provide a method for vertical adjustment of 360-degree panoramic images based on deep learning. It takes a non-vertical image as input through an end-to-end neural network and directly outputs the adjusted panoramic vertical image. This solves the problem in the prior art where the limitation of two-dimensional projection causes fragmentation and failure to converge at the edges, resulting in cracks in the vertically adjusted panoramic image.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A deep learning-based method for vertical adjustment of 360-degree panoramic images includes the following steps:

[0007] S101: Expand the boundaries of the panoramic image:

[0008] The input non-vertical panoramic image is sliced ​​and stitched using image processing software, extending n pixels in all directions to obtain an expanded panoramic image.

[0009] S102: Transform the one-dimensional tilt feature into a spherical coordinate mapping matrix, and output a three-dimensional coordinate mapping matrix:

[0010] The augmented panoramic image obtained from S101 is input into the encoder. The deep convolutional neural network in the encoder is used to extract latent spatial features. The latent spatial features are input into the L2M module in the encoder to convert the latent spatial features into unit spherical coordinates, obtain coordinate mapping estimates, and output a three-dimensional coordinate mapping matrix.

[0011] S103: Obtain the upsampled 3D coordinate mapping matrix:

[0012] The 3D mapping coordinate matrix obtained in S102 is input into the decoder. After grouping convolution by the decoder, it is upsampled by several upsampling modules in the manner of PixelShuffle, gradually expanding the size of the mapping coordinate matrix, and outputting several upsampled 3D coordinate mapping matrices.

[0013] S104: Map the input non-upright panoramic image to the upsampled 3D coordinate mapping matrix to obtain the upright panoramic image:

[0014] The size of the input non-upright panoramic image is adjusted to correspond to the size of the upsampled three-dimensional coordinate mapping matrix obtained in S103, resulting in an adjusted non-upright panoramic image. The three-dimensional coordinates of the upsampled three-dimensional coordinate mapping matrix are converted into two-dimensional coordinates. Based on the coordinate information provided by the two-dimensional coordinates, the pixel values ​​of the adjusted non-upright panoramic image of the corresponding size are filled into the specified positions to obtain upright panoramic images of different sizes.

[0015] Furthermore, in S101, the vertical and horizontal extensions are 32 pixels each; the input non-vertical panoramic image size is 256×512.

[0016] Furthermore, in S102, the deep convolutional neural network is a ConvNext-tiny network, the latent spatial features are one-dimensional vectors of length 768, and the output three-dimensional coordinate mapping matrix has a size of 16×32.

[0017] Furthermore, in S103, the three-dimensional mapping coordinate matrix obtained in S102 is subjected to four layers of upsampling using the PixelShuffle method after grouped convolution. The first convolutional feature map has a size of 16×32 and a kernel size of 3; the second convolutional feature map has a size of 32×64 and a kernel size of 5; the third convolutional feature map has a size of 64×128 and a kernel size of 7; and the fourth convolutional feature map has a size of 128×256 and a kernel size of 9.

[0018] Furthermore, in S104, the expression for converting the three-dimensional coordinates of the three-dimensional coordinate mapping matrix obtained in S103, which has the same size as the input non-upright panoramic image, into two-dimensional coordinates is as follows:

[0019]

[0020] In the above expression, W represents the image width, H represents the image height, and θ represents the zenith angle on the sphere corresponding to the three-dimensional coordinates. This represents the azimuth angle on the sphere corresponding to the three-dimensional coordinates.

[0021] Furthermore, in S104, the upright panoramic image of the corresponding size obtained in S104 and the target upright image after size adjustment are subjected to multi-level image loss constraints.

[0022] The L1 loss formula is:

[0023]

[0024] In the above formula, x i Represents the ground truth pixel values ​​in a dataset of upright target images. Represents the pixel values ​​of an upright panoramic image;

[0025] The network loss expression is:

[0026] loss py =loss 32 +loss 64 +loss 128 +loss 256

[0027] loss = loss py +loss grid

[0028] In the above expressions, the L1 loss formula is used. py This represents the image loss constraint at different sizes, loss grid represents the truth constraint, and loss represents the total network loss.

[0029] Beneficial effects of the technical solution:

[0030] 1. Since two-dimensional panoramic images are discontinuous, but their left and right boundaries are continuous in essence, this invention extends the image to compensate for the continuity problem. That is, by cutting and rotating the non-vertical part of the image, the size of the original non-vertical panoramic image is enlarged, the boundary part of the panoramic image is correctly extended, and the convolutional network is prevented from not finding the corresponding adjacent points at the image boundary. This allows the network to extract more continuous features at the boundary of the non-vertical panoramic image and improve the accuracy of image mapping constraints.

[0031] 2. This invention utilizes a three-channel matrix to transform latent features into unit spherical coordinates, reshaping a one-dimensional vector into three-dimensional x, y, z coordinates, and normalizing the three-dimensional x, y, z coordinates to spherical coordinates to obtain coordinate mapping estimation. The output is a 16×32 three-dimensional coordinate mapping matrix, thereby transforming one-dimensional tilt features into a spherical coordinate mapping matrix, and thus achieving tilt estimation in three-dimensional space. By using a three-dimensional coordinate mapping matrix to constrain the network, reshaping two-dimensional image coordinates into three-dimensional spherical coordinates, and performing upsampling in three-dimensional space, this invention solves the problem of discontinuous learning images and ambiguity, improving image accuracy.

[0032] 3. This invention designs an upsampling method that is more consistent with coordinate mapping, namely a network decoder, which incorporates grouped convolution to prevent the network decoder from confusing the obtained 3D coordinate information, and introduces pyramid-shaped multi-level image constraints to ensure the stability of convergence.

[0033] 4. This invention proposes to perform coordinate constraints in three-dimensional space, reorganizing the original two-dimensional mapping coordinate matrix into a three-dimensional mapping coordinate matrix, which ultimately solves the problem of cracks in the generated images in the past and improves the image quality. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of an upright panoramic image;

[0035] Figure 2 To and Figure 1 Corresponding non-vertical panoramic image illustration;

[0036] Figure 3 This is a flowchart of a deep learning-based method for vertical adjustment of 360-degree panoramic images according to the present invention.

[0037] Figure 4 This is a schematic diagram of the boundary of the expanded panoramic image in step S101 of the deep learning-based 360-degree panoramic image vertical adjustment method of the present invention.

[0038] Figure 5 This is a flowchart of the process of converting the one-dimensional tilt feature into a spherical coordinate mapping matrix in step S102 and upsampling the three-dimensional coordinate mapping matrix in step S103 of the present invention.

[0039] Figure 6 This is a panoramic imaging pattern diagram of a deep learning-based 360-degree panoramic image upright adjustment method according to the present invention.

[0040] Figure 7This is a flowchart illustrating the process of mapping the input non-vertical panoramic image to the upsampled three-dimensional coordinate mapping matrix in step S104 of the present invention, a method for adjusting the upright position of a 360-degree panoramic image based on deep learning.

[0041] Figure 8 This is a schematic diagram of pyramid image constraints for a 360-degree panoramic image upright adjustment method based on deep learning according to the present invention. Detailed Implementation

[0042] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments:

[0043] like Figure 3 As shown, a deep learning-based method for vertical adjustment of 360-degree panoramic images includes the following steps:

[0044] S101: Expand the boundaries of the panoramic image:

[0045] The input non-vertical panoramic image is sliced ​​and stitched using image processing software, extending 32 pixels in all directions to obtain an expanded panoramic image. The size of the input non-vertical panoramic image is 256×512, where the image size refers to the resolution size in pixels.

[0046] Specific expansion methods are as follows: Figure 4 As shown, Figure 4 Image (a) is the unextended image. Figure 4 Image (b) is a schematic diagram showing the extension of the left and right parts. The left part of the cut image is directly translated and spliced ​​onto the image. Figure 4 Similarly, on the right side of (a), the other side is spliced ​​to extend the left and right boundaries; Figure 4 Image (d) is a schematic diagram of vertical extension. The image, after horizontal extension, is stitched together vertically. The top image is sliced, rotated 180°, and then stitched onto the top of the image. Similarly, the bottom extension is performed to complete the vertical extension of the image. After vertical, horizontal, and lateral extensions are completed, the desired result is obtained. Figure 4 The extended image shown in (c) is shown in the middle.

[0047] S102: Transform the one-dimensional tilt feature into a spherical coordinate mapping matrix, and output a three-dimensional coordinate mapping matrix:

[0048] The augmented panoramic image obtained from S101 is input into the encoder. The ConvNext-tiny network within the encoder extracts latent spatial features, which are one-dimensional vectors of length 768. These latent spatial features are then input into the L2M (one-dimensional latent space to three-dimensional coordinate mapping matrix) module within the encoder, as follows: Figure 5As shown in (b), the latent spatial features are integrated through a fully connected layer to output the required tensor with 1536 channels. The latent spatial features are converted into unit spherical coordinates to obtain coordinate mapping estimates and output a three-dimensional coordinate mapping matrix.

[0049] In particular, considering that different non-vertical images, although the scenes are different, have the same tilt angle, then the imaging on panoramic images will have a uniform pattern, such as... Figure 6 As shown, when two vertical images of different scenes are rotated by the same angle (pitch angle 45° and roll angle 45°), similar curved shapes will appear in the images. Based on this characteristic, existing methods use two-dimensional coordinate matrices for mapping. However, this method has a defect in mapping panoramic images. Panoramic images are essentially continuous spaces on both sides. Mapping to a two-dimensional matrix ignores the continuity of the panoramic image. Due to the limitations of two-dimensional projection, a break occurs. During the learning process, the network cannot connect the left and right sides, resulting in large error oscillations at the edges, leading to ambiguity. Points on the leftmost and rightmost sides may appear different. Points are adjacent scenes. Due to the existence of errors, convergence is not possible at the edges, resulting in cracks in the final panoramic image after vertical adjustment. However, the mapping coordinates of the panoramic image are continuous in three-dimensional space and there is no fragmentation problem. This invention uses a three-channel matrix to convert latent features into unit spherical coordinates, reshape the one-dimensional vector into three-dimensional x, y, z coordinates, and normalize the three-dimensional x, y, z coordinates to spherical coordinates to obtain coordinate mapping estimation. The output is a three-dimensional coordinate mapping matrix with a resolution of 16×32 pixels. This module can convert one-dimensional tilt features into a spherical coordinate mapping matrix, thereby realizing tilt estimation in three-dimensional space.

[0050] S103: Obtain the upsampled 3D coordinate mapping matrix:

[0051] In step S102, the tilt feature information (3D coordinate mapping matrix) of the image has been obtained, with a resolution of 16×32 pixels. Since it needs to be mapped to the input non-vertical panoramic image with a resolution of 256×512 pixels, upsampling is required using a decoder. During the upsampling of the coordinate mapping matrix, there is no strict continuity constraint between each coordinate in the matrix and its neighboring coordinates. Figure 6As shown, after correction, point P in (b) will be located at point P' in (a), and its adjacent points will be located at point P” after correction. Therefore, the three-dimensional mapping coordinate matrix obtained in S102 is input into the decoder. After group convolution by the decoder, it is upsampled by four upsampling modules using the PixelShuffle (Sub-Pixel Convolutional Neural Network) method to gradually expand the size of the mapping coordinate matrix and output the four upsampled three-dimensional coordinate mapping matrices.

[0052] Specifically, the 3D coordinate mapping matrix obtained from the encoder is convolved and then upsampled using PixelShuffle. A low-resolution H×W input image is transformed into a high-resolution rH×rW image through Sub-pixel convolution. Considering that the x, y, and z coordinates are also independent, this invention employs grouped convolution during the convolution process to prevent confusion of feature information in the three channels. This ensures that pheromones are generated independently between channels, improving the accuracy of the coordinate information. The specific process is as follows: Figure 5 As shown in (c), after four upsampling modules, a 3D coordinate mapping matrix with the same resolution as the input non-vertical panoramic image is obtained. To ensure the stability of upsampling at each layer, the convolutional kernel size of each layer is adjusted. The first layer's convolutional feature map size is 16×32, and the kernel size is set to 3; the second layer's convolutional feature map size is 32×64, and the kernel size is set to 5; the third layer's convolutional feature map size is 64×128, and the kernel size is set to 7; the fourth layer's convolutional feature map size is 128×256, and the kernel size is set to 9. This keeps the workload of each layer of the network at the same level, and the 3D coordinate mapping matrix obtained by each upsampling module is used as the output of the decoder. Each layer first performs convolution and then upsampling, so the size of the convolutional feature map is one size smaller than the size of the decoder's output image. The size of the convolutional feature map refers to the resolution size, in pixels.

[0053] S104: Map the input non-upright panoramic image to the upsampled 3D coordinate mapping matrix to obtain the upright panoramic image:

[0054] The size of the input non-upright panoramic image is adjusted to correspond to the size of the upsampled three-dimensional coordinate mapping matrix obtained in S103, resulting in an adjusted non-upright panoramic image. The three-dimensional coordinates of the upsampled three-dimensional coordinate mapping matrix are converted into two-dimensional coordinates. Based on the coordinate information provided by the two-dimensional coordinates, the pixel values ​​of the adjusted non-upright panoramic image of the corresponding size are filled into the specified positions to obtain upright panoramic images of different sizes.

[0055] The upsampled 3D coordinate mapping matrix needs to be converted from 3D to 2D coordinates through mathematical operations before image mapping. The specific steps are as follows: First, the 2D coordinates are converted into 3D coordinates. Based on the length and width of the panoramic image, the zenith angle θ and azimuth angle on the corresponding sphere can be mapped. Then by θ i Calculate pixel P(u) P ,v P The orthogonal coordinates (x, y, z) on the unit sphere are given by the following two formulas:

[0056]

[0057] In the above formula, W represents the image width and H represents the image height;

[0058] The expression for converting the three-dimensional coordinates obtained using the inverse functions of (1) and (2) into two-dimensional coordinates is as follows:

[0059]

[0060] In the above expression, W represents the image width, H represents the image height, and θ represents the zenith angle on the sphere corresponding to the three-dimensional coordinates. This represents the azimuth angle on the sphere corresponding to the three-dimensional coordinates.

[0061] From (3) and (4), the three-dimensional coordinates (Δx) estimated by the network can be obtained. P ,Δy P ,Δz P The input image pixel values ​​are then converted into corresponding two-dimensional coordinates P(Δu,Δv). Subsequently, based on the coordinate information provided by the two-dimensional mapping coordinates, the input image pixel values ​​are filled into the specified positions, such as... Figure 7 As shown, the coordinate information of the upright panoramic image obtained by the decoder is mapped to obtain the output upright panoramic image; by adjusting the size of the input non-upright panoramic image, it can be mapped with the three-dimensional coordinate mapping matrices of different sizes obtained by the decoder to obtain output upright panoramic images of different sizes.

[0062] In the upright panoramic image generation network of step S104, in order to ensure that the upright panoramic image generated by the network achieves good results, the present invention uses L1 loss to constrain the network. The L1 loss formula is as follows:

[0063]

[0064] In the above formula, x i Represents the ground truth pixel value from the target upright dataset. This represents the pixel values ​​of the image generated by the network. However, images generated by a single L1 constraint have defects in detail. If image constraints are only added at the end, the network cannot converge. This method uses multi-level image constraints to ensure the stability of network convergence. After resizing the target upright image, image constraints are applied to the output upright panoramic image of the corresponding resolution, as shown in Figure 8. This shape resembles a pyramid and is denoted as the pyramid constraint.

[0065] loss py =loss 32 +loss 64 +loss 128 +loss 256 (6)

[0066] Meanwhile, since this task involves spatial variations, and convolutional networks are inherently difficult to handle spatial variations, simply applying constraints at the network's ends is unlikely to achieve convergence. Therefore, this method employs an intermediate mapping map constraint. Because a two-dimensional mapping map cannot establish a relationship between the left and right sides of an image when reflecting a 360-degree space, this method transforms the two-dimensional coordinates into a three-dimensional coordinate mapping matrix based on a spherical model, thus ensuring spatial continuity. This truth constraint is denoted as the loss. grid ;

[0067] The total network loss is expressed as:

[0068] loss = loss py +loss grid (7)

[0069] In expressions (6) and (7) above, the L1 loss formula is used. py This represents the image loss constraint at different sizes, i.e., loss. 32 loss 64 loss 128 loss 256 Both refer to the L1 loss formula, loss grid represents the truth constraint, and loss represents the total network loss.

[0070] In summary, this invention designs an upsampling method that better conforms to coordinate mapping, namely a network decoder. It incorporates grouped convolution to prevent the network decoder from confusing the obtained 3D coordinate information, and introduces pyramid-style multi-level image constraints to ensure the stability of convergence. By performing coordinate constraints in 3D space, the original 2D mapped coordinate matrix is ​​reorganized into a 3D mapped coordinate matrix, ultimately solving the problem of image cracks in previous technologies and improving image quality.

[0071] The above descriptions are merely embodiments of the present invention, and common knowledge regarding specific technical solutions or characteristics is not elaborated upon here. It should be noted that those skilled in the art can make various modifications and improvements without departing from the technical solutions of the present invention, and these should also be considered within the scope of protection of the present invention. These modifications and improvements will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A method for vertical adjustment of 360-degree panoramic images based on deep learning, characterized in that, Includes the following steps: S101: Expand the boundaries of the panoramic image: The input non-vertical panoramic image is sliced ​​and stitched using image processing software, extending n pixels in all directions to obtain an expanded panoramic image. S102: Transform the one-dimensional tilt feature into a spherical coordinate mapping matrix, and output a three-dimensional coordinate mapping matrix: The augmented panoramic image obtained from S101 is input into the encoder. The deep convolutional neural network in the encoder is used to extract latent spatial features. The latent spatial features are input into the L2M module in the encoder to convert the latent spatial features into unit spherical coordinates, obtain coordinate mapping estimates, and output a three-dimensional coordinate mapping matrix. S103: Obtain the upsampled 3D coordinate mapping matrix: The 3D mapping coordinate matrix obtained in S102 is input into the decoder. After grouping convolution by the decoder, it is upsampled by several upsampling modules in the manner of PixelShuffle, gradually expanding the size of the mapping coordinate matrix, and outputting several upsampled 3D coordinate mapping matrices. S104: Map the input non-upright panoramic image to the upsampled 3D coordinate mapping matrix to obtain the upright panoramic image: The size of the input non-upright panoramic image is adjusted to correspond to the size of the upsampled three-dimensional coordinate mapping matrix obtained in S103, resulting in an adjusted non-upright panoramic image. The three-dimensional coordinates of the upsampled three-dimensional coordinate mapping matrix are converted into two-dimensional coordinates. Based on the coordinate information provided by the two-dimensional coordinates, the pixel values ​​of the adjusted non-upright panoramic image of the corresponding size are filled into the specified positions to obtain upright panoramic images of different sizes.

2. The method for vertical adjustment of 360-degree panoramic images based on deep learning according to claim 1, characterized in that: In S101, the vertical and horizontal extensions are 32 pixels; the input non-vertical panoramic image size is 256×512.

3. The method for vertical adjustment of 360-degree panoramic images based on deep learning according to claim 1, characterized in that: In S102, the deep convolutional neural network is a ConvNext-tiny network, the latent spatial features are one-dimensional vectors of length 768, and the output three-dimensional coordinate mapping matrix has a size of 16×32.

4. The method for vertical adjustment of 360-degree panoramic images based on deep learning according to claim 1, characterized in that: In S103, the 3D mapping coordinate matrix obtained in S102 is subjected to grouped convolution and then upsampled in four layers using the PixelShuffle method. The first convolutional feature map has a size of 16×32 and a kernel size of 3; the second convolutional feature map has a size of 32×64 and a kernel size of 5; the third convolutional feature map has a size of 64×128 and a kernel size of 7; and the fourth convolutional feature map has a size of 128×256 and a kernel size of 9.

5. The method for vertical adjustment of 360-degree panoramic images based on deep learning according to claim 1, characterized in that: In S104, the expression for converting the three-dimensional coordinates of the three-dimensional coordinate mapping matrix obtained in S103, which has the same size as the input non-upright panoramic image, into two-dimensional coordinates is as follows: In the above expression, W represents the image width, H represents the image height, and θ represents the zenith angle on the sphere corresponding to the three-dimensional coordinates. This represents the azimuth angle on the sphere corresponding to the three-dimensional coordinates.

6. The method for vertical adjustment of 360-degree panoramic images based on deep learning according to claim 1, characterized in that: In S104, the upright panoramic image of the corresponding size obtained in S104 and the target upright image after size adjustment are subjected to multi-level image loss constraints. The L1 loss formula is: In the above formula, x i Represents the ground truth pixel values ​​in a dataset of upright target images. Represents the pixel values ​​of an upright panoramic image; The network loss expression is: loss py =loss 32 +loss 64 +loss 128 +loss 256 loss=loss py +loss grid In the above expressions, the L1 loss formula is used. py This represents the image loss constraint at different sizes, loss grid represents the truth constraint, and loss represents the total network loss.

Images