360-degree panoramic image upright adjustment method based on deep learning

Abstract

The invention belongs to the technical field of panoramic image upright adjustment, and discloses a 360-degree panoramic image upright adjustment method based on deep learning, and the method comprises the steps: S101, expanding the boundary of a panoramic image; s102, converting the one-dimensional inclination characteristics into a spherical coordinate mapping matrix, and outputting a three-dimensional coordinate mapping matrix; s103, obtaining a three-dimensional coordinate mapping matrix after up-sampling; s104, performing mapping on an input non-upright panoramic image and the three-dimensional coordinate mapping matrix after up-sampling to obtain an upright panoramic image; according to the method, the non-upright image is input through the end-to-end neural network, and the adjusted panoramic upright image is directly output, so that the problem that in the prior art, due to limitation of two-dimensional projection, splitting is caused, convergence cannot be achieved at the edge, and consequently the panoramic image subjected to upright adjustment cracks is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of upright adjustment of panoramic images, and in particular to a method for upright adjustment of a 360-degree panoramic image based on deep learning. Background Art

[0002] Nowadays, commercial 360-degree panoramic cameras are common. Unlike ordinary cameras with limited field of view, this camera can capture the surrounding 360-degree scene in all directions and finally form a 360-degree panoramic image. Figure 1 As shown in the figure, the scene is captured by the camera in an upright state. Although the upper and lower ends of the image will be distorted when the three-dimensional space is projected onto a two-dimensional plane image, the scene is easy to understand as a whole. However, in actual scenes, it is difficult for users to ensure that the camera is absolutely upright, so a tilted camera will produce a non-upright 360-degree panoramic image, that is, Figure 2 As shown, it can be seen that non-upright images are more difficult to understand than upright images.

[0003] The traditional method of uprighting panoramic images is to first estimate the tilt angle, and then compensate for the estimated tilt by resampling the non-upright image and generate an upright image. This calculation involves a spherical mathematical model, which is complex and time-consuming. In addition, there is a certain loss of accuracy during the resampling process, resulting in jagged images. In recent years, a better approach is to use deep learning networks to treat the panoramic image uprighting task as an end-to-end two-dimensional image, but existing approaches often ignore its three-dimensional spatial continuity properties, such as Figure 1 As shown, the left and right ends of the image should be continuous scenes in the three-dimensional scene, but the two-dimensional image separates them, resulting in poor quality of the generated image and cracks. Summary of the invention

[0004] The present invention aims to provide a 360-degree panoramic image upright adjustment method based on deep learning, which inputs a non-upright image through an end-to-end neural network and directly outputs an adjusted panoramic upright image, so as to solve the problem in the prior art that due to the limitation of two-dimensional projection, splitting is caused and convergence at the edges cannot be achieved, resulting in cracks in the panoramic image after upright adjustment.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A 360-degree panoramic image upright adjustment method based on deep learning comprises the following steps:

[0007] S101: Expanding the border of the panoramic image:

[0008] Use image processing software to slice and splice the input non-erect panoramic image, extending it by n pixels up, down, left, and right, to obtain an expanded panoramic image;

[0009] S102: Convert the one-dimensional tilt feature into a spherical coordinate mapping matrix, and output a three-dimensional coordinate mapping matrix:

[0010] Input the expanded panoramic image obtained in S101 into the encoder, use the deep convolutional neural network in the encoder to extract the latent space features, input the latent space features into the L2M module in the encoder, convert the latent space features into unit spherical coordinates, obtain the coordinate mapping estimate, and output the three-dimensional coordinate mapping matrix;

[0011] S103: Obtain the upsampled three-dimensional coordinate mapping matrix:

[0012] The three-dimensional mapping coordinate matrix obtained in S102 is input into the decoder, and after performing group convolution by the decoder, up-sampled by several up-sampling modules in a PixelShuffle manner, the size of the mapping coordinate matrix is ​​gradually expanded, and several up-sampled three-dimensional coordinate mapping matrices are output;

[0013] S104: Mapping the input non-upright panoramic image and the upsampled three-dimensional coordinate mapping matrix to obtain an upright panoramic image:

[0014] The size of the input non-upright panoramic image is adjusted to a size corresponding to the upsampled three-dimensional coordinate mapping matrix obtained in S103 to obtain an adjusted non-upright panoramic image, and the three-dimensional coordinates of the upsampled three-dimensional coordinate mapping matrix are converted into two-dimensional coordinates. The pixel values ​​of the adjusted non-upright panoramic image of the corresponding size are filled into the specified positions according to the coordinate information provided by the two-dimensional coordinates to obtain upright panoramic images of different sizes.

[0015] Furthermore, in S101 , the top, bottom, left, and right sides are all extended by 32 pixels; the input non-erect panoramic image size is 256×512.

[0016] Further, in S102, the deep convolutional neural network is a ConvNext-tiny network, the latent space feature is a one-dimensional vector with a length of 768, and the size of the output three-dimensional coordinate mapping matrix is ​​16×32.

[0017] Furthermore, in S103, the three-dimensional mapping coordinate matrix obtained in S102 is subjected to group convolution and then up-sampled in four layers using PixelShuffle. The size of the first convolution feature map is 16×32, and the convolution kernel size is 3; the size of the second convolution feature map is 32×64, and the convolution kernel size is 5; the size of the third convolution feature map is 64×128, and the convolution kernel size is 7; the size of the fourth convolution feature map is 128×256, and the convolution kernel size is 9.

[0018] Further, in S104, the three-dimensional coordinates of the three-dimensional coordinate mapping matrix obtained in S103 and having the same size as the input non-erect panoramic image are converted into two-dimensional coordinates by the expression:

[0019]

[0020] In the above expression, W represents the image width, H represents the image height, and θ represents the zenith angle on the spherical surface corresponding to the three-dimensional coordinates. Indicates the azimuth of the three-dimensional coordinates on the spherical surface.

[0021] Further, in S104, the upright panoramic image of the corresponding size obtained in S104 and the resized target upright image are subjected to multi-level image loss constraints;

[0022] The L1 loss formula is:

[0023]

[0024] In the above formula, x i represents the true pixel value of the target upright image dataset, Represents the pixel value of the upright panoramic image;

[0025] The network loss expression is:

[0026] loss py =loss 32 +loss 64 +loss 128 +loss 256

[0027] loss=loss py +loss grid

[0028] In the above expressions, the L1 loss formula is used. py Represents the image loss constraints at different sizes, loss grid Represents the truth constraint, and loss represents the total loss of the network.

[0029] Beneficial effects of the technical solution:

[0030] 1. Since the two-dimensional panoramic image is discontinuous, but its left and right boundaries are essentially continuous, in order to compensate for the continuity problem, the present invention extends the image, that is, by cutting and rotating the non-upright image part to expand the size of the original non-upright panoramic image, correctly extending the border part of the panoramic image, and avoiding the convolution network from finding the corresponding adjacent points at the image boundary, so that the network can extract more continuous features at the boundary of the non-upright panoramic image, and improve the accuracy of the image mapping constraint;

[0031] 2. The present invention uses a three-channel matrix to convert potential features into unit spherical coordinates, reshapes one-dimensional vectors into x, y, z three-dimensional coordinates, and normalizes x, y, z three-dimensional coordinates to spherical coordinates to obtain coordinate mapping estimates, and outputs a three-dimensional coordinate mapping matrix with a size of 16×32, thereby converting one-dimensional tilt features into a spherical coordinate mapping matrix, and then realizing tilt estimation in three-dimensional space; the three-dimensional coordinate mapping matrix is ​​used to constrain the network, reshape the two-dimensional image coordinates into three-dimensional spherical coordinates, and upsample in three-dimensional space, which solves the problem of discontinuity and ambiguity in network learning images and improves image accuracy;

[0032] 3. The present invention designs an upsampling method that is more in line with coordinate mapping, namely, a network decoder, in which grouped convolution is added so that the network decoder does not confuse the obtained three-dimensional coordinate information, and a pyramid-like multi-level image constraint is introduced to ensure the smoothness of convergence.

[0033] 4. The present invention proposes to perform coordinate constraints in three-dimensional space, and reorganize the original two-dimensional mapping coordinate matrix into a three-dimensional mapping coordinate matrix, which ultimately solves the problem of cracks in the generated images in the past technology and improves the image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a schematic diagram of an upright panoramic image;

[0035] Figure 2 For Figure 1 Schematic diagram of the corresponding non-erect panoramic image;

[0036] Figure 3 This is a flow chart of a 360-degree panoramic image upright adjustment method based on deep learning of the present invention;

[0037] Figure 4 A schematic diagram of the boundary of an expanded panoramic image in step S101 of a method for upright adjustment of a 360-degree panoramic image based on deep learning according to the present invention;

[0038] Figure 5 The flowchart of the present invention is as follows: Step S102 converts the one-dimensional tilt feature into a spherical coordinate mapping matrix and Step S103 upsamples the three-dimensional coordinate mapping matrix;

[0039] Figure 6 A panoramic imaging regularity diagram of a 360-degree panoramic image upright adjustment method based on deep learning of the present invention;

[0040] Figure 7This is a flow chart of mapping the input non-erect panoramic image and the up-sampled three-dimensional coordinate mapping matrix in step S104 of a 360-degree panoramic image upright adjustment method based on deep learning of the present invention;

[0041] Figure 8 A schematic diagram of pyramid image constraints of a 360-degree panoramic image upright adjustment method based on deep learning of the present invention; DETAILED DESCRIPTION

[0042] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments:

[0043] like Figure 3 As shown, a 360-degree panoramic image upright adjustment method based on deep learning includes the following steps:

[0044] S101: Expanding the border of the panoramic image:

[0045] Use image processing software to slice and stitch the input non-erect panoramic image, extending it by 32 pixels in the top, bottom, left, and right directions to obtain an expanded panoramic image, where the size of the input non-erect panoramic image is 256×512, where the image size refers to the resolution size in pixels;

[0046] Specific expansion methods such as Figure 4 As shown, Figure 4 (a) is the image without stretching. Figure 4 (b) is a schematic diagram of the extension of the left and right parts. The left part of the cut image is directly translated and spliced ​​on Figure 4 On the right side of (a), the other side is spliced ​​in the same way to extend the left and right boundaries; Figure 4 (d) is a schematic diagram of the up and down extension. The image that has been extended left and right is spliced ​​up and down. The top image is sliced ​​and rotated 180° to splice it at the top of the image. Similarly, the bottom extension is performed to complete the up and down extension of the image. After the up, down, left and right extension is completed, Figure 4 The extended image shown in (c);

[0047] S102: Convert the one-dimensional tilt feature into a spherical coordinate mapping matrix, and output a three-dimensional coordinate mapping matrix:

[0048] The expanded panoramic image obtained in S101 is input into the encoder, and the ConvNext-tiny network in the encoder is used to extract the latent space feature, which is a one-dimensional vector of length 768. The latent space feature is input into the L2M (one-dimensional latent space to three-dimensional coordinate mapping matrix) module in the encoder, as shown in Figure 5As shown in (b), the latent space features are integrated through the fully connected layer, and the required tensor is output. The number of channels is 1536. The latent space features are converted into unit spherical coordinates to obtain the coordinate mapping estimate and output the three-dimensional coordinate mapping matrix.

[0049] Among them, considering different non-upright images, although the scenes are different, but the tilt angle is the same, then the imaging on the panoramic image will have a unified rule, such as Figure 6 As shown in the figure, when two upright images of different scenes are rotated at the same angle (pitch angle 45° and roll angle 45°), similar curve shapes will be shown in the images; based on this feature, the existing method uses a two-dimensional coordinate matrix for mapping, but this method has defects in the mapping of panoramic images. In essence, the left and right sides of a panoramic image should be continuous spaces. Mapping to a two-dimensional matrix ignores the continuity of the panoramic image. Due to the limitation of two-dimensional projection, the network cannot connect the left and right sides during the learning process. The error oscillation span is too large at the edge, resulting in ambiguity. The leftmost point and the rightmost point are not the same. The points are adjacent scenes. Due to the existence of errors, they cannot converge at the edges, resulting in cracks in the final mapped panoramic image after upright adjustment. However, the mapping coordinates of the panoramic image are continuous in three-dimensional space and there is no splitting problem. The present invention uses a three-channel matrix to convert potential features into unit spherical coordinates, reshape the one-dimensional vector into x, y, z three-dimensional coordinates, and normalize the x, y, z three-dimensional coordinates to spherical coordinates to obtain coordinate mapping estimation, and output a three-dimensional coordinate mapping matrix with a resolution of 16×32 pixels. This module can convert one-dimensional tilt features into a spherical coordinate mapping matrix, thereby realizing tilt estimation in three-dimensional space;

[0050] S103: Obtain the upsampled three-dimensional coordinate mapping matrix:

[0051] In step S102, the image tilt feature information (3D coordinate mapping matrix) has been obtained, and the resolution size is 16×32 pixels. Since it needs to be mapped with the input non-erect panoramic image with a resolution size of 256×512 pixels, it is necessary to use the decoder for upsampling. In the process of upsampling the coordinate mapping matrix, each coordinate in the matrix has no strict continuity constraints with its adjacent coordinates, such as Figure 6As shown, after correction, point P in (b) will be at point P' in (a), and its adjacent point will be at point P" after correction; therefore, the three-dimensional mapping coordinate matrix obtained in S102 is input into the decoder, and after group convolution by the decoder, up-sampled by four up-sampling modules in the manner of PixelShuffle (Sub-Pixel Convolutional Neural Network, pixel reorganization), and the size of the mapping coordinate matrix is ​​gradually expanded, and four up-sampled three-dimensional coordinate mapping matrices are output;

[0052] Specifically, the three-dimensional coordinate mapping matrix obtained by the encoder is upsampled by PixelShuffle after convolution, and a H×W low-resolution input image is converted into an rH×rW high-resolution image through Sub-pixel convolution operation. Considering that the three coordinates xyz are also independent of each other, in order to prevent the feature information of the three channels from being confused during the convolution process, the present invention adopts group convolution to ensure that the pheromones between each other are generated independently, thereby improving the accuracy of the coordinate information. The specific process is as follows: Figure 5 As shown in (c); after four upsampling modules, a three-dimensional coordinate mapping matrix with the same resolution as the input non-upright panoramic image is obtained. In order to ensure the stability of upsampling at each layer, the present invention also adjusts the convolution kernel size of each layer. The size of the first convolution feature map is 16×32, and the convolution kernel size is set to 3 at this time; the size of the second convolution feature map is 32×64, and the convolution kernel size is set to 5; the size of the third convolution feature map is 64×128, and the convolution kernel size is set to 7; the size of the fourth convolution feature map is 128×256, and the convolution kernel size is set to 9, so that the workload of each layer of the network is maintained at the same level, and the three-dimensional coordinate mapping matrix obtained by each upsampling module is used as the output of the decoder; wherein, each layer is convolved first and then upsampled, so when convolving, the size is one size smaller than the size of the decoder output image; the convolution feature map size refers to the resolution size, in pixels.

[0053] S104: Mapping the input non-upright panoramic image and the upsampled three-dimensional coordinate mapping matrix to obtain an upright panoramic image:

[0054] The size of the input non-erect panoramic image is adjusted to a size corresponding to the up-sampled three-dimensional coordinate mapping matrix obtained in S103 to obtain an adjusted non-erect panoramic image, and the three-dimensional coordinates of the up-sampled three-dimensional coordinate mapping matrix are converted into two-dimensional coordinates, and the pixel values ​​of the adjusted non-erect panoramic image of the corresponding size are filled into the specified position according to the coordinate information provided by the two-dimensional coordinates to obtain upright panoramic images of different sizes;

[0055] Among them, the three-dimensional coordinate mapping matrix obtained after upsampling needs to be converted into two-dimensional coordinates through mathematical operations before image mapping. The specific steps are as follows: first, the two-dimensional coordinates are converted into three-dimensional coordinates. According to the length and width of the panorama, the zenith angle θ and azimuth angle on the corresponding spherical surface can be mapped Then by θ i Calculate the pixel point P(u P ,v P ) on the orthogonal coordinates (x, y, z) on the unit sphere, the two formulas are as follows:

[0056]

[0057] In the above formula, W represents the image width and H represents the image height;

[0058] The expression for converting the three-dimensional coordinates obtained by using the inverse functions of (1) and (2) into two-dimensional coordinates is:

[0059]

[0060] In the above expression, W represents the image width, H represents the image height, and θ represents the zenith angle on the spherical surface corresponding to the three-dimensional coordinates. Indicates the azimuth angle on the spherical surface corresponding to the three-dimensional coordinates;

[0061] According to (3) and (4), the three-dimensional coordinate point (Δx P ,Δy P ,Δz P ), and then the input image pixel value is filled into the specified position according to the coordinate information provided by the two-dimensional mapping coordinates, such as Figure 7 As shown, the coordinate information of the upright panoramic image obtained by the decoder is mapped to obtain an output upright panoramic image; the input non-upright panoramic image can be resized and mapped with the three-dimensional coordinate mapping matrices of different sizes obtained by the decoder after sampling to obtain output upright panoramic images of different sizes;

[0062] In the upright panoramic image generation network of step S104, in order to achieve a good effect of the upright panoramic image generated by the network, the present invention uses L1 loss to constrain the network, and the L1 loss formula is:

[0063]

[0064] In the above formula, x i represents the true pixel value from the target upright dataset, Represents the pixel value of the image generated by the network; however, the image generated by a single L1 constraint has defects in details. If only the image constraint is added at the end, the network cannot converge. This method uses multi-level image constraints to ensure the stability of network convergence. After the target upright image is resized, it is constrained with the output upright panoramic image of the corresponding resolution, as shown in 8. It is pyramid-like and is recorded as a pyramid constraint.

[0065] loss py =loss 32 +loss 64 +loss 128 +loss 256 (6)

[0066] At the same time, since this task involves spatial changes, and convolutional networks are inherently difficult to handle spatial changes, it is difficult to converge the network by only constraining the network end. Therefore, this method uses an intermediate mapping constraint. Because the two-dimensional mapping cannot establish a correlation between the left and right sides of the image when reflecting a 360-degree space, this method converts the two-dimensional coordinates into a three-dimensional coordinate mapping matrix based on the spherical model, thus ensuring spatial continuity. This part of the truth constraint is recorded as loss grid ;

[0067] The total network loss is expressed as:

[0068] loss=loss py +loss grid (7)

[0069] In the above expressions (6) and (7), the L1 loss formula is used. py Represents the image loss constraints at different sizes, i.e., loss 32 、loss 64 、loss 128 、loss 256 Both refer to the L1 loss formula, loss grid Represents the truth constraint, and loss represents the total loss of the network.

[0070] In summary, the present invention designs an upsampling method that is more in line with coordinate mapping, namely, a network decoder, in which grouped convolution is added so that the network decoder does not confuse the obtained three-dimensional coordinate information, and introduces a pyramid-like multi-level image constraint to ensure the smoothness of convergence; coordinate constraints are performed in three-dimensional space, and the original two-dimensional mapping coordinate matrix is ​​reorganized into a three-dimensional mapping coordinate matrix, which ultimately solves the problem of cracks in the generated image in past technologies and improves image quality.

[0071] The above is only an embodiment of the present invention, and the common knowledge such as the known specific technical solutions or characteristics in the solution is not described in detail here. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the technical solution of the present invention, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A 360-degree panoramic image upright adjustment method based on deep learning, characterized in that: The following steps are involved: S101: Expanding the border of the panoramic image: Use image processing software to slice and splice the input non-erect panoramic image, extending it by n pixels up, down, left, and right, to obtain an expanded panoramic image; S102: Convert the one-dimensional tilt feature into a spherical coordinate mapping matrix, and output a three-dimensional coordinate mapping matrix: Input the expanded panoramic image obtained in S101 into the encoder, use the deep convolutional neural network in the encoder to extract the latent space features, input the latent space features into the L2M module in the encoder, convert the latent space features into unit spherical coordinates, obtain the coordinate mapping estimate, and output the three-dimensional coordinate mapping matrix; S103: Obtain the upsampled three-dimensional coordinate mapping matrix: The three-dimensional mapping coordinate matrix obtained in S102 is input into the decoder, and after performing group convolution by the decoder, up-sampled by several up-sampling modules in a PixelShuffle manner, the size of the mapping coordinate matrix is ​​gradually expanded, and several up-sampled three-dimensional coordinate mapping matrices are output; S104: Mapping the input non-upright panoramic image and the upsampled three-dimensional coordinate mapping matrix to obtain an upright panoramic image: The size of the input non-upright panoramic image is adjusted to a size corresponding to the upsampled three-dimensional coordinate mapping matrix obtained in S103 to obtain an adjusted non-upright panoramic image, and the three-dimensional coordinates of the upsampled three-dimensional coordinate mapping matrix are converted into two-dimensional coordinates. The pixel values ​​of the adjusted non-upright panoramic image of the corresponding size are filled into the specified positions according to the coordinate information provided by the two-dimensional coordinates to obtain upright panoramic images of different sizes.

2. The method for upright adjustment of 360-degree panoramic images based on deep learning according to claim 1, characterized in that: In S101, the top, bottom, left, and right sides are all extended by 32 pixels; the input non-erect panoramic image has a size of 256×512.

3. The method for upright adjustment of 360-degree panoramic images based on deep learning according to claim 1, characterized in that: In S102, the deep convolutional neural network is a ConvNext-tiny network, the latent space feature is a one-dimensional vector with a length of 768, and the size of the output three-dimensional coordinate mapping matrix is ​​16×32.

4. The method for upright adjustment of 360-degree panoramic images based on deep learning according to claim 1, characterized in that: In S103, the three-dimensional mapping coordinate matrix obtained in S102 is subjected to group convolution and then up-sampled in four layers using PixelShuffle. The size of the first convolution feature map is 16×32, and the convolution kernel size is 3; the size of the second convolution feature map is 32×64, and the convolution kernel size is 5; the size of the third convolution feature map is 64×128, and the convolution kernel size is 7; the size of the fourth convolution feature map is 128×256, and the convolution kernel size is 9.

5. The method for upright adjustment of 360-degree panoramic images based on deep learning according to claim 1, characterized in that: In S104, the three-dimensional coordinates of the three-dimensional coordinate mapping matrix obtained in S103 and consistent with the input non-erect panoramic image size are converted into two-dimensional coordinates by the expression: In the above expression, W represents the image width, H represents the image height, and θ represents the zenith angle on the spherical surface corresponding to the three-dimensional coordinates. Indicates the azimuth of the three-dimensional coordinates on the spherical surface.

6. The method for upright adjustment of 360-degree panoramic images based on deep learning according to claim 1, characterized in that: In S104, the upright panoramic image of the corresponding size obtained in S104 and the resized target upright image are subjected to multi-level image loss constraints; The L1 loss formula is: In the above formula, x i represents the true pixel value of the target upright image dataset, Represents the pixel value of the upright panoramic image; The network loss expression is: loss py =loss 32 +loss 64 +loss 128 +loss 256 loss=loss py +loss grid In the above expressions, the L1 loss formula is used. py Represents the image loss constraints at different sizes, loss grid Represents the truth constraint, and loss represents the total loss of the network.

Images