A panoramic image style transfer method, system and storage medium

Abstract

The application discloses a panoramic image style migration method and system and a storage medium, and comprises the following steps: performing multi-view projection on a panoramic image to obtain a plurality of equirectangular projection images; performing cube projection on each equirectangular projection image; and then, according to a specific sequence, splicing different cube projection images into a target image; constructing a panoramic image style migration network model, wherein the network model comprises a multi-scale attention content encoder, a style encoder, a global feature sharing module and a decoder; restoring and performing equirectangular projection on the stylized target image to obtain a plurality of stylized panoramic images of different views; then, performing multi-view fusion; and finally, obtaining a finished product. The embodiment can process panoramic images of any resolution, effectively solves the problems of distortion of the pole position, inconsistent styles of the boundary position and the like after panoramic image style migration, and improves the migration quality of the panoramic image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a panoramic image style transfer method, system, and storage medium. Background Technology

[0002] Image style transfer is the process of transforming a content image based on the feature information of a style image. Essentially, it is about transferring feature distribution, ultimately achieving a visual effect that combines a new style with the original content. This allows the synthesized image to retain the structure, shape, and other information of the content image while also incorporating the color, texture, and other information of the style image.

[0003] With the development of deep learning technology, image style transfer methods have made great progress, achieving high-quality stylization effects. However, because panoramic images have a completely different structure from ordinary planar graphics, directly processing panoramic images using existing image style transfer methods will result in problems such as extremum distortion, inconsistent edges, and poor stylization of local details in the stylized panoramic images. Furthermore, due to current limitations in computing resources and memory, ordinary image style transfer methods also struggle to handle high-resolution panoramic images. Summary of the Invention

[0004] To address the issue of unsatisfactory results in panoramic image processing using existing image style transfer methods, this invention provides a panoramic image style transfer method, system, and storage medium. This invention not only eliminates extreme distortions in stylized panoramic images and maintains visual consistency in edge style, but also generates satisfactory stylization effects, enhancing the immersive experience of panoramic images.

[0005] The present invention adopts the following technical solution:

[0006] A panoramic image style transfer method, including

[0007] Multi-view projection is performed on the original panoramic image to obtain panoramic images from different perspectives;

[0008] Cube projection is performed on panoramic images from different perspectives to obtain multiple cube projection images, which are then combined into a single target image in the order of reconstruction.

[0009] The target image is input into the panoramic image style transfer network to obtain a stylized target image; the panoramic image style transfer network includes: a multi-scale attention content encoder, a style encoder, a global feature sharing module, and a decoder;

[0010] The stylized target image is segmented, and the resulting cube projection is subjected to equal rectangular projection to obtain multiple stylized panoramic images from different perspectives.

[0011] Stylized panoramic images from different perspectives are fused to obtain stylized panoramic images.

[0012] Furthermore, the step of performing multi-view projection on the original panoramic image to obtain panoramic images from different perspectives includes:

[0013] A three-dimensional cylindrical coordinate system is established with the user's viewing position as the center, and the 360-degree panoramic image presented to the user under the head-mounted display is projected onto the surface of the sphere.

[0014] Based on the characteristics of the panoramic image content, the panoramic image is rotated multiple times. In the horizontal direction, the content at the cut edge is rotated to the center area, and then a rectangular projection is performed to obtain multiple two-dimensional panoramic images from different perspectives.

[0015] Furthermore, the restoration order is a splicing method that combines pixels in a way that achieves maximum continuity.

[0016] Furthermore, the target image is input into a panoramic image style transfer network to obtain a stylized target image, specifically:

[0017] The target image is input into a multi-scale attention-based content encoder to obtain content features;

[0018] The style image is input into the style encoder to obtain style features;

[0019] The content features and style features are input into the global feature sharing module, and the decoder is connected to its output to obtain the stylized target image.

[0020] Furthermore, the multi-view fusion includes:

[0021] Pixel-by-pixel linear overlay of stylized panoramic images from multiple different perspectives;

[0022] During the overlay process, the weights of the panoramic images from each viewpoint are averaged to obtain the final stylized panoramic image.

[0023] Furthermore, the panoramic image style transfer network specifically comprises:

[0024] The multi-scale attention content encoder consists of several convolutional layers, aggregation blocks, and attention modules. It is used to stitch content feature maps of different levels in the channel direction, extract important information in the spatial and channel dimensions, and perform multi-scale attention feature fusion.

[0025] A style encoder is used to extract style information from a style image;

[0026] The global feature sharing module, including an adaptive decomposition module, a style unification module, and a combination module, is used to segment the target image into multiple sub-images for stylization and to stitch the multiple stylized sub-images together to form a stylized target image.

[0027] A decoder is used to convert a stylized target image from the feature map domain to the image domain.

[0028] Furthermore, the global feature sharing module obtains stylized sub-images with consistent visual effects by sharing the global style features of the target image thumbnail during the stylization sub-image process.

[0029] Furthermore, the rotation is a vertical rotation.

[0030] A system for panoramic image style transfer, the system comprising:

[0031] Multi-view rotation module: Projects the original panoramic image from multiple perspectives to obtain panoramic images from different viewpoints;

[0032] Cube Combination Module: Performs cube projection on panoramic images from different perspectives to obtain multiple cube projection images, and combines them into a target image in the restoration order;

[0033] Style transfer module: Inputs the target image into the panoramic image style transfer network to obtain a stylized target image; the panoramic image style transfer network includes: a multi-scale attention content encoder, a style encoder, a global feature sharing module, and a decoder;

[0034] Panoramic Image Restoration Module: The stylized target image is segmented, and the resulting cube projection image is subjected to equal rectangular projection to obtain multiple stylized panoramic images from different perspectives;

[0035] Multi-view fusion module: Performs multi-view fusion on stylized panoramic images from different perspectives to obtain stylized panoramic images.

[0036] A computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the panoramic image style transfer processing method.

[0037] The beneficial effects of this invention are as follows:

[0038] (1) This invention is aimed at the task of style transfer of panoramic images. By combining multi-view projection and cube projection on panoramic images, the distortion at the extreme points of the stylized panoramic image and the style discontinuity at the edges are eliminated, giving the viewer a deep sense of immersion.

[0039] (2) This invention proposes a global feature sharing method to improve the resolution of processable panoramic images, constrain decoder training, and make the style of each sub-image after segmentation visually consistent, thereby achieving good style transfer.

[0040] (3) This invention proposes a multi-scale attention content encoder, which extracts and fuses important information in the spatial and channel dimensions of content image features at different scales, thereby enhancing the feature extraction effect on the details of panoramic images. Attached Figure Description

[0041] Figure 1 This is a flowchart of the process of this invention;

[0042] Figure 2 This is a schematic diagram of the rectangular projection method for panoramic images in this invention;

[0043] Figure 3 This is a schematic diagram of the panoramic image cube projection method in this invention;

[0044] Figure 4 This is a schematic diagram of the structure of the panoramic image style transfer network model of the present invention;

[0045] Figure 5 This is a schematic diagram of the structure of the multi-scale attention content encoder network model of the present invention.

[0046] Figure 6 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0047] The present invention will be further described below with reference to the accompanying drawings and embodiments, including various details of the embodiments to aid understanding. These details should be considered merely exemplary and not as a basis for limiting the invention. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are simplified in the following description.

[0048] Example 1

[0049] Figure 1 An exemplary flow diagram of an embodiment of the panoramic image style transfer method obtained according to the present invention is shown. The above-described panoramic image style transfer method includes the following steps:

[0050] S1 performs multi-view projection on the original panoramic image to obtain panoramic images from different perspectives;

[0051] Currently, panoramic images used in VR devices are projected in three dimensions, with the most common two-dimensional planar projection method being rectangular projection. One difference between rectangular images and traditional images is that the pixels at their left and right edges are continuous in content. If existing style transfer methods are directly applied to rectangular images, the left and right edges will have different styles. Therefore, to alleviate the inconsistency in style between the left and right edges, this invention designs a multi-view projection operation. For example... Figure 2 As shown, let the radius of the sphere be r, and the longitude coordinates be r. The original longitude coordinates are Let the latitude coordinate be θ, the original latitude coordinate be θ0, and x and y be the horizontal and vertical coordinates of the rectangular image in the planar coordinate system, respectively. The calculation method for projecting the panoramic image from the cylindrical coordinate system to the planar coordinate system is as follows:

[0052]

[0053] y = r(θ - θ0)

[0054] Furthermore, with the user's position as the center, the sphere is rotated N times at different angles along the vertical axis, rotating the content from the cut-off edges on both sides to the central view area, ensuring that the content of each area has appeared in the central view area. Each rotation involves a rectangular projection, resulting in a total of N two-dimensional panoramic images from different perspectives.

[0055] S2 performs cube projection on panoramic images from different perspectives to obtain N cube projection images, which are then combined into a target image in the order of restoration.

[0056] Because the field of view of a rectangular image is much larger than that of a two-dimensional image, the content near the poles of the image is severely distorted. Existing common style transfer methods do not take this into account, causing the style in this region to greatly affect the original content, resulting in pole distortion when projected back into a spherical panoramic image. To address this problem, this invention combines the advantages and characteristics of both methods, integrating the cubic projection of the spherical panoramic image into the rectangular projection, thus solving the pole distortion problem. Figure 3 As shown, cube projection of a spherical panoramic image is similar to the decomposition of panoramic image content. First, a cube of appropriate size is added outside the sphere, with the cube circumscribed in the sphere. Then, using a linear mapping method, objects on the entire surface of the sphere are mapped onto the six faces of the cube: front, right, back, left, top, and bottom. Each face can be treated as a separate image.

[0057] Specifically, panoramic images from different perspectives are directly converted into N cube projection images, and then combined into a target image by stitching them together in a way that maximizes the continuity of pixels.

[0058] The restoration order combination is specifically a method of maximizing continuity, which involves combining the six faces of a cube into an image, resulting in a pixel-continuous image.

[0059] S3 inputs the target image into the panoramic image style transfer network to obtain the stylized target image;

[0060] Because panoramic images produce the same visual effect as regular images in all directions, their resolution is extremely high (e.g., 10000*5000 pixels). Conventional image style transfer methods generally only process small images (e.g., 500*500 pixels) and cannot perform style transfer on ultra-high resolution panoramic images. Furthermore, high resolution means the network needs strong detail capture capabilities to achieve better visual results. Therefore, this invention designs a high-resolution panoramic image style transfer network with good detail effects. The panoramic image style transfer network is as follows... Figure 4 As shown, it includes: a multi-scale attention content encoder, a style encoder, a global feature sharing module, and a decoder;

[0061] Existing style transfer methods typically use the VGG19 network directly, which fails to capture image details effectively, leading to style distortion in these areas. Therefore, this invention designs a multi-scale attention-based content encoder to improve the network's ability to extract detailed features. (See also...) Figure 5 The multi-scale attention content encoder consists of several convolutional layers (each containing a 3*3 two-dimensional convolution and a ReLU activation function), aggregation blocks (1*1 two-dimensional convolutions), and attention modules (composed of spatial attention modules and channel attention modules connected in series) for multi-scale content features.

[0062] Specifically, after inputting the target image, the multi-scale attention content encoder concatenates feature maps of different levels along the channel direction, uses upsampling to unify the size of feature maps of different levels, and then concatenates the feature maps of the same size along the channel dimension, finally obtaining a size of H*W*(C1+C2+...+C n The target image content features are combined with the low-level geometric information representation and the high-level semantic information representation. Then, the features are extracted and fused in the attention module to extract and fuse important information in the spatial and channel dimensions, and output multi-scale features with strong information expression capabilities.

[0063] The style encoder is based on the VGG19 network and includes several convolutional layers (each layer contains a 3*3 two-dimensional convolution and a ReLU activation function) to extract style information from the style image.

[0064] To achieve style transfer of high-resolution panoramic images with limited memory, a global feature sharing module is proposed. This module consists of an adaptive decomposition module, a style unification module, and a combination module, and is used to perform target image stylization.

[0065] Specifically, the adaptive decomposition module takes the target image feature map from the previous layer as input and adaptively segments it into multiple computer-processable sub-image feature maps based on the size of the input feature map. The style consistency module takes the multiple sub-image feature maps and style image feature maps from the previous layer as input. By sharing the content features of the target image thumbnail and stylizing them using the same style features, a visually consistent stylization effect is achieved across all sub-images. Finally, the combination module sequentially concatenates the multiple stylized sub-images and inputs the stylized target image.

[0066] Furthermore, the target image thumbnail refers to a downsampling operation performed on the target image, reducing its resolution to the same level as the sub-image, so that the content features of the target image can be removed from each sub-image in the stylization consistency module.

[0067] The decoder includes several adaptive instance normalization layers, upsampling layers, and residual blocks. Through the convolutional layer at the end, the stylized target image is transformed from the feature map domain to the image domain, thus completing the stylization process.

[0068] S4 segments the stylized target image and performs an equal rectangular projection on the resulting cube projection image to obtain N stylized panoramic images from different perspectives.

[0069] S4 is the reverse operation of S2; the principle is the same, but the operation steps are reversed. Both must be used simultaneously to solve the extremum distortion problem caused by style transfer of panoramic images. Specifically, the input stylized target image is cut according to different cube projection planes. Each stylized panoramic image from each viewpoint yields six equal-rectangular projection images (the front, right, back, left, top, and bottom projections of the cube). Then, equal-rectangular projections are performed on the equal-rectangular projection images from each viewpoint to obtain N stylized panoramic images from different viewpoints.

[0070] S5 performs multi-view fusion on stylized panoramic images from different perspectives to obtain a stylized panoramic image. Similarly, S5 is the reverse operation of S1; the principle is the same, but the operation steps are reversed. Both must be used simultaneously to solve the problem of discontinuous edge styles after style transfer of panoramic images. The multi-view fusion method refers to linearly superimposing N stylized panoramic images from different perspectives pixel by pixel, where the weight of each image is 1 / N, to obtain the final stylized panoramic image.

[0071] Example 2

[0072] refer to Figure 6 As an implementation of the methods shown in the above figures, the present invention provides an embodiment of a panoramic image style transfer system, the system comprising:

[0073] Multi-view rotation module: Projects the original panoramic image from multiple perspectives to obtain panoramic images from different viewpoints;

[0074] Cube Combination Module: Performs cube projection on panoramic images from different perspectives to obtain multiple cube projection images, and combines them into a single target image in the order of restoration;

[0075] Style transfer module: Inputs the target image into the panoramic image style transfer network to obtain a stylized target image; the panoramic image style transfer network includes: a multi-scale attention content encoder, a style encoder, a global feature sharing module, and a decoder;

[0076] Panoramic Image Restoration Module: The stylized target image is segmented, and the resulting cube projection image is subjected to equal rectangular projection to obtain multiple stylized panoramic images from different perspectives;

[0077] Multi-view fusion module: Performs multi-view fusion on stylized panoramic images from different perspectives to obtain stylized panoramic images.

[0078] Example 3

[0079] According to embodiments of the present invention, the present invention also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the panoramic image style transfer processing method.

[0080] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A panoramic image style transfer method, characterized in that, include Multi-view projection is performed on the original panoramic image to obtain panoramic images from different perspectives; Cube projection is performed on panoramic images from different perspectives to obtain multiple cube projection images, which are then combined into a single target image in the order of reconstruction. The target image is input into a panoramic image style transfer network to obtain a stylized target image; The panoramic image style transfer network includes: a multi-scale attention content encoder, a style encoder, a global feature sharing module, and a decoder; The stylized target image is segmented, and the resulting cube projection is subjected to equal rectangular projection to obtain multiple stylized panoramic images from different perspectives. Stylized panoramic images from different perspectives are fused to obtain stylized panoramic images. The process of projecting the original panoramic image from multiple perspectives to obtain panoramic images from different viewpoints includes: A three-dimensional cylindrical coordinate system is established with the user's viewing position as the center, and the 360-degree panoramic image presented to the user under the head-mounted display is projected onto the surface of the sphere. Based on the characteristics of the panoramic image content, the panoramic image is rotated multiple times. In the horizontal direction, the content at the cut edge is rotated to the center area, and then a rectangular projection is performed to obtain multiple two-dimensional panoramic images from different perspectives.

2. The panoramic image style transfer method according to claim 1, characterized in that, The restoration order is a splicing method that combines pixels in a way that achieves maximum continuity.

3. The panoramic image style transfer method according to claim 1, characterized in that, The target image is input into a panoramic image style transfer network to obtain a stylized target image, specifically: The target image is input into a multi-scale attention-based content encoder to obtain content features; The style image is input into the style encoder to obtain style features; The content features and style features are input into the global feature sharing module, and the decoder is connected to its output to obtain the stylized target image.

4. The panoramic image style transfer method according to any one of claims 1-3, characterized in that, The multi-view fusion includes: Pixel-by-pixel linear overlay of stylized panoramic images from multiple different perspectives; During the overlay process, the weights of the panoramic images from each viewpoint are averaged to obtain the final stylized panoramic image.

5. The panoramic image style transfer method according to claim 3, characterized in that, The panoramic image style transfer network is specifically as follows: The multi-scale attention content encoder consists of several convolutional layers, aggregation blocks, and attention modules. It is used to stitch content feature maps of different levels in the channel direction, extract important information in the spatial and channel dimensions, and perform multi-scale attention feature fusion. A style encoder is used to extract style information from a style image; The global feature sharing module, including an adaptive decomposition module, a style unification module, and a combination module, is used to segment the target image into multiple sub-images for stylization and to stitch the multiple stylized sub-images together to form a stylized target image. A decoder is used to convert a stylized target image from the feature map domain to the image domain.

6. The panoramic image style transfer method according to claim 5, characterized in that, The global feature sharing module obtains stylized sub-images with consistent visual effects by sharing the global style features of the target image thumbnail during the stylization process.

7. The panoramic image style transfer method according to claim 1, characterized in that, The rotation is a vertical rotation.

8. A system for implementing the panoramic image style transfer method according to any one of claims 1-7, characterized in that, The system includes: Multi-view rotation module: Projects the original panoramic image from multiple perspectives to obtain panoramic images from different viewpoints; Cube Combination Module: Performs cube projection on panoramic images from different perspectives to obtain multiple cube projection images, and combines them into a target image in the restoration order; Style transfer module: Inputs the target image into the panoramic image style transfer network to obtain a stylized target image; the panoramic image style transfer network includes: a multi-scale attention content encoder, a style encoder, a global feature sharing module, and a decoder; Panoramic Image Restoration Module: The stylized target image is segmented, and the resulting cube projection image is subjected to equal rectangular projection to obtain multiple stylized panoramic images from different perspectives; Multi-view fusion module: Performs multi-view fusion on stylized panoramic images from different perspectives to obtain stylized panoramic images.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processing device, it implements the panoramic image style transfer method according to any one of claims 1-7.

Images