A block-based panoramic image generation method

Abstract

The present invention provides a block-based panoramic image generation method, comprising: obtaining global latent codes, image block latent features, and image block coordinates, wherein the image blocks are pre-scaled to partial spherical blocks according to the coordinates of the global panoramic image and divided into regular grids in the spherical tangent space so that the image blocks mapped onto the sphere are evenly distributed; inputting the above content into multiple spherical distortion-aware convolution blocks for block-based spherical convolution to obtain local embeddings of the image blocks; inputting the local embeddings of the image blocks into a stylized head layer for stylized head processing to obtain intermediate panoramic content and block texture features; inputting the intermediate panoramic content and block texture features into multiple distortion-aware upsampling blocks to obtain a panoramic image generation block; inputting the panoramic image generation block into a discriminator for comparison with real panoramic image blocks; and outputting a panoramic image when comparison conditions are met. By implementing this method, high-quality and consistent panoramic images can be generated.

Description

Technical Field

[0001] The present invention belongs to the technical field of panoramic image generation, and in particular relates to a block-based panoramic image generation method. Background Art

[0002] With the rapid development of virtual reality (VR) and augmented reality (AR) technologies, panoramic images have become a key medium for achieving immersive visual experiences. Panoramic images are typically captured using a high-resolution fisheye camera array, covering a 360°×180° field of view (FOV). Due to the specific format requirements of panoramic content, capturing and displaying panoramic images requires specially designed equipment, while high bandwidth support is also required to store and transmit massive amounts of data. Therefore, the technical difficulties in acquiring, transmitting, and displaying panoramic images highlight the necessity of using artificial intelligence generated content (AIGC) to enrich panoramic image generation to meet diverse immersive experience needs.

[0003] Deep probabilistic generative models (PGMs) have made significant progress in generating realistic content, particularly achieving breakthrough results on the task of generating planar images. However, for panoramic image generation, existing methods typically map panoramic formats to planar images and then generate them using planar PGMs, which are regularized using cylindrical coordinate priors and a spherical weighted loss. However, due to the presence of rectangular image edges, such methods can only generate pseudo-panoramic images and fail to ensure image edge connectivity, resulting in inconsistent content across panoramic viewpoints. To address this inconsistency, some methods have introduced global hard constraints, such as using multi-view and holistic panoramic cues as conditions to guide generation. More advanced techniques include introducing horizontally shifted circular padding in generative adversarial networks (GANs) and using latent recombination in diffusion models. While these methods alleviate some of the issues by ensuring consistency between the leftmost and rightmost edges of the image, they require simultaneous generation of the entire image during training and inference, resulting in significant computational overhead, especially when high-resolution panoramic image generation is required. More importantly, most existing methods rely on additional information such as single / multi-view and rich contextual clues, which are not always available in practical panoramic image generation scenarios. Therefore, developing an efficient and fundamental unconditional panoramic image generation method has become an urgent need to meet the emerging high-quality immersive experience. Summary of the Invention

[0004] In view of this, an object of the present invention is to provide a block-based panoramic image generation method to meet the demand for generating high-quality panoramic images.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] The embodiment of the present invention provides a block-based panoramic image generation method, comprising: obtaining a global latent code, an image block latent feature, and an image block coordinate, wherein the image block is pre-scaled to a partial spherical block according to the coordinates of the global panoramic image, and a regular grid is used in the spherical tangent space so that the image blocks presented on the spherical surface are evenly distributed; inputting the global latent code, the image block latent feature, and the image block coordinate into a plurality of spherical distortion-aware convolution blocks in a spherical local embedding module to perform spherical convolution based on the image block to obtain a local embedding of the image block; and inputting the local embedding of the image block into the spherical content The stylized head layer of the generator is subjected to stylized head processing to obtain intermediate panoramic content and block texture features. The intermediate panoramic content and block texture features are input into multiple distortion-aware upsampling blocks of the spherical content generator to obtain a panoramic image generation block. The distortion-aware upsampling block consists of two main branches, one for performing convolution processing on the block texture features, and the other for performing image block-based spherical convolution processing on the intermediate panoramic content and fusing the convolution-processed block texture features. The panoramic image generation block is input into the discriminator and compared with the real block of the panoramic image. When the comparison conditions are met, the panoramic image is output.

[0007] Optionally, the global latent code, the image block latent features and the image block coordinates are input into the multiple spherical distortion-aware convolution blocks in the spherical local embedding module to perform spherical convolution based on the image block to obtain the local embedding of the image block, including: for the lth spherical distortion-aware convolution block, the image block latent features output by l-1 spherical distortion-aware convolution blocks are extended feature point interpolation operation according to the image block coordinates to obtain spherical neighborhood grid points, and the spherical neighborhood grid points are merged with the position code in the coordinate channel; the spherical neighborhood grid points are convolved with the convolution kernel to obtain a first convolution result, the parameters in the convolution kernel are modulated according to the style code, and the style code is obtained by transforming the global latent code by the mapping network; the first convolution result is added to the image block latent feature to form a residual structure, and the residual structure is modulated and convolved according to the style code to obtain a second convolution result, and the second convolution result is used as the image block latent feature output by the lth spherical distortion-aware convolution block.

[0008] Optionally, the image blocks are pre-scaled to partial spherical blocks according to the coordinates of the global panoramic image, and a regular grid is used in the spherical tangent space to make the image blocks mapped onto the spherical surface uniformly distributed. The division process includes: inverse spherical projection of the tangent space grid, mapping it back to the spherical surface, and determining the spherical area and the tangent space area according to the following formula: Each corresponding adjacent grid coordinate p i,j ={θ i ,ψ j}:

[0009]

[0010] Among them, θ and ψ are the Coincident coordinate points, Represents the interval of the grid in the tangent space, which is a constant;

[0011] Obtain the panoramic image construction range, determine the scaling ratio of the panoramic image coordinates to the local spherical block according to the panoramic image construction range; determine the grid point coordinates of the local spherical block according to the coincident coordinate points in the spherical coordinates, the panoramic image construction range and the scaling ratio; determine the grid point coordinates of the local spherical block according to the coincident coordinate points in the spherical coordinates and the tangent space; The corresponding adjacent grid coordinates p i,j ={θ i ,ψ j} and the grid point coordinates of the local spherical block to determine the tangent space The grid point coordinates are projected back to the spherical surface; based on the multiple grid point coordinates, the spherical surface is divided into multiple image blocks, and the multiple image blocks are evenly distributed on the spherical surface.

[0012] Optionally, convolving the spherical neighborhood grid points with the convolution kernel to obtain a first convolution result includes:

[0013]

[0014] in, represents the first convolution result of the image block potential feature convolution based on the output of the l-1th spherical distortion-aware convolution block, SPConv[·] represents the spherical convolution based on the image block, represents the spherical neighborhood grid points obtained by performing extended feature point interpolation operations based on the potential features of the image block output by the l-1th spherical distortion-aware convolution block, {i, j}, {m, n} are the local grid point indices on the tangent space, Represents the convolution kernel in the image block-based spherical convolution, and the convolution kernel size is k×k.

[0015] Optionally, the adding of the first convolution result and the potential features of the image block to form a residual structure, and performing modulated convolution on the residual structure according to the style coding to obtain the second convolution result includes:

[0016]

[0017] in, represents the second convolution result of the lth spherical distortion perception convolution block, Represented by the style code g s Modulated convolution, while Conv 1×1 [·] represents the 1×1 convolution of the residual structure, According to the first convolution result of the potential feature convolution of the image block output by the l-1th spherical distortion perception convolution block, represents the latent features of the image patch output by the l-1th spherical distortion-aware convolutional block.

[0018] Optionally, the intermediate panoramic content and block texture features are input into a plurality of distortion-aware upsampling blocks of the spherical content generator to obtain a panoramic image generation block, and the distortion-aware upsampling block processes the block texture features, including:

[0019]

[0020] in, represents the block texture feature output by the lth distortion-aware upsampling block, Represented by the style code g′ s Modulated convolution, represents the block texture feature output by the l-1th distortion-aware upsampling block, and denote the two learnable parameters of the lth distortion-aware upsampling block, and σ1 and σ2 denote the Gaussian noise inserted during the generation process.

[0021] Optionally, the intermediate panoramic content and the block texture features are input to a plurality of distortion-aware upsampling blocks of the spherical content generator to obtain a panoramic image generation block, and the processing of the intermediate panoramic content by the distortion-aware upsampling block includes: for the lth distortion-aware upsampling block, processing the intermediate panoramic content output by the l-1 distortion-aware upsampling block; Perform interpolation of extended feature points and perform spherical convolution on the intermediate panoramic content after interpolation to obtain the image block content Block texture features Perform modulated convolution based on style coding and adjust the image block content Perform deconvolution operation; add the block texture features after style coding modulation convolution to the image block content after deconvolution operation to obtain the intermediate panoramic content output by the lth distortion-aware upsampling block

[0022] Optionally, the block texture features after the style coding modulation convolution are added to the image block content after the deconvolution operation to obtain the intermediate panoramic content output by the lth distortion-aware upsampling block include:

[0023]

[0024] in, represents the intermediate panoramic content output by the l-th distortion-aware upsampling block, IntP represents the intermediate panoramic content output by the l-1th distortion-aware upsampling block, k [·] represents the interpolation operation of the extended feature points, SPConv[·] represents the spherical convolution based on the image block, and DeConv[·] represents the deconvolution operation. represents the block texture feature output by the lth distortion-aware upsampling block, represents a 1×1 convolution, whose modulation is g′ s Weighted.

[0025] Optionally, the spherical local embedding module, the spherical content generator, and the discriminator constitute a panoramic image block generative adversarial network, and the loss function of the panoramic image block generative adversarial network is as follows:

[0026] L Gen =-L Dis +λ Div L Div +λ Pat L Pat +λ Co L Co ,

[0027] Among them, L Dis is the loss function from the discriminator, which is expressed as L Dis =L Adv +λ R1 L R1 +λ Co L Co ,L Adv is a non-saturated adversarial loss used to fit the probability distribution of the input, L R1 is the R1 regularization loss, which is used to enhance stability when training the discriminator, and λ R1 is the intensity control parameter of the regularization loss of the discriminator, L Div is a regularization loss to encourage generation diversity, L Pat is the path length regularization loss used to make the generated panoramic image patch content consistent with the global latent code g z Keep consistent, and L Co represents the l1 loss when the discriminator predicts the coordinates of the input patch, λ Div is the intensity control parameter of the patch diversity loss; Pat is the path regularization loss strength control parameter in StyleGAN, λ Co is the coordinate prediction loss intensity control parameter.

[0028] Optionally, the convolution kernel size of the spherical convolution based on the image block in the panoramic image block generation adversarial network is a weight k Initialized with the following values:

[0029]

[0030] in, represents the zero-value matrix of size k×k, except that the center value is equal to 1; η represents the random number used for initialization.

[0031] The present invention provides a block-based panoramic image generation method, which constructs two core modules: Spherical Local Embedding (SLE) and Spherical Content Synthesizer (SCS). The purpose of the SLE module is to embed the features and positions of local image blocks into the base texture, which is then aggregated by the SCS module to generate more detailed image block content. Because the SLE and SCS modules operate based on the features of local spherical image blocks, the panoramic image block generative adversarial network (SP-GAN) can generate continuous and consistent spherical image blocks. The SP-GAN proposed in this embodiment can generate high-quality and consistent panoramic images. In addition, the image block-based spherical convolution proposed in this embodiment acts on local spherical image blocks. The image block division method ensures a uniform grid distribution on the sphere, allowing the spherical convolution to naturally handle the deformation problem of panoramic images. In addition, thanks to the block-by-block processing strategy, the SP-GAN can be flexibly expanded to image generation of different resolutions based on a single training model. The SP-GAN proposed in this embodiment demonstrates excellent performance in terms of generation quality, computational efficiency, and high-resolution generalization capability.

[0032] Other advantages, objectives and features of the present invention will be described in the following description and will be apparent to those skilled in the art to some extent, or those skilled in the art can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to make the purpose, technical solutions and beneficial effects of the present invention more clear, the present invention provides the following drawings for illustration:

[0034] Figure 1 This is a specific example flow chart of a block-based panoramic image generation method in the present invention;

[0035] Figure 2 Schematic diagram of the network architecture of the panoramic image patch generative adversarial network (SP-GAN) in the present invention;

[0036] Figure 3 A conversion diagram of the convolution kernel from the ERP image to the panoramic image during the deformation between the panoramic image and the planar format mentioned in the present invention;

[0037] Figure 4 It is the mapping diagram of the convolution kernel from the tangent space to the panoramic image and the ERP image in the present invention;

[0038] Figure 5 In the present invention, Schematic diagram of 3×3SPConv centered at ;

[0039] Figure 6 Schematic diagram of the process of generating an overall panoramic image from panoramic image blocks in the present invention;

[0040] Figure 7 This is a principle block diagram of a specific example of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0042] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; internal connections between two components; wireless connections or wired connections. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0043] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0044] The embodiment of the present invention provides a method for generating a panoramic image based on a block, such as Figure 1 As shown, including:

[0045] S101, obtaining a global latent code, an image block latent feature, and image block coordinates, wherein the image block is pre-scaled to a partial spherical block according to the coordinates of the global panoramic image, and a regular grid is used in the spherical tangent space so that the image blocks mapped onto the spherical surface are evenly distributed;

[0046] S102, inputting the global latent code, the latent features of the image block, and the coordinates of the image block into multiple spherical distortion-aware convolution blocks in a spherical local embedding module to perform spherical convolution based on the image block to obtain a local embedding of the image block;

[0047] S103, inputting the local embedding of the image block into the stylized head layer of the spherical content generator for stylized head processing to obtain intermediate panoramic content and block texture features;

[0048] S104: Inputting the intermediate panoramic content and the block texture features into multiple distortion-aware upsampling blocks of the spherical content generator to obtain a panoramic image generation block. The distortion-aware upsampling block consists of two main branches: one branch for performing convolution processing on the block texture features, and the other branch for performing image block-based spherical convolution processing on the intermediate panoramic content and fusing the convolution-processed block texture features.

[0049] S105: Input the generated panoramic image block into the discriminator and compare it with the real panoramic image block. When the comparison condition is met, the panoramic image is output.

[0050] For example, this embodiment proposes to use a panoramic image patch generative adversarial network (SP-GAN) including a spherical local embedding module (SLE module), a spherical content generator (SCS module) and a discriminator. The network framework structure is as follows: Figure 2 shown.

[0051] Since the input of the panoramic image block generative adversarial network mentioned above is the global latent code, image block latent features and image block coordinates, and convolution processing is required for this, the image blocks need to be divided first. However, the panoramic image has a curved structure, and the existing two-dimensional convolution cannot process it directly. The currently commonly used method is to use ERP to project the panoramic image into a plane format, and then perform subsequent processing on the ERP image. However, when processing plane images, the essence of two-dimensional convolution is to aggregate information on a regular grid of image pixels. Unfortunately, due to the deformation between the panoramic image and the plane format, the regular grid in the ERP image cannot guarantee a uniform distribution of the grid on the sphere, such as Figure 3 Therefore, when applying two-dimensional convolution to the ERP image, this embodiment needs to sample on an irregular grid to ensure that the grid of the panoramic image is evenly distributed on the spherical surface, as shown in FIG. Figure 4 shown.

[0052] The specific sampling method, that is, the division method of image blocks is as follows:

[0053] Based on the local similarity between spherical manifold and Euclidean space, this embodiment uses a regular grid on the spherical tangent space as a proxy for the uniformly distributed grid near each point on the sphere. Specifically, for each point p = {θ, ψ} on the sphere, use τp Represent its tangent space, using represents the local grid on the tangent space. Correspondingly, the convolution kernel on the tangent space is expressed as The two-dimensional convolution operation on the tangent space is defined as:

[0054]

[0055] Among them, {i,j} and {m,n} represent and The index of describes the convolution operation on the tangent space of a given spherical point p.

[0056] To approximate the grid in tangent space The content of the image is mapped back to the sphere using inverse spherical projection, i.e., the panoramic image. It coincides with the point p={θ,ψ} on the sphere, so the point on the sphere and in the tangent space can be calculated. Each corresponding adjacent grid coordinate point p i,j ={θ i ,ψ j}, the relationship is given by the following formula:

[0057]

[0058] in represents the interval of the grid on the tangent space. is a constant. From formula (2), we can see that when i→0 and j→0, θ i →θ and ψ j →ψ. More importantly, when i=0 and j=0, This shows that the points on the tangent space It completely coincides with the point p={θ,ψ} on the sphere.

[0059] Due to the block-based nature of SP-GAN, the projection coordinates p in (2) i,j ={θ i ,ψ j} does not inherently guarantee the existence of panoramic image pixels. More importantly, when a larger interval is incorrectly chosen When p i,j ={θ i ,ψ j} may even exceed the range of the provided sphere blocks; this will lead to inter-block inconsistencies, i.e., blocking artifacts in the generated panoramic image. Instead, choose a smaller The value may result in a limited receptive field during training. To address this issue, this embodiment rescales the coordinates of the global panoramic image to local spherical blocks.

[0060] More specifically, when the spherical block constructed from the panoramic image has a range of [θ s ,θ e ] and [ψ s ,ψ e ], the scaling ratio of the panoramic image coordinates to the local spherical block is determined according to the range of the spherical block constructed by the panoramic image. Specifically, and Get the scaling ratio of longitude and latitude. Determine the grid point coordinates of the local spherical block based on the coincident coordinate points in the spherical coordinates, the panoramic image construction range, and the scaling ratio. The specific formula is as follows:

[0061]

[0062] Based on formula (3), by keeping the same deviation in the tangent space as in the global coordinates, we can obtain the deviation from the tangent space τ p Grid point coordinates projected back to the sphere Specifically, according to the spherical surface and the tangent space The corresponding adjacent grid coordinates p i,j ={θ i ,ψ j} and the grid point coordinates of the local spherical block Determine the tangent space τ p in Grid point coordinates projected back to the sphere The specific formula is as follows:

[0063]

[0064] Due to s θ <1 and s ψ <1, essentially expanding the coordinates in the local spherical block, thereby reducing the possibility of projecting beyond the spherical block from the tangent space. Therefore, this embodiment does not empirically select But will Set to a constant and adaptively adjust the projection according to the ratio of the image block size to the entire panoramic image, that is, by s θ and s ψ Then, bilinear interpolation and boundary filling strategies are used to approximate each point of the given local image patch coordinates. Since SP-GAN only requires image patches for training and inference, rescaling to local coordinates ensures training stability and continuity between generated image patches.

[0065] In the implementation, the local coordinates on a given spherical block are For point p, the image block-based spherical convolution (SPConv) proposed in this embodiment has the formula:

[0066]

[0067] in Indicates that it is located The spherical grid point at position , which is given by formula (4). In addition, in formula (5), Centered on p, K is a convolution kernel that can be learned during training. Based on the proposed SPConv, the deformation problem of panoramic patch images can be solved and panoramic images can be generated based on patches.

[0068] In addition to addressing the problem of panoramic image deformation, the SP-GAN proposed in this embodiment also ensures continuity between the generated panoramic image patches. More importantly, the generation process requires gradually increasing the resolution through convolution operations, ultimately generating high-resolution panoramic image patches. Therefore, this embodiment constructs a new SP-GAN for panoramic image patch generation based on the proposed SPConv. When its underlying spherical features are connected, the network is able to generate continuous and consistent panoramic image patches.

[0069] SP-GAN, such as Figure 2 As shown in Figure 5, the spherical local embedding module includes multiple spherical distortion-aware convolution blocks, each of which contains a block-based spherical convolution module. The block-based spherical convolution module performs block-based spherical convolution on the input data as shown in formula (5). By continuously stacking spherical distortion-aware convolution blocks, the spherical local embedding module can convert the features of the potential panoramic image block into the basic spherical local embedding P s , while ensuring that the generated embedding can accurately reflect the distortion of the panoramic image. The local embedding of the image block P s As the input of the spherical content generator, it is further processed. The spherical content generator consists of two parts. The first part is the stylized head layer, which is used to embed the local P s Stylized head processing is performed to obtain intermediate panoramic content and block texture features. The second part is multiple distortion-aware upsampling blocks. Each distortion-aware upsampling block consists of two main branches, which are used to gradually synthesize panoramic intermediate features and content. One branch convolves the block texture features through convolution and deconvolution, and the other branch is used to perform spherical convolution processing based on image blocks on the intermediate panoramic content and fuse the block texture features after convolution processing.

[0070] In the SP-GAN proposed in this embodiment, on the one hand, the SLE module is responsible for embedding local spherical content, and accepts global latent codes, image block latent features and corresponding coordinates as input. The global latent code is used to control the overall style and ensure consistency between blocks. The image block latent features are closely related to the local structure of the generated block and are controlled by the corresponding image block coordinates. On the other hand, the SCS module is responsible for generating fine textures based on local embedding and global latent codes. It is worth noting that in order to ensure the continuity of the block content in the generated panoramic image, no padding is used in each layer of convolution operations in SP-GAN. This approach allows SP-GAN to generate high-quality blocks in a coarse-to-fine manner, while cleverly adapting to the curvature characteristics of the spherical blocks.

[0071] The discriminator evaluates the quality of the generated patches to ensure their consistency with real image patches. If the generated patches meet the comparison criteria, they are stitched together to form a complete panoramic image. In SP-GAN, the discriminator and the generator are trained in an adversarial manner. The generator produces increasingly realistic image patches to deceive the discriminator, while the discriminator strives to improve its ability to distinguish between real and generated data. This adversarial process promotes the joint progress of the generator and discriminator, resulting in the final panoramic image output, which should be visually indistinguishable from the real scene while maintaining structural and semantic accuracy.

[0072] This embodiment provides a block-based panoramic image generation method, which constructs two core modules: Spherical Local Embedding (SLE) and Spherical Content Synthesizer (SCS). The purpose of the SLE module is to embed the features and positions of local image blocks into the base texture, which is then aggregated by the SCS module to generate more detailed image block content. Because the SLE and SCS modules operate based on the features of local spherical image blocks, SP-GAN can generate continuous and consistent spherical image blocks. The SP-GAN proposed in this embodiment can generate high-quality and consistent panoramic images. In addition, the image block-based spherical convolution proposed in this embodiment of the present invention acts on local spherical image blocks. The division method of these image blocks ensures a uniform distribution of grids on the sphere, allowing spherical convolution to naturally handle the deformation problem of panoramic images. In addition, thanks to the block-by-block processing strategy, SP-GAN can be flexibly expanded to image generation of different resolutions based on a single training model. The SP-GAN proposed in this embodiment demonstrates excellent performance in terms of generation quality, computational efficiency, and high-resolution generalization capability.

[0073] The method of this example is verified on the MatterPort3D dataset and compared with the algorithm that can generate panoramas. Due to the rectangular characteristics of panoramic images, the experimental indicators select the center, cross-border, random square area image blocks and the image in the dataset to calculate the FID and IS values, which are recorded as FID cent 、FID bord 、FID rand and ISIS cent 、IS bord 、IS rand , where the lower the FID, the better, and the higher the IS, the better. The experimental results compared with the performance of the previous best block panoramic image generation method are shown in the following table:

[0074] method FID IS <![CDATA[FID cent ]]> <![CDATA[IS cent ]]> <![CDATA[FID bord ]]> <![CDATA[IS bord ]]> <![CDATA[FID rand ]]> <![CDATA[IS rand ]]> This example 27.55 4.08 26.74 4.07 29.51 4.09 26.40 4.07 Previous best practices 30.66 3.69 29.82 3.71 32.59 3.67 29.56 3.70

[0075] As an optional implementation, the global latent code, the image block latent features, and the image block coordinates are input into multiple spherical distortion-aware convolution blocks in the spherical local embedding module to perform spherical convolution based on the image block to obtain a local embedding of the image block, including:

[0076] For the lth spherical distortion-aware convolution block, the latent features of the image block output by the l-1 spherical distortion-aware convolution blocks are extended with feature point interpolation according to the image block coordinates to obtain spherical neighborhood grid points. The spherical neighborhood grid points are combined with position codes in the coordinate channel.

[0077] Convolving the spherical neighborhood grid points with a convolution kernel to obtain a first convolution result, wherein parameters in the convolution kernel are modulated according to a style code, wherein the style code is obtained by transforming a global latent code by a mapping network;

[0078] The first convolution result is added to the potential features of the image block to form a residual structure, and the residual structure is modulated and convolved according to the style coding to obtain a second convolution result, which is used as the potential features of the image block output by the lth spherical distortion perception convolution block.

[0079] Exemplarily, the input of the SLE module includes the global latent code g z , image block potential feature P z and image patch coordinates C p , then output the local embedding P of the block s In addition, the spherical local embedding module consists of multiple spherical distortion-aware convolution (DAC) blocks. Each DAC block is built based on the SPConv operation proposed in the above formula (5), which is then merged with the global style information. Specifically, it is assumed that the coordinate range of the panoramic image block to be generated is [θ s ,θe ] and latitude [ψ s ,ψ e ] Then, the image block potential feature P z (i,j) is given by coordinate θ s ≤θ i ≤θ e and ψ s ≤ψ j ≤ψ e Then, the image block potential feature P z with C p Connect together by channel and input into the DAC block of the SLE module.

[0080] like Figure 5 As shown, Figure 5 Indicates Schematic diagram of a 3×3 SPConv centered on . Specifically, for the lth DAC block, the latent features of the input image block are first interpolated for the subsequent SPConv The spherical neighborhood grid points are obtained. The interpolation operation in SPConv is centered at each point (i, j). The principle is shown in the following formula:

[0081]

[0082] Among them, IntP k [·] represents the interpolation operation of the extended feature points. Zoom in k×k times. In order to embed the position information, use {θ i ,ψ j} position code. It should be noted that in this embodiment The position code is merged into the coordinate channel of . Then, the spherical neighborhood grid points obtained by interpolation are calculated according to formula (5). Perform element-by-element product operation with the convolution kernel of size k×k to obtain the first convolution result. In this way, the output of SPConv is the first convolution result and are the same size, that is:

[0083]

[0084] in Indicates that SPConv has a learnable convolution kernel with parameters More importantly, W gs Also influenced by image style encoding g s The modulation, and style coding g s The global latent code g is pre-coded by the mapping network z Transformed.

[0085] Then, the first convolution result and image patch latent features Add them together to form a residual structure, and then modulate the residual structure according to the style code to obtain the second convolution result This process can be expressed by the following formula:

[0086]

[0087] in Represented by the image style code g s Modulated convolution, while Conv 1×1 [·] represents the 1×1 convolution of the residual structure. In fact, the global latent code g z First processed by the mapping network, it is expressed as a style encoding Similar to the style encoding in Style-GAN2.

[0088] This embodiment provides a block-based panoramic image generation method that combines global latent codes with image block latent features, allowing the model to strike a balance between local details and global context, helping to generate image blocks that are visually coherent and match the global scene. The style code obtained by transforming the global latent code through a mapping network can modulate the parameters in the convolution kernel, so that the generated image blocks are not only consistent in content but also match the target scene in style, enhancing the realism and diversity of the generated images.

[0089] As an optional implementation, the intermediate panoramic content and block texture features are input into multiple distortion-aware upsampling blocks of the spherical content generator to obtain a panoramic image generation block. The distortion-aware upsampling block processes the block texture features, including:

[0090]

[0091] For example, given the local embedding P in the SLE module s , the proposed SCS module is designed to be based on P s and image style code g′ s Generate detailed panoramic content. More specifically, the local embedding P s First, the stylization head (SH) processes the initial intermediate panoramic content I p and block texture feature P t The SH architecture is the same as that in Style-GAN2. The stylized head layer proposed in this embodiment also incorporates the image style encoding g′ s To ensure the content consistency of the entire generated panoramic image. It should be noted that the image style encoding g′ of the SCS module sis obtained through an additional mapping network, whose architecture is the same as the mapping network in the SLE module, but the weights and training process are different. Then, the intermediate panoramic content I p and block texture feature P t It is input into multiple Distortion-Aware Upsampling (DAU) blocks, which consists of two main branches, one for gradually synthesizing panoramic intermediate features and the other for gradually synthesizing content.

[0092] For the lth DAU block, use Denotes the intermediate panoramic content output by the l-1th distortion-aware upsampling block, and is expressed as Represents the block texture feature output by the l-1th distortion-aware upsampling block. For the block texture feature output by the lth distortion-aware upsampling block It can be expressed by the following formula:

[0093]

[0094] Where DeConv[·] represents the deconvolution operation; Represented by the style code g′ s Modulated convolution, and Represents the two learnable parameters of the l-th DAU block. At the same time, Gaussian noise σ1 and σ2 are inserted during the generation process to enhance the diversity of generated content.

[0095] As an optional implementation, the intermediate panoramic content and block texture features are input into multiple distortion-aware upsampling blocks of the spherical content generator to obtain a panoramic image generation block. The distortion-aware upsampling block consists of two main branches, one for performing convolution processing on the block texture features, and the other for performing image block-based spherical convolution processing on the intermediate panoramic content and fusing the convolution-processed block texture features, including:

[0096] For the l-th distortion-aware upsampling block, the intermediate panoramic content output by the l-1 distortion-aware upsampling blocks is Perform interpolation of extended feature points and perform spherical convolution on the intermediate panoramic content after interpolation to obtain the image block content

[0097] Block texture features Perform modulated convolution based on style coding and adjust the image block content Perform deconvolution operation;

[0098] The block texture features after style coding modulation convolution are added to the image block content after deconvolution operation to obtain the intermediate panoramic content output by the lth distortion-aware upsampling block.

[0099] For example, due to the intermediate panoramic content The generated panoramic image is gradually depicted and becomes the final output of SP-GAN. This embodiment proposes a new branch to further solve the spherical distortion problem by combining SP-GAN. Specifically, first, for the l-th distortion-aware upsampling block, the intermediate panoramic content output by the l-1 distortion-aware upsampling block is Perform interpolation of extended feature points and perform spherical convolution on the intermediate panoramic content after interpolation to obtain the image block content The specific formula is as follows:

[0100]

[0101] It should be noted that in the SCS module, the feature already contains structural information and position clues. Therefore, the position embedding operation is skipped to avoid misleading the process of generating details. However, SPConv is still necessary to further enhance the panoramic content details and handle panoramic distortion. In addition, in order to aggregate the texture features from the block Information on block texture features Perform modulated convolution based on style coding and adjust the image block content Perform deconvolution operation, add the block texture features after style coding modulation convolution and the image block content after deconvolution operation, and obtain the intermediate panoramic content output by the lth distortion-aware upsampling block The obtained intermediate panoramic content Combined with block texture features, the image quality is higher. The specific formula is as follows:

[0102]

[0103] in, represents a 1×1 convolution, whose modulation is g′ s The weighting aims to change the number of channels to match the number of channels of the output block content. With the help of the proposed DAU block, arrive The process improves the texture and details generated within the image block. By stacking DAU blocks, the SCS module is able to generate correct and realistic panoramic content in a block-by-block and gradual generation manner from low resolution to high resolution.

[0104] This embodiment provides a block-based panoramic image generation method that adds block texture features after style coding modulation convolution to the image block content after deconvolution. In other words, in each distortion-aware upsampling block, the intermediate panoramic content and block texture features are aggregated. This fusion of different feature levels combines the semantic information of deep features with the texture details of shallow features, helping to generate richer and more realistic image blocks, resulting in high-quality panoramic images.

[0105] As an optional implementation, the spherical local embedding module, the spherical content generator, and the discriminator constitute a panoramic image block generative adversarial network, and the loss function of the panoramic image block generative adversarial network is as follows:

[0106] L Gen =-L Dis +λ Div L Div +λ Pat L Pat +λ Co L Co ,

[0107] Among them, L Dis is the loss function from the discriminator, which is expressed as L Dis =L Adv +λ R1 L R1 +λ Co L Co ,L Adv is a non-saturated adversarial loss used to fit the probability distribution of the input, L R1 is the R1 regularization loss, which is used to enhance stability when training the discriminator, and λ R1 is the intensity control parameter of the regularization loss of the discriminator, L Div is a regularization loss to encourage generation diversity, L Pat is the path length regularization loss used to make the generated panoramic image patch content consistent with the global latent code g z Keep consistent, and L Co represents the l1 loss when the discriminator predicts the coordinates of the input patch, λ Div is the intensity control parameter of the patch diversity loss; Pat is the path regularization loss strength control parameter in StyleGAN, λ Co is the coordinate prediction loss intensity control parameter.

[0108] For example, since SP-GAN only requires local panoramic image patches for training, when all operations are restricted to executing within the local patches, the continuity between the generated panoramic image patches can be guaranteed at a basic level. In this way, the latent features of the connected local patches can generate seamless image content. More importantly, the non-padding design and the local coordinate-based SPConv essentially constrain the scope of the convolution operation to the local patch, ensuring the content continuity of SP-GAN when generating panoramic images based on image patches. Therefore, the remaining key task is to ensure that the content at different locations in the panoramic image can transition smoothly.

[0109] To solve this problem, SP-GAN combines adversarial training (based on Style-GAN2) and coordinate prediction loss for optimization. The loss function in SP-GAN is designed as follows:

[0110] L Gen =-L Dis +λ Div L Div +λ Pat L Pat +λ Co L Co , (12)

[0111] Among them, L Dis is the loss function from the discriminator, L Div is the regularization loss to encourage generation diversity, λ Div is the intensity control parameter of the patch diversity loss, L Pat is the path length regularization loss used to make the generated panoramic image patch content consistent with the global latent code g z Keep consistent, Pat is the path regularization loss strength control parameter in StyleGAN, and L Co represents the l1 loss when the discriminator predicts the coordinates of the input patch, λ Co is the coordinate prediction loss intensity control parameter. In addition, the method of training the discriminator is similar:

[0112] L Dis =L Adv +λ R1 L R1 +λ Co L Co , (13)

[0113] Among them, L Adv is a non-saturated adversarial loss used to fit the probability distribution of the input, L R1 is the R1 regularization loss, λ R1is a parameter controlling the strength of the discriminator's regularization loss, used to enhance stability when training the Style-GAN2 discriminator. It should also be clarified in this example that the development of the discriminator architecture is similar to that in Style-GAN2, and the entire training process is based entirely on panoramic image patches. Therefore, while ensuring high-quality generation, SP-GAN can also efficiently utilize fewer computational resources to generate high-resolution panoramic images.

[0114] More importantly, a special weight initialization method is designed for SP-GAN when training it in an end-to-end manner. The weight W of the convolution kernel size k for the spherical convolution based on the image patch in the panoramic image patch generation adversarial network is gs Initialized with the following values:

[0115]

[0116] in, denotes a zero-value matrix of size k × k, except for a center value equal to 1; η represents a random number used for initialization. With this weight initialization, SP-GAN starts with almost no effect and gradually learns to handle panoramic distortion during training. This initialization strategy can improve training stability in practice.

[0117] During the training phase, in order to generate continuous image patches, the latent features of a spherical patch are sampled from the complete latent sphere, such as Figure 6 As shown, the boxes with different lines represent different spherical blocks, and the boxes in the latent space and the image domain correspond one to one. More importantly, these latent block features overlap with each other, further enhancing the continuity between the generated blocks, while also providing flexibility for the resolution of the synthetic panorama. It should be pointed out that the complete latent feature is pure Gaussian noise, which is used for unconditional generation. On this basis, the SP-GAN proposed in this embodiment can also be extended to other application scenarios and generated in combination with auxiliary information such as text, views, and sketches. When the latent block features are connected from the left and right sides, the closed-loop characteristics are also naturally guaranteed.

[0118] The present application also provides an electronic device, such as Figure 7 As shown, a processor 501 and a memory 502 , wherein the processor 501 and the memory 502 may be connected via a bus or other means.

[0119] The processor 501 may be a central processing unit (CPU). The processor 501 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0120] Memory 502, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as the program instructions / modules corresponding to the block-based panoramic image generation method in the embodiments of the present invention. The processor executes the non-transitory software programs, instructions, and modules stored in the memory to perform various processor functions and data processing.

[0121] The memory 502 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 502 may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0122] The one or more modules are stored in the memory 502 and when executed by the processor 501, perform the following steps: Figure 1 A block-based panoramic image generation method in the illustrated embodiment.

[0123] For details of the above electronic equipment, please refer to Figure 1 The corresponding descriptions and effects in the embodiments shown can be understood and will not be repeated here.

[0124] This embodiment further provides a computer storage medium storing computer-executable instructions capable of executing the block-based panoramic image generation method of any of the above-described method embodiments. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); the storage medium may also include a combination of the above-described types of memory.

[0125] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A block-based panoramic image generation method, characterized in that: include: Obtaining global latent codes, latent features of image blocks, and coordinates of the image blocks, wherein the image blocks are pre-scaled to partial spherical blocks according to the coordinates of the global panoramic image, and a regular grid is used on the spherical tangent space so that the image blocks mapped onto the spherical surface are evenly distributed; The global latent code, the latent features of the image block, and the image block coordinates are input into multiple spherical distortion-aware convolution blocks in the spherical local embedding module to perform spherical convolution based on the image block to obtain the local embedding of the image block; The local embedding of the image block is input into the stylized head layer of the spherical content generator for stylized head processing to obtain the intermediate panoramic content and block texture features; Inputting the intermediate panoramic content and block texture features into multiple distortion-aware upsampling blocks of the spherical content generator to obtain a panoramic image generation block, wherein the distortion-aware upsampling block consists of two main branches: one branch is used to convolve the block texture features, and the other branch is used to perform image block-based spherical convolution on the intermediate panoramic content and fuse the convolved block texture features; The generated panoramic image block is input into the discriminator and compared with the real panoramic image block. When the comparison conditions are met, the panoramic image is output; The step of inputting the global latent code, the image block latent features, and the image block coordinates into a plurality of spherical distortion-aware convolution blocks in the spherical local embedding module to perform spherical convolution based on the image block to obtain a local embedding of the image block includes: For the Spherical distortion-aware convolution blocks, The latent features of the image block output by the spherical distortion perception convolution block are extended according to the image block coordinates to obtain spherical neighborhood grid points, and the spherical neighborhood grid points are combined with the position code in the coordinate channel; Convolving the spherical neighborhood grid points with a convolution kernel to obtain a first convolution result, wherein parameters in the convolution kernel are modulated according to a style code, wherein the style code is obtained by transforming a global latent code by a mapping network; The first convolution result is added to the potential features of the image block to form a residual structure, and the residual structure is modulated and convolved according to the style coding to obtain the second convolution result, which is used as the first convolution result. The potential features of the image patch output by the spherical distortion-aware convolution block; The convolution operation on the spherical neighborhood grid points and the convolution kernel to obtain a first convolution result includes: in, Indicates that according to The first convolution result of the potential feature convolution of the image block output by the spherical distortion perception convolution block, represents the spherical convolution based on image blocks, Indicates that according to The spherical neighborhood grid points are obtained by performing extended feature point interpolation operation on the potential features of the image block output by the spherical distortion perception convolution block. 、 is the local grid point index on the tangent space, Represents the convolution kernel in the spherical convolution based on the image block, and the convolution kernel size is .

2. The method for generating a panoramic image based on a block according to claim 1, wherein: The image blocks are pre-scaled to partial spherical blocks according to the coordinates of the global panoramic image, and are divided into regular grids on the spherical tangent space so that the image blocks mapped onto the spherical surface are evenly distributed. The division process includes: Project the tangent space grid back onto the sphere and determine the difference between the tangent space grid and the sphere grid according to the following formula: Each corresponding adjacent grid coordinate : in, and On the sphere and in the tangent space respectively Coincident coordinate points, , Represents the interval of the grid in the tangent space, which is a constant; Obtaining a panoramic image construction range, and determining a scaling ratio for scaling the panoramic image coordinates to the local spherical block according to the panoramic image construction range; Determine the grid point coordinates of the local spherical block according to the coincident coordinate points in the spherical coordinates, the panoramic image construction range, and the scaling ratio; According to the spherical surface and the tangent space Corresponding adjacent grid coordinates and the grid point coordinates of the local spherical block to determine the tangent space The coordinates of the grid points projected back onto the sphere; The sphere is divided into a plurality of image blocks according to the coordinates of a plurality of grid points, and the plurality of image blocks are evenly distributed on the sphere.

3. The method for generating a panoramic image based on a block according to claim 1, wherein: The method of adding the first convolution result to the potential features of the image block to form a residual structure, and performing modulated convolution on the residual structure according to the style coding to obtain a second convolution result includes: in, Indicates the The second convolution result of the spherical distortion-aware convolution block, Representation is encoded by style modulated convolution, and Residual structure convolution, According to The first convolution result of the potential feature convolution of the image block output by the spherical distortion perception convolution block, Indicates the The latent features of the image patch output by the spherical distortion-aware convolution block.

4. The method for generating a panoramic image based on a block according to claim 1 or 2, wherein: The intermediate panoramic content and block texture features are input into a plurality of distortion-aware upsampling blocks of a spherical content generator to obtain a panoramic image generation block, and the distortion-aware upsampling block processes the block texture features, including: in, Indicates the The block texture features output by the distortion-aware upsampling block, Representation is encoded by style Modulated convolution, Indicates the The block texture features output by the distortion-aware upsampling block, and Indicates the Two learnable parameters of the distortion-aware upsampling block, 、 Indicates that Gaussian noise is inserted during the generation process.

5. The method for generating a panoramic image based on a block according to claim 4, wherein: The intermediate panoramic content and block texture features are input into multiple distortion-aware upsampling blocks of the spherical content generator to obtain a panoramic image generation block. The distortion-aware upsampling block processes the intermediate panoramic content, including: For the Distortion-aware upsampling block pairs The intermediate panoramic content output by the distortion-aware upsampling block Perform interpolation of extended feature points and perform spherical convolution on the intermediate panoramic content after interpolation to obtain the image block content ; Block texture features Perform modulated convolution based on style coding and adjust the image block content Perform deconvolution operation; The block texture features after style coding modulation convolution are added to the image block content after deconvolution operation to obtain the first The intermediate panoramic content output by the distortion-aware upsampling block .

6. The method for generating a panoramic image based on a block according to claim 5, wherein: The block texture features after style coding modulation convolution are added to the image block content after deconvolution operation to obtain the first The intermediate panoramic content output by the distortion-aware upsampling block ,include: in, Indicates the The intermediate panoramic content output by the distortion-aware upsampling block, , Indicates the The intermediate panoramic content output by the distortion-aware upsampling block, represents the interpolation operation of extended feature points, represents the spherical convolution based on image blocks, represents the deconvolution operation, Indicates the The block texture features output by the distortion-aware upsampling block, express Convolution, whose modulation is Weighted.

7. The method for generating a panoramic image based on a block according to claim 1 or 2, characterized in that: The spherical local embedding module, the spherical content generator, and the discriminator constitute a panoramic image block generative adversarial network. The loss function of the panoramic image block generative adversarial network is as follows: in, is the loss function from the discriminator, which is expressed as is a non-saturated adversarial loss used to fit the probability distribution of the input, yes Regularization loss, used to enhance stability when training the discriminator, is the intensity control parameter of the regularization loss of the discriminator, is a regularization loss to encourage generation diversity, is the path length regularization loss used to align the generated panoramic image patch content with the global latent code Stay consistent, and represents when the discriminator predicts the coordinates of the input block loss, is the intensity control parameter of patch diversity loss; is the path regularization loss strength control parameter in StyleGAN, is the coordinate prediction loss intensity control parameter.

8. The method for generating a panoramic image based on a block according to claim 7, wherein: The convolution kernel size of the spherical convolution based on the image block in the panoramic image block generation adversarial network is Weight Initialized with the following values: ； in, Indicates that the size of the zero value matrix is , except that the center value is equal to ; Indicates the random number used for initialization.

Images