Block-based panoramic image generation method

Abstract

The invention provides a block-based panoramic image generation method, which comprises the following steps: acquiring global potential codes, image block potential features and image block coordinates, scaling image blocks to partial spherical blocks in advance according to the coordinates of a global panoramic image, and performing regular grid division on a spherical tangent space, the image blocks mapped to the spherical surface are uniformly distributed; inputting the content into a plurality of spherical distortion sensing convolution blocks for spherical convolution based on the image blocks to obtain local embedding of the image blocks; inputting local embedding of the image blocks into a stylization header layer for stylization header processing to obtain middle panoramic content and block texture features; and inputting the middle panoramic content and the block texture features into a plurality of distortion perception up-sampling blocks to obtain panoramic image generation blocks, inputting the panoramic image generation blocks into a discriminator, comparing the panoramic image generation blocks with panoramic image real blocks, and outputting a panoramic image when a comparison condition is satisfied. By implementing the method, a high-quality and consistent panoramic image can be generated.

Description

Technical Field

[0001] The present invention belongs to the technical field of panoramic image generation, and in particular relates to a block-based panoramic image generation method. Background Art

[0002] With the rapid development of virtual reality (VR) and augmented reality (AR) technologies, panoramic images have become a key medium for achieving immersive visual experience. Panoramic images are usually captured by a high-resolution fisheye camera array, covering a 360°×180° panoramic field of view (FOV). Due to the specific format requirements of panoramic content, specially designed equipment is required to capture and display panoramic images, and high bandwidth support is also required to cope with the storage and transmission of massive data. Therefore, the technical difficulties in the acquisition, transmission and display of panoramic images highlight the necessity of using artificial intelligence generated content (AIGC) to enrich the generation of panoramic images to meet the diverse needs of immersive experience.

[0003] Deep probabilistic generative models (PGMs) have made significant progress in generating realistic content, especially in the task of generating planar images. However, for panoramic image generation, existing methods usually map the panoramic format to a planar image and then use a planar PGM for generation, which is regularized by using a cylindrical coordinate prior and a spherical weighted loss. However, due to the existence of rectangular image edges, such methods can only generate pseudo-panoramic images in nature and cannot ensure image edge connectivity, resulting in inconsistent content under panoramic perspectives. To address this inconsistency, some methods introduce global hard constraints, such as using multi-view and overall panoramic cues as conditions to guide generation. More advanced techniques include introducing horizontally shifted circular padding in generative adversarial networks (GANs) and using latent recombination in diffusion models. Although these methods alleviate some of the problems by ensuring consistency between the leftmost and rightmost edges of the image, they require the generation of the entire image at the same time during training and inference, resulting in huge consumption of computational resources, especially when panoramic image generation requires high resolution. More importantly, most existing methods rely on additional information such as single / multi-view and rich contextual cues, which are not always available in practical panoramic image generation scenarios. Therefore, developing an efficient and fundamental unconditional panoramic image generation method has become an urgent need to meet the emerging high-quality immersive experience. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a block-based panoramic image generation method to meet the demand for generating high-quality panoramic images.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] The embodiment of the present invention provides a block-based panoramic image generation method, comprising: obtaining a global latent code, an image block latent feature, and an image block coordinate, wherein the image block is pre-scaled to a partial spherical block according to the coordinate of the global panoramic image, and a regular grid is used for division on the spherical tangent space, so that the image blocks presented on the spherical surface are evenly distributed; inputting the global latent code, the image block latent feature, and the image block coordinate into a plurality of spherical distortion perception convolution blocks in a spherical local embedding module to perform spherical convolution based on the image block to obtain a local embedding of the image block; and inputting the local embedding of the image block into a spherical content The stylized head layer of the generator is subjected to stylized head processing to obtain intermediate panoramic content and block texture features; the intermediate panoramic content and block texture features are input into multiple distortion-aware upsampling blocks of the spherical content generator to obtain a panoramic image generation block, wherein the distortion-aware upsampling block is composed of two main branches, one branch is used to perform convolution processing on the block texture features, and the other branch is used to perform spherical convolution processing based on the image block on the intermediate panoramic content and fuse the block texture features after the convolution processing; the panoramic image generation block is input into the discriminator to be compared with the real block of the panoramic image, and when the comparison condition is met, the panoramic image is output.

[0007] Optionally, the step of inputting the global latent code, the image block latent features and the image block coordinates into the multiple spherical distortion-aware convolution blocks in the spherical local embedding module to perform spherical convolution based on the image block to obtain the local embedding of the image block includes: for the lth spherical distortion-aware convolution block, performing an extended feature point interpolation operation on the image block latent features output by l-1 spherical distortion-aware convolution blocks according to the image block coordinates to obtain spherical neighborhood grid points, wherein the spherical neighborhood grid points are merged with the position code in the coordinate channel; convolving the spherical neighborhood grid points with the convolution kernel to obtain a first convolution result, wherein the parameters in the convolution kernel are modulated according to the style code, and the style code is obtained by transforming the global latent code through a mapping network; adding the first convolution result to the image block latent feature to form a residual structure, modulating and convolving the residual structure according to the style code to obtain a second convolution result, and the second convolution result is used as the image block latent feature output by the lth spherical distortion-aware convolution block.

[0008] Optionally, the image blocks are pre-scaled to partial spherical blocks according to the coordinates of the global panoramic image, and a regular grid is used in the spherical tangent space to make the image blocks mapped to the spherical surface uniformly distributed. The division process includes: inverse spherical projection of the tangent space grid, mapping it back to the spherical surface, and determining the spherical surface and the tangent space according to the following formula: Each corresponding adjacent grid coordinate p i,j ={θ i ,ψ j}:

[0009]

[0010] Among them, θ and ψ are the The coordinate points of coincidence, Represents the spacing of the grid in the tangent space, which is a constant;

[0011] Obtain the panoramic image construction range, and determine the scaling ratio of the panoramic image coordinates to the local spherical block according to the panoramic image construction range; determine the grid point coordinates of the local spherical block according to the coincident coordinate points in the spherical coordinates, the panoramic image construction range, and the scaling ratio; determine the grid point coordinates of the local spherical block according to the coincident coordinate points in the spherical coordinates, the panoramic image construction range, and the scaling ratio; The corresponding adjacent grid coordinates p i,j ={θ i ,ψ j} and the grid point coordinates of the local spherical block to determine the tangent space The grid point coordinates are projected back to the spherical surface; the spherical surface is divided into a plurality of image blocks according to the plurality of grid point coordinates, and the plurality of image blocks are evenly distributed on the spherical surface.

[0012] Optionally, the convolving the spherical neighborhood grid points and the convolution kernel to obtain a first convolution result includes:

[0013]

[0014] in, represents the first convolution result of the image block potential feature convolution based on the output of the l-1th spherical distortion-aware convolution block, SPConv[·] represents the spherical convolution based on the image block, represents the spherical neighborhood grid points obtained by performing extended feature point interpolation operations based on the potential features of the image block output by the l-1th spherical distortion-aware convolution block, {i, j}, {m, n} are the local grid point indices on the tangent space, Represents the convolution kernel in the image block-based spherical convolution, and the convolution kernel size is k×k.

[0015] Optionally, the adding of the first convolution result and the potential features of the image block to form a residual structure, and performing modulated convolution on the residual structure according to the style coding to obtain the second convolution result includes:

[0016]

[0017] in, represents the second convolution result of the lth spherical distortion-aware convolution block, Represented by the style code g s Modulated convolution, while Conv 1×1 [·] represents a 1×1 convolution of the residual structure, According to the first convolution result of the image block potential feature convolution output by the l-1th spherical distortion aware convolution block, Represents the latent features of the image patch output by the l-1th spherical distortion-aware convolutional block.

[0018] Optionally, the intermediate panoramic content and the block texture features are input into a plurality of distortion-aware upsampling blocks of the spherical content generator to obtain a panoramic image generation block, and the distortion-aware upsampling block processes the block texture features, including:

[0019]

[0020] in, represents the block texture feature output by the lth distortion-aware upsampling block, Represented by the style code g′ s Modulated convolution, represents the block texture feature output by the l-1th distortion-aware upsampling block, and denote the two learnable parameters of the lth distortion-aware upsampling block, and σ1 and σ2 denote the insertion of Gaussian noise during the generation process.

[0021] Optionally, the intermediate panoramic content and the block texture features are input to a plurality of distortion-aware upsampling blocks of the spherical content generator to obtain a panoramic image generation block, and the processing of the intermediate panoramic content by the distortion-aware upsampling block includes: for the lth distortion-aware upsampling block, processing the intermediate panoramic content output by the l-1th distortion-aware upsampling block; Perform interpolation operation on extended feature points, and perform spherical convolution based on image blocks on the intermediate panoramic content after interpolation operation to obtain the image block content Block texture features Perform style-coded modulated convolution and apply it to the image block content Perform deconvolution operation; add the block texture features after style coding modulation convolution to the image block content after deconvolution operation to obtain the intermediate panoramic content output by the lth distortion-aware upsampling block

[0022] Optionally, the block texture features after the style coding modulation convolution are added to the image block content after the deconvolution operation to obtain the intermediate panoramic content output by the lth distortion-aware upsampling block include:

[0023]

[0024] in, represents the intermediate panoramic content output by the lth distortion-aware upsampling block, IntP represents the intermediate panoramic content output by the l-1th distortion-aware upsampling block, k [·] represents the interpolation operation of extended feature points, SPConv[·] represents the spherical convolution based on image blocks, and DeConv[·] represents the deconvolution operation. represents the block texture feature output by the lth distortion-aware upsampling block, represents a 1×1 convolution, whose modulation is g′ s Weighted.

[0025] Optionally, the spherical local embedding module, the spherical content generator and the discriminator constitute a panoramic image block generation adversarial network, and the loss function of the panoramic image block generation adversarial network is as follows:

[0026] L Gen =-L Dis +λ Div L Div +λ Pat L Pat +λ Co L Co ,

[0027] Among them, L Dis is the loss function from the discriminator, which is expressed as L Dis =L Adv +λ R1 L R1 +λ Co L Co ,L Adv is a non-saturated adversarial loss used to fit the probability distribution of the input, L R1 is the R1 regularization loss, which is used to enhance stability when training the discriminator, and λ R1 is the intensity control parameter of the regularization loss of the discriminator, L Div is a regularization loss to encourage generation diversity, L Pat is the path length regularization loss used to make the generated panoramic image patch content consistent with the global latent code g z Keep consistent, and L Co represents the l1 loss when the discriminator predicts the coordinates of the input patch, λ Div is the intensity control parameter of the patch diversity loss; Pat is the path regularization loss strength control parameter in StyleGAN, λ Co is the coordinate prediction loss intensity control parameter.

[0028] Optionally, the convolution kernel size of the spherical convolution based on the image block in the panoramic image block generation adversarial network is a weight k Initialized with the following values:

[0029]

[0030] in, represents the zero-value matrix of size k×k, except that the central value is equal to 1; η represents the random number used for initialization.

[0031] The present invention provides a block-based panoramic image generation method, which constructs two core modules: Spherical Local Embedding (SLE) and Spherical Content Synthesizer (SCS). The purpose of the SLE module is to embed the features and positions of local image blocks into the basic texture, and then the SCS module aggregates and generates more refined image block content. Since the SLE and SCS modules operate on the basis of local spherical image block features, the panoramic image block generation adversarial network (SP-GAN) can generate continuous and consistent spherical image blocks. The SP-GAN proposed in this embodiment can generate high-quality and consistent panoramic images. In addition, the image block-based spherical convolution proposed in the embodiment of the present invention acts on the local spherical image block. The division method of the image block ensures that the grid on the sphere is evenly distributed, so that the spherical convolution can naturally handle the deformation problem of the panoramic image. In addition, thanks to the block-by-block processing strategy, the SP-GAN can be flexibly expanded to image generation of different resolutions based on a single training model. The SP-GAN proposed in this embodiment shows excellent performance in terms of generation quality, computational efficiency and high-resolution generalization ability.

[0032] Other advantages, objectives and features of the present invention will be described in the following description and will be apparent to those skilled in the art to some extent, or those skilled in the art may be taught from the practice of the present invention. The objectives and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:

[0034] Figure 1 is a specific example flow chart of a method for generating a panoramic image based on a block in the present invention;

[0035] Figure 2 Schematic diagram of the network architecture of the panoramic image block generative adversarial network (SP-GAN) in the present invention;

[0036] Figure 3 is a conversion diagram of the convolution kernel from the ERP image to the panoramic image during the deformation process between the panoramic image and the planar format mentioned in the present invention;

[0037] Figure 4 It is a mapping diagram of the convolution kernel from the tangent space to the panoramic image and the ERP image in the present invention;

[0038] Figure 5 In the present invention, Schematic diagram of 3×3SPConv centered at ;

[0039] Figure 6 A schematic diagram of a process of generating an overall panoramic image from panoramic image blocks in the present invention;

[0040] Figure 7 It is a principle block diagram of a specific example of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, it can also be the internal connection of two components, it can be a wireless connection, or it can be a wired connection. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0043] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0044] The embodiment of the present invention provides a method for generating a panoramic image based on a block, such as Figure 1 As shown, including:

[0045] S101, obtaining a global latent code, an image block latent feature, and an image block coordinate, wherein the image block is pre-scaled to a partial spherical block according to the coordinate of the global panoramic image, and a regular grid is used in the spherical tangent space so that the image blocks presented when mapped to the spherical surface are evenly distributed;

[0046] S102, inputting the global latent code, the latent features of the image block, and the coordinates of the image block into a plurality of spherical distortion-aware convolution blocks in a spherical local embedding module to perform spherical convolution based on the image block to obtain a local embedding of the image block;

[0047] S103, inputting the local embedding of the image block into the stylized head layer of the spherical content generator for stylized head processing to obtain intermediate panoramic content and block texture features;

[0048] S104, inputting the intermediate panoramic content and the block texture features into a plurality of distortion-aware upsampling blocks of a spherical content generator to obtain a panoramic image generation block, wherein the distortion-aware upsampling block is composed of two main branches, one branch is used to perform convolution processing on the block texture features, and the other branch is used to perform spherical convolution processing based on image blocks on the intermediate panoramic content and fuse the block texture features after the convolution processing;

[0049] S105, input the panoramic image generation block into the discriminator, and compare it with the panoramic image real block. When the comparison condition is met, the panoramic image is output.

[0050] For example, this embodiment proposes to use a panoramic image block generative adversarial network (SP-GAN) including a spherical local embedding module (SLE module), a spherical content generator (SCS module) and a discriminator. The network framework structure is as follows: Figure 2 shown.

[0051] Since the input of the panoramic image block generative adversarial network mentioned above is the global latent code, image block latent features and image block coordinates, and convolution processing is required, the image blocks need to be divided first. However, the panoramic image has a curved structure, and the existing two-dimensional convolution cannot process it directly. The current commonly used method is to use ERP to project the panoramic image into a plane format, and then perform subsequent processing on the ERP image. However, when processing plane images, the essence of two-dimensional convolution is to aggregate information on a regular grid of image pixels. Unfortunately, due to the deformation between the panoramic image and the plane format, the regular grid in the ERP image cannot guarantee a uniform distribution of the grid on the sphere, such as Figure 3 Therefore, when applying two-dimensional convolution to the ERP image, this embodiment needs to sample on an irregular grid to ensure that the grid of the panoramic image on the spherical surface is evenly distributed, as shown in FIG. Figure 4 shown.

[0052] The specific sampling method, that is, the division method of the image blocks is as follows:

[0053] Based on the local similarity between spherical manifold and Euclidean space, this embodiment uses a regular grid in the spherical tangent space as a proxy for the uniformly distributed grid near each point on the sphere. Specifically, for each point p = {θ, ψ} on the sphere, use τp Denote its tangent space by represents the local grid on the tangent space. Correspondingly, the convolution kernel on the tangent space is expressed as The two-dimensional convolution operation on the tangent space is defined as:

[0054]

[0055] Among them, {i,j} and {m,n} represent and The index of describes the convolution operation performed on the tangent space of a given spherical point p.

[0056] To approximate the grid in tangent space The content of the image is projected back to the sphere using inverse spherical projection, i.e., the panoramic image. coincides with the point p = {θ, ψ} on the sphere, so the point on the sphere and in the tangent space can be calculated Each corresponding adjacent grid coordinate point p i,j ={θ i ,ψ j}, the relationship is given by the following formula:

[0057]

[0058] in represents the interval of the grid in the tangent space. is a constant. From formula (2), we can see that when i→0 and j→0, θ i →θ and ψ j →ψ. More importantly, when i=0 and j=0, we have This shows that the points on the tangent space It completely coincides with the point p={θ,ψ} on the sphere.

[0059] Due to the block-based nature of SP-GAN, the projection coordinates p in (2) i,j ={θ i ,ψ j} does not inherently guarantee the existence of panoramic image pixels. More importantly, when a larger interval is incorrectly chosen When p i,j ={θ i ,ψ j} may even exceed the range of the provided spherical block; this will lead to inter-block inconsistencies, i.e., blocking artifacts in the generated panoramic image. Instead, choose a smaller The value may result in a limited receptive field during training. To solve this problem, this embodiment rescales the coordinates of the global panoramic image to local spherical blocks.

[0060] More specifically, when the spherical block constructed from the panoramic image has a range of [θ s ,θ e ] and [ψ s ,ψ e ], according to the spherical block range constructed by the panoramic image, the scaling ratio of the panoramic image coordinates to the local spherical block is determined. Specifically, and Get the scaling ratio of longitude and latitude. Determine the grid point coordinates of the local spherical block based on the coincident coordinate points in the spherical coordinates, the panoramic image construction range, and the scaling ratio. The specific formula is as follows:

[0061]

[0062] Based on formula (3), by keeping the same deviation in the tangent space as in the global coordinates, we can obtain the deviation from the tangent space τ p The grid point coordinates projected back to the sphere Specifically, according to the spherical surface and the tangent space The corresponding adjacent grid coordinates p i,j ={θ i ,ψ j} and the grid point coordinates of the local spherical block Determine the tangent space τ p In The grid point coordinates projected back to the sphere The specific formula is as follows:

[0063]

[0064] Due to s θ <1 and s ψ <1, essentially expanding the coordinates in the local spherical block, thereby reducing the possibility of projecting beyond the spherical block from the tangent space. Therefore, this embodiment does not empirically select Instead, is set to a constant and the projection is adaptively adjusted according to the ratio of the image block size to the entire panoramic image, i.e., by s θ and ψ Then, bilinear interpolation and boundary filling strategies are used to approximate each point of the given local image patch coordinates. Since SP-GAN only requires image patches for training and inference, rescaling to local coordinates ensures the stability of training and continuity between generated image patches.

[0065] In the implementation, the local coordinates on the given spherical block are For point p, the image block-based spherical convolution (SPConv) proposed in this embodiment has the formula:

[0066]

[0067] in Indicates that it is located The spherical grid point at the position is given by formula (4). In addition, in formula (5), Centered on p, K is a convolution kernel that can be learned during training. Based on the proposed SPConv, the deformation problem of panoramic block images can be solved and panoramic images can be generated based on blocks.

[0068] In addition to addressing the deformation problem of panoramic images, the SP-GAN proposed in this embodiment also achieves continuity between the generated panoramic image blocks. More importantly, the generation process needs to gradually increase the resolution through convolution operations to ultimately generate high-resolution panoramic image blocks. Therefore, this embodiment constructs a new SP-GAN for panoramic image block generation based on the proposed SPConv. When its potential spherical features are connected, the network is able to generate continuous and consistent panoramic image blocks.

[0069] SP-GAN, such as Figure 2 As shown in FIG. 1 , the spherical local embedding module includes multiple spherical distortion-aware convolution blocks, each of which includes an image block-based spherical convolution module. The image block-based spherical convolution module performs image block-based spherical convolution on the input data as shown in formula (5). By continuously stacking spherical distortion-aware convolution blocks, the spherical local embedding module can convert the features of the potential panoramic image block into the basic spherical local embedding P s , while ensuring that the generated embedding can accurately reflect the distortion of the panoramic image. The local embedding of the image block P s As the input of the spherical content generator, it is further processed. The spherical content generator consists of two parts. The first part is the stylized head layer, which is used to embed the local s The stylized head is processed to obtain the intermediate panoramic content and block texture features. The second part is multiple distortion-aware upsampling blocks. Each distortion-aware upsampling block consists of two main branches, which are used to gradually synthesize the panoramic intermediate features and content. One branch convolves the block texture features through convolution and deconvolution, and the other branch is used to perform spherical convolution processing on the intermediate panoramic content based on image blocks and fuse the block texture features after convolution processing.

[0070] In the SP-GAN proposed in this embodiment, on the one hand, the SLE module is responsible for embedding local spherical content, and accepts global latent codes, image block latent features, and corresponding coordinates as input. The global latent code is used to control the overall style and ensure consistency between blocks. The image block latent features are closely related to the local structure of the generated block and are controlled by the corresponding image block coordinates. On the other hand, the SCS module is responsible for generating fine textures based on local embedding and global latent codes. It is worth noting that in order to ensure the continuity of the block content in the generated panoramic image, no padding is used in each layer of convolution operations in SP-GAN. This approach allows SP-GAN to generate high-quality blocks in a coarse-to-fine manner, while cleverly adapting to the curvature characteristics of the spherical blocks.

[0071] The quality of the generated blocks is evaluated by the discriminator to ensure their consistency with the real image blocks. If the generated blocks meet the comparison conditions, the blocks are spliced ​​into a complete panoramic image. In SP-GAN, the discriminator is trained adversarially with the generator, where the generator generates increasingly realistic image blocks to deceive the discriminator, while the discriminator strives to improve its discrimination ability to distinguish between real and generated data. This adversarial process promotes the joint progress of the generator and the discriminator, so that the final output panoramic image should be visually indistinguishable from the real scene while maintaining structural and semantic accuracy.

[0072] This embodiment provides a block-based panoramic image generation method, and constructs two core modules: Spherical Local Embedding (SLE) and Spherical Content Synthesizer (SCS). The purpose of the SLE module is to embed the features and positions of local image blocks into the base texture, and then the SCS module aggregates and generates more refined image block content. Since the SLE and SCS modules operate on the basis of local spherical image block features, SP-GAN can generate continuous and consistent spherical image blocks. The SP-GAN proposed in this embodiment can generate high-quality and consistent panoramic images. In addition, the image block-based spherical convolution proposed in the embodiment of the present invention acts on local spherical image blocks. The division method of the image block ensures that the grid on the sphere is evenly distributed, so that the spherical convolution can naturally handle the deformation problem of the panoramic image. In addition, thanks to the block-by-block processing strategy, SP-GAN can be flexibly expanded to image generation of different resolutions based on a single training model. The SP-GAN proposed in this embodiment shows excellent performance in terms of generation quality, computational efficiency, and high-resolution generalization ability.

[0073] The method of this example is verified on the MatterPort3D dataset and compared with an algorithm that can generate a panorama. Due to the rectangular characteristics of the panoramic image, the experimental indicators select the image blocks in the center, across the boundary, and random square areas and the images in the dataset to calculate the FID and IS values, which are recorded as FID cent 、FID bord 、FID rand and IS cent , IS bord , IS rand , where the lower the FID, the better, and the higher the IS, the better. The experimental results of the performance comparison with the best previous block panoramic image generation method are shown in the following table:

[0074] method FID IS <![CDATA[FID cent ]]> <![CDATA[IS cent ]]> <![CDATA[FID bord ]]> <![CDATA[IS bord ]]> <![CDATA[FID rand ]]> <![CDATA[IS rand ]]> This example 27.55 4.08 26.74 4.07 29.51 4.09 26.40 4.07 Previous best practices 30.66 3.69 29.82 3.71 32.59 3.67 29.56 3.70

[0075] As an optional implementation, the global latent code, the image block latent features, and the image block coordinates are input into a plurality of spherical distortion-aware convolution blocks in the spherical local embedding module to perform spherical convolution based on the image block to obtain a local embedding of the image block, including:

[0076] For the lth spherical distortion-aware convolution block, an extended feature point interpolation operation is performed on the image block potential features output by the l-1 spherical distortion-aware convolution blocks according to the image block coordinates to obtain spherical neighborhood grid points, wherein the spherical neighborhood grid points are combined with position codes in the coordinate channel;

[0077] Convolving the spherical neighborhood grid points with a convolution kernel to obtain a first convolution result, wherein parameters in the convolution kernel are modulated according to a style code, and the style code is obtained by transforming a global latent code through a mapping network;

[0078] The first convolution result is added to the potential feature of the image block to form a residual structure, and the residual structure is modulated and convolved according to the style coding to obtain a second convolution result, which is used as the potential feature of the image block output by the lth spherical distortion perception convolution block.

[0079] Exemplarily, the input of the SLE module includes the global latent code g z , image block potential feature P z and the image patch coordinates C p , then output the local embedding P of the block s In addition, the spherical local embedding module consists of multiple spherical distortion-aware convolution (DAC) blocks. Each DAC block is built based on the SPConv operation proposed in the above formula (5), which is then merged with the global style information. Specifically, it is assumed that the coordinate range of the panoramic image block to be generated is [θ s ,θe ] and latitude [ψ s ,ψ e ] Then, the image block potential feature P z (i,j) is given by the coordinate θ s ≤θ i ≤θ e and ψ s ≤ψ j ≤ψ e Then, the image block potential feature P z With C p Connect together by channel and input into the DAC block of the SLE module.

[0080] like Figure 5 As shown, Figure 5 Indicates Schematic diagram of a 3×3 SPConv with center. Specifically, for the lth DAC block, the latent features of the input image block are first interpolated for the subsequent SPConv The spherical neighborhood grid points are obtained. The interpolation operation in SPConv is centered on each point (i, j). The principle is shown in the following formula:

[0081]

[0082] Among them, IntP k [·] represents the interpolation operation of extending feature points. Zoom in k×k times. In order to embed the position information, use {θ i ,ψ j} position code. It should be noted that in this embodiment The position code is merged into the coordinate channel of . Then, the interpolated spherical neighborhood grid points are calculated according to formula (5). Perform element-by-element product operation with the convolution kernel of size k×k to obtain the first convolution result. In this way, the output of SPConv is the first convolution result and are of the same size, namely:

[0083]

[0084] in It means that SPConv has a learnable convolution kernel, whose parameters are More importantly, W gs Also influenced by image style encoding g s The modulation, while the style coding g s The global latent code g is pre-coded by the mapping network z Transformed.

[0085] Then, the first convolution result With the latent features of the image patch Add them together to form a residual structure, and then modulate the residual structure according to the style code to obtain the second convolution result This process can be expressed by the following formula:

[0086]

[0087] in Represented by the image style code g s Modulated convolution, while Conv 1×1 [·] represents a 1×1 convolution of the residual structure. In fact, the global latent code g z First processed by the mapping network, it is expressed as a style encoding Similar to the style encoding in Style-GAN2.

[0088] This embodiment provides a block-based panoramic image generation method, which combines the global latent coding with the latent features of the image block, so that the model can strike a balance between local details and global context, which helps to generate image blocks that are visually coherent and match the global scene. The style coding obtained by transforming the global latent coding through the mapping network can modulate the parameters in the convolution kernel, so that the generated image blocks are not only consistent in content, but also match the target scene in style, thereby enhancing the realism and diversity of the generated images.

[0089] As an optional implementation, the intermediate panoramic content and the block texture features are input into a plurality of distortion-aware upsampling blocks of the spherical content generator to obtain a panoramic image generation block, and the processing of the block texture features by the distortion-aware upsampling block includes:

[0090]

[0091] For example, given the local embedding P in the SLE module s , the proposed SCS module is designed to be based on P s and image style encoding g′ s Generate detailed panoramic content. More specifically, the local embedding P s First, it is processed by the stylisation head (SH) to initially form the intermediate panoramic content I p and block texture feature P t The SH architecture is the same as that in Style-GAN2. The stylized head layer proposed in this embodiment also incorporates the image style encoding g′ s To ensure the content consistency of the entire generated panoramic image. It should be noted that the image style encoding g′ of the SCS module sis obtained through an additional mapping network, whose architecture is the same as the mapping network in the SLE module, but with different weights and training procedures. Then, the intermediate panoramic content I p and block texture feature P t It is input into multiple Distortion-Aware Upsampling (DAU) blocks, which consists of two main branches, respectively, for gradually synthesizing panoramic intermediate features and content.

[0092] For the lth DAU block, use represents the intermediate panoramic content output by the l-1th distortion-aware upsampling block, and is expressed as represents the block texture feature output by the l-1th distortion-aware upsampling block. For the block texture feature output by the lth distortion-aware upsampling block It can be expressed by the following formula:

[0093]

[0094] Where DeConv[·] represents the deconvolution operation; Represented by the style code g′ s Modulated convolution, and Represents two learnable parameters of the lth DAU block. At the same time, Gaussian noise σ1 and σ2 are inserted in the generation process to enhance the diversity of generated content.

[0095] As an optional implementation, the intermediate panoramic content and block texture features are input into multiple distortion-aware upsampling blocks of the spherical content generator to obtain a panoramic image generation block, wherein the distortion-aware upsampling block is composed of two main branches, one branch is used to perform convolution processing on the block texture features, and the other branch is used to perform spherical convolution processing based on image blocks on the intermediate panoramic content and fuse the block texture features after the convolution processing, including:

[0096] For the l-th distortion-aware upsampling block, the intermediate panoramic content output by the l-1 distortion-aware upsampling blocks Perform interpolation operation on extended feature points, and perform spherical convolution based on image blocks on the intermediate panoramic content after interpolation operation to obtain the image block content

[0097] Block texture features Perform style-coded modulated convolution and apply it to the image block content Perform deconvolution operation;

[0098] The block texture features after style coding modulation convolution are added to the image block content after deconvolution operation to obtain the intermediate panoramic content output by the lth distortion-aware upsampling block

[0099] For example, due to the intermediate panoramic content The generated panoramic image is gradually depicted and becomes the final output of SP-GAN. This embodiment proposes a new branch to further solve the spherical distortion problem by combining SP-GAN. Specifically, first, for the l-th distortion-aware upsampling block, the intermediate panoramic content output by the l-1 distortion-aware upsampling block is Perform interpolation operation on extended feature points, and perform spherical convolution based on image blocks on the intermediate panoramic content after interpolation operation to obtain the image block content The specific formula is as follows:

[0100]

[0101] It should be noted that in the SCS module, the feature already contains structural information and position clues. Therefore, the position embedding operation is skipped to avoid misleading the process of generating details. However, SPConv is still necessary to further enhance the panoramic content details and handle panoramic distortion. In addition, in order to aggregate the texture features from the block Information on block texture features Perform style-coded modulated convolution and apply it to the image block content Perform a deconvolution operation, add the block texture features after the style coding modulation convolution to the image block content after the deconvolution operation, and obtain the intermediate panoramic content output by the lth distortion-aware upsampling block The obtained intermediate panoramic content Combined with block texture features, the image quality is higher. The specific formula is as follows:

[0102]

[0103] in, represents a 1×1 convolution, whose modulation is g′ s The weighting aims to change the number of channels to match the number of channels of the output block content. With the help of arrive The process improves the texture and details generated within the image block. By stacking DAU blocks, the SCS module is able to generate correct and realistic panoramic content in a block-by-block and gradual generation manner from low resolution to high resolution.

[0104] This embodiment provides a block-based panoramic image generation method, which adds the block texture features after style coding modulation convolution and the image block content after deconvolution operation, that is, in each distortion-aware upsampling block, the intermediate panoramic content and the block texture features are aggregated. This fusion of different feature levels combines the semantic information of deep features and the texture details of shallow features, which helps to generate richer and more realistic image blocks and obtain high-quality panoramic images.

[0105] As an optional implementation, the spherical local embedding module, the spherical content generator and the discriminator constitute a panoramic image block generation adversarial network, and the loss function of the panoramic image block generation adversarial network is as follows:

[0106] L Gen =-L Dis +λ Div L Div +λ Pat L Pat +λ Co L Co ,

[0107] Among them, L Dis is the loss function from the discriminator, which is expressed as L Dis =L Adv +λ R1 L R1 +λ Co L Co ,L Adv is a non-saturated adversarial loss used to fit the probability distribution of the input, L R1 is the R1 regularization loss, which is used to enhance stability when training the discriminator, and λ R1 is the intensity control parameter of the regularization loss of the discriminator, L Div is a regularization loss to encourage generation diversity, L Pat is the path length regularization loss used to make the generated panoramic image patch content consistent with the global latent code g z Keep consistent, and L Co represents the l1 loss when the discriminator predicts the coordinates of the input patch, λ Div is the intensity control parameter of the patch diversity loss; Pat is the path regularization loss strength control parameter in StyleGAN, λ Co is the coordinate prediction loss intensity control parameter.

[0108] For example, since SP-GAN only needs local panoramic image blocks for training, when all operations are restricted to local blocks, the continuity between the generated panoramic image blocks can be guaranteed at a basic level. In this way, the connected local block potential features can generate seamless image content. More importantly, the non-padding design and SPConv based on local coordinates essentially constrain the scope of the convolution operation within the local block, ensuring the content continuity of SP-GAN when generating panoramic images based on image blocks. Therefore, the remaining key task is to ensure that the content at different locations in the panoramic image can transition smoothly.

[0109] To solve this problem, SP-GAN combines adversarial training (based on Style-GAN2) and coordinate prediction loss for optimization. The loss function in SP-GAN is designed as follows:

[0110] L Gen =-L Dis +λ Div L Div +λ Pat L Pat +λ Co L Co , (12)

[0111] Among them, L Dis is the loss function from the discriminator, L Div is a regularization loss to encourage generation diversity, λ Div is the intensity control parameter of the patch diversity loss, L Pat is the path length regularization loss used to make the generated panoramic image patch content consistent with the global latent code g z Keep consistent, Pat is the path regularization loss strength control parameter in StyleGAN, and L Co represents the l1 loss when the discriminator predicts the coordinates of the input patch, λ Co is the coordinate prediction loss strength control parameter. In addition, the method of training the discriminator is similar:

[0112] L Dis =L Adv +λ R1 L R1 +λ Co L Co , (13)

[0113] Among them, L Adv is a non-saturated adversarial loss used to fit the probability distribution of the input, L R1 is the R1 regularization loss, λ R1is the intensity control parameter of the regularization loss of the discriminator, which is used to enhance stability when training the discriminator of Style-GAN2. It should also be clarified in this embodiment that the development of the discriminator architecture is similar to that in Style-GAN2, and the entire training process is completely based on panoramic image patches. Therefore, while ensuring high-quality generation, SP-GAN can also efficiently utilize fewer computing resources to generate high-resolution panoramic images.

[0114] More importantly, a special weight initialization method is designed for SP-GAN when training it in an end-to-end manner. Weight W of the convolution kernel size k for the spherical convolution based on the image patch in the panoramic image patch generation adversarial network gs Initialized with the following values:

[0115]

[0116] in, denotes a zero-value matrix of size k×k, except that the center value is equal to 1; η denotes a random number used for initialization. With the above weight initialization, SP-GAN starts with almost no effect and gradually learns to handle panoramic distortion during training. This initialization strategy can improve training stability in practice.

[0117] In the training phase, in order to generate continuous image patches, the latent features of a spherical patch are sampled from the complete latent sphere, such as Figure 6 As shown, the boxes of different lines represent different spherical blocks, and the boxes in the latent space and image domain correspond one to one. More importantly, these potential block features overlap with each other, further enhancing the continuity between the generated blocks, while also providing flexibility for the resolution of the synthetic panorama. It should be pointed out that the complete potential feature is pure Gaussian noise, which is used for unconditional generation. On this basis, the SP-GAN proposed in this embodiment can also be extended to other application scenarios and generated in combination with auxiliary information such as text, views, and sketches. When the potential block features are connected from the left and right sides, the closed-loop characteristics are also naturally guaranteed.

[0118] The present application also provides an electronic device, such as Figure 7 As shown, a processor 501 and a memory 502, wherein the processor 501 and the memory 502 may be connected via a bus or other means.

[0119] The processor 501 may be a central processing unit (CPU). The processor 501 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0120] The memory 502 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the block-based panoramic image generation method in the embodiment of the present invention. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory.

[0121] The memory 502 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 502 may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0122] The one or more modules are stored in the memory 502, and when executed by the processor 501, the following is performed: Figure 1 A block-based panoramic image generation method in the illustrated embodiment.

[0123] For details of the above electronic equipment, please refer to Figure 1 The corresponding related descriptions and effects in the illustrated embodiments can be understood and will not be repeated here.

[0124] This embodiment also provides a computer storage medium, which stores computer executable instructions, and the computer executable instructions can execute the block-based panoramic image generation method in any of the above method embodiments. The storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

[0125] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A block-based panoramic image generation method, characterized in that: include: Acquire global latent codes, image block latent features and image block coordinates, wherein the image blocks are pre-scaled to partial spherical blocks according to the coordinates of the global panoramic image, and regular grid division is adopted on the spherical tangent space so that the image blocks presented on the spherical surface are evenly distributed; The global latent code, the latent features of the image block, and the coordinates of the image block are input into a plurality of spherical distortion-aware convolution blocks in the spherical local embedding module to perform spherical convolution based on the image block to obtain a local embedding of the image block; The local embedding of the image block is input into the stylized head layer of the spherical content generator for stylized head processing to obtain the intermediate panoramic content and block texture features; Inputting the intermediate panoramic content and the block texture features into a plurality of distortion-aware upsampling blocks of a spherical content generator to obtain a panoramic image generation block, wherein the distortion-aware upsampling block consists of two main branches, one branch is used to perform convolution processing on the block texture features, and the other branch is used to perform image block-based spherical convolution processing on the intermediate panoramic content and fuse the convolution-processed block texture features; The panoramic image generation block is input into the discriminator and compared with the real block of the panoramic image. When the comparison conditions are met, the panoramic image is output.

2. The method for generating a panoramic image based on a block according to claim 1, characterized in that: The method includes inputting the global latent code, the image block latent features and the image block coordinates into a plurality of spherical distortion-aware convolution blocks in a spherical local embedding module to perform spherical convolution based on the image block to obtain a local embedding of the image block, including: For the lth spherical distortion-aware convolution block, an extended feature point interpolation operation is performed on the image block potential features output by the l-1 spherical distortion-aware convolution blocks according to the image block coordinates to obtain spherical neighborhood grid points, wherein the spherical neighborhood grid points are combined with position codes in the coordinate channel; Convolving the spherical neighborhood grid points with a convolution kernel to obtain a first convolution result, wherein parameters in the convolution kernel are modulated according to a style code, and the style code is obtained by transforming a global latent code through a mapping network; The first convolution result is added to the potential feature of the image block to form a residual structure, and the residual structure is modulated and convolved according to the style coding to obtain a second convolution result, which is used as the potential feature of the image block output by the lth spherical distortion perception convolution block.

3. The method for generating a panoramic image based on a block according to claim 2, characterized in that: The image blocks are pre-scaled to partial spherical blocks according to the coordinates of the global panoramic image, and are divided into regular grids on the spherical tangent space so that the image blocks mapped onto the spherical surface are evenly distributed. The division process includes: The tangent space mesh is inversely projected onto the spherical surface and mapped back onto the sphere. The following formula is used to determine the Each corresponding adjacent grid coordinate p i,j ={θ i ,ψ j }: Among them, θ and ψ are the The coordinate points of coincidence, Represents the spacing of the grid in the tangent space, which is a constant; Obtaining a panoramic image construction range, and determining a scaling ratio of the panoramic image coordinates to the local spherical block according to the panoramic image construction range; Determine the grid point coordinates of the local spherical block according to the coincident coordinate points in the spherical coordinates, the panoramic image construction range, and the scaling ratio; According to the spherical surface and the tangent space The corresponding adjacent grid coordinates p i,j ={θ i ,ψ j } and the grid point coordinates of the local spherical block to determine the tangent space The coordinates of the grid points projected back onto the spherical surface; According to the coordinates of multiple grid points, the spherical surface is divided into multiple image blocks, and the multiple image blocks are evenly distributed on the spherical surface.

4. A block-based panoramic image generation method according to claim 2 or 3, characterized in that: The convolution of the spherical neighborhood grid points and the convolution kernel to obtain a first convolution result includes: in, represents the first convolution result of the image block potential feature convolution based on the output of the l-1th spherical distortion-aware convolution block, SPConv[·] represents the spherical convolution based on the image block, represents the spherical neighborhood grid points obtained by performing extended feature point interpolation operations based on the potential features of the image block output by the l-1th spherical distortion-aware convolution block, {i, j}, {m, n} are the local grid point indices on the tangent space, Represents the convolution kernel in the image block-based spherical convolution, and the convolution kernel size is k×k.

5. The method for generating a panoramic image based on a block according to claim 4, characterized in that: The first convolution result is added to the potential feature of the image block to form a residual structure, and the residual structure is modulated and convolved according to the style coding to obtain a second convolution result, including: in, represents the second convolution result of the lth spherical distortion-aware convolution block, Represented by the style code g s Modulated convolution, while Conv 1×1 [·] represents a 1×1 convolution of the residual structure, According to the first convolution result of the image block potential feature convolution output by the l-1th spherical distortion aware convolution block, Represents the latent features of the image patch output by the l-1th spherical distortion-aware convolutional block.

6. A method for generating a panoramic image based on a block according to any one of claims 1 to 3, characterized in that: The intermediate panoramic content and block texture features are input into a plurality of distortion-aware upsampling blocks of a spherical content generator to obtain a panoramic image generation block, and the distortion-aware upsampling block processes the block texture features, including: in, represents the block texture feature output by the lth distortion-aware upsampling block, Represented by the style code g′ s Modulated convolution, represents the block texture feature output by the l-1th distortion-aware upsampling block, and denote the two learnable parameters of the lth distortion-aware upsampling block, and σ1 and σ2 denote the insertion of Gaussian noise during the generation process.

7. The method for generating a panoramic image based on a block according to claim 6, characterized in that: The intermediate panoramic content and block texture features are input into a plurality of distortion-aware upsampling blocks of a spherical content generator to obtain a panoramic image generation block, and the distortion-aware upsampling block processes the intermediate panoramic content, including: For the l-th distortion-aware upsampling block, the intermediate panoramic content output by the l-1 distortion-aware upsampling blocks Perform interpolation operation on extended feature points, and perform spherical convolution based on image blocks on the intermediate panoramic content after interpolation operation to obtain the image block content Block texture features Perform style-coded modulated convolution and apply it to the image block content Perform deconvolution operation; The block texture features after style coding modulation convolution are added to the image block content after deconvolution operation to obtain the intermediate panoramic content output by the lth distortion-aware upsampling block 8. The method for generating a panoramic image based on a block according to claim 7, characterized in that: The block texture features after style coding modulation convolution are added to the image block content after deconvolution operation to obtain the intermediate panoramic content output by the lth distortion-aware upsampling block include: in, represents the intermediate panoramic content output by the lth distortion-aware upsampling block, IntP represents the intermediate panoramic content output by the l-1th distortion-aware upsampling block, k [·] represents the interpolation operation of extended feature points, SPConv[·] represents the spherical convolution based on image blocks, and DeConv[·] represents the deconvolution operation. represents the block texture feature output by the lth distortion-aware upsampling block, represents a 1×1 convolution, whose modulation is g′ s Weighted.

9. A method for generating a panoramic image based on a block according to any one of claims 1 to 3, characterized in that: The spherical local embedding module, the spherical content generator and the discriminator constitute a panoramic image block generation adversarial network, and the loss function of the panoramic image block generation adversarial network is as follows: L Gen =-L Dis +λ Div L Div +λ Pat L Pat +λ Co L Co , Among them, L Dis is the loss function from the discriminator, which is expressed as L Dis =L Adv +λ R1 L R1 +λ Co L Co ,L Adv is a non-saturated adversarial loss used to fit the probability distribution of the input, L R1 is the R1 regularization loss, which is used to enhance stability when training the discriminator, and λ R1 is the intensity control parameter of the regularization loss of the discriminator, L Div is a regularization loss to encourage generation diversity, L Pat is the path length regularization loss used to make the generated panoramic image patch content consistent with the global latent code g z Keep consistent, and L Co represents the l1 loss when the discriminator predicts the coordinates of the input patch, λ Div is the intensity control parameter of the patch diversity loss; Pat is the path regularization loss strength control parameter in StyleGAN, λ Co is the coordinate prediction loss intensity control parameter.

10. The method for generating a panoramic image based on a block according to claim 9, characterized in that: The weight of the convolution kernel size k of the spherical convolution based on the image block in the panoramic image block generation adversarial network Initialized with the following values: in, represents the zero-value matrix of size k×k, except that the central value is equal to 1; η represents the random number used for initialization.

Images