Panorama generation method and device, electronic equipment and storage medium

Abstract

The invention provides a panorama generation method and device, electronic equipment and a storage medium. The panorama generation method comprises the following steps: generating a first panorama with a first resolution based on an image description text; redrawing and amplifying the first panorama and adding image details to obtain a second panorama with a second resolution; wherein the first resolution ratio is smaller than the second resolution ratio. According to the method, image details are added while redrawing amplification is carried out on a full-incoming image, so that real local details are ensured while high resolution is achieved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a panoramic image generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] In the field of extended reality technology (such as virtual reality, augmented reality, and mixed reality), creating comfortable and immersive scenes can improve the user experience. With the development of large-scale model technology, users can generate images from text, enabling the creation of custom images.

[0003] In immersive extended reality scenarios, unlike ordinary text-to-image generation, 360° panoramic images are required, rather than single-view photographic images or images of a single object. Current 360° panoramic image generation technologies suffer from issues such as insufficient image resolution. Summary of the Invention

[0004] This disclosure provides a panoramic image generation method, apparatus, electronic device, and storage medium.

[0005] The following technical solution is adopted in this disclosure.

[0006] In some embodiments, this disclosure provides a panoramic image generation method, including:

[0007] Generate a first panoramic image with a first resolution based on image description text;

[0008] The first panoramic image is redrawn, enlarged, and image details are added to obtain a second panoramic image with a second resolution.

[0009] Wherein, the first resolution is smaller than the second resolution.

[0010] In some embodiments, this disclosure provides a panoramic image generation apparatus, including:

[0011] The text-based image unit is used to generate a first panoramic image with a first resolution based on the image-description text.

[0012] The control unit is used to redraw and enlarge the first panoramic image and add image details to obtain a second panoramic image with a second resolution.

[0013] Wherein, the first resolution is smaller than the second resolution.

[0014] In some embodiments, this disclosure provides an electronic device, including: at least one memory and at least one processor;

[0015] The memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the above method.

[0016] In some embodiments, this disclosure provides a computer-readable storage medium for storing program code that, when run by a processor, causes the processor to perform the methods described above.

[0017] The panoramic image generation method provided in this disclosure achieves high resolution while ensuring realistic local details by redrawing, enlarging, and supplementing details of the generated panoramic image. Attached Figure Description

[0018] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0019] Figure 1 This is a flowchart of a panoramic image generation method according to an embodiment of the present disclosure.

[0020] Figure 2 This is a flowchart of a panoramic image generation method according to an embodiment of the present disclosure.

[0021] Figure 3 This is a schematic diagram of a panoramic image generation method according to an embodiment of the present disclosure.

[0022] Figure 4 This is a schematic diagram of top and bottom repair of a panoramic image according to an embodiment of this disclosure.

[0023] Figure 5 This is a panoramic view of an embodiment of the present disclosure in ERP and CMP formats.

[0024] Figure 6 This is a schematic diagram illustrating the training of the first model according to an embodiment of this disclosure.

[0025] Figure 7 This is a schematic diagram illustrating the color adjustment of the base according to an embodiment of this disclosure.

[0026] Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0027] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0028] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0029] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0030] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0031] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0032] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0033] It should be understood that the various steps described in the method embodiments of this disclosure can be performed in sequence and / or in parallel. Furthermore, method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0034] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0035] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0036] It should be noted that the use of the word "a" in this disclosure is illustrative rather than restrictive, and those skilled in the art should understand that it should be understood as "one or more" unless otherwise expressly indicated in the context.

[0037] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0038] The solutions provided by the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0039] In the field of extended reality, such as virtual reality and augmented reality, immersing users in a space constructed within extended reality requires the creation of a comfortable extended reality space (which can be a purely virtual space or a space that blends virtual and real elements). The environmental images displayed in the extended reality space are 360° panoramic views, unlike a single photographic perspective or a single object. Related technologies for generating 360° panoramic images suffer from problems such as low resolution, low local realism, and a lack of stylistic diversity.

[0040] like Figure 1 As shown, Figure 1 This is a flowchart of a panoramic image generation method according to an embodiment of the present disclosure, which includes the following steps.

[0041] S11. Generate a first panoramic image with a first resolution based on the image description text.

[0042] In some embodiments, the executor of the method proposed in this disclosure may be a terminal, such as an extended reality device. Figure 3 As shown, the image description text (scene description text) can be text entered or selected by the user of the extended reality device. The image description text is used to describe the characteristics of the first panoramic image that the user wishes to generate, such as: a grassland at night with a full moon and stars in the sky. It can be that the image description text is input into the text-generated image model, or that the image model is generated from the text (e.g., ...). Figure 3 The dedicated text-based 360-degree image diffusion model is used to generate a first panoramic image with a first resolution. Figure 3 The initial image (in the text) is used by the text-to-image model to generate a panoramic image that matches the input text. This can be a diffusion model. The panoramic image can be a 360° spherical image. The first resolution can be a 720P image. For example... Figure 5 As shown, the format of the first panoramic image can be Equirectangular (ERP) format.

[0043] S12. Redraw and enlarge the first panoramic image and add image details to obtain a second panoramic image with a second resolution.

[0044] In some embodiments, after obtaining the first panoramic image at a first resolution, the first panoramic image will be redrawn and enlarged. During the redrawing and enlargement, in order to ensure the integrity of the image structure, image details need to be added to obtain a second panoramic image at a second resolution. The first resolution is smaller than the second resolution, and the second resolution can be 2K resolution. Figure 3 The initial image was redrawn and enlarged to 2K resolution using a general text-based image diffusion model.

[0045] In this embodiment, by magnifying and adding details to the generated panoramic image, high resolution is achieved while preserving realistic local details.

[0046] like Figure 2 As shown, Figure 2 This is a flowchart of a panoramic image generation method according to an embodiment of the present disclosure, which further includes the following after steps S11 and S12:

[0047] S13. Perform image restoration on the two opposite ends of the second panoramic image in the first direction.

[0048] In some embodiments, the first direction can be an axial direction passing through the center of the second panoramic image, which is also a spherical 360° image, and the first direction can be an axial direction from the North and South Poles. By performing image inpainting, imperfections in the second panoramic image can be removed, improving the overall image quality.

[0049] In some embodiments of this disclosure, a flowchart method for generating panoramic images further includes S14 and S15.

[0050] S14. The second panoramic image is magnified based on the generative adversarial network to obtain a third panoramic image with a third resolution.

[0051] In some embodiments, the second panoramic image is further magnified using a Generative Adversarial Network (GAN), specifically ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks). GAN's image magnification function can increase image sharpness while maintaining the second resolution, which is lower than the third resolution, which can be between 6K and 8K resolution.

[0052] S15. The third panoramic image is redrawn in blocks to obtain the fourth panoramic image.

[0053] In some embodiments, the image magnification function of GAN can maintain image sharpness, but lacks local realism. Therefore, it is necessary to introduce block-based high-definition redrawing, which can be achieved through... Figure 3 The general text diffusion model is used to redraw the image in blocks, thereby improving the local realism. The fourth panoramic image has the same resolution as the third panoramic image, and the local areas are more realistic.

[0054] Panoramic image generation schemes in related technologies can only generate panoramic images with a resolution of 1024×512, and lack high-definition magnification capabilities. Alternatively, they may use a single-segment magnification method to directly magnify to 6K resolution, resulting in a lack of detail and realism. In some embodiments of this disclosure, the above problems are considered, and a multi-stage processing flow is adopted. Each step has a specific goal and function, ensuring that the initial low-resolution first panoramic image is gradually improved to a high-quality fourth panoramic image. During the magnification process, by using text-to-image models, GAN models, etc., rich details can be added while maintaining the integrity of the image structure. Imperfections are removed by repairing the ends, improving the overall image quality. Block-based redrawing ensures the local realism of the image. In some embodiments of this disclosure, the multi-stage method of generating panoramic images achieves high resolution while ensuring realistic local details, resulting in a high-quality and detail-rich panoramic image effect.

[0055] In some embodiments of this disclosure, generating a first panoramic image with a first resolution based on the image description text includes: receiving input or selected image description text; displaying one or more candidate images generated based on the image description text; and, in response to a selection operation on a candidate image, using the selected candidate image as the first panoramic image.

[0056] In some embodiments, the method proposed in this disclosure can be used in extended reality devices to allow users to select and set a virtual scene of the extended real world within the device's settings. In the virtual scene settings, users can select one or more existing tags as image description text, or they can manually or verbally input the image description text. Tags are pre-set specific descriptive terms, such as: aerial view, flatland view, night view, day view, etc. After determining the image description text, a text-to-image model will generate one or more candidate images based on the image description text. The candidate images are panoramic images at a first resolution. One or more candidate images can be generated using a single text-to-image model, or multiple candidate images can be generated using multiple text-to-image models. The styles of the multiple candidate images can be different, and the user can choose their preferred candidate image, which will be used as the first panoramic image. By providing candidate images, the final generated panoramic image can not only satisfy the image description text but also best match the user's preferred style.

[0057] In some embodiments of this disclosure, redrawing and enlarging the first panoramic image and adding image details to obtain a second panoramic image with a second resolution includes: redrawing and enlarging the first panoramic image using a diffusion model, wherein, during the redrawing and enlarging process, a first value of denoising intensity is used to enlarge the first panoramic image, and the first value is greater than 0.5 and less than 0.7.

[0058] In some embodiments, the first panoramic image is redrawn and enlarged using a diffusion model, with denoising strength being a key parameter in the generated image. After enlarging the first panoramic image, it is encoded into a latent space image, noise is added to it, and the denoising strength is set to control how much noise is added to the latent space image; 0 indicates no noise is added, and 1 indicates adding the maximum amount of noise to make it a completely random tensor in the latent space image. Then, a noise predictor inputs the latent space image and image description text into the diffusion model and provides the predicted noise tensor to be subtracted. The noise tensor is subtracted from the latent space image to obtain a new image. The steps of predicting and subtracting the noise tensor are repeated a specific number of times, and then the latent space image is restored to a pixel space image to obtain the second panoramic image. If the denoising strength is 1, the obtained panoramic image is equivalent to the result of running a text-to-image tool using pure text instructions, i.e., generating an image from pure noise. By using a denoising strength of 0.65, most of the features of the first panoramic image are preserved, and details that match the image description text are automatically added.

[0059] In some embodiments of this disclosure, the first panoramic image is in ERP (Equirectangular) format; image restoration is performed on the two opposite ends of the second panoramic image in a first direction, including: converting the first panoramic image in ERP format to an image in CMP (Cubemap) format, restoring the top and bottom regions of the CMP format image, and then converting it back to ERP format.

[0060] In some embodiments, such as Figure 5 As shown, the ERP format projects latitude and longitude lines from a sphere onto a cylindrical surface, then unfolds the image along a generatrix of the cylinder. For example, unfolding a world map of the Earth onto a flat image based on latitude and longitude yields the corresponding ERP map. ERP and CMP formats are mutually convertible. Specifically, this can be achieved through... Figure 3 The image inpainting model in the text is used to perform top and bottom image restoration on CMP format images. The image inpainting model can be LAMA (large masking painting) model used to perform top and bottom image restoration on a second panorama. Figure 3Image restoration (such as top and bottom repair) can remove generated imperfections through image restoration, such as... Figure 4 As shown, Figure 4 The image on the left is a CMP file without top and bottom region restoration. Figure 4 The top left is the top area. Figure 4 The bottom left is the bottom area. Figure 4 The right side shows the CMP format area after top and bottom region repair. You can see that... Figure 4 The flaws in the center of the upper left and lower left images have been removed through restoration, making the images look more natural.

[0061] In some embodiments of this disclosure, the third panoramic image is redrawn in blocks, including: dividing the third panoramic image into n rows × m columns of local blocks, wherein adjacent local blocks have an overlapping area of ​​a first number of pixels; performing a second value of denoising intensity amplification operation on each local block, and performing overlapping fusion of the local blocks in the denoising step, wherein the second value is greater than 0.3 and less than 0.5.

[0062] In some embodiments, the third panoramic image can be in ERP format. For the third panoramic image, it is divided into n rows × m columns (e.g., 2 rows × 6 columns) of local patches. The local patches can maintain an overlap of 256 pixels to achieve a smooth transition. Each local patch is magnified with a denoising intensity of, for example, 0.4. The local patches are overlapped and fused in each denoising step (one step in the denoising iteration). This ensures a smooth transition while also supplementing details during magnification. Furthermore, because the second value is neither greater than 0.5 nor less than 0.3, details are guaranteed to change to a certain extent without significant alteration. In this embodiment, the panoramic image is divided into multiple local patches, and each local patch is magnified independently while preserving the overlap between local patches to ensure a smooth transition and overall consistency.

[0063] In some embodiments of this disclosure, the two local block images at both ends in the row and / or column directions have overlapping areas. In some embodiments, to ensure seamless connection between the leftmost and rightmost edges of the panoramic image in the row direction and the top and bottom edges in the column direction, when segmenting the local block images, it is ensured that the two local block images at both ends in the row and / or column direction still have overlapping areas. That is, when a local block image at one end is extracted, the overlapping area of ​​the local block image at the other end in the row and / or column direction is also included, thereby achieving seamless magnification. In this embodiment, the seam processing mechanism at both ends in the row and / or column direction ensures the continuity of the image and the visual effect.

[0064] To better illustrate the embodiments presented in this disclosure, the following is combined with... Figure 3To improve the resolution of the panoramic image and address issues such as unrealistic details, a multi-stage panoramic image linking method was employed, using multiple zoom-in steps to resolve these problems.

[0065] After the user inputs scene description text (image description text), a dedicated text-generated 360-degree image diffusion model is used to generate an initial ERP format skybox (first panoramic image) with a resolution of 720P.

[0066] The initial image is redrawn and enlarged using a general ERP (Epoch Times Per Second) diffusion model. To ensure the quality of the enlargement process, a denoising strength of 0.65 is used to obtain a 2K image (second panorama), which adds additional details while ensuring the correct ERP structure.

[0067] The second panoramic image in ERP format is converted to Cubemap format, and the central area of ​​the top and bottom (top and bottom) of the skybox is repaired using an image inpainting model (such as LAMA model) to remove the generated imperfections, and then converted back to ERP format.

[0068] Use ESRGAN to upscale GAN-based images to 6K or 8K.

[0069] Because GAN's image upscaling function can maintain image sharpness while lacking local realism, the algorithm also introduces a block-based redrawing function (similar to MultiDiffusion, but with added left and right seam functionality). The ERP image upscaling block approach is as follows: For a 6K or 8K panoramic image, it is divided into 2×6 local blocks (2 rows and 6 columns). A 256-pixel overlap is maintained between blocks for smooth transitions. Each block is upscaled with a 0.4 denoising intensity, and the blocks are overlapped and blended during each denoising step. To ensure seamless transition between the left and right sides of the panoramic image, the algorithm ensures that the rightmost and leftmost blocks still have an overlap when segmenting (meaning that when the rightmost block is extracted, it includes the leftmost overlapping area), thus achieving seamless upscaling.

[0070] In some embodiments of this disclosure, generating a first panoramic image with a first resolution based on image description text includes: inputting the image description text into a first model to generate a first panoramic image with a target style; prior to generating the first panoramic image with the first resolution based on the image description text, the method further includes: training the first model using a styled panoramic reference image to enable the first model to generate a styled panoramic image.

[0071] In some embodiments, the target style can be the style described in the image description text, or it can be a style automatically selected by the first model. The target style can be pastoral, urban, classical, etc. The first model can be a diffusion model. The first model needs to be trained before use. By using a styled panoramic reference image (the styled panoramic reference image is an existing panoramic image), the first model can generate a styled panoramic image, thus solving the problem of the panoramic image lacking style.

[0072] In some embodiments of this disclosure, a styled panoramic reference image is generated by: acquiring an existing panoramic reference image (which does not have a style at this time) and a style reference image with the highest feature similarity to the panoramic reference image; inputting the depth estimation result of the panoramic reference image into a depth controller, and inputting the style reference image into a style adapter; inputting the panoramic reference image, the output of the depth controller, and the output of the style adapter into the first model for graphic redrawing to obtain a styled panoramic reference image.

[0073] In some embodiments, such as Figure 6 As shown, in the first model ( Figure 6 During the training process of the controllable 360-degree image model, a panoramic reference image is first acquired. Figure 6 The panoramic reference image (without style) is used. Then, from an existing style image library, combined with content matching retrieval based on CLIP (Contrastive Language-Image Pre-training) feature similarity, the closest style reference image is selected as the style adapter (IP-Adapter). The style adapter is the adapter for the first model. The panoramic reference image is used for depth estimation (e.g., using Depth-Anything estimation), and the depth estimation result is input to the depth controller (e.g., ControlNet). The output of the depth controller and the output of the style adapter are the residual features of the intermediate layers of the diffusion denoising network (first network). Additional residuals allow the denoising network to achieve specific effects. These outputs are input to the first model for image redrawing, redrawing the panoramic reference image into a panoramic reference image with the style of the style reference image. A first numerical denoising intensity can be used during redrawing. The resolution of the panoramic reference image can be higher than the first resolution; by increasing the resolution during the training phase, the resolution of the generated style-rich first panoramic image is guaranteed. In this embodiment, a style-themed panoramic reference image is generated using the above method and used to train the first model, thereby obtaining a first model that can achieve multiple styles.

[0074] In some embodiments of this disclosure, the method further includes: generating a base; displaying a target panoramic image and displaying the base in the bottom area of ​​the target panoramic image; wherein the target panoramic image is a second panoramic image, a third panoramic image, or a fourth panoramic image.

[0075] In some embodiments, the target panoramic image can be a panoramic image displayed in the extended reality device, which can be the panoramic image viewed by the user when using the extended reality device. Any of the second, third, or fourth panoramic images can be used as the target panoramic image; therefore, the steps of generating the base and displaying the target panoramic image and the base can be performed after steps S12, S13, S14, or S15. By displaying the base when displaying the panoramic image, the user has a foothold when using the extended reality device and viewing the extended reality world. In some embodiments, the base can be a three-dimensional model. Although the target panoramic image has a bottom area, it is not a model but an image, so it cannot directly interact with the user's footsteps, resulting in a slightly less realistic experience. Displaying a three-dimensional model base in the extended reality world allows the user to interact with the base, making the experience more realistic.

[0076] In some embodiments of this disclosure, before generating the base, the method further includes: determining the viewpoint of the target panoramic image, determining whether to generate the base based on the viewpoint of the target panoramic image, and performing the step of generating the base in response to the viewpoint of the target panoramic image being a preset type of viewpoint.

[0077] In some embodiments, before generating the base, it is determined whether the target image to be displayed is a preset type of viewpoint, such as a high-altitude viewpoint. If the base is not displayed, the user would be suspended in the extended reality world, potentially causing acrophobia or dizziness. Therefore, when the target panoramic image's viewpoint is a preset type of viewpoint, the base needs to be generated and displayed; if the target panoramic image's viewpoint is not a preset type of viewpoint, the base does not need to be generated. In some embodiments, such as... Figure 7 As shown, the target panoramic image may be a panoramic view of a high-altitude environment (skybox). If this panoramic image is directly used as the background image of the extended reality world, it may cause feelings of acrophobia or dizziness, resulting in user discomfort. To at least partially solve this problem, in this embodiment, the base serves as the user's station in the extended reality space, improving the user experience.

[0078] In some embodiments, generating the base includes: dividing some or all of the pixels in the bottom region of the target panoramic image into buckets in HSV space; determining the HSV value of the interval with the most pixels in the bucket as the target HSV value; and adjusting the base color to the target HSV value.

[0079] In some embodiments, a target panoramic image is displayed around the user in extended reality. A portion (e.g., 25%) or all pixels in the bottom region of the target panoramic image are converted to HSV (Hue, Saturation, Value) and binned. The binning process can involve dividing each of the three HSV dimensions into multiple intervals (e.g., 30 equally divided intervals), forming three-dimensional bins. Each pixel is assigned to a corresponding bin, allowing for the counting of pixels within each bin. The bin containing the most pixels is identified; this bin represents the most common color feature at the bottom of the target panoramic image. An HSV value is selected from the interval corresponding to this bin as the target HSV value. The base (original base) can be a pre-set virtual base that was not originally present in the target panoramic image. The difference between the base color and the target HSV value can then be calculated as an offset. This offset is applied to the base's color map to adjust the overall base color, making it more harmonious with the color at the bottom of the target panoramic image. The adjusted base color is combined with the base model to create a base that better matches the bottom of the target panoramic image, thus creating a more color-matched virtual environment. In this way, the color of the virtual base is consistent with the color of the bottom of the panoramic image, reducing discomfort such as fear of heights and dizziness for users in the immersive experience and improving the quality of the user's virtual experience.

[0080] In this embodiment of the disclosure, a multi-stage panoramic image generation scheme is adopted: in order to achieve higher resolutions such as 6K or 8K and realistic local details, a multi-stage panoramic image generation scheme is designed to achieve high-quality and detailed generation effects while ensuring the correctness of the structure.

[0081] To address the lack of style data for panoramic images, a batch of style-specific panoramic image data was generated based on panoramic reference images and style reference images, thereby enabling the training of a first model with multiple styles.

[0082] The base design, based on the panoramic immersive scene, addresses issues like dizziness caused by directly viewing high-altitude scenery. It serves as a station for the user within an extended real-world space, thus creating the final immersive scene. To achieve a good color match between the base and the panoramic image, the HSV colors of the base are adjusted using the panoramic image's bottom HSV extraction function.

[0083] This disclosure also proposes a panoramic image generation apparatus, comprising:

[0084] The text-based image unit is used to generate a first panoramic image with a first resolution based on the image-description text.

[0085] The control unit is used to redraw and enlarge the first panoramic image and add image details to obtain a second panoramic image with a second resolution.

[0086] Wherein, the first resolution is smaller than the second resolution.

[0087] In some embodiments, the control unit is also configured to perform image restoration on two opposite ends of the second panoramic image in a first direction.

[0088] In some embodiments, the control unit is also configured to magnify the second panoramic image based on a generative adversarial network to obtain a third panoramic image with a third resolution;

[0089] The third panoramic image is redrawn in blocks to obtain the fourth panoramic image;

[0090] The second resolution is smaller than the third resolution.

[0091] In some embodiments, generating a first panoramic image with a first resolution based on the image description text includes:

[0092] Receive the input or selected image description text;

[0093] Display one or more candidate images generated based on the image description text;

[0094] In response to the selection operation of the candidate image, the selected candidate image is used as the first panoramic image.

[0095] In some embodiments, the first panoramic image is redrawn, enlarged, and image details are added to obtain a second panoramic image with a second resolution, including:

[0096] The first panoramic image is redrawn and enlarged using a diffusion model, wherein a first value of denoising intensity is used to enlarge the first panoramic image during the redrawing and enlargement process, and the first value is greater than 0.5 and less than 0.7.

[0097] In some embodiments, the first panoramic image is in ERP (Equirectangular) format;

[0098] Image restoration is performed on the two opposite ends of the second panoramic image in the first direction, including: converting the first panoramic image in ERP format to an image in CMP (Cubemap, stereoscopic texture projection) format, restoring the top and bottom areas of the CMP format image, and then converting it back to ERP format.

[0099] In some embodiments, the third panoramic image is redrawn in blocks, including:

[0100] The third panoramic image is divided into n rows × m columns of local block images, wherein adjacent local block images have an overlapping area of ​​a first number of pixels.

[0101] Each of the local image libraries is subjected to a second value of denoising intensity amplification operation, and the local block images are overlapped and fused in the denoising step; wherein the second value is greater than 0.3 and less than 0.5.

[0102] In some embodiments, the two local block diagrams at both ends in the row and / or column directions have overlapping areas.

[0103] In some embodiments, generating a first panoramic image with a first resolution based on image description text includes: inputting the image description text into a first model to generate a first panoramic image with a target style;

[0104] In some embodiments, the control unit is further configured to train the first model using a styled panoramic reference image before generating a first panoramic image with a first resolution based on the image description text, so that the first model is able to generate a styled panoramic image.

[0105] In some embodiments, the stylized panoramic reference image is generated in the following ways:

[0106] Obtain existing panoramic reference images and style reference images with the highest feature similarity to the panoramic reference images;

[0107] The depth estimation result of the panoramic reference map is input into the depth controller, and the style reference map is input into the style adapter;

[0108] The panoramic reference image, the output of the depth controller, and the output of the style adapter are input into the first model to perform graphic redrawing and obtain a styled panoramic reference image.

[0109] In some embodiments, the control unit is further configured to: generate a base; display a target panoramic view and display the base in the bottom area of ​​the target panoramic view;

[0110] The target panoramic image is a second panoramic image, a third panoramic image, or a fourth panoramic image.

[0111] In some embodiments, the control unit is further configured to: determine the viewpoint of the target panoramic image before generating the base, and determine whether to generate the base based on the viewpoint of the target panoramic image;

[0112] In some embodiments, the control unit is configured to: in response to the target panoramic view having a view of a preset type, perform the step of generating the base.

[0113] In some embodiments, the generation base includes:

[0114] Bucket some or all of the pixels in the bottom area of ​​the target panoramic image in HSV space;

[0115] The HSV value of the interval with the most pixels in the bucket is determined as the target HSV value;

[0116] Adjust the base color to the target HSV value.

[0117] For embodiments of the apparatus, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments. The apparatus embodiments described above are merely illustrative, and the modules described as separate modules may or may not be separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0118] The methods and apparatus of this disclosure have been described above based on embodiments and application examples. Furthermore, this disclosure also provides an electronic device and a computer-readable storage medium, which are described below.

[0119] The following is for reference. Figure 8 The figure illustrates a structural schematic of an electronic device (e.g., a terminal device or server) 800 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in the figure is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present disclosure.

[0120] Electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 802 or a program loaded from storage device 808 into random access memory (RAM) 803. RAM 803 also stores various programs and data required for the operation of electronic device 800. The processing device 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0121] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although an electronic device 800 with various devices is shown in the figure, it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0122] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure.

[0123] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0124] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0125] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0126] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods of the present disclosure.

[0127] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0129] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0130] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0131] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0132] According to one or more embodiments of this disclosure, a panoramic image generation method is provided, comprising:

[0133] Generate a first panoramic image with a first resolution based on image description text;

[0134] The first panoramic image is redrawn, enlarged, and image details are added to obtain a second panoramic image with a second resolution.

[0135] Wherein, the first resolution is smaller than the second resolution.

[0136] According to one or more embodiments of the present disclosure, a panoramic image generation method is provided, further comprising: performing image restoration on two opposite ends of the second panoramic image in a first direction;

[0137] According to one or more embodiments of the present disclosure, a panoramic image generation method is provided, which further includes: enlarging a second panoramic image based on a generative adversarial network to obtain a third panoramic image with a third resolution;

[0138] The third panoramic image is redrawn in blocks to obtain the fourth panoramic image;

[0139] The second resolution is smaller than the third resolution.

[0140] According to one or more embodiments of this disclosure, a panoramic image generation method is provided, which generates a first panoramic image with a first resolution based on the image description text, including:

[0141] Receive the input or selected image description text;

[0142] Display one or more candidate images generated based on the image description text;

[0143] In response to the selection operation of the candidate image, the selected candidate image is used as the first panoramic image.

[0144] According to one or more embodiments of this disclosure, a panoramic image generation method is provided, which redraws and enlarges a first panoramic image and adds image details to obtain a second panoramic image with a second resolution, including:

[0145] The first panoramic image is redrawn and enlarged using a diffusion model, wherein a first value of denoising intensity is used to enlarge the first panoramic image during the redrawing and enlargement process, and the first value is greater than 0.5 and less than 0.7.

[0146] According to one or more embodiments of this disclosure, a panoramic image generation method is provided, wherein the first panoramic image is in ERP (Equirectangular) format;

[0147] Image restoration is performed on the two opposite ends of the second panoramic image in the first direction, including: converting the first panoramic image in ERP format to an image in CMP (Cubemap, stereoscopic texture projection) format, restoring the top and bottom areas of the CMP format image, and then converting it back to ERP format.

[0148] According to one or more embodiments of this disclosure, a panoramic image generation method is provided, which redraws the third panoramic image in blocks, including:

[0149] The third panoramic image is divided into n rows × m columns of local block images, wherein adjacent local block images have an overlapping area of ​​a first number of pixels.

[0150] Each of the local image libraries is subjected to a second value of denoising intensity amplification operation, and the local block images are overlapped and fused in the denoising step; wherein the second value is greater than 0.3 and less than 0.5.

[0151] According to one or more embodiments of this disclosure, a panoramic image generation method is provided, wherein two local block images at both ends in the row and / or column directions have overlapping areas.

[0152] According to one or more embodiments of this disclosure, a panoramic image generation method is provided, which generates a first panoramic image with a first resolution based on image description text, including: inputting the image description text into a first model to generate a first panoramic image with a target style;

[0153] Before generating a first panoramic image with a first resolution based on image description text, the method further includes: training the first model using a styled panoramic reference image so that the first model can generate a styled panoramic image.

[0154] According to one or more embodiments of this disclosure, a panoramic image generation method is provided, wherein a styled panoramic reference image is generated in the following manner:

[0155] Obtain existing panoramic reference images and style reference images with the highest feature similarity to the panoramic reference images;

[0156] The depth estimation result of the panoramic reference map is input into the depth controller, and the style reference map is input into the style adapter;

[0157] The panoramic reference image, the output of the depth controller, and the output of the style adapter are input into the first model to perform graphic redrawing and obtain a styled panoramic reference image.

[0158] According to one or more embodiments of the present disclosure, a panoramic image generation method is provided, further comprising: generating a base;

[0159] Display a panoramic view of the target and display the base in the bottom area of ​​the panoramic view;

[0160] The target panoramic image is a second panoramic image, a third panoramic image, or a fourth panoramic image.

[0161] According to one or more embodiments of this disclosure, a panoramic image generation method is provided, which further includes, before generating the base, determining the viewpoint of the target panoramic image and determining whether to generate the base based on the viewpoint of the target panoramic image.

[0162] In response to the target panoramic image having a viewpoint of a preset type, the step of generating the base is performed.

[0163] According to one or more embodiments of this disclosure, a panoramic image generation method is provided, wherein the generation base includes:

[0164] Bucket some or all of the pixels in the bottom area of ​​the target panoramic image in HSV space;

[0165] The HSV value of the interval with the most pixels in the bucket is determined as the target HSV value;

[0166] Adjust the base color to the target HSV value.

[0167] According to one or more embodiments of this disclosure, a panoramic image generation apparatus is provided, comprising:

[0168] The text-based image unit is used to generate a first panoramic image with a first resolution based on the image-description text.

[0169] The control unit is used to redraw and enlarge the first panoramic image and add image details to obtain a second panoramic image with a second resolution.

[0170] Wherein, the first resolution is smaller than the second resolution.

[0171] According to one or more embodiments of the present disclosure, an electronic device is provided, including: at least one memory and at least one processor;

[0172] The at least one memory is used to store program code, and the at least one processor is used to call the program code stored in the at least one memory to execute the method described in any one of the above.

[0173] According to one or more embodiments of the present disclosure, a computer-readable storage medium is provided for storing program code that, when executed by a processor, causes the processor to perform the methods described above.

[0174] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0175] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0176] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A panoramic image generation method characterized by comprising: The method comprises: generating a first panorama with a first resolution based on an image description text; redrawing and enlarging the first panorama and adding image details to obtain a second panorama with a second resolution; wherein the first resolution is less than the second resolution.

2. The method of claim 1, wherein, The method further comprises: image inpainting on two opposite ends of the second panorama in a first direction.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: enlarging the second panorama based on a generative adversarial network to obtain a third panorama with a third resolution; block redrawing of the third panorama to obtain a fourth panorama; wherein the second resolution is less than the third resolution.

4. The method of claim 1, wherein, Generating a first panorama with a first resolution based on an image description text comprises: receiving an input or selected image description text; displaying one or more candidate images generated based on the image description text; in response to a selection operation on a candidate image, taking the selected candidate image as the first panorama.

5. The method of claim 1, wherein, Redrawing and enlarging the first panorama and adding image details to obtain a second panorama with a second resolution comprises: using a diffusion model to redraw and enlarge the first panorama, wherein a first value of denoising intensity is used to enlarge the first panorama during redrawing and enlargement, and the first value is greater than 0.5 and less than 0.

7.

6. The method of claim 2, wherein: the first panorama is in an ERP (Equirectangular, Equirectangular Projection) format; image inpainting on two opposite ends of the second panorama in a first direction comprises: converting the first panorama in the ERP format to an image in the CMP (Cubemap, Cubemap Projection) format, inpainting the top and bottom areas of the image in the CMP format, and converting back to the ERP format.

7. The method of claim 3, wherein, The block redrawing of the third panorama comprises: dividing the third panorama into n rows and m columns of local block images, wherein adjacent local block images have an overlapping area of a first number of pixels; performing an enlargement operation on each local block image with a second value of denoising intensity, and performing overlapping fusion of the local block images in the denoising step; wherein the second value is greater than 0.3 and less than 0.

5.

8. The method of claim 7, wherein: the two local block images at the ends in the row direction and / or the column direction have overlapping areas.

9. The method of claim 1, wherein: generating a first panorama with a first resolution based on an image description text comprises: inputting the image description text into a first model to generate a first panorama with a target style; before generating a first panorama with a first resolution based on an image description text, the method further comprises: training the first model using a panorama reference image with a style, so that the first model can generate a panorama with the style.

10. The method of claim 9, wherein, The panorama reference image with a style is generated by: obtaining an existing panorama reference image and a style reference image with the highest feature similarity to the panorama reference image; inputting a depth estimation result of the panorama reference image into a depth controller and inputting the style reference image into a style adapter; Input the panoramic reference picture, the output of the depth controller and the output of the style adapter into the first model for graphic redrawing to obtain a panoramic reference picture with a style.

11. The method according to any one of claims 1 to 10, characterized in that, Further comprising: generating a base; displaying a target panoramic picture and displaying the base in a bottom region of the target panoramic picture; wherein the target panoramic picture is a second panoramic picture, a third panoramic picture or a fourth panoramic picture.

12. The method of claim 11, wherein, before the generating the base, further comprising: determining a perspective of the target panoramic picture, and determining whether to generate the base based on the perspective of the target panoramic picture; in response to the perspective of the target panoramic picture being a preset type of perspective, performing the step of generating the base.

13. The method of claim 11, wherein, The generating the base comprises: bucketing part or all of pixels in a bottom region of the target panoramic picture in HSV space; determining an HSV value of an interval with the most pixels in the bucket as a target HSV value; coloring the base to the target HSV value.

14. A panorama generation apparatus characterized by comprising: comprising: a text-to-picture unit configured to generate a first panoramic picture with a first resolution based on image description text; a control unit configured to redrawing and enlarging the first panoramic picture and adding image details to obtain a second panoramic picture with a second resolution; wherein the first resolution is less than the second resolution.

15. An electronic device comprising: at least one memory and at least one processor; wherein the at least one memory is configured to store program code, and the at least one processor is configured to invoke the program code stored in the at least one memory to execute the method of any one of claims 1-13.

16. A computer-readable storage medium configured to store program code, which when executed by a processor, causes the processor to execute the method of any one of claims 1-13.

Images