Panoramic image generation method and device, electronic equipment and storage medium

Abstract

The invention discloses a panoramic image generation method and apparatus, an electronic device and a storage medium. The panoramic image generation method comprises the steps of obtaining text description information of a to-be-generated target panoramic image; and generating a target panoramic image based on a pre-trained diffusion repair model and the text description information, the diffusion repair model being obtained by performing small sample fine tuning or zero sample fine tuning on a target diffusion model based on a control network ControlNet. According to the method, the panoramic image required by the user can be conveniently generated.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application No. 63 / 578,830, entitled “DATA-EFFICIENT TEXT TO PANORAMA GENERATION,” filed with the U.S. Patent Office on August 25, 2023, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of image processing technology, and more specifically, to a method, device, electronic device and storage medium for generating a panoramic image. Background Art

[0004] With the advancement of science and technology, panoramic images are increasingly used in various scenarios, such as displaying panoramic images containing content required by users in virtual reality (VR), augmented reality (AR), and mixed reality (MR) scenarios to provide users with an immersive user experience. In related technologies, when generating a panoramic image required by a user, it is usually necessary to shoot multiple images in the corresponding scene and then stitch the multiple images together, which is inconvenient when generating a panoramic image required by a user. Summary of the invention

[0005] The present application proposes a method, device, electronic device and storage medium for generating a panoramic image, which can conveniently generate a panoramic image required by a user.

[0006] In a first aspect, an embodiment of the present application provides a method for generating a panoramic image, the method comprising: obtaining text description information of a target panoramic image to be generated; generating a target panoramic image based on a pre-trained diffusion restoration model and the text description information, wherein the diffusion restoration model is obtained by performing small sample fine-tuning or zero sample fine-tuning on a target diffusion model based on a control network ControlNet.

[0007] In second aspect, an embodiment of the present application provides a device for generating a panoramic image, the device comprising: an information acquisition module and an image generation module, wherein the information acquisition module is used to acquire text description information of a target panoramic image to be generated; the image generation module is used to generate a target panoramic image based on a pre-trained diffusion restoration model and the text description information, wherein the diffusion restoration model is obtained by performing small sample fine-tuning or zero sample fine-tuning on a target diffusion model based on a control network ControlNet.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the panoramic image generation method provided in the first aspect above.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the method for generating a panoramic image provided in the first aspect.

[0010] The solution provided by the present application generates a target panoramic image by acquiring text description information of a target panoramic image to be generated, based on the text description information and a diffusion repair model obtained by performing small sample fine-tuning or zero sample fine-tuning on a target diffusion model based on a control network ControlNet, thereby generating a panoramic image that meets user needs based on the text description information, thereby improving the convenience of panoramic image generation, and the diffusion repair model used is obtained by performing small sample fine-tuning or zero sample fine-tuning on a target diffusion model based on a control network ControlNet, thereby reducing the requirement for the amount of training data for a model used to generate a panoramic image, thereby reducing the implementation cost and complexity of generating a panoramic image based on text. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A schematic flow chart of a method for generating a panoramic image according to an embodiment of the present application is shown.

[0013] Figure 2 A schematic flow chart of a method for generating a panoramic image according to another embodiment of the present application is shown.

[0014] Figure 3 A schematic diagram of an image block provided in an embodiment of the present application is shown.

[0015] Figure 4 A schematic diagram of an image block after distortion processing provided in an embodiment of the present application is shown.

[0016] Figure 5A schematic diagram showing a mask image corresponding to an image block after distortion processing provided in an embodiment of the present application is shown.

[0017] Figure 6 A schematic diagram of an image block after restoration processing provided by an embodiment of the present application is shown.

[0018] Figure 7 A schematic diagram showing an effect of generating a panoramic image provided by an embodiment of the present application is shown.

[0019] Figure 8 A schematic diagram showing a principle of generating a panoramic image provided by an embodiment of the present application is shown.

[0020] Fig. 9 A schematic flow chart of a method for generating a panoramic image according to yet another embodiment of the present application is shown.

[0021] Fig.10 A schematic flow chart of a method for generating a panoramic image according to yet another embodiment of the present application is shown.

[0022] Fig.11 A schematic flow chart of a method for generating a panoramic image according to yet another embodiment of the present application is shown.

[0023] Fig.12 A schematic diagram showing the principle of obtaining panoramic description information of a sample panoramic image.

[0024] Fig.13 A block diagram of a device for generating a panoramic image according to an embodiment of the present application is shown.

[0025] Fig.14 It is a block diagram of an electronic device for executing the method for generating a panoramic image according to an embodiment of the present application.

[0026] Fig.15 It is a storage unit of an embodiment of the present application for storing or carrying a program code for implementing a method for generating a panoramic image according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0028] As an emerging technology, panoramic image stitching has developed rapidly in recent years and has attracted more and more attention from researchers. Currently, panoramic images have become a hot topic and key technology in computer simulation, computer vision simulation, image processing, computer special effects, and virtual reality research, and play an important role in geological surveys, military reconnaissance, medical minimally invasive surgery, aerospace, video conferencing and many other fields.

[0029] Compared with flat images, panoramic images can record 360° images around the required scene, giving people a strong sense of reality and three-dimensionality, so they have broad application prospects in many industries such as film and television, tourism, and real estate. At present, most of the methods for generating panoramic images are synthesized using multiple images from different angles, and the production process is complex, cumbersome, and inefficient.

[0030] Although, with the development of artificial intelligence (AI) technology, it is possible to generate the required panoramic images through AI to reduce the cost, but in the related technology, when training the model for generating panoramic images, a large number of panoramic images are needed as sample data to train the model, so there is also inconvenience.

[0031] In response to the above problems, the inventors proposed the panoramic image generation method, device, electronic device and storage medium provided in the embodiments of the present application, which can realize the generation of panoramic images required by users based on text description information, improve the convenience of panoramic image generation, and the diffusion repair model used is obtained by fine-tuning the target diffusion model based on the control network ControlNet with small samples or zero samples, so it can reduce the training data volume requirement of the model used to generate panoramic images, thereby reducing the implementation cost and complexity of generating panoramic images based on text. Among them, the specific panoramic image generation method is described in detail in the subsequent embodiments.

[0032] The method for generating a panoramic image provided by an embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0033] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for generating a panoramic image provided by an embodiment of the present application. In a specific embodiment, the method for generating a panoramic image is applied to Fig.13 The panoramic image generating device 400 and the electronic device 100 equipped with the panoramic image generating device 400 are shown in FIG. Fig.14 ). Figure 1 The process shown in FIG. 1 is described in detail, and the method for generating a panoramic image may specifically include the following steps:

[0034] Step S110: obtaining text description information of the target panoramic image to be generated.

[0035] In the embodiment of the present application, the text description information may be text information describing the target panoramic image to be generated, and may also be understood as panoramic description information of the target panoramic image. For example, the text description information may include text information such as the real scene corresponding to the target panoramic image required, the scene content included in the target panoramic image required, and the image style.

[0036] In some implementations, the electronic device may obtain the input information and obtain the above text description information based on the input information. The input information may be information in various modalities, such as text description, user-drawn images, depth images, videos, etc.

[0037] In a possible implementation, the electronic device may display an information input interface and detect an operation in the information input interface; when an input operation in the information input interface is detected, the input text description information is determined according to the input operation.

[0038] In one possible implementation, the electronic device may acquire images or videos from other devices; then, a visual language model, such as a pre-trained model for unified visual language understanding and generation (Bootstrapping Language-Image Pre-training, BLIP), may be used to acquire text description information of a target panoramic image to be generated for the acquired images or videos.

[0039] Step S120: Generate a target panoramic image based on a pre-trained diffusion restoration model and the text description information, wherein the diffusion restoration model is obtained by performing small sample fine-tuning or zero sample fine-tuning on a target diffusion model based on a control network ControlNet.

[0040] After obtaining the above text description information, the electronic device can generate a target panoramic image based on a pre-trained diffusion repair model and the text description information. When generating the target panoramic image, an image of one of the perspectives can be generated based on the text description information, and then, based on the generated image, multiple perspectives of the panorama (360 degrees) can be obtained through iterative distortion and repair; in each iteration, the generated perspective is distorted to a new perspective, and then the distorted image is repaired based on the above text description information and the diffusion repair model, so that the missing part of the image is repaired, thereby obtaining an image of a new perspective; finally, multiple perspectives are pre-stitched to obtain the required target panoramic image.

[0041] In the embodiment of the present application, considering that the diffusion model has been extensively trained on many perspectives, only a small amount of data is needed to ensure that the generated image meets the characteristics of the panorama. Therefore, the above diffusion repair model is obtained by fine-tuning the target diffusion model based on the control network ControlNet with a small sample or zero sample, thereby reducing the demand for panoramic data when fine-tuning the model, thereby reducing the implementation cost and complexity of generating panoramic images based on text. In addition, in order to ensure the consistency of each part of the generated panorama, ControlNet is used to control the scene content that needs to be generated by the diffusion repair model at each perspective.

[0042] Among them, the diffusion model is a progressive denoising network, which needs to gradually restore the image from the noisy image, so as to achieve the effect of gradually generating the image from the random noise. When applied to the panoramic image generation method provided in the embodiment of the present application, the diffusion model is mainly used to repair the missing content in the distorted image. When using the diffusion model to repair the distorted image, the control network ControlNet can be combined to further control the diffusion, so that the diffusion model can accept text description information and input conditions, thereby controlling the content repaired by the diffusion model to ensure consistency with images from other perspectives.

[0043] The method for generating a panoramic image provided in an embodiment of the present application obtains text description information of a target panoramic image to be generated, generates a target panoramic image based on the text description information and a diffusion repair model obtained by performing small sample fine-tuning or zero sample fine-tuning on a target diffusion model based on a control network ControlNet, thereby enabling generation of a panoramic image that meets user needs based on the text description information, thereby improving the convenience of generating a panoramic image. The diffusion repair model used is obtained by performing small sample fine-tuning or zero sample fine-tuning on a target diffusion model based on a control network ControlNet, thereby reducing the requirement for the amount of training data for a model for generating a panoramic image, thereby reducing the implementation cost and complexity of generating a panoramic image based on text.

[0044] See also Figure 2 , Figure 2 A schematic diagram of a method for generating a panoramic image according to another embodiment of the present application is shown. The method for generating a panoramic image is applied to the above electronic device. Figure 2 The process shown in FIG. 1 is described in detail, and the method for generating a panoramic image may specifically include the following steps:

[0045] Step S210: obtaining text description information of the target panoramic image to be generated.

[0046] In the embodiment of the present application, step S210 can refer to the contents of the aforementioned embodiment and will not be described again here.

[0047] Step S220: Generate an initial image block based on the pre-trained image generation model and the text description information.

[0048] In an embodiment of the present application, when a target panoramic image is generated based on a pre-trained diffusion restoration model and text description information, an initial image block can be generated based on a pre-trained image generation model and the above text description information, so that image blocks at other positions can be generated based on the initial image block, thereby obtaining multiple image blocks for splicing into the target panoramic image. The above image generation model can be a model such as a conditional diffusion model that can generate an image based on text description information.

[0049] In some embodiments, when generating the initial image block based on the above image generation model, the target prompt information for inputting into the image generation model can be determined based on the text description information, and the target prompt information is used to prompt the image generation model to generate an image block at the center position of the scene image that is consistent with the text description information; the target prompt information is input into the image generation model to obtain the image block output by the image generation model as the initial image block. In other words, the generated image block can be an image block at the center position of the panorama to be generated, so that the initial image block can be used as a key frame, and then iterative distortion and repair in different directions based on the key frame are performed to obtain all the image blocks used for splicing into the above target panoramic image.

[0050] In the above implementation, the above target prompt information can be used as a prompt (prompt word) required to be input into the image generation model when implementing generative artificial intelligence (Artificial Intelligence Generated Content, AIGC). After providing the prompt to the image generation model, the image generation model can understand the user intention of the prompt and generate an image that matches the user intention.

[0051] Step S230: Based on the diffusion restoration model and the text description information, and taking the initial image block as a starting position, generate a plurality of first image blocks along a first direction.

[0052] In the embodiment of the present application, after acquiring the above initial image block, the electronic device can generate multiple first image blocks along the first direction based on the above diffusion restoration model and text description information, with the initial image block as the starting position. In other words, multiple image blocks along the first direction can be generated through iterative distortion and restoration based on the above initial image block.

[0053] In some embodiments, when generating multiple first image blocks along the first direction, it can be achieved in the following way: taking the initial image block as the target image block, performing a first distortion process on the target image block to obtain the next image block along the first direction as the current image block; based on the diffusion restoration model and the text description information, performing a restoration process on the current image block to obtain the next first image block along the first direction; taking the next first image block after the restoration process as the target image block, returning to the step of performing a first distortion process on the target image block to obtain the next image block along the first direction, until the last first image block along the first direction is obtained.

[0054] In the above implementation, the first distortion processing of the target image block can be understood as distortion of the target image block to the next position in the first direction, so as to achieve distortion of the obtained target image block to the next viewing angle. After iterative first distortion processing and repair processing, it is possible to achieve development in the first direction based on the initial image block and extend to one end of the ring.

[0055] The following describes the first distortion processing and the restoration processing performed on the target image block by taking an example.

[0056] For example, please also refer to Figure 3 , Figure 4 , Figure 5 and Figure 6 , in response to Figure 3 After the first distortion process is performed on the target image block shown, we can get Figure 4 As shown in the distorted image block, it can be seen that there is a content missing area (the black area in the figure) in the distorted image block; then for the distorted image block, obtain Figure 5 The mask image block corresponding to the distorted image block shown in FIG. 1 is used to indicate the area to be repaired (i.e., the black area) in the distorted image block. Then, through the above implementation method, the distorted image block is repaired by using the mask image block, thereby obtaining Figure 6 The inpainted image patches are shown.

[0057] Step S240: Based on the diffusion restoration model and the text description information, and taking the initial image block as a starting position, generate a plurality of second image blocks along a second direction, wherein the first direction is opposite to the second direction.

[0058] In the embodiment of the present application, after acquiring the above initial image block, the electronic device can also generate multiple second image blocks along the second direction based on the above diffusion restoration model and text description information, with the initial image block as the starting position, and the second direction is opposite to the above first direction, for example, the first direction and the second direction can be two opposite directions in the horizontal direction. In other words, multiple image blocks along the second direction can be generated through iterative distortion and restoration based on the above initial image block.

[0059] In some embodiments, when generating multiple second image blocks along the second direction, it can be achieved in the following manner: taking the initial image block as the target image block, performing a second warping process on the target image block to obtain the next image block along the second direction as the current image block; based on the diffusion repair model and the text description information, performing a repair process on the current image block to obtain the next second image block along the second direction; taking the next second image block after the repair process as the target image block, returning to the step of performing a second warping process on the target image block to obtain the next image block along the second direction, until the last second image block along the second direction is obtained.

[0060] In the above implementation, performing the second distortion processing on the target image block can be understood as distorting the target image block to the next position in the second direction, so as to achieve distortion of the obtained target image block to the next viewing angle. After iterative second distortion processing and repair processing, it is possible to achieve development in the second direction based on the initial image block and extend to the other end of the ring.

[0061] It should be noted that the execution order between the above steps S230 and S240 may not be limited. For example, the above step S230 may be executed first and then the above step S240, or the above step S240 may be executed first and then the above step S230, or the above step S230 and step S240 may be executed simultaneously.

[0062] Step S250: splicing the initial image block, the multiple first image blocks, and the multiple second image blocks to obtain the target panoramic image.

[0063] In the embodiment of the present application, after obtaining the above initial image block, and obtaining multiple first image blocks and second image blocks obtained by performing distortion processing and repair processing in the first direction and the second direction on the basis of the initial image block, the above initial image block, multiple first image blocks and multiple second image blocks can be spliced ​​to obtain a target panoramic image. In this way, in the process of generating a panoramic image, a two-way generation method can be used to generate image blocks at various positions (different viewing angles), thereby alleviating the potential problems of incoherence and artifacts in the generation process and ensuring the overall consistency of the generated content.

[0064] In some embodiments, when the initial image block, multiple first image blocks, and multiple second image blocks are spliced, the above image blocks can be spliced ​​according to the position of each image block (the position in the first direction and the second direction) to obtain the required target panoramic image.

[0065] For example, see Figure 7 For the initial image block at position "0", by extending the generation path in the first direction and the second direction, image blocks in two directions are generated, and then all the image blocks are spliced ​​to obtain the target panoramic image that needs to be generated.

[0066] In some embodiments, when the electronic device generates the first image block and the second image block based on the initial image block, the generation path may include both the first direction and the second direction in the horizontal direction, and the first direction and the second direction in the vertical direction. Thus, a method may be adopted in both the vertical and horizontal directions, that is, image blocks at other positions are extended from the center to the periphery, and a spherical panoramic image may be generated subsequently. For details, please refer to Figure 8 On the panoramic sphere, firstly, a vertical position is selected, and a corresponding image block is generated along a circular path in the horizontal direction; in the horizontal generation process, based on the initial image block, a plurality of first image blocks and second image blocks arranged in a circular manner are obtained using the generation paths in the first direction and the second direction.

[0067] The panoramic image generation method provided in the embodiment of the present application can realize the generation of panoramic images that meet user needs based on text description information, thereby improving the convenience of panoramic image generation. The diffusion repair model used is obtained by fine-tuning a target diffusion model based on a control network ControlNet with a small sample or a zero sample. Therefore, the requirement for the amount of training data for the model used to generate panoramic images can be reduced, thereby reducing the implementation cost and complexity of generating panoramic images based on text. In addition, in the process of generating panoramic images, a bidirectional generation method is used to generate image blocks at various positions (different perspectives), thereby alleviating the potential problems of incoherence and artifacts in the generation process, ensuring the overall consistency of the generated content, and improving the image quality of the generated panoramic image.

[0068] See also Fig. 9 , Fig. 9 A schematic diagram of a method for generating a panoramic image according to another embodiment of the present application is shown. The method for generating a panoramic image is applied to the above electronic device. Fig. 9 The process shown in FIG. 1 is described in detail, and the method for generating a panoramic image may specifically include the following steps:

[0069] Step S310: Obtain text description information of the target panoramic image to be generated.

[0070] Step S320: Generate an initial image block based on the pre-trained image generation model and the text description information.

[0071] Step S330: taking the initial image block as a target image block, performing a first distortion process on the target image block to obtain a next image block along the first direction as a current image block.

[0072] In the embodiment of the present application, steps S310 to S330 may refer to the contents of the aforementioned embodiment and will not be described in detail here.

[0073] Step S340: Based on the reference image block and the text description information, and through the ControlNet and the target diffusion model, the current image block is repaired to obtain the next first image block along the first direction, and the reference image block is the currently obtained image block used to splice into the target panoramic image.

[0074] In an embodiment of the present application, when the current image block is repaired based on the diffusion repair model and text description information, a reference image block can be determined based on the currently obtained image block for splicing into the target panoramic image, and then the current image block is repaired based on the reference image block and the above text description information, and through ControlNet and the target diffusion model. It can be understood that when using the target diffusion model based on ControlNet for repair processing, since ControlNet can accept input conditions to further control the diffusion model, the currently generated image block can be used as a newly introduced condition to ensure the consistency of each part in the generated panorama. Among them, the currently obtained image blocks for splicing into the target panoramic image may include the above initial image blocks, the first image blocks and the second image blocks in the embodiment of the present application, etc.

[0075] In some embodiments, the above target diffusion model can be a stable diffusion (SD) model. The SD model is a diffusion model based on latent. Its principle is to diffuse a picture forward and add noise to obtain a normally distributed vector z, fuse text features and then diffuse it backward to obtain the picture. In the inference stage, the forward diffusion and noise process is abandoned, and the text features and random normally distributed vectors are directly diffused backward to obtain the picture corresponding to the text features. The SD model in the embodiment of the present application can be a model of versions such as stable diffusion v1.5 and table diffusion v2.1.

[0076] ControlNet is obtained by training all layers in the UNet module in the SD model with constraints. ControlNet can be regarded as a plug-in of the SD model, which enables the SD model to obtain more input conditions to constrain the SD model to generate the required image content for the image area to be repaired when repairing the image block. The ControlNet model further controls the overall behavior of the entire neural network by manipulating the input conditions of the neural network block. "Network block" refers to a group of neural layers that are combined as a frequently used unit to build a neural network, such as "resnet" block, "conv-bn-relu" block, multi-head attention block, Transformer block, etc. The network structure of the ControlNet model can be divided into two parts, namely locked copy and trainable copy. Lockedcopy fixes the original weights of stablediffusion and retains the image generation ability that stablediffusion has learned; the parameters of the trainable copy are initialized to the parameters of the corresponding trainable copy, and the parameters of this part can be updated when training the ControlNet model.

[0077] In some embodiments, when the current image block is repaired based on the reference image block and text description information and through ControlNet and the target diffusion model, the content of the currently generated panoramic image can be determined from the image blocks currently obtained for splicing into the target panoramic image as the first condition information; the second image block at the target position in the second direction is obtained from the second image block as the second condition information, and the target position is symmetrical to the position of the current image block; the first condition information and the second condition information are used as input information of the ControlNet model, the reference image block and the text description information are used as input information of the target diffusion model, and the current image block is repaired through the pre-trained control network ControlNet model and the target diffusion model to obtain the next first image block along the first direction.

[0078] In the above implementation, in order to enhance the overall consistency of the generated panoramic image and effectively solve the problems related to inconsistency and artifacts, a dual-condition method can be implemented in ControlNet, that is, two different types of conditions are used in the process of generating a single image block. Among them, the first type of condition is the above-mentioned first condition information, and the first condition information uses the content currently generated on the panoramic image itself, that is, the above-mentioned currently generated panoramic image content. The first condition information is used as the condition input of ControlNet, that is, based on the panoramic condition, the global guidance is introduced into the generation process, so as to ensure that the image block being repaired can generate content consistent with the overall structure and style of the existing panorama; the second type of condition is the above-mentioned second condition information, and the second condition information utilizes the symmetry along the generation path, that is, the image block at the symmetrical position on the opposite path (that is, the second direction) is used as the condition input of ControlNet. This method can be understood as a condition based on symmetry, so that the coherence between the content generated on different paths (that is, the first direction and the second direction) can be enhanced by integrating information from symmetrical points along the generation path.

[0079] Step S350: taking the next first image block after the repair process as the target image block, returning to the step of performing the first distortion process on the target image block to obtain the next image block along the first direction, until the last first image block along the first direction is obtained.

[0080] Step S360: Based on the diffusion restoration model and the text description information, and taking the initial image block as a starting position, a plurality of second image blocks along a second direction are generated, wherein the first direction is opposite to the second direction.

[0081] In some embodiments, the initial image block is used as a target image block, and a second warping process is performed on the target image block to obtain a next image block along the second direction as the current image block; the content of the currently generated panoramic image is determined from the image blocks currently obtained for stitching into the target panoramic image as the first condition information; the first image block at the target position in the first direction is obtained from the first image block as the second condition information, and the target position is symmetrical to the position of the current image block; the first condition information and the second condition information are used as input information of the ControlNet model, the reference image block and the text description information are used as input information of the target diffusion model, and the current image block is repaired through the pre-trained control network ControlNet model and the target diffusion model to obtain the next second image block along the first direction; the next second image block after the repair process is used as the target image block, and the step of performing a second warping process on the target image block to obtain the next image block along the second direction is returned until the last second image block along the second direction is obtained. Among them, the way in which the electronic device generates the second image block can refer to the way in which the second image block is generated in the above steps S330 to S350, and will not be repeated here. Therefore, when generating the first image block and the second image block, by combining the above first condition information and the second condition information, it is possible to promote panoramic coherence by combining the above first condition information and the second condition information, that is, combining the global guidance of the existing panoramic image content and the conditions based on symmetry, which together contribute to the output of a visually consistent and attractive panoramic image, thereby improving the quality of the final result.

[0082] Step S370: splicing the initial image block, the multiple first image blocks, and the multiple second image blocks to obtain the target panoramic image.

[0083] In the embodiment of the present application, step S370 can refer to the contents of other embodiments and will not be repeated here.

[0084] The method for generating a panoramic image provided in the embodiment of the present application can generate a panoramic image that meets user needs based on text description information, thereby improving the convenience of panoramic image generation. The diffusion repair model used is obtained by fine-tuning a target diffusion model based on a control network ControlNet with a small sample or a zero sample. Therefore, the requirement for the amount of training data for the model used to generate the panoramic image can be reduced, thereby reducing the implementation cost and complexity of generating a panoramic image based on text. In addition, in the process of generating the panoramic image, a bidirectional generation method is used to generate image blocks at various positions (different perspectives), thereby alleviating potential problems of incoherence and artifacts in the generation process, ensuring the overall consistency of the generated content, and improving the image quality of the generated panoramic image. In addition, when repairing image blocks at different positions, a reference image block is introduced as a condition to control the diffusion model, thereby better ensuring the overall consistency of the generated content and improving the image quality of the generated panoramic image.

[0085] See also Fig.10 , Fig.10 A schematic diagram of a method for generating a panoramic image provided by another embodiment of the present application is shown. The method for generating a panoramic image is applied to the above electronic device. Fig.10 The process shown in FIG. 1 is described in detail, and the method for generating a panoramic image may specifically include the following steps:

[0086] Step S410: Acquire text description information of the target panoramic image to be generated.

[0087] Step S420: Generate an initial image block based on the pre-trained image generation model and the text description information.

[0088] Step S430: Based on the diffusion restoration model and the text description information, and taking the initial image block as a starting position, generate a plurality of first image blocks along a first direction.

[0089] Step S440: Based on the diffusion restoration model and the text description information, and taking the initial image block as a starting position, a plurality of second image blocks along a second direction are generated, wherein the first direction is opposite to the second direction.

[0090] In the embodiment of the present application, steps S410 to S440 can refer to the contents of the aforementioned embodiment and will not be described in detail here.

[0091] Step S450: Obtain a depth map corresponding to the target panoramic image to be generated.

[0092] In an embodiment of the present application, after obtaining the above initial image blocks, the first image blocks and the second image blocks for stitching into a panoramic image, the electronic device can also obtain a depth map corresponding to the target panoramic image to be generated, so as to stitch the above image blocks according to the depth map, thereby guiding the depth map to be consistent with each image block.

[0093] In some embodiments, when obtaining a depth map corresponding to a target panoramic image to be generated, depth information corresponding to an initial image block, each first image block, and each second image block can be obtained based on a zero-sample depth estimation model; then, based on the depth information corresponding to the initial image block, each first image block, and each second image block, the depth map corresponding to the target panoramic image is determined.

[0094] In the above implementation, since the panoramic image can be divided into separate perspective blocks, each perspective block can be regarded as an independent unit, and the depth information of these blocks can be predicted using a zero-sample depth estimation method without the need for a model specially trained for depth-related data. In this method, the depth value of each individual image block is inferred using pre-existing data and a model trained for the depth estimation task. After obtaining the depth information of each of the above image blocks, the depth map of the entire panorama can be reconstructed. Among them, when reconstructing the depth map of the entire panorama, the estimated depths of each block can be merged and accurately aligned to establish a coherent depth representation of the entire panoramic scene. Thus, it can be achieved that by using this zero-sample depth estimation method for the perspective image block, a depth estimate of the complete panorama can be obtained without the need for specialized training data or models that focus specifically on the panoramic depth, thereby helping to improve the quality of the panoramic depth map and helping to improve accuracy and visual appeal during fine-tuning.

[0095] Step S460: Based on the depth map, the initial image block, the multiple first image blocks, and the multiple second image blocks are stitched together to obtain the target panoramic image.

[0096] In an embodiment of the present application, after acquiring the above depth map, the electronic device can splice the above-obtained initial image block, multiple first image blocks, and multiple second image blocks based on the depth map to obtain the required target panoramic image.

[0097] In some application scenarios, a depth map can be obtained in the above manner, and the obtained initial image blocks, multiple first image blocks, and multiple second image blocks can be spliced ​​based on the depth map to obtain an immersive video, a three-dimensional point cloud, etc. of the required scene. For example, the panoramic image generation method provided in the embodiment of the present application can be effectively applied to VR, AR, and MR scenes, allowing users to more conveniently create immersive scenes. Users can enter their own text descriptions to generate personalized target panoramic images as background images, thereby enhancing the user's immersive experience and enabling users to customize the environment according to their preferences without the need for a large number of panoramic data sets or a complex training process.

[0098] The panoramic image generation method provided in the embodiment of the present application can generate a panoramic image that meets user needs based on text description information, thereby improving the convenience of panoramic image generation. The diffusion restoration model used is obtained by performing small sample fine-tuning or zero sample fine-tuning on a target diffusion model based on a control network ControlNet. Therefore, the requirement for the amount of training data for the model used to generate the panoramic image can be reduced, thereby reducing the implementation cost and complexity of generating the panoramic image based on the text. In addition, the depth map of the panoramic image can be determined by zero-sample depth estimation, and the image blocks at different positions can be spliced ​​based on the depth map to obtain the panoramic image, thereby guiding the depth map to be consistent with each image block, thereby further improving the quality of the generated panoramic image.

[0099] See also Fig.11 , Fig.11 A schematic diagram of a method for generating a panoramic image provided by another embodiment of the present application is shown. The method for generating a panoramic image is applied to the above electronic device. Fig.11 The process shown in FIG. 1 is described in detail, and the method for generating a panoramic image may specifically include the following steps:

[0100] Step S510: Acquire sample panoramic images, where the number of the sample panoramic images is less than the target number.

[0101] In an embodiment of the present application, a training method for the above diffusion repair model is also provided. When the above diffusion repair model is trained, a small sample fine-tuning can be performed based on the target diffusion model of the control network ControlNet. When performing small sample fine-tuning, a sample panoramic image with a number less than the target number can be obtained.

[0102] Step S520: dividing the sample panoramic image into a plurality of image blocks in the perspective image.

[0103] In an embodiment of the present application, after acquiring the above sample panoramic image, the sample panoramic image can be segmented into a plurality of image blocks in a perspective view so as to perform supervised training based on these image blocks.

[0104] Step S530: Obtain panoramic description information of the sample panoramic image.

[0105] In an embodiment of the present application, during the process of fine-tuning a small sample, panoramic description information of the sample panoramic image can also be obtained for the above sample panoramic image in order to obtain input information for model training.

[0106] In some implementations, a key descriptor of each image block in a plurality of image blocks may be obtained; and panoramic description information of the sample panoramic image may be determined based on a pre-trained large language model and the key descriptor of each image block.

[0107] In the above implementation, the above large language model can be ChatGPT, see Fig.12 , a pre-trained BLIP model can be used to capture key descriptors, and then ChatGPT is used to summarize the key descriptors of all image patches into a comprehensive panoramic description information.

[0108] Step S540: Based on the multiple image blocks and the panoramic description information, fine-tune the target diffusion model based on the control network ControlNet to obtain the diffusion repair model.

[0109] In an embodiment of the present application, when fine-tuning the target diffusion model based on the control network ControlNet based on the above multiple image blocks and panoramic description information, the fine-tuning process can correspond to the way of generating the first image block and the second image block in the aforementioned embodiment. That is to say, for each of the above image blocks, it can be distorted in the first direction or the second direction to obtain a distorted image block, and the distorted image block can be repaired based on the above ControlNet-based target diffusion model and panoramic description information to obtain a repaired image block. For the repaired image block, its corresponding true value is the image block at the next position corresponding to the above image block before distortion in the first direction or the second direction, so supervision can be achieved when fine-tuning the model.

[0110] It should be noted that, for the diffusion restoration model in the embodiment of the present application, its training process can be carried out in advance, and then the diffusion restoration model can be directly used each time a panoramic image is generated, without the need to train the model each time a panoramic image is generated.

[0111] In addition, when the above diffusion inpainting model is trained by zero-shot fine-tuning, a set of images can be generated using the pre-trained diffusion model; then, distortion is applied to these images based on the panoramic projection relationship; and the corresponding pixels of the distorted images are used as the true values ​​to achieve a self-supervised fine-tuning process.

[0112] Step S550: Obtain text description information of the target panoramic image to be generated.

[0113] Step S560: Generate a target panoramic image based on a pre-trained diffusion restoration model and the text description information, wherein the diffusion restoration model is obtained by performing small sample fine-tuning or zero sample fine-tuning on a target diffusion model based on a control network ControlNet.

[0114] In the embodiment of the present application, step S550 and step S560 can refer to the contents of the aforementioned embodiment and will not be described in detail here.

[0115] The panoramic image generation method provided in the embodiment of the present application can generate a panoramic image that meets user needs based on text description information, thereby improving the convenience of panoramic image generation; in addition, a diffusion repair model training method is provided to perform small sample fine-tuning on a target diffusion model based on a control network ControlNet, thereby reducing the requirement for the amount of training data for the model used to generate panoramic images, thereby reducing the implementation cost and complexity of generating panoramic images based on text.

[0116] See also Fig.13 , which shows a structural block diagram of a panoramic image generation device 600 provided in an embodiment of the present application. The panoramic image generation device 600 applies the above-mentioned electronic device, and the panoramic image generation device 600 includes: an information acquisition module 610 and an image generation module 620. Among them, the information acquisition module 610 is used to obtain text description information of the target panoramic image to be generated; the image generation module 620 is used to generate the target panoramic image based on a pre-trained diffusion repair model and the text description information, and the diffusion repair model is obtained by small sample fine-tuning or zero sample fine-tuning of the target diffusion model based on the control network ControlNet.

[0117] In some embodiments, the image generation module 620 can be specifically used to: generate an initial image block based on a pre-trained image generation model and the text description information; generate multiple first image blocks along a first direction based on the diffusion repair model and the text description information, with the initial image block as the starting position; generate multiple second image blocks along a second direction based on the diffusion repair model and the text description information, with the initial image block as the starting position, wherein the first direction is opposite to the second direction; and splice the initial image block, the multiple first image blocks, and the multiple second image blocks to obtain the target panoramic image.

[0118] In a possible implementation, the image generation module 620 may also be used to: take the initial image block as the target image block, perform a first warping process on the target image block, and obtain the next image block along the first direction as the current image block; based on the diffusion repair model and the text description information, perform a repair process on the current image block to obtain the next first image block along the first direction; take the next first image block after the repair process as the target image block, and return to the step of performing a first warping process on the target image block to obtain the next image block along the first direction, until the last first image block along the first direction is obtained.

[0119] Optionally, the image generation module 620 can also be used to: based on the reference image block and the text description information, and through the ControlNet and the target diffusion model, repair the current image block to obtain the next first image block along the first direction, and the reference image block is the currently obtained image block used to splice into the target panoramic image.

[0120] Optionally, the image generation module 620 can also be used to: determine the content of the currently generated panoramic image from the image blocks currently obtained for stitching into the target panoramic image, as the first condition information; obtain the second image block at the target position in the second direction from the second image block, as the second condition information, and the target position is symmetrical to the position of the current image block; use the first condition information and the second condition information as input information of the ControlNet model, use the reference image block and the text description information as input information of the target diffusion model, and repair the current image block through the pre-trained control network ControlNet model and the target diffusion model to obtain the next first image block along the first direction.

[0121] In a possible implementation, the image generation module 620 can also be used to: determine target prompt information for input into the image generation model based on the text description information, the target prompt information being used to prompt the image generation model to generate an image block at the center position of the scene image that is consistent with the text description information; input the target prompt information into the image generation model to obtain the image block output by the image generation model as the initial image block.

[0122] In a possible implementation, the image generation module 620 can also be used to: obtain a depth map corresponding to the target panoramic image to be generated; based on the depth map, splice the initial image block, the multiple first image blocks and the multiple second image blocks to obtain the target panoramic image.

[0123] Optionally, the image generation module 620 can also be used to: obtain the depth information corresponding to the initial image block, each of the first image blocks and each of the second image blocks based on a zero-sample depth estimation model; and determine the depth map based on the depth information corresponding to the initial image block, each of the first image blocks and each of the second image blocks.

[0124] In some embodiments, the panoramic image generation device 600 may also include a sample image acquisition module, an image segmentation module, a sample description acquisition module and a model fine-tuning module, the sample image acquisition module is used to acquire sample panoramic images, the number of which is less than the target number; the image segmentation module is used to segment the sample panoramic image into multiple image blocks in the perspective image; the sample description acquisition module is used to acquire panoramic description information of the sample panoramic image; the model fine-tuning module is used to fine-tune the target diffusion model based on the control network ControlNet based on the multiple image blocks and the panoramic description information to obtain the diffusion repair model.

[0125] In a possible implementation, the sample description acquisition module may be specifically used to: obtain a key descriptor of each image block among the multiple image blocks; and determine panoramic description information of the sample panoramic image based on a pre-trained large language model and the key descriptor of each image block.

[0126] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0127] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0128] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0129] In summary, the solution provided by the present application generates a target panoramic image by acquiring text description information of the target panoramic image to be generated, based on the text description information and a diffusion repair model obtained by small sample fine-tuning or zero sample fine-tuning of a target diffusion model based on a control network ControlNet, thereby achieving generation of a panoramic image that meets user needs based on the text description information, thereby improving the convenience of panoramic image generation, and the diffusion repair model used is obtained by small sample fine-tuning or zero sample fine-tuning of a target diffusion model based on a control network ControlNet, thereby reducing the requirement for the amount of training data for the model used to generate the panoramic image, thereby reducing the implementation cost and complexity of generating panoramic images based on text.

[0130] Please refer to Fig.14 , which shows a structural block diagram of an electronic device provided in an embodiment of the present application. The electronic device 100 may be a head-mounted display device, a smart phone, a tablet computer, a laptop computer, a server, or other device capable of running applications. The electronic device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more applications are configured to execute the method described in the aforementioned method embodiment.

[0131] The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0132] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 100 during use.

[0133] Please refer to Fig.15 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 800 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0134] The computer readable storage medium 800 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer readable storage medium 800 has storage space for program code 810 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 810 can be compressed, for example, in an appropriate form.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating a panoramic image, characterized in that: The method comprises: Obtaining text description information of the target panoramic image to be generated; Based on a pre-trained diffusion restoration model and the text description information, a target panoramic image is generated, wherein the diffusion restoration model is obtained by performing small sample fine-tuning or zero sample fine-tuning on a target diffusion model based on a control network ControlNet.

2. The method according to claim 1, characterized in that The step of generating a panoramic image based on the diffusion restoration model and the text description information includes: Generate an initial image block based on a pre-trained image generation model and the text description information; Based on the diffusion restoration model and the text description information, and taking the initial image block as a starting position, generating a plurality of first image blocks along a first direction; Based on the diffusion restoration model and the text description information, and taking the initial image block as a starting position, generating a plurality of second image blocks along a second direction, wherein the first direction is opposite to the second direction; The initial image block, the multiple first image blocks, and the multiple second image blocks are spliced ​​to obtain the target panoramic image.

3. The method according to claim 2, characterized in that The step of generating a plurality of first image blocks along a first direction based on the diffusion restoration model and the text description information and taking the initial image block as a starting position includes: Taking the initial image block as a target image block, performing a first distortion process on the target image block to obtain a next image block along the first direction as a current image block; Based on the diffusion restoration model and the text description information, the current image block is restored to obtain a next first image block along a first direction; The next first image block after the repair process is used as the target image block, and the step of performing the first distortion process on the target image block to obtain the next image block along the first direction is returned to, until the last first image block along the first direction is obtained.

4. The method according to claim 3, characterized in that The step of performing a restoration process on the current image block based on the diffusion restoration model and the text description information to obtain a next first image block along a first direction includes: Based on the reference image block and the text description information, and through the ControlNet and the target diffusion model, the current image block is repaired to obtain the next first image block along the first direction, and the reference image block is the currently obtained image block used to splice into the target panoramic image.

5. The method according to claim 4, characterized in that The method of performing a repair process on the current image block based on the reference image block and the text description information and by using the ControlNet and the target diffusion model to obtain a next first image block along the first direction includes: Determining the content of the currently generated panoramic image from the currently obtained image blocks used for stitching into the target panoramic image as the first condition information; Acquire, from the second image block, the second image block at a target position in the second direction as second condition information, wherein the target position is symmetrical to a position of the current image block; The first condition information and the second condition information are used as input information of the ControlNet model, the reference image block and the text description information are used as input information of the target diffusion model, and the current image block is repaired through the pre-trained control network ControlNet model and the target diffusion model to obtain the next first image block along the first direction.

6. The method according to claim 2, characterized in that The generating of the initial image block based on the pre-trained image generation model and the text description information includes: Based on the text description information, determining target prompt information for input into the image generation model, the target prompt information being used to prompt the image generation model to generate an image block at a center position in the scene image that matches the text description information; The target prompt information is input into the image generation model to obtain an image block output by the image generation model as the initial image block.

7. The method according to claim 2, characterized in that The step of stitching the initial image block, the plurality of first image blocks, and the plurality of second image blocks to obtain the target panoramic image includes: Obtaining a depth map corresponding to the target panoramic image to be generated; Based on the depth map, the initial image block, the multiple first image blocks, and the multiple second image blocks are spliced ​​to obtain the target panoramic image.

8. The method according to claim 7, characterized in that The obtaining of a depth map corresponding to the target panoramic image to be generated includes: Based on a zero-sample depth estimation model, acquiring depth information corresponding to the initial image block, each of the first image blocks, and each of the second image blocks; The depth map is determined based on depth information corresponding to the initial image block, each of the first image blocks, and each of the second image blocks.

9. The method according to any one of claims 1 to 8, characterized in that: The diffusion repair model is trained in the following way: Acquire sample panoramic images, wherein the number of the sample panoramic images is less than the target number; dividing the sample panoramic image into a plurality of image blocks in a perspective image; Obtaining panoramic description information of the sample panoramic image; Based on the multiple image blocks and the panoramic description information, a target diffusion model based on a control network ControlNet is fine-tuned to obtain the diffusion repair model.

10. The method according to claim 9, characterized in that The obtaining of panoramic description information of the sample panoramic image includes: Obtaining a key descriptor of each image block in the plurality of image blocks; Based on the pre-trained large language model and the key descriptors of each image block, the panoramic description information of the sample panoramic image is determined.

11. A device for generating a panoramic image, characterized in that: The device comprises: an information acquisition module and an image generation module, wherein: The information acquisition module is used to acquire text description information of the target panoramic image to be generated; The image generation module is used to generate a target panoramic image based on a pre-trained diffusion restoration model and the text description information. The diffusion restoration model is obtained by performing small sample fine-tuning or zero sample fine-tuning on a target diffusion model based on a control network ControlNet.

12. An electronic device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 10.

Images