Image generation methods, apparatus, systems, storage media and program products

Abstract

This application discloses an image generation method, apparatus, system, storage medium, and program product, belonging to the field of computer technology. The method includes: acquiring panoramic description information, which describes the content of a target panoramic image to be generated; generating multiple view descriptions based on the panoramic description information and multi-view prompts, wherein the viewing angles of the multiple views are contained within the viewing angle range of the target panoramic image, and the multiple view descriptions correspond one-to-one with the multiple views; and generating the target panoramic image using an image generation model based on the multiple view descriptions. Since the multi-view prompts indicate the division of the multiple views and the relationships between them, the multiple view descriptions generated according to the multi-view prompts have spatial consistency, thereby reducing the possibility of redundant and / or omitted objects in the target panoramic image generated by the image generation model, and improving the generation quality of the panoramic image.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image generation method, apparatus, system, storage medium, and program product. Background Technology

[0002] Text-generated images are a technique for generating images from text. Text-generated panoramic images are an application of text-generated image technology in the field of panoramic imaging, used to generate panoramic images from text.

[0003] In related technologies, a multi-view, multi-branch generation strategy is used to generate panoramic images based on the panoramic description text provided by the user. This process includes: dividing the 360° panoramic view into N equally wide views (or sub-views), ensuring that each view covers a different scene area and that adjacent views overlap to a certain extent. Then, based on the N views, the panoramic image generation task is decomposed into N parallel branch tasks, each branch generating a scene image for its corresponding viewpoint based on the panoramic description text. After the N branches output their respective scene images, a coordinate transformation algorithm is used to fuse the scene images output by the N branches to obtain a complete panoramic image.

[0004] However, directly using the panoramic description text as the description text for N views will result in redundant objects in the generated panoramic image and / or omission of some objects mentioned in the panoramic description text. Summary of the Invention

[0005] This application provides an image generation method, apparatus, system, storage medium, and program product, which can reduce the possibility of redundant and / or omitted objects in the target panoramic image generated by the image generation model based on multiple view description information, thereby improving the quality of the panoramic image. The technical solution is as follows:

[0006] Firstly, an image generation method is provided, the method comprising: acquiring panoramic description information, the panoramic description information being used to describe the content of a target panoramic image to be generated; generating multiple view description information based on the panoramic description information and multiple view prompts, the multiple view prompts indicating the division method of the multiple views and the association relationship between the multiple views, the viewing angle range of the multiple views being included within the viewing angle range of the target panoramic image, and the multiple view description information being associated with the multiple view prompts. Figure 1 One-to-one correspondence; based on the multiple view description information, the target panoramic image is generated through an image generation model.

[0007] The panoramic description information can be information input by the user or information automatically acquired or generated by the computer system based on the actual application scenario and user needs. This application does not restrict the method of acquiring panoramic description information.

[0008] Because multi-view cue words indicate the division of multiple views and the relationships between them, the view descriptions generated according to these cue words possess spatial consistency. This reduces the likelihood of redundant and / or omitted objects in the target panoramic image generated by the image generation model based on multi-view descriptions. Therefore, generating a target panoramic image based on multi-view descriptions significantly improves the quality of the generated panoramic image.

[0009] In one possible implementation, generating multiple view description information based on the panoramic description information and the multi-view prompts includes: inputting the panoramic description information and the first prompts into an information generation model to obtain content element information output by the information generation model, wherein the first prompts are used to guide the information generation model to determine the key elements of the target panoramic image, and the content element information includes multiple key elements in the target panoramic image and the positional relationships between the multiple key elements; and generating the multiple view description information based on the content element information and the multi-view prompts.

[0010] It should be understood that multiple view descriptions and multiple views Figure 1 There is a one-to-one correspondence, meaning that each view description corresponds to one view, and this view description describes the scene within the view's field of view. The scenes of multiple views may or may not overlap. When the scenes of multiple views overlap, their corresponding view descriptions will also have a certain degree of similarity.

[0011] It should be noted that when the viewing angles of multiple views do not overlap, the viewing angle of the target panoramic image is equal to the sum of the viewing angles of the multiple views; however, when the viewing angles of multiple views overlap, the viewing angle of the target panoramic image is slightly smaller than the sum of the viewing angles of the multiple views, and the viewing angle of the overlapping part needs to be subtracted from the sum of the viewing angles of the multiple views.

[0012] In one possible implementation, generating the multiple view description information based on the content element information and the multi-view prompt words includes: inputting the content element information, the multi-view prompt words, and the second prompt word into the information generation model to obtain multiple initial description information output by the information generation model, wherein the second prompt word is used to guide the information generation model to split the content element information according to the multiple views; and determining the multiple view description information based on the multiple initial description information.

[0013] In one possible implementation, the multi-view cue includes the center position, rotation angle, and number of rotations of the target panoramic image.

[0014] In one possible implementation, multiple initial description information pieces are directly determined as multiple view description information pieces.

[0015] In one possible implementation, determining the plurality of view description information based on the plurality of initial description information includes: inputting the plurality of initial description information and a third prompt word into the information generation model to obtain the plurality of view description information output by the information generation model, wherein the third prompt word is used to guide the information generation model to correct the plurality of initial description information.

[0016] In one possible implementation, the third prompt word includes spatial constraints, which include at least one of the following:

[0017] The same key element maintains a consistent appearance across different views;

[0018] The same key element exists in both adjacent views;

[0019] Ignore key elements located at the edge of the view;

[0020] The number of key elements in a single view is less than the threshold.

[0021] In one possible implementation, the third prompt word further includes text optimization conditions, which include at least one of the following:

[0022] The description information of the multiple views needs to be semantically consistent;

[0023] The descriptions of the same key element in the multiple view descriptions must be consistent;

[0024] Based on the aforementioned spatial constraints, key elements that do not belong to the corresponding view are deleted.

[0025] It should be noted that the process of revising multiple initial description information can be performed once or multiple times, and in the case of multiple revisions, the input information used for these revisions can be the same. This application does not impose any restrictions on this.

[0026] In one possible implementation, determining the plurality of view description information based on the plurality of initial description information includes: acquiring at least one view description sample, the view description sample including a plurality of sample view description information corresponding to a sample panoramic image; inputting the at least one view description sample, the plurality of initial description information and a fourth prompt word into the information generation model to obtain the plurality of view description information output by the information generation model, wherein the fourth prompt word is used to guide the information generation model to correct the plurality of initial description information with the at least one view description sample as a reference.

[0027] It should be understood that at least one view description sample serves as an example provided to the information generation model for learning, enabling the model to learn the patterns of that at least one view description sample. That is, based on at least one view description sample, the information generation model can be guided to refine multiple initial pieces of information with minimal prompting.

[0028] The scene corresponding to the view description sample differs from the scenes corresponding to the multiple initial description information sets, but they share similarities. For example, if the scene of the target panoramic image is a living room, the scene corresponding to the view description sample could also be an indoor scene, such as a bathroom, study, or exhibition hall. Furthermore, the number of sample view description information sets included in the view description sample can be equal to the number of initial description information sets, providing a better reference for the information generation model.

[0029] In one possible implementation, the panoramic description information includes any of the following: the name of the scene in which the target panoramic image is located, a set of information describing the scene, and a single-view image;

[0030] The single-view image is used to describe the scene area covered by a single view in the target panoramic image.

[0031] In one possible implementation, the panoramic description information includes the single-view image; the step of inputting the panoramic description information and the first prompt word into the information generation model to obtain the content element information output by the information generation model includes: performing image-to-text conversion processing on the single-view image to obtain the view description information corresponding to the single-view image; inputting the view description information corresponding to the single-view image and the first prompt word into the information generation model to obtain the content element information output by the information generation model.

[0032] In one possible implementation, if the panoramic description information is the name of the scene, then the first prompt word is used to guide the information generation model to expand the content of the panoramic description information to obtain the key elements of the target panoramic image; if the panoramic description information is a set of information describing the scene, then the first prompt word is used to guide the information generation model to extract features from the panoramic description information to obtain the key elements of the target panoramic image.

[0033] Secondly, an image generation apparatus is provided, which has the function of implementing the image generation method described in the first aspect. The image generation apparatus includes at least one module for implementing the image generation method provided in the first aspect.

[0034] Thirdly, a computer system is provided, comprising a processor and a memory, the memory being used to store a computer program for performing the method provided in the first aspect. The processor is configured to execute the computer program stored in the memory to implement the image generation method described in the first aspect.

[0035] Fourthly, a computer-readable storage medium is provided, wherein the storage medium stores instructions that, when executed on a computer, cause the computer to perform the image generation method described in the first aspect.

[0036] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, cause the computer to perform the image generation method described in the first aspect. Alternatively, a computer program is provided that, when executed on a computer, causes the computer to perform the image generation method described in the first aspect.

[0037] The technical effects achieved by the second to fifth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of a system architecture for MVDiffusion provided in an embodiment of this application;

[0039] Figure 2 This is a schematic diagram illustrating a related technology for generating panoramic images based on the MVDiffusion model, as provided in an embodiment of this application.

[0040] Figure 3 This is a schematic diagram of the structure of a computer system provided in an embodiment of this application;

[0041] Figure 4This is a schematic flowchart of an image generation method provided in an embodiment of this application;

[0042] Figure 5 This is a schematic diagram of a process for determining output information based on a thought chain, provided in an embodiment of this application;

[0043] Figure 6 This is a flowchart illustrating another image generation method provided in an embodiment of this application;

[0044] Figure 7 This is a schematic diagram of the structure of an image generation device provided in an embodiment of this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0046] To facilitate understanding, before explaining the image generation method provided in this application, the terminology, application scenarios, and implementation environment involved in the embodiments of this application will be introduced first.

[0047] First, the terminology used in the embodiments of this application will be introduced.

[0048] 1. Text-to-image

[0049] Text-to-image (TPE) is a technique for generating images from text. It primarily relies on deep learning and natural language processing (NLP) technologies. It extracts key information by analyzing the input text description and then uses a deep learning model to generate an image corresponding to the text description. The core of TPE lies in combining high-quality algorithms from NLP and computer vision (CV), enabling machines to understand text semantics and transform them into visual information.

[0050] 2. Panoramic images

[0051] A panoramic image is an image that can display a wide field of view, usually obtained through special shooting techniques or software processing. Because panoramic images can present a scene with a 360° or wider field of view, providing users with an immersive visual experience, panoramic images have a wide range of applications in virtual reality (VR), augmented reality (AR), tourism promotion, interior design and decoration, real estate display, game development, and other fields.

[0052] In some applications, panoramic images can be represented by combining scene images from multiple perspectives. That is, by stitching together scene images from multiple small perspectives, a panoramic image with a wider field of view can be obtained.

[0053] 3. Large Language Model (LLM)

[0054] Large language models refer to natural language processing models trained on large-scale text data using deep learning techniques, particularly the Transformer architecture. Large language models are capable of understanding, generating, and processing natural language, possessing powerful language understanding and generation capabilities.

[0055] 4. Prompt Engineering (PE)

[0056] Cue engineering refers to the process of designing and optimizing inputs for Natural Language Processing (NLP) models, especially Large Language Models (LLMs), to obtain the desired response. Through well-designed cue words (also known as cue text), the model can be guided to more accurately understand the user's needs, thereby generating more relevant, creative, or accurate answers. This is crucial for improving the quality of interactions with artificial intelligence (AI).

[0057] Typically, prompting engineering includes few-shot prompting and role-prompting. Few-shot prompting is a method based on a small number of samples. By showing the model a few examples, it allows the model to learn patterns in these examples and generate similar output. Role-prompting is a method that guides the model to generate text that matches the characteristics of that role by assigning a specific role or identity to it. By providing the model with a role description, it enables the model to generate output that matches the characteristics of that role.

[0058] 5. Chain of Thoughts (CoT)

[0059] The core idea of ​​the thinking chain is to break down a complex problem into a series of smaller, more manageable logical steps. These steps are usually arranged in a logical order, forming a clear chain of reasoning. By guiding the model through these steps, it can more systematically understand and analyze the input information, thereby deriving more accurate and reliable outputs.

[0060] Secondly, the application scenarios of the embodiments of this application will be introduced.

[0061] Text-to-image generation technology refers to the technology of generating corresponding images based on input text descriptions. The development of text-to-image generation technology mainly relies on deep learning techniques, such as generative adversarial networks (GANs) and diffusion models.

[0062] Generative Adversarial Networks (GANs) consist of two neural networks: a generator and a discriminator. The generator's task is to generate corresponding images based on the input text, while the discriminator's task is to determine whether the generated images are realistic. Through adversarial training of these two networks, GANs can generate high-quality images. Diffusion models are generative models that aim to achieve good results by continuously removing noise. They learn the denoising process through a forward process (gradually adding Gaussian noise to the data until it becomes random noise) and a backward process (continuously denoising from random noise to restore the original data), thus enabling the generation of various types of data.

[0063] In recent years, diffusion models have made significant breakthroughs in the application of text-to-image generation, especially the stable diffusion (SD) series of models. These models are text-to-image generation models based on latent diffusion models (LDM), which combine the concepts of diffusion models with the advantages of latent space representation, resulting in better performance in terms of image generation quality and speed.

[0064] It should be noted that the latent space representation refers to the space after image compression and dimensionality reduction. In the steady-state diffusion model, the image is compressed into the latent space, which can significantly reduce the amount of computation.

[0065] LDM (Latent Derivative Model) is a generative model that combines the advantages of variational autoencoders (VAEs) and diffusion models. It achieves high-quality image generation by mapping the input image to a representation in the latent space, progressively adding noise to this representation, and learning how to remove the noise to recover a clear image. When generating a new image, LDM starts with pure noise, then progressively generates a denoised latent space representation. Finally, the decoder part of the VAE transforms the denoised latent representation back into the original image space, thus generating a clear image. Because LDM balances the quality of the generated image with the efficiency of the generation process, it is widely used in image generation, artistic creation, advertising design, and other fields.

[0066] While LDM performs well on conventional texturing image tasks, it performs poorly on texturing panoramic images. Besides the differences in data format (e.g., resolution, field of view) between panoramic and conventional 2D images, the more significant reason is the greater complexity of panoramic image generation compared to conventional 2D images. This complexity primarily manifests in:

[0067] (1) Panoramic images need to ensure smoothness and consistency when transitioning between different viewpoints. For example, if a user moves their viewpoint from left to right in a panoramic image, they should see smooth and continuous changes in scenery, rather than sudden jumps or breaks in the image. This requires the model to take into account the spatial relationships between different viewpoints when generating images and to ensure their coherence.

[0068] (2) The generation of panoramic images also needs to take into account the spatial continuity of adjacent viewpoints and the consistency between image content and text description. This means that the model must not only generate image content that matches the text description, but also ensure that this content remains consistent and coherent across different viewpoints. For example, if the text describes a transitional scene from a beach to a forest, the model should be able to generate a panoramic image that gradually transitions from the beach to the forest, while ensuring that both scenes are visually and semantically consistent with the text description.

[0069] In view of the complexity of the panoramic image generation process, the academic and industry communities have proposed some technical solutions for generating panoramic images, such as using multi-view diffusion (MVDiffusion in this embodiment) to generate panoramic images.

[0070] MVDiffusion, based on existing text-based image models (such as the stablediffusion series), provides a simple and effective method for generating multi-view scene images. Models that rely on MVDiffusion to generate images are called MVDiffusion models. The MVDiffusion model employs a multi-view, multi-branch generation process. First, the 360° view is divided into N (N is an integer greater than 1) views, each covering a different angle, with some overlap between adjacent views. Naturally, each branch generates only its corresponding view. Overlapping pixels between adjacent views can be communicated through a corresponding-aware attention (CAA) module, ensuring continuity at the junctions of adjacent views. After all views have been generated, a simple coordinate transformation is used to fuse the multi-view images into a complete panoramic image.

[0071] See Figure 1, Figure 1 This is a schematic diagram of the MVDiffusion system architecture. The MVDiffusion system architecture mainly includes two types of modules: one type consists of modules already present in stablediffusion, such as the text encoder, image encoder / decoder, and latent space denoising module; the other type consists of newly added modules in MVDiffusion, such as the corresponding perceptual attention module and the multi-view fusion module. The functions of each module will be introduced below in conjunction with the process of generating panoramic images.

[0072] (1) Text Encoder: The text encoder transforms the text describing view i (1≤i≤N, where i is an integer) from a string into a vector representation that can be understood by the model, thus obtaining the text representation of view i. Taking stablediffusion as an example, the text encoder it uses is the contrastive language-image pretraining (CLIP) model, which is an image-text joint encoding model.

[0073] (2) Image Encoder / Decoder: The function of the image encoder is to convert the input image into a representation in the latent space, while the image decoder is responsible for converting the latent representation back into the original image space to generate the final image. Specifically, the image encoder extracts key features from the noisy image of view i, while the image decoder reconstructs the denoised image of view i, ensuring that the generated image has high quality and detail. Taking stablediffusion as an example, its commonly used image encoder / decoder is the variational autoencoder (VAE).

[0074] (3) Latent Space Denoising Module: The function of the latent space denoising model is to gradually recover a clear image representation by removing noise from the latent representation of view N. Taking stablediffusion as an example, its latent space denoising module is based on a diffusion model structure. The specific implementation typically uses a variant of the U-Net (a convolutional neural network CNN based on deep learning) architecture. This architecture can effectively process multi-level feature information and preserve image details and structure during the denoising process. The input to the latent space denoising module includes a noisy latent representation and a text feature representation, for example... Figure 1 The latent representations of view i (image and text) are shown, and the output is a denoised, clear latent representation, for example... Figure 1 The image shown is the denoised latent representation of view i. After multiple denoising steps, the output of the latent space denoising module can be used as input to the image decoder to obtain a high-quality image. Figure 1 The image shown is the denoised image of view i.

[0075] (4) Corresponding Perception Attention Module (CAA): This module enhances the interaction between the source feature map and the target feature map (i.e., the local neighborhood) by introducing the location information of the local neighborhood of the multi-view, thereby improving the spatial consistency of the multi-view generated by the model.

[0076] (5) Multi-view Fusion Module: This module converts the N views generated by the model into an equidistant rectangular panoramic image. The process of generating the panoramic image is as follows: First, determine the width and height of the panoramic image. Typically, the width is twice the height to maintain the proportions of the equidistant rectangle. Then, determine the field of view (FOV) of each view, converting the position of each pixel in the view into coordinates in the panoramic image. This process usually involves geometric transformation calculations of the coordinates. After mapping all pixels, the final equidistant rectangular panoramic image can be synthesized.

[0077] based on Figure 1 The system architecture shown typically takes a general descriptive text (also known as panoramic description text) as input to the MVDiffusion model, which includes the objects in the panoramic image, their appearance features, the positional relationships between different objects, and background information. The strategy employed by the MVDiffusion model to generate the description text for each view is relatively simple, such as... Figure 2 As shown, the panoramic description text is copied N times and used as the description text for each view to obtain... Figure 1 The view shown Figure 1 Description text, view Figure 2 Description text...View N describes the text, then the view... Figure 1 The -N description text is input into the MVDiffusion model to output a panoramic image.

[0078] However, the MVDiffusion model directly uses the panoramic description text as the description text for each view, resulting in identical description text for every view. The model attempts to include all objects and their positional relationships in each view, increasing the likelihood of redundant objects in the panoramic image. Simultaneously, CAA imposes strong spatial constraints on local regions, which often conflict with the description text. This makes it extremely difficult to ensure spatial consistency between adjacent views while generating all objects in each view. This conflict may cause some objects to be ignored or lost in the panoramic image output by the MVDiffusion model.

[0079] As an example, suppose the panoramic description text is: The living room is a comfortable, sun-drenched space with soft, neutral walls and large windows. A comfortable sofa and colorful cushions face a stylish coffee table, while a plush rug provides a tranquil space for yoga. Framed artwork and a wall clock add personal charm, and a shoe cabinet near the entrance keeps the area tidy. However, the panoramic image output by the MVDiffusion model based on the above panoramic description text will contain redundant objects, such as repeated appearances of the wall clock, sofa, coffee table, and rug; and the panoramic image will also omit some objects, such as the shoe cabinet located at the edge.

[0080] It is evident that the panoramic images output by the MVDiffusion model contain redundant objects and missing objects. It is weak in aligning descriptive text with view content. The root cause is that the descriptive text of each view lacks spatial consistency, leading to a conflict between spatial continuity and text description, which in turn reduces the quality of the generated panoramic images.

[0081] Based on this, embodiments of this application provide an image generation method. Based on panoramic description information describing the content of a target panoramic image to be generated, multiple view description information is generated using multi-view prompts, and then the target panoramic image is generated based on the multiple view description information. The multiple view description information and multiple view prompts are used to generate multiple panoramic images. Figure 1 This one-to-one correspondence ensures better alignment between multiple view descriptions and view content when generating the target panoramic image. Furthermore, since the multi-view cue words indicate the division of multiple views and the relationships between them, the view descriptions generated according to these cue words possess spatial consistency. This reduces the likelihood of redundant and / or omitted objects in the target panoramic image generated by the image generation model based on multiple view descriptions. Thus, generating a target panoramic image based on multiple view descriptions significantly improves the quality of the generated panoramic image.

[0082] The image generation method provided in this application can be applied to any scenario related to panoramic images, such as VR, AR, tourism promotion, interior design and decoration, real estate display, game development, etc. This application does not limit this.

[0083] As an example, in the virtual tourism field, users can input text describing a scenic spot and automatically generate a panoramic image of that spot for a virtual travel experience. In the real estate showcasing industry, developers can use this technology to generate panoramic images of the interior or surrounding environment of a property, providing customers with a more intuitive home-buying experience. In the game development field, game developers can use Wensheng panoramic imaging technology to quickly generate panoramic images of game scenes, enhancing the richness and realism of the game visuals.

[0084] With the development of three-dimensional (3D) technology, the image generation method provided in this application embodiment can also be applied to the rational planning and automatic generation of 3D content. By extending the technical solution provided in this application embodiment to the field of 3D content generation, greater commercial value can be created in fields such as film creation and game production.

[0085] For example, based on the 3D scene description information input by the user, observation points are set at multiple latitude and longitude lines; around each observation point, multiple view description information corresponding to that observation point is generated according to the 3D scene description information and multi-view prompts; based on the multiple view description information corresponding to each rotating observation point, a panoramic image of that observation point is generated through an image generation model; and a 3D scene is constructed based on the foreground images of multiple observation points.

[0086] With the development and evolution of the image generation model, given that multiple view descriptions are determined for each observation point, the image generation model can be replaced with a 3D content generation model. Then, the multiple view descriptions corresponding to multiple rotating observation points can be input into the 3D content generation model, and the constructed 3D scene can be output through the 3D content generation model.

[0087] It should be understood that, when the image generation method provided in the embodiments of this application is extended to the field of three-dimensional content generation, the following... Figure 4 The implementation methods shown in the examples can all be combined with and supplemented to the process of generating 3D content to improve the quality and speed of building 3D scenes.

[0088] It should be noted that the above application scenarios are merely examples. In actual applications, the image generation method provided in this application embodiment can also be applied to other scenarios. This application embodiment does not limit these scenarios, and will not provide further examples here.

[0089] Finally, the implementation environment of the embodiments of this application will be described.

[0090] The image generation method provided in this application embodiment can be implemented by a computer system, which can be a single physical device such as a terminal device or a server, or it can be composed of multiple distributed physical devices.

[0091] Please refer to Figure 3 , Figure 3 The diagram shows the structure of a computer system, which includes at least one processor 301, a communication bus 302, a memory 303, and at least one communication interface 304.

[0092] Processor 301 can be a general-purpose central processing unit (CPU), graphics processing unit (GPU), neural-network processing unit (NPU), network processor (NP), microprocessor, or one or more integrated circuits for implementing the solutions of this application, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0093] The communication bus 302 is used to transmit information between the aforementioned components. The communication bus 302 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0094] The memory 303 may be a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disc (including a compact disc read-only memory (CD-ROM), a compressed optical disc, a laser disc, a digital versatile optical disc, a Blu-ray disc, etc.), a magnetic disk storage medium, or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but not limited thereto. The memory 303 may exist independently and be connected to the processor 301 via a communication bus 302. The memory 303 may also be integrated with the processor 301.

[0095] Communication interface 304 uses any transceiver-like device for communicating with other devices or communication networks. Communication interface 304 includes a wired communication interface and may also include a wireless communication interface. The wired communication interface may be, for example, an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless communication interface may be a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0096] As an example, processor 301 may include one or more NPUs, for example Figure 3 The NPU0 and NPU1 shown are illustrated. Of course... Figure 3 The example only uses NPU as the processor, but in practical applications, it can be any of the types mentioned above.

[0097] As an example, a computer system may include multiple processors, such as Figure 3 The processors 301 and 305 are shown. Each of these processors may be a single-core processor or a multi-core processor. A processor here may refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0098] In some embodiments, the computer system may further include output devices and input devices. The output device communicates with the processor 301 and can display information in various ways. For example, the output device may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device communicates with the processor 301 and can receive user input in various ways. For example, the input device may be a mouse, a keyboard, a touchscreen device, or a sensing device.

[0099] In some embodiments, memory 303 is used to store program code 310 for executing the scheme of this application, and processor 301 can execute the program code 310 stored in memory 303. The program code 310 may include one or more software modules, and the computer system can implement the following by using processor 301 and the program code 310 in memory 303. Figure 4 The image generation method provided in the embodiment.

[0100] It should be noted that the application scenarios and implementation environments described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the emergence of new application scenarios and the evolution of implementation environments, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0101] Next, the image generation method provided in the embodiments of this application will be explained in detail.

[0102] Figure 4 This is a flowchart of an image generation method provided in an embodiment of this application, which is applied to the above-mentioned... Figure 3 In the computer system shown, for example, the image generation method is executed by the processor of the computer system. Please refer to... Figure 4 The method includes the following steps.

[0103] Step 401: Obtain panoramic description information, which is used to describe the content of the target panoramic image to be generated.

[0104] The panoramic description information can be information input by the user or information automatically acquired or generated by the computer system based on the actual application scenario and user needs. This application embodiment does not limit the method of acquiring panoramic description information.

[0105] In some embodiments, the panoramic description information includes any one of the following: the name of the scene in which the target panoramic image is located, a set of information describing the scene in which the target panoramic image is located, and a single-view image. The single-view image is used to describe the scene area covered by a single view in the target panoramic image.

[0106] Taking the panoramic description information as the name of the scene where the target panoramic image is located as an example, the content of the panoramic description information can be a living room, bedroom, cozy home, cinema, beach, aquarium, etc.

[0107] Taking panoramic description information as the set of information describing the scene in which the target panoramic image is located, and taking the living room as an example, the panoramic description information can be: In the morning, Mary glanced at the wall clock in the living room; Mary sat down on the sofa in the living room and started reading; Mary ate a salad and drank a cup of coffee on the coffee table in the living room; Mary started doing yoga on the rug next to the coffee table; Mary walked to the shoe cabinet in the entrance hall of the living room, started changing her shoes, and prepared to go out.

[0108] Taking a panoramic description as a single-view image as an example, this single-view image can be a regular two-dimensional image. Taking a living room as an example, this single-view image can be a scene image centered on the living room sofa and facing directly forward.

[0109] It should be noted that the panoramic description information can also include multiple pieces of information as shown in the example above. When the panoramic description information includes multiple types of information (such as information in text format and information in image format), the format of the panoramic description information needs to be adjusted to ensure that the format of each piece of information in the panoramic description information is consistent, so as to facilitate subsequent processing of the panoramic description information.

[0110] In some embodiments, if the panoramic description information is in text format, after obtaining the panoramic description information, the obtained panoramic description text can be optimized to obtain the processed panoramic description text, and then the following step 402 can be performed based on the processed panoramic description text.

[0111] As an example, text optimization processing includes, but is not limited to: expanding the panoramic description text; refining the language of the panoramic description text so that it accurately conveys the scene features of the target panoramic image to be generated; and reducing the content of the panoramic description text to make its expression more concise and clear.

[0112] In some embodiments, if the panoramic description information is in image format, after obtaining the panoramic description information, image optimization processing can be performed on the obtained single-view image to obtain processed panoramic description information, and then the following steps can be performed based on the processed panoramic description information.

[0113] As an example, image optimization processing includes, but is not limited to: noise reduction processing to improve the clarity and quality of single-view images; resolution adjustment to meet the display effects in specific application scenarios; color optimization processing, i.e., adjusting parameters such as color saturation, brightness, and contrast of single-view images to improve the visual effect of single-view images; and image enhancement processing, i.e., using techniques such as contrast enhancement, histogram equalization, and sharpening algorithms to process single-view images to improve their clarity and contrast.

[0114] Step 402: Based on the panoramic description information and multi-view cue words, generate multiple view description information. The multi-view cue words indicate the division of multiple views and the relationship between multiple views. The viewing angle range of multiple views is included within the viewing angle range of the target panoramic image. The multiple view description information and multiple view... Figure 1 One-to-one correspondence.

[0115] For the target panoramic image, the number of view descriptions can be flexibly set according to actual needs. For example, considering that the information generation model generates descriptions with higher accuracy within a small field of view, the number of view descriptions can be set as large as possible, so that each view only focuses on scene information within a small range. Conversely, a large number of view descriptions can reduce the efficiency of generating the target panoramic image. Therefore, while ensuring the completeness of information within multiple field of view, the number of view descriptions can be set as small as possible.

[0116] As an example, the number of view descriptions can be any number, such as 6, 8, or 10.

[0117] It should be understood that multiple view descriptions and multiple views Figure 1 There is a one-to-one correspondence, meaning that each view description corresponds to one view, and this view description describes the scene within the view's field of view. The scenes of multiple views may or may not overlap. When the scenes of multiple views overlap, their corresponding view descriptions will also have a certain degree of similarity.

[0118] It should be noted that when the viewing angles of multiple views do not overlap, the viewing angle of the target panoramic image is equal to the sum of the viewing angles of the multiple views; however, when the viewing angles of multiple views overlap, the viewing angle of the target panoramic image is slightly smaller than the sum of the viewing angles of the multiple views, and the viewing angle of the overlapping part needs to be subtracted from the sum of the viewing angles of the multiple views.

[0119] As an example, if the viewing angles of multiple views are all 90° and the number of views is 8, then when the viewing angles of multiple views do not overlap, the viewing angle of the target panoramic image is 360°; while when the viewing angles of multiple views overlap, the viewing angle of the target panoramic image is less than 360°.

[0120] In some embodiments, step 402 can be implemented by inputting panoramic description information, multi-view prompts and target prompts into the information generation model to obtain multiple view description information output by the information generation model.

[0121] Among them, the target prompt words are used to guide the information generation model to determine multiple views based on the multi-view prompt words, and generate description information for each view according to the panoramic description information.

[0122] As an example, the information generation model could be a large language model. For instance, this large language model could be a Transformer architecture.

[0123] In other embodiments, based on panoramic description information and multi-view prompts, a thought chain approach can be used to guide the information generation model step by step in generating multiple view descriptions by setting different prompts. That is, using panoramic description information and multi-view prompts as input information, the information generation model interacts with the model in multiple rounds, enabling the model to output multiple view descriptions after multiple rounds of guidance.

[0124] The core idea of ​​the thinking chain is to break down a problem into a series of logical steps, enabling the information generation model to more clearly understand and deduce the corresponding answer. (See also...) Figure 5 For input information, the thought chain guides the information generation model to perform a multi-step reasoning process based on the input information to determine the output information.

[0125] In one possible implementation, multi-step reasoning based on a thought chain-guided information generation model can be achieved in the following ways.

[0126] The first method involves providing example guidance. This means providing the information generation model with several examples, each containing a multi-step reasoning process. Inputting these examples along with the input information into the information generation model guides it through these multi-step reasoning steps.

[0127] The second approach involves using prompts to guide the information generation model through multi-step reasoning. For example, prompts such as "Please think step by step" can be added to guide the model in performing this process.

[0128] The third approach involves breaking down complex problems. First, the information generation model is guided to break down the complex problem into sub-problems. Then, the model answers each sub-problem one by one, using the answers as context for the next question, until all sub-problems are answered, thus arriving at the final answer.

[0129] Based on this, the present application also provides a scheme for generating multiple view description information using a thought chain. When generating multiple view description information, the implementation process of this technical solution may include the following steps (1)-(2).

[0130] (1) Input the panoramic description information and the first prompt word into the information generation model to obtain the content element information output by the information generation model.

[0131] The first cue word is used to guide the information generation model to determine the key elements of the target panoramic image. The content element information includes multiple key elements in the target panoramic image and the positional relationships between these key elements.

[0132] It should be noted that the content of the first prompt word will vary slightly depending on the format of the panoramic description information. Based on the three types of panoramic description information exemplified above, the first prompt word input into the information generation model also includes three scenarios.

[0133] In the first case, if the panoramic description information is the name of the scene where the target panoramic image is located, then the first prompt word is used to guide the information generation model to expand the content of the panoramic description information in order to obtain the key elements of the target panoramic image.

[0134] As an example, assuming the panoramic description is "living room", the first prompt could be: "Based on the provided scene name, please list several key elements that may appear in this scene, and the positional relationships between these key elements."

[0135] Based on this, the content element information output by the information generation model can include: Key elements: wall clock, sofa, coffee table, rug, shoe cabinet. The positional relationships between key elements are: wall clock, located on the wall of the living room; sofa, located in the center of the living room; coffee table, located in front of the sofa; rug, located next to the coffee table; shoe cabinet, located in the entryway of the living room.

[0136] In the second scenario, if the panoramic description information is a set of information describing the scene in which the target panoramic image is located, then the first prompt word is used to guide the information generation model to extract features from the panoramic description information in order to obtain the key elements of the target panoramic image.

[0137] As an example, suppose the panoramic description is: In the morning, Mary glanced at the wall clock in the living room; Mary sat down on the sofa in the living room and began to read; Mary ate a salad and drank a cup of coffee at the coffee table in the living room; Mary began to do yoga on the rug next to the coffee table; Mary walked to the shoe cabinet in the entryway of the living room, began to change her shoes, and prepared to go out. Correspondingly, the first prompt could be: Based on the provided text set, please summarize the scene that appears in this text set, and give the multiple key elements existing in the scene in the text set, as well as the positional relationship between the multiple key elements.

[0138] Based on this, the content element information output by the information generation model can include: Key elements: wall clock, sofa, coffee table, rug, shoe cabinet. The positional relationships between key elements are: wall clock, located on the wall of the living room; sofa, located in the center of the living room; coffee table, located in front of the sofa; rug, located next to the coffee table; shoe cabinet, located in the entryway of the living room.

[0139] It should be noted that although the first prompt words are different in the first and second scenarios, the content element information output by the information generation model may be completely the same, partially the same, or completely different, depending on the context. This application's embodiments do not impose any limitations on this.

[0140] In the third case, if the panoramic description information is a single-view image, then the process of determining the content element information in step (1) above is as follows: perform image-text conversion processing on the single-view image to obtain the view description information corresponding to the single-view image; input the view description information corresponding to the single-view image and the first prompt word into the information generation model to obtain the content element information output by the information generation model.

[0141] In one possible implementation, the process of converting a single-view image into text can be as follows: input the single-view image into an image annotation model, and use the image annotation model to convert the single-view image into text information to obtain the view description information corresponding to the single-view image.

[0142] As an example, the image annotation model can be a bidirectional language-image pre-training (BLIP) model. For instance, the image annotation model can be a BLIP-2 model.

[0143] As an example, assuming the panoramic description is an image of a living room, the text information obtained through image-to-text conversion would be: A cozy living room with a sofa, a wooden coffee table in front of the sofa, and a rug on the floor next to the coffee table. Correspondingly, the first prompt could be: "There is a view description of a living room scene from one perspective. Based on this view description, please provide several key elements that may exist in this living room scene, and the positional relationships between these key elements."

[0144] Based on this, the content element information output by the information generation model can be: Key elements: sofa, coffee table, rug. The positional relationships between the key elements are: sofa, located in the living room; coffee table, located in front of the sofa; rug, located next to the coffee table.

[0145] In some embodiments, the first cue word may further include a range of the number of key elements. Based on this, the information generation model can output multiple key elements in the target panoramic image according to this range, wherein the total number of the multiple key elements falls within the aforementioned range.

[0146] As an example, if the first prompt word defines that the number of key elements output by the information generation model should be between 6 and 10, then the content element information output by the information generation model includes at least 6 key elements, and the number of key elements does not exceed 10.

[0147] It should be noted that the content of the first prompt word may vary due to differences in people's expression habits, the format of panoramic description information, and output requirements. As long as the first prompt word can guide the information generation model to determine the key elements of the target panoramic image, this application embodiment does not limit the language expression, specific content, or level of detail of the first prompt word.

[0148] (2) Generate multiple view description information based on content element information and multi-view prompts.

[0149] In some embodiments, the multi-view cue words include the center position of the target panoramic image, the rotation angle, and the number of rotations.

[0150] As an example, a multi-view prompt could be: Imagine you are standing in the center of the scene. Rotate your body at a 45° angle. After 8 rotations, you will see 8 views, for a total 360° field of view.

[0151] Since the rotation angle and the number of rotations together determine the range of the panoramic view, multi-view prompts can also include the center position of the target panoramic image, the panoramic view of the target panoramic image, and the rotation angle. Assuming the panoramic view is 360° and the rotation angle is 45°, then based on this rotation angle and panoramic view, the number of rotations can be determined to be 8.

[0152] It should be noted that, when the multi-view prompts include the above-mentioned items, the embodiments of this application do not limit the specific content or language expression of the multi-view prompts. The above examples are provided for reference only and do not constitute a limitation on the multi-view prompts in the embodiments of this application.

[0153] As an example, using the content element information generated for the living room scene mentioned earlier, if eight views are defined by the multi-view tooltip, the description information for the eight views generated based on the content element information can be as follows:

[0154] See Figure 1 View description information (angle deviating 0° from the center of the target panoramic image): The key elements of the view are a wall clock, a sofa, and a coffee table. The spatial relationship of the key elements is as follows: the wall clock is located directly in front and mounted on the wall; the sofa is against the wall, facing the coffee table; the coffee table is located in front of the sofa, with a salad and a cup of coffee on it. Appearance and quantity of key elements: 1 wall clock (round, with numbers), 1 sofa (large, with cushions), 1 coffee table (wooden).

[0155] See Figure 2 View description information (45° off-center from the target panoramic image): The key elements of the view are a sofa, coffee table, and rug. The spatial relationships of the key elements are: the sofa is still in the field of view and faces the coffee table; the coffee table is located in the center of the field of view, slightly to the left; the rug is visible on the floor and close to the coffee table. The appearance and quantity of the key elements are: 1 sofa, 1 coffee table, and 1 rug (soft, rectangular).

[0156] See Figure 3 View description information (90° off-center from the target panoramic image): The key elements of the view are a coffee table, a rug, and an entrance area (shoe cabinet). The spatial relationship of the key elements is: the coffee table is in the center, with the rug visible underneath; the entrance area is now visible on the right, showing the shoe cabinet. The appearance and quantity of the key elements are: 1 coffee table, 1 rug, and 1 shoe cabinet (wooden, holding multiple pairs of shoes).

[0157] See Figure 4 View description information (135° angle off from the center of the target panoramic image): The key elements of the view are carpet, entrance area (shoe cabinet), and wall. The spatial relationships of the key elements are: the carpet is clearly visible in the foreground, extending into the entrance area; the shoe cabinet is now more prominent, directly in the center of the view; the living room wall is visible behind the shoe cabinet. Appearance and quantity of key elements: 1 carpet, 1 shoe cabinet.

[0158] See Figure 5View description information (180° off-center from the target panoramic image): The key elements of the view are a wall and an entrance area (shoe cabinet). The spatial relationship of the key elements is: the wall is in the foreground, the entrance is on the left; the shoe cabinet occupies the left side, and shoes are arranged in the shoe cabinet. The appearance and quantity of the key elements: 1 wall, 1 shoe cabinet, multiple pairs of shoes (of different styles).

[0159] See Figure 6 View description information (225° angle off from the center of the target panoramic image): The key elements of the view are the entrance area (shoe cabinet) and the wall. The spatial relationship of the key elements is: the shoe cabinet is still relatively prominent in the field of view, but now it is more to the left; the wall still serves as the background of the view. Appearance and quantity of key elements: 1 shoe cabinet, multiple pairs of shoes.

[0160] See Figure 7 View description information (270° angle off from the center of the target panoramic image): The key elements of the view are the entrance area (shoe cabinet) and the carpet. The spatial relationship of the key elements is: the carpet is visible in the foreground and leads to the entrance; the shoe cabinet is on one side. The appearance and quantity of the key elements: 1 carpet, 1 shoe cabinet.

[0161] View description information for View 8 (315° off-center from the target panoramic image): The key elements contained in the view are a carpet, an entrance area (shoe cabinet), and a wall clock (blurred view). The spatial relationship of the key elements is: the carpet is in the center of the field of view and leads to the shoe cabinet; the wall clock is visible in the background as a secondary object. The appearance and quantity of the key elements are: 1 carpet, 1 shoe cabinet, and 1 wall clock (blurred view from this angle).

[0162] It should be noted that the eight view descriptions in the above example are merely illustrative of the content of view descriptions based on the key elements contained in the view, the spatial relationships between the key elements, and the appearance and quantity of the key elements. In practical applications, more or less information can be used to describe a view to obtain the corresponding view description information, and this application embodiment does not impose any limitations on this. For example, view description information may also include color information, brand information, size information, etc., of the key elements.

[0163] In some embodiments, the process of generating multiple view description information based on content element information and multi-view prompt words in step (2) above is as follows: input the content element information, multi-view prompt words and the second prompt word into the information generation model to obtain multiple initial description information output by the information generation model. The second prompt word is used to guide the information generation model to split the content element information according to multiple views. Based on the multiple initial description information, multiple view description information is determined.

[0164] In this system, there is a one-to-one correspondence between multiple views, multiple initial descriptions, and multiple view descriptions; that is, one view corresponds to one initial description and one view description. The initial descriptions are intermediate results that can be further processed. The view descriptions are the final results, used to perform step 403 below.

[0165] As an example, the second prompt could be: Please identify multiple views according to the multi-view prompt; and determine the key elements contained in each view and the positional relationship between the key elements based on the content element information, thereby generating view description information corresponding to each view.

[0166] As another example, the second prompt could be: Please provide view descriptions for multiple views based on the multi-view prompt, requiring that each view description includes the key elements of that view.

[0167] In some embodiments, determining multiple view description information based on multiple initial description information includes the following four implementation methods, which will be described in turn.

[0168] The first implementation method is to directly determine multiple initial description information as multiple view description information.

[0169] The second implementation involves inputting multiple initial descriptions and a third prompt word into the information generation model to obtain multiple view descriptions output by the information generation model. The third prompt word is used to guide the information generation model to correct the multiple initial descriptions.

[0170] As an example, the third prompt includes a spatial constraint that includes at least one of the following:

[0171] The same key element maintains a consistent appearance across different views;

[0172] The same key element exists in both adjacent views;

[0173] Ignore key elements located at the edge of the view;

[0174] The number of key elements in a single view is less than the threshold.

[0175] Taking a scenario where the third cue word includes spatial constraints and the number of views is eight, as an example, the third cue word could be: Each view contains one or two key elements located in the center of the field of view; key elements at the edges can be ignored. Furthermore, adjacent views should contain common key elements. The first and eighth views are also adjacent and should contain common key elements. The same key element can only appear in the view description information of adjacent views, not in the view description information of two non-adjacent views. Additionally, as the view changes, some key elements will appear, and some key elements will disappear.

[0176] As another example, the third cue word may also include text optimization conditions, which include at least one of the following:

[0177] Multiple view descriptions need to maintain semantic consistency;

[0178] Multiple view descriptions must be consistent in their descriptions of the same key element;

[0179] Deleting key elements that do not belong to the corresponding view based on spatial constraints.

[0180] Taking the third prompt, which includes text optimization conditions, as an example, the third prompt could be: When revising multiple initial descriptions, it's necessary to check for semantic inconsistencies. If any are found, correct them accordingly. Simultaneously, based on spatial logic, check for key elements in the initial descriptions that shouldn't appear in this view; if so, correct them accordingly. Furthermore, based on the principle of simplicity, check for redundant key elements in the initial descriptions; if so, correct them accordingly. Based on these corrections, the final view descriptions should cover all key elements.

[0181] It should be noted that, in cases where the third prompt word includes spatial constraints and / or text optimization conditions, the embodiments of this application do not impose restrictions on the specific content or language expression of the third prompt word.

[0182] The third implementation involves obtaining at least one view description sample, which includes multiple sample view description information corresponding to the sample panoramic image; inputting the at least one view description sample, multiple initial description information, and a fourth prompt word into the information generation model to obtain multiple view description information output by the information generation model. The fourth prompt word guides the information generation model to correct the multiple initial description information using the at least one view description sample as a reference.

[0183] It should be understood that at least one view description sample serves as an example provided to the information generation model for learning, enabling the model to learn the patterns of that at least one view description sample. That is, based on at least one view description sample, the information generation model can be guided to refine multiple initial pieces of information with minimal prompting.

[0184] The scene corresponding to the view description sample differs from the scenes corresponding to the multiple initial description information sets, but they share similarities. For example, if the scene of the target panoramic image is a living room, the scene corresponding to the view description sample could also be an indoor scene, such as a bathroom, study, or exhibition hall. Furthermore, the number of sample view description information sets included in the view description sample can be equal to the number of initial description information sets, providing a better reference for the information generation model.

[0185] As an example, assuming multiple initial descriptions are generated based on a living room scene, the corresponding view description samples could be multiple sample view descriptions for the bathroom. The bathroom view description samples could include the following eight sample view descriptions:

[0186] See Figure 1 View description information: A bathroom with a mirror;

[0187] See Figure 2 View description information: A bathroom with a mirror and a vase;

[0188] See Figure 3 View description information: A bathroom with a vase and a toilet;

[0189] See Figure 4 View description information: A bathroom with a toilet;

[0190] See Figure 5 View description information: A bathroom with a toilet and a shower;

[0191] See Figure 6 View description information: A bathroom with a shower;

[0192] See Figure 7 View description information: A bathroom with a shower and a mirror;

[0193] View description information for View 8: A bathroom with a mirror and a vase.

[0194] Correspondingly, the fourth prompt could be: Please refer to the language expression, spatial relationships, and descriptions of key elements in the provided view description sample to correct multiple initial descriptions.

[0195] It should be noted that, given that the fourth prompt word can guide the information generation model to correct the initial description information, this application embodiment does not limit the specific content or language expression of the fourth prompt word.

[0196] The fourth implementation involves, after obtaining multiple initial description information, inputting at least one view description sample and a prompt word into the information generation model to obtain multiple view description information output by the information generation model. Here, the prompt word is a combination of the third and fourth prompt words mentioned above.

[0197] For explanations regarding the view description sample, the third prompt word, and the fourth prompt word, please refer to the three implementation methods mentioned above; they will not be repeated here.

[0198] It should be noted that, for the three implementation methods described above for modifying multiple initial description information to obtain multiple view description information, the modification process can be performed once or multiple times, and in the case of multiple modifications, the input information used for these multiple modifications can be the same. This application embodiment does not impose any limitations on this.

[0199] Step 403: Generate a panoramic image of the target based on multiple view description information using an image generation model.

[0200] In one possible implementation, the image generation model can adopt an existing text-based image model, such as the MVDiffusion model. In this case, step 403 is implemented as follows: multiple view description information are used as input to the MVDiffusion model, and the target panoramic image is output through the MVDiffusion model.

[0201] The implementation logic for generating panoramic images using the MVDiffusion model can be found in industry references or previous articles. Figure 1-2 The explanation will not be repeated here.

[0202] Of course, with the development of text-generated image technology / models or text-generated panoramic image technology / models, the above image generation model can also be implemented using other network models, and this application embodiment does not limit this.

[0203] In this embodiment, based on panoramic description information describing the content of the target panoramic image to be generated, multiple view description information is generated using multi-view prompts, and then the target panoramic image is generated based on the multiple view description information. The multiple view description information and multiple view prompts are used to generate multiple view description information. Figure 1This one-to-one correspondence ensures better alignment between multiple view descriptions and view content when generating the target panoramic image. Furthermore, since the multi-view cue words indicate the division of multiple views and the relationships between them, the view descriptions generated according to these cue words possess spatial consistency. This reduces the likelihood of redundant and / or omitted objects in the target panoramic image generated by the image generation model based on multiple view descriptions. Thus, generating a target panoramic image based on multiple view descriptions significantly improves the quality of the generated panoramic image.

[0204] Based on the above description, for ease of understanding, the following will be combined with the appendix. Figure 6 This document provides an exemplary description of the overall implementation process for generating the target panoramic image using the technical solution of this application. See [link to relevant documentation]. Figure 6 The image generation method provided in this application embodiment may include the following steps (1)-(4) in its implementation process.

[0205] (1) Determine the content element information.

[0206] Step (1) is performed after preprocessing the panoramic description information input by the user. If the panoramic description information input by the user is text, preprocessing is not required. If the panoramic description information input by the user is an image, the image annotation model is needed to obtain the view description information corresponding to the image during preprocessing.

[0207] In step (1), the content input to the information generation model includes preprocessed panoramic description information and a first prompt word. The first prompt word is used to guide the information generation model to summarize multiple key elements in the target panoramic image to be generated, as well as the positional relationships between the multiple key elements, based on the description information input by the user.

[0208] (2) Multi-view planning.

[0209] Step (2) is executed after the content element information is obtained. In step (2), the multiple key elements and their positional relationships obtained in the previous step are used as input to the information generation model. At the same time, the second prompt word input in step (2) guides the information generation model to establish a rotation hypothesis based on the given center position, divide the 360° view into N views, and generate corresponding descriptive information for the key elements and their positional relationships in each view based on the multiple key elements and their positional relationships obtained in the previous step, thus obtaining multiple view description information.

[0210] In this embodiment of the application, taking N=8 as an example, each view covers a 90° field of view, and adjacent views overlap by 45°. Ultimately, the 8 views can cover a 360° field of view.

[0211] (3) Consistency correction and a few prompts.

[0212] Step (3) is executed after step (2) is completed. Its purpose is to correct the multiple view description information output by step (2) using spatial constraints and small sample examples.

[0213] Consistency correction mainly includes the following operations: checking whether the semantics of multiple view descriptions remain consistent: the appearance of the same key element should remain consistent across different views; based on spatial logical relationships, checking for key elements that should not appear in this view. Under normal circumstances, the same key element will only appear in adjacent views and will not span multiple views; based on the principle of simplicity, checking for redundant objects in the view descriptions. Some special objects, such as carpets, are prone to appearing in multiple views, but are located at the edge of the field of view in some views. Based on the principle of simplicity, such objects should be as concise as possible in the multi-view text descriptions. If there are redundant objects, they can be selectively ignored in some view descriptions.

[0214] Secondly, some reference samples are provided as examples, requiring the information generation model to mimic multiple sample view descriptions in the view description samples, while satisfying spatial constraints, and generate multiple view descriptions that conform to contextual coherence. This step clarifies the format of the information generation model's output, making the randomness of the output view descriptions less.

[0215] It should be noted that steps (1)-(3) above are all performed in the dialogue interface of the information generation model. Therefore, the previous dialogue history will serve as the context information for the next round of dialogue. Step (3) can be executed multiple times to obtain better output through repeated corrections by the information generation model.

[0216] (4) Input multiple view description information into the Wensheng panoramic image model to generate the target panoramic image.

[0217] Step (4) is executed after step (3) is completed. Since step (3) has obtained multiple view description information, it can be input into the Wensheng panoramic image model to obtain the corresponding target panoramic image.

[0218] In this example, the MVDiffusion model is selected as the model for textured panoramic images. The system architecture of the MVDiffusion model can be found by referring to... Figure 1The N view descriptions and N random noise images obtained in step (3) are input into the MVDiffusion model. After a multi-step denoising process of the multi-view-multi-branch network, and after decoding and multi-view fusion by the variational autoencoder, the target panoramic image can finally be obtained.

[0219] It should be understood that the above exemplary description is consistent with the above... Figure 4 The embodiments belong to the same technical concept, and the technical features involved are all included in the above. Figure 4 The method embodiments may serve as supplementary explanations to the above embodiments. The corresponding implementation methods and beneficial effects can also be referred to the foregoing description, and will not be elaborated here.

[0220] Next, the image generation apparatus involved in the embodiments of this application will be described.

[0221] Figure 7 This is a schematic diagram of the structure of an image generation device provided in an embodiment of this application. The image generation device can be implemented as part or all of a computer system by software, hardware, or a combination of both. This computer system can be... Figure 3 The system shown. See also Figure 7 The image generation device includes: an information acquisition module 701, a multi-view description module 702, and a panoramic image generation module 703.

[0222] The information acquisition module 701 is used to acquire panoramic description information, which describes the content of the target panoramic image to be generated; for detailed implementation process, please refer to the relevant description in step 401 above, which will not be repeated here.

[0223] The multi-view description module 702 is used to generate multiple view description information based on panoramic description information and multi-view prompts. The multi-view prompts indicate the division of the multiple views and the relationship between the multiple views. The viewing angle range of the multiple views is included within the viewing angle range of the target panoramic image. The multiple view description information and the multiple view prompts are related. Figure 1 One-to-one correspondence; for detailed implementation process, please refer to the relevant description in step 402 above, which will not be repeated here.

[0224] The panoramic image generation module 703 is used to generate a target panoramic image based on multiple view description information through an image generation model; for detailed implementation process, please refer to the relevant description in step 403 above, which will not be repeated here.

[0225] In one possible implementation, the multi-view description module 702 includes:

[0226] The content parsing unit is used to input panoramic description information and first prompt words into the information generation model to obtain content element information output by the information generation model. The first prompt words are used to guide the information generation model to determine the key elements of the target panoramic image. The key element information includes multiple key elements in the target panoramic image and the positional relationship between multiple key elements.

[0227] The description information generation unit is used to generate multiple view description information based on content element information and multi-view prompts.

[0228] In one possible implementation, the information generation unit describes the following:

[0229] The first information generation subunit is used to input content element information, multi-view prompt words, and second prompt words into the information generation model to obtain multiple initial description information output by the information generation model. The second prompt word is used to guide the information generation model to split the content element information according to multiple views.

[0230] The second information generation subunit is used to determine multiple view description information based on multiple initial description information.

[0231] In one possible implementation, the multi-view cue words include the center position of the target panoramic image, the rotation angle, and the number of rotations.

[0232] In one possible implementation, the second information generation subunit is specifically used for:

[0233] Multiple initial descriptions and a third prompt word are input into the information generation model to obtain multiple view descriptions output by the information generation model. The third prompt word is used to guide the information generation model to correct the multiple initial descriptions.

[0234] In one possible implementation, the third prompt word includes spatial constraints, which include at least one of the following:

[0235] The same key element maintains a consistent appearance across different views;

[0236] The same key element exists in both adjacent views;

[0237] Ignore key elements located at the edge of the view;

[0238] The number of key elements in a single view is less than the threshold.

[0239] In one possible implementation, the third prompt word also includes text optimization conditions, which include at least one of the following:

[0240] Multiple view descriptions need to maintain semantic consistency;

[0241] Multiple view descriptions must be consistent in their descriptions of the same key element;

[0242] Deleting key elements that do not belong to the corresponding view based on spatial constraints.

[0243] In one possible implementation, the second information generation subunit is specifically used for:

[0244] Obtain at least one view description sample, which includes multiple sample view description information corresponding to the sample panoramic image;

[0245] Input at least one view description sample, multiple initial description information, and a fourth prompt word into the information generation model to obtain multiple view description information output by the information generation model. The fourth prompt word is used to guide the information generation model to revise the multiple initial description information with at least one view description sample as a reference.

[0246] In one possible implementation, the panoramic description information includes any of the following: the name of the scene in which the target panoramic image is located, a set of information describing the scene, or a single-view image;

[0247] Among them, single-view images are used to describe the scene area covered by a single view in the target panoramic image.

[0248] In one possible implementation, the panoramic description information includes a single-view image; and a content parsing unit, specifically used for:

[0249] Perform image-to-text conversion processing on a single-view image to obtain the view description information corresponding to the single-view image;

[0250] The view description information corresponding to the single-view image and the first prompt word are input into the information generation model to obtain the content element information output by the information generation model.

[0251] In one possible implementation, if the panoramic description information is the name of a scene, then the first prompt word is used to guide the information generation model to expand the content of the panoramic description information in order to obtain the key elements of the target panoramic image; if the panoramic description information is a set of information describing the scene, then the first prompt word is used to guide the information generation model to extract features from the panoramic description information in order to obtain the key elements of the target panoramic image.

[0252] In this embodiment, based on panoramic description information describing the content of the target panoramic image to be generated, multiple view description information is generated using multi-view prompts, and then the target panoramic image is generated based on the multiple view description information. The multiple view description information and multiple view prompts are used to generate multiple view description information. Figure 1This one-to-one correspondence ensures better alignment between multiple view descriptions and view content when generating the target panoramic image. Furthermore, since the multi-view cue words indicate the division of multiple views and the relationships between them, the view descriptions generated according to these cue words possess spatial consistency. This reduces the likelihood of redundant and / or omitted objects in the target panoramic image generated by the image generation model based on multiple view descriptions. Thus, generating a target panoramic image based on multiple view descriptions significantly improves the quality of the generated panoramic image.

[0253] It should be noted that the image generation apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when generating target panoramic images. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the image generation apparatus and the image generation method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0254] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the image generation method shown in the above embodiments.

[0255] This application also provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the image generation method shown in the above embodiments.

[0256] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital versatile disc (DVD)), or a semiconductor medium (e.g., solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.

[0257] It should be understood that "multiple" as mentioned herein refers to two or more. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and the terms "first," "second," etc., do not necessarily imply that they are different.

[0258] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0259] The above descriptions are embodiments provided in this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image generation method characterized by, The method includes: Obtain panoramic description information, which is used to describe the content of the target panoramic image to be generated; Based on the panoramic description information and multi-view prompts, multiple view description information is generated. The multi-view prompts indicate the division method of the multiple views and the relationship between the multiple views. The viewing range of the multiple views is included within the viewing range of the target panoramic image. The multiple view description information corresponds one-to-one with the multiple views. Based on the multiple view descriptions, the target panoramic image is generated using an image generation model.

2. The method of claim 1, wherein, The process of generating multiple view descriptions based on the panoramic description information and multi-view prompts includes: The panoramic description information and the first prompt word are input into the information generation model to obtain the content element information output by the information generation model. The first prompt word is used to guide the information generation model to determine the key elements of the target panoramic image. The content element information includes multiple key elements in the target panoramic image and the positional relationship between the multiple key elements. Based on the content element information and the multi-view prompts, the multi-view description information is generated.

3. The method of claim 2, wherein, The step of generating the multiple view description information based on the content element information and the multiple view prompts includes: The content element information, the multi-view prompt words, and the second prompt word are input into the information generation model to obtain multiple initial description information output by the information generation model. The second prompt word is used to guide the information generation model to split the content element information according to the multiple views. Based on the multiple initial description information, the multiple view description information is determined.

4. The method according to any one of claims 1 to 3, characterized in that, The multi-view prompts include the center position, rotation angle, and number of rotations of the target panoramic image.

5. The method of claim 3 or 4, wherein, The step of determining the plurality of view description information based on the plurality of initial description information includes: The multiple initial description information and the third prompt word are input into the information generation model to obtain the multiple view description information output by the information generation model. The third prompt word is used to guide the information generation model to correct the multiple initial description information.

6. The method of claim 5, wherein, The third prompt word includes spatial constraints, which include at least one of the following: The same key element maintains a consistent appearance across different views; The same key element exists in both adjacent views; Ignore key elements located at the edge of the view; The number of key elements in a single view is less than the threshold.

7. The method of claim 6, wherein, The third prompt word also includes text optimization conditions, which include at least one of the following: The description information of the multiple views needs to be semantically consistent; The descriptions of the same key element in the multiple view descriptions must be consistent; Based on the aforementioned spatial constraints, key elements that do not belong to the corresponding view are deleted.

8. The method of claim 3 or 4, wherein, The step of determining the plurality of view description information based on the plurality of initial description information includes: Obtain at least one view description sample, the view description sample including multiple sample view description information corresponding to the sample panoramic image; The at least one view description sample, the plurality of initial description information, and the fourth prompt word are input into the information generation model to obtain the plurality of view description information output by the information generation model. The fourth prompt word is used to guide the information generation model to revise the plurality of initial description information with the at least one view description sample as a reference.

9. The method according to any one of claims 2 to 8, wherein, The panoramic description information includes any of the following: the name of the scene where the target panoramic image is located, a set of information describing the scene, or a single-view image; The single-view image is used to describe the scene area covered by a single view in the target panoramic image.

10. The method of claim 9, wherein, The panoramic description information includes the single-view image; The step of inputting the panoramic description information and the first prompt word into the information generation model to obtain the content element information output by the information generation model includes: The single-view image is subjected to image-to-text conversion processing to obtain the view description information corresponding to the single-view image; The view description information corresponding to the single-view image and the first prompt word are input into the information generation model to obtain the content element information output by the information generation model.

11. The method as described in claim 9, characterized in that, If the panoramic description information is the name of the scene, then the first prompt word is used to guide the information generation model to expand the content of the panoramic description information in order to obtain the key elements of the target panoramic image; If the panoramic description information is a set of information describing the scene, then the first prompt word is used to guide the information generation model to extract features from the panoramic description information in order to obtain the key elements of the target panoramic image.

12. An image generation apparatus characterized by comprising: The device includes: The information acquisition module is used to acquire panoramic description information, which describes the content of the target panoramic image to be generated. The multi-view description module is used to generate multiple view description information based on the panoramic description information and the multi-view prompt words. The multi-view prompt words indicate the division method of the multiple views and the relationship between the multiple views. The viewing angle range of the multiple views is included in the viewing angle range of the target panoramic image. The multiple view description information corresponds one-to-one with the multiple views. The panoramic image generation module is used to generate the target panoramic image based on the multiple view description information and through an image generation model.

13. The apparatus of claim 12, wherein, The multi-view description module includes: The content parsing unit is used to input the panoramic description information and the first prompt word into the information generation model to obtain the content element information output by the information generation model. The first prompt word is used to guide the information generation model to determine the key elements of the target panoramic image. The key element information includes multiple key elements in the target panoramic image and the positional relationship between the multiple key elements. The description information generation unit is used to generate the multiple view description information based on the content element information and the multi-view prompt words.

14. The apparatus of claim 13, wherein, The description information generation unit includes: The first information generation subunit is used to input the content element information, the multi-view prompt words, and the second prompt words into the information generation model to obtain multiple initial description information output by the information generation model. The second prompt words are used to guide the information generation model to split the content element information according to the multiple views. The second information generation subunit is used to determine the multiple view description information based on the multiple initial description information.

15. The apparatus of any one of claims 12-14, wherein, The multi-view prompts include the center position, rotation angle, and number of rotations of the target panoramic image.

16. The apparatus of claim 14 or 15, wherein, The second information generation subunit is specifically used for: The multiple initial description information and the third prompt word are input into the information generation model to obtain the multiple view description information output by the information generation model. The third prompt word is used to guide the information generation model to correct the multiple initial description information.

17. The apparatus of claim 16, wherein, The third prompt word includes spatial constraints, which include at least one of the following: The same key element maintains a consistent appearance across different views; The same key element exists in both adjacent views; Ignore key elements located at the edge of the view; The number of key elements in a single view is less than the threshold.

18. The apparatus of claim 17, wherein, The third prompt word also includes text optimization conditions, which include at least one of the following: The description information of the multiple views needs to be semantically consistent; The descriptions of the same key element in the multiple view descriptions must be consistent; Based on the aforementioned spatial constraints, key elements that do not belong to the corresponding view are deleted.

19. The apparatus of claim 14 or 15, wherein, The second information generation subunit is specifically used for: Obtain at least one view description sample, the view description sample including multiple sample view description information corresponding to the sample panoramic image; The at least one view description sample, the plurality of initial description information, and the fourth prompt word are input into the information generation model to obtain the plurality of view description information output by the information generation model. The fourth prompt word is used to guide the information generation model to revise the plurality of initial description information with the at least one view description sample as a reference.

20. The apparatus of any one of claims 13-19, wherein, The panoramic description information includes any of the following: the name of the scene where the target panoramic image is located, a set of information describing the scene, or a single-view image; The single-view image is used to describe the scene area covered by a single view in the target panoramic image.

21. The apparatus of claim 20, wherein, The panoramic description information includes the single-view image; the content parsing unit is specifically used for: The single-view image is subjected to image-to-text conversion processing to obtain the view description information corresponding to the single-view image; The view description information corresponding to the single-view image and the first prompt word are input into the information generation model to obtain the content element information output by the information generation model.

22. The apparatus as claimed in claim 20, characterized in that, If the panoramic description information is the name of the scene, then the first prompt word is used to guide the information generation model to expand the content of the panoramic description information in order to obtain the key elements of the target panoramic image; If the panoramic description information is a set of information describing the scene, then the first prompt word is used to guide the information generation model to extract features from the panoramic description information in order to obtain the key elements of the target panoramic image.

23. A computer system, characterized in that The computer system includes a memory and a processor, the memory for storing a computer program and the processor for executing the computer program to implement the steps of the method according to any one of claims 1-11.

24. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on the computer, cause the computer to perform the method according to any one of claims 1-11.

25. A computer program product comprising instructions, wherein: When the instructions are executed on a computer, the computer causes the computer to perform the method according to any one of claims 1-11.

Images