Language model assisted generation of coherent images

Abstract

Systems, apparatuses, and methods may provide a technique that performs a morphing operation on an input image to obtain a changed view and a mask indicating a missing region in the changed view associated with the morphing operation, performing, by a first pre-trained artificial intelligence (AI) model, a repair operation on the missing region according to a text output from a second pre-trained AI model, where the text output is a response to one or more cues, and repeating the deformation operation and the repair operation to obtain a plurality of generated views. The technique may also fuse a plurality of generated views into a 360 DEG panoramic view.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 599,669, filed November 16, 2023. Background Technology

[0003] Generating panoramic scenes from a single input image can be challenging. For example, traditional solutions may produce low-standard outputs with recurring objects (such as multiple beds in a bedroom) or require time-consuming manual text input for each view used to produce the panoramic scene. Attached Figure Description

[0004] Those skilled in the art will understand the various advantages of the embodiments by reading the following description and appended claims, and by referring to the following drawings, wherein:

[0005] Figure 1 This is an illustrated description of a process example according to an embodiment;

[0006] Figure 2 This is an illustrated explanation of an example of a list of pseudocode based on an embodiment;

[0007] Figure 3 This is an example illustration comparing conventional image-to-panorama visualization results with enhanced image-to-panorama visualization results according to an embodiment;

[0008] Figure 4 This is a flowchart illustrating a method example for enhancing the operational performance of a computing system according to an embodiment;

[0009] Figure 5 This is a block diagram of an example of a performance-enhanced computing system according to an embodiment;

[0010] Figure 6 This is an illustrated description of an example of a semiconductor packaging apparatus according to an embodiment;

[0011] Figure 7 This is a block diagram of an example processor according to an embodiment;

[0012] Figure 8 This is a block diagram of an example of a multiprocessor-based computing system according to an embodiment; Detailed Implementation

[0013] The challenge of generating panoramic scenes (“panoramic images”) from a single input image involves the fact that most of the training data used for scene generation is based on images with limited field of view and inconsistent environmental data. Existing methods for expanding the field of view may not be sufficient.

[0014] More specifically, some methods may treat the panorama as a single equidistant rectangular image and learn to generate the image in a single operation through a generative model (e.g., a latent diffusion model). This approach struggles to close loops at both ends of the generated equidistant rectangular image.

[0015] Other methods create panoramas by generating multiple perspective views using a more robust pre-trained diffusion model (trained on large-scale perspective data). These methods may achieve multi-view functionality through deformation or by using additional multi-view attention modules fine-tuned on small datasets. Figure 1 Consistency. However, these methods struggle to generate diverse 360° views.

[0016] The technique described in this paper provides Language Model Assisted Generation of Images with Coherence (L-MAGIC), which utilizes large language models (LLMs) to guide the generation of coherent views in a 360° panoramic scene. More specifically, the technique described in this paper generates a coherent 360° panoramic scene given a single seed image. Examples utilize large language models based on seed images to guide the generation of diverse and coherent views. The term "coherent" in this paper refers not only to smooth view transitions but also to the ability to create scenes without unnecessary repetition of objects (e.g., multiple beds in a bedroom) across multiple views. The output quality and resolution of the panoramic scene are further enhanced by super-resolution and multi-view fusion techniques.

[0017] In fact, the technique described in this paper can generate 360° panoramic views with higher quality and more coherent scene layouts. The technique is effective for both "real-world" scene input images and synthetic data. Furthermore, the technique can be combined with mature conditional generative artificial intelligence (AI) models to create panoramic views based on diverse input modalities, such as textual descriptions of the scene, perspective sketches of the scene, depth maps, etc. This characteristic allows the technique to be used in applications involving more than a single image input. In addition, realistic and diverse panoramic scenes can be used to create 3D point clouds and immersive videos for applications such as video games, virtual reality, and 3D environments.

[0018] When the techniques described herein are implemented, redundant objects (e.g., multiple beds in a bedroom) are adaptively eliminated from the scene. Furthermore, embodiments use a large language model to generate a scene description, which is then used as cue words to generate panoramic or environmental scenes. Moreover, the generated scene generally matches the description of the large language model.

[0019] like Figure 1 As shown, the L-MAGIC process 10 uses an iterative warp and inpainting process to generate a panoramic scene. Given a starting image, warp operation 12 generates a changed perspective (e.g., viewpoint rotation) and a mask indicating the missing regions caused by the change. Inpainting operation 14 responds to automated prompts and, with the aid of text output from a large language model (LLM, e.g., a pre-trained AI model) 16, uses an image generation model to fill in the missing regions. These text outputs from the LLM 16 guide the generated regions to become more diverse and realistic scenes. The LLM 16 is also used to monitor each inpainting operation 14 so that for special cases where the inpainting does not follow the guidance from the LLM 16, the inpainting operation 14 can be rerun to ensure the effectiveness of the guidance from the LLM 16.

[0020] The final panorama is created by fusing / merging the generated views 20 into a 360° panoramic view 22 and then subjecting it to some post-processing aimed at enhancing quality and resolution. Furthermore, process 10 uses techniques to differentiate scene layout regions (such as, foreground and background) to identify objects and uses information from LLM-guided scene generation with positive and negative cue words to enhance human expectations of scene layout diversity and consistency. In the example shown, the panoramic view 22 is used to generate an immersive video 24 and / or a three-dimensional (3D) point cloud 26.

[0021] More specifically, the input to process 10 is image I, which is either captured by camera 28 (e.g., in the real world) or synthesized by a conditional diffusion model 30 (e.g., ControlNet). Multiple novel views are generated through iterative deformation and inpainting to construct the 360° panoramic view 22. A pre-trained diffusion model, assisted by a pre-trained language model, is used to generate views with both high-quality local textures and a coherent 360° layout. Further quality enhancement techniques ensure the smooth fusion of multiple views into a high-resolution panoramic scene. Process 10 can also generate panoramic view 22, immersive video 24, and 3D point cloud 26 from various types of input (e.g., text input 32, synthesized image 34, and / or sketch 36).

[0022] Figure 2 The diagram shows a list of 40 pseudocode snippets illustrating how a panoramic view / scene is generated from a natural image through iterative warping and inpainting. Warping operations produce incomplete perspective views and masks of missing regions. Inpainting operations, aided by a language model, restore the masked regions to their full extent. The final panoramic view is created by fusing the generated views with post-processing steps aimed at enhancing quality and resolution.

[0023] More specifically, during the warp operation, L-MAGIC uses image warping to transform an existing view into a novel view, where the novel view has a relative rotation compared to the existing view and covers incomplete areas of the 360° panoramic scene. Each warp operation takes the entire existing view, as well as the relative camera rotation between the novel view and the existing views, as input, and produces an image of the novel view and a binary mask indicating whether each pixel of the novel view is complete, where complete pixels are covered by the existing view. Since only a panoramic perspective view is generated, no camera translation is used during the warp operation.

[0024] During each deformation operation, all complete views are projected onto a unit sphere representing the panoramic scene. Then, based on the relative camera pose, the next incomplete view is submitted for repair. To project the image onto the unit sphere, a mesh is first constructed by defining vertices V at each image pixel and creating edges E between adjacent pixels. Then, the vertices are projected onto the unit sphere using equation (1):

[0025]

[0026] Where K is the intrinsic parameter matrix, v∈V are the homogeneous coordinates of the pixels, and v sp This refers to the projected positions. To transform the complete view A into a novel view B, each projected vertex v of A... sp v rot =Rv sp Rotate, where R is the rotation matrix from A to B, then perform rasterization-based rendering (I,M) = rasterize(V rot ,E,K′), where V rot I is the set of rotated vertices, and K′ is the intrinsic parameter matrix of image B. The output I is the deformed image, and M is a binary mask indicating whether to perform inpainting on pixels (e.g., obtained by detecting each pixel to determine whether the ray projection intersects with the effective mesh face).

[0027] To ensure consistent local restoration across each perspective, a large field of view (FoV) is used, and the rotation angle is adjusted so that both known and unknown regions are sufficiently large after deformation. In practice, a 100-degree field of view with approximately 40° rotation between adjacent views provides favorable results. To further reduce the accumulation of errors during iteration, the scene is expanded alternately from both sides of the input image, rather than from a single direction. To ensure a smooth 360° loop closure, the rotation angle is adjusted so that the final view has a relatively large incomplete area at the center, producing a series of...

[0028] A view of rotation angles {0°, 41°, -41°, 82°, -82°, 123°, 200.5° (e.g., for closed loops)}.

[0029] During the inpainting process, L-MAGIC uses two-dimensional (2D) image inpainting models, such as, for example, the Stable Diffusion v2 inpainting model (e.g., the first pre-trained AI model), to complete missing regions in the novel view, where missing regions refer to pixels whose output binary mask of the deformation operation is zero.

[0030] More specifically, the repair operation completes the deformed view with a consistent local style and a coherent 360° scene layout. In one example, the Stable Diffusion Version 2 repair model effectively infers large missing areas in each deformed view while maintaining a consistent local style. However, naive repair without any prior information (e.g., contextual information related to the layout) can produce severe patching artifacts. For example, a common prior approach might operate based on a textual description of the scene provided by the user or an input image. However, because perspective repair methods lack a mechanism to segment the layout into different views, using the same description in different views can result in duplicate objects, such as multiple beds in a bedroom. To address these issues, a language model is used. (e.g., ChatGPT4, a second pre-trained AI model) to guide the repair process, and use a visual language model. (For example, bootstrapping language-image pre-training (BLIP-2), a third pre-trained model).

[0031] Pseudocode list 40 indicates that before deformation and repair, First, you are prompted to generate a description d for the input image I. I (For example, line 2). Two questions are posed such that d I It includes both coarse and fine-level descriptions. Next, it requires... Based on d I Imagine the global scene layout d 360 (For example, in line 3), where d 360 Each line corresponds to a description of a specific view. To avoid duplicate objects, each view should be described concisely without mentioning objects in other views.

[0032] variable d 360 This mainly includes objects from various views. Using such descriptions as repair tips may lead to inconsistent perspective styles. Therefore, it is required... From d IRemove objects and obtain the final scene-level description d. scene (For example, "a bedroom with a wooden bed" becomes "bedroom" (e.g., line 4)). Variable d scene Subsequently with d 360 Use them together to ensure a consistent multi-view style. Although d scene Ensuring consistent style across multiple views is important, but training data bias in the diffusion model may still result in generating objects that are typically associated with a specific scene, even in d. 360 It's not explicitly mentioned in the documentation. For example, if the prompt contains the word "bedroom," it often generates a bed, resulting in duplicate beds in multiple views. To address this, allow... Automatically determine if there are any objects in the scene that need to be avoided from being repeated (e.g., line 5).

[0033] After each morphing operation, the output from lines 2 to 5 is used to automatically generate prompts for text conditional repair (e.g., line 12). Specifically, for a morphed view rotated by 0° (i=1), d scene Used as a text conditional repair prompt (d i (line 7). For other views (e.g., line 13), if in d repeat If there is no object in the list (e.g., it does not involve avoiding duplicates), then use the prompt word " <d scene > The outer view, in which we see <d 360 The corresponding description in the text > "Execute repair. If in d repeat If any object exists, use the positive prompt word " <d scene The outer view of > where we only see <d 360 The corresponding description in the text >, and uses the negative warning word "any type" <d repeat The object in the middle > (for example, used for d) repeat A single sentence for each object in the environment. Positive prompt word templates prevent the stable diffusion of common objects in the environment (e.g., the bed in a bedroom). Negative prompt word templates avoid... repeat The objects mentioned in the text are repeated.

[0034] The training data for the diffusion model contains a bias—images with the title "Bedroom" mostly contain beds. Therefore, even with the constraint of the cue word, duplicate objects may still be generated. To further mitigate this problem, use... To detect each repaired image I i Does it contain d? repeat The objects mentioned in the text (e.g., line 15). If the answer is "yes", rerun the repair until the answer becomes "no", or the maximum number of trials c is reached (e.g., 20).

[0035] As mentioned earlier, one or more cue words may be used to enable the language model to automatically control the diffusion model during the repair process. For line 2 of pseudocode list 40, it's possible to direct the visual language model... The following two questions are raised:

[0036] Question Q1 BLIP What place is this (describe it in fewer than 5 words)?

[0037] answer:

[0038] Question Q2 BLIP Please describe the foreground and background in detail.

[0039] answer:

[0040] These two issues allow the model to output both coarse and fine-grained scene-level descriptions without focusing on centrally located objects, which is beneficial for inferring the global scene layout in the third line. The final d I This is the answer to two questions.

[0041] To obtain the scene layout description d of each view 360 In the third line, to the language model The following questions are raised:

[0042] Q1 GPT Given that <Q1 BLIP The answer > scenario, in which we saw earlier <Q2 BLIP > Answer. Generate 6 rotated views to describe what you see in this location, where the camera in each view is rotated 60 degrees to the right (you don't have to describe the original view; the first of the 6 views you need to describe is the one with a 60-degree rotation). Avoid redundant descriptions; only describe the content of each view. Also, don't repeat the same objects in different views. Don't reference previously generated views. Generate concise (<10 words) and varied content for each view. Begin each sentence with: View xxx (view number, from 1-6): We see…

[0043] Although the language model is required to output results in the order of 60 => 120..., it may simply use prompt words in the order of, for example, 60 => -60 => 120 => -120... Furthermore, the 60-degree value is only a rough number, allowing the language model to easily understand the question (for example, in deformation and repair operations, a value closer to 40 degrees is used instead of 60 degrees based on the actual angle rotation).

[0044] As mentioned earlier, language models (such as ChatGPT) sometimes do not perfectly follow Q1. GPTThe formatting requirements in Q1 may cause auto-suggestion generation to fail. To avoid such failures, perform a check to confirm Q1. GPT The output checks if it has the required number of lines (e.g., 6 lines) and if each line begins with "View XXX (line number): We see...". If any condition is violated, line 3 of pseudocode listing 40 is rerun. This method ensures that the language model understands the question and meets all formatting requirements.

[0045] After all the 360° perspective views are generated, these views are / merged into the final panorama. To ensure the smooth merging of multiple overlapping images into a panoramic image and to create a high-resolution panorama, smooth blending and super-resolution techniques are utilized.

[0046] More specifically, the adjacent pixels at the image center have a larger angular distance than the pixels at the image edges. When the complete view is warped into a novel view, the original central region becomes the edge region, resulting in a blurred rendered image due to interpolation. Simultaneously, because the stable diffusion model outputs a resolution of 512*512, the panoramas created from these images have relatively low resolution. To address these two issues, super-resolution is applied to the output I of each restoration operation. i , will I i The resolution is increased to 2048*2048. Then, the high-resolution image is warped into a novel low-resolution view, eliminating the need for (strong) interpolation. After performing all warping and restoration operations, the super-resolution images are simply fused to generate a high-resolution panorama.

[0047] During the deformation and panorama generation process, multiple perspective views may overlap in the same area. To avoid sharp edges when merging images, a weighted average is performed. For example, given multiple colors of the same location, c... i The deformed pixels, the final merged pixel color is Where the weight w i The distance to the nearest image boundary of the original view i is calculated. This strategy effectively reduces the weight of pixels near the deformation boundary, ensuring a smooth transition during the multi-view fusion process.

[0048] To create the final panorama (e.g., line 22), each view is first projected onto a unit sphere. Then, an equidistant rectangular projection is performed to transform the multiple projected views into the same equidistant rectangular plane, and the projected views are merged into a single equidistant rectangular image.

[0049] To remove object-level information, line 4 of pseudocode listing 40 might require:

[0050] Q2 GPT Modify the following statements: <Q1 BLIPThe answer > allows us to remove all objects from the description (e.g., "a bedroom with a bed" might become "bedroom". If the description is just one object, then don't change the sentence). Only output the modified sentence.

[0051] To adaptively determine whether to avoid duplicate objects, line 5 of pseudocode listing 40 might ask the following two questions:

[0052] Q3 GPT Given with <Q1 BLIP The scene in front of us is the answer to the question. <Q2 BLIP The answer is: What are the two main foreground objects we're looking at? Describe them using two lines, each formatted as "We see: xxx (an object, do not describe details, use only one word for the object. Start with the most likely object. Do not mention background objects such as items on the wall, ceiling, or floor.)"

[0053] Q4 GPT We often have <Q1 BLIP I saw multiple [answers] in the scenario. <Q3 GPT Does every object in the answer >? Only answer with a lowercase 'yes' or 'no'.

[0054] d repeat The final state of the variable is Q3. GPT The collection of objects in Q4 makes Q4 GPT The corresponding answer is 'No'.

[0055] Quantitatively, the implementation consistently outperforms traditional solutions in both human evaluation metrics and standard evaluation metrics. The results of the techniques described in this paper are favored more frequently than those of traditional solutions (e.g., by more than 50%). The implementation also achieves a higher Inception Score (IS), indicating more diverse multi-view layouts and higher perspective rendering quality.

[0056] Figure 3The images show a comparison of the image-to-panorama visualizations. In the examples shown, the first set of images 50 (50a, 50b, e.g., Stable Diffusion v2) indicates that the solution cannot achieve 360° loop closure (e.g., sharp edges in the center of the panorama). The second set of images 52 (52a, 52b, e.g., Text2room) and the third set of images 54 (54a, 54b, MVDiffusion) indicate that these solutions cannot avoid repeating objects across multiple views. In contrast, the fourth set of images 56 (56a, 56b, e.g., L-MAGIC) shows that the techniques described in this paper generate panoramas with high perspective rendering quality and reasonable scene layout.

[0057] Figure 4 A method 60 for improving the operational performance of a computing system is illustrated. Method 60 can be implemented in one or more modules as a set of logic instructions stored in a machine-readable or computer-readable storage medium, such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), firmware, flash memory, etc. Method 60 can be implemented in hardware, or in any combination thereof. For example, hardware implementation may include configurable logic, fixed-function logic, or any combination thereof. Examples of configurable logic (e.g., configurable hardware) include appropriately configured programmable logic arrays (PLAs), field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and general-purpose microprocessors. Examples of fixed-function logic (e.g., fixed-function hardware) include appropriately configured application-specific integrated circuits (ASICs), combinational logic circuits, and sequential logic circuits. Configurable logic or fixed-function logic can be implemented using complementary metal oxide semiconductor (CMOS) logic circuits, transistor-transistor logic (TTL) logic circuits, or other circuits.

[0058] The computer program code used to perform the operations shown in method 60 may be written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as JAVA, SMALLTALK, C++, etc.) and traditional procedural programming languages ​​(such as the "C" programming language or similar programming languages). Furthermore, the logic instructions may include assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, status setting data, integrated circuit configuration data, and status information for the electronic circuitry and / or other structural components inherent to the individual hardware (e.g., host processor, central processing unit / CPU, microcontroller, etc.).

[0059] The illustrated processing box 62 determines whether the input image receives camera output. If not, box 64 generates an input image based on a pre-trained diffusion model and one or more of text input, sketch input, or synthetic images. Box 66 performs a warp operation on the input image to obtain a modified view and a mask, where the mask indicates missing regions in the modified view associated with the warp operation. In one example, box 66 alternately expands the field of view from both sides of the input image. Box 66 may also adjust the rotation angle to generate an incomplete region at the center of the final view. Box 68 specifies that a repair operation is performed on the missing regions by a first pre-trained AI model (e.g., a diffusion model) based on the text output of a second pre-trained AI model (e.g., a language model), where the text output is a response to one or more cue words. In one example, box 68 generates cue words at least in part based on a description of the input image from a third pre-trained AI model (e.g., a visual language model). In embodiments, the cue words apply distinctions between scene layout regions (e.g., background, foreground). Furthermore, the cue words may include negative and positive cue words. Additionally, box 68 may apply super-resolution to the output of the repair operation.

[0060] At box 70, it is determined whether boxes 66 and 68 will be repeated. In one example, the determination at box 70 considers whether the generated views are sufficient to complete a 360° panoramic scene. If the warp and repair operations will be repeated, method 60 returns to box 66 and obtains multiple generated views. Otherwise, box 72 merges the multiple generated views into a 360° panoramic view. Furthermore, box 74 can generate one or more of an immersive video or a 3D point cloud based on the 360° panoramic view.

[0061] To generate video with pure camera rotation, image warping is used, projecting the panorama onto individual frames based on a rotation matrix. To generate video with camera translation, depth-based warping is used, where a depth estimation model (e.g., Zoe-Depth or other arbitrary model) first upscales the pixels to 3D, and then warps the 3D point cloud onto individual frames based on camera extrinsic matrices (e.g., including rotation and translation).

[0062] To generate a 3D point cloud, a depth estimation model is first applied to each perspective of the panorama. Then, multi-view depth is optimized to force overlapping pixels in multiple views to have the same depth. Next, the multi-view depth is boosted to a 3D point cloud, which is then fused into a single coordinate system to produce the final 360-degree point cloud.

[0063] Therefore, Method 60 enhances performance to at least this extent, using a second pre-trained AI model to guide the inpainting operation to produce higher-quality images and more coherent scene layouts. Furthermore, the ability to start with camera output (e.g., “real-world” input images), synthetic images, perspective sketches, and text input makes Method 60 more scalable and suitable for a wider range of applications. In fact, realistic and diverse scenes can be used to create 3D point clouds and immersive videos for applications such as video games, virtual reality (VR), and 3D environments.

[0064] Now go to Figure 5 The diagram illustrates a performance-enhanced computing system 280. System 280 can typically be a component of an electronic device / platform with the following functions: computing capabilities (e.g., personal digital assistant / PDA, laptop, tablet, convertible tablet, edge node, server, cloud computing infrastructure), communication capabilities (e.g., smartphone), imaging capabilities (e.g., camera, camcorder), media playback capabilities (e.g., smart TV), wearable capabilities (e.g., watch, glasses, headwear, footwear, jewelry), vehicle capabilities (e.g., car, truck, motorcycle), robotic capabilities (e.g., autonomous robot), Internet of Things (IoT) capabilities, drone capabilities, etc., or any combination thereof.

[0065] In the illustrated example, system 280 includes a host processor 282 (e.g., a central processing unit / CPU) with an integrated memory controller (IMC) 284 coupled to system memory 286 (e.g., a dual inline memory module / DIMM, which includes dynamic RAM / DRAM). In this embodiment, an I / O (input / output) module 288 is coupled to the host processor 282. The illustrated I / O module 288 communicates with devices such as: a display 290 (e.g., a touchscreen, liquid crystal display / LCD, light emitting diode / LED display), a camera 304, mass storage 302 (e.g., a hard disk drive / HDD, optical disc, solid state drive / SSD), and a network controller 292 (e.g., wired and / or wireless). The host processor 282 can be combined with the I / O module 288, the graphics processor 294, and the AI ​​accelerator 296 (e.g., a dedicated processor) to form a system on chip (SoC) 298.

[0066] In an embodiment, the AI ​​accelerator 296 and / or the host processor 282 executes instructions 300 (e.g., executable program instructions) retrieved from system memory 286 and / or mass storage 302 to perform the method 60 already discussed. Figure 4 One or more aspects of the image. Therefore, execution of instruction 300 causes AI accelerator 296, host processor 282, and / or computing system 280 to perform a warping operation on the input image to obtain a modified view and a mask, wherein the mask indicates missing regions in the modified view associated with the warping operation. Furthermore, execution of instruction 300 causes AI accelerator 296, host processor 282, and / or computing system 280 to perform a repair operation on the missing regions based on the text output of a second pre-trained AI model using a first pre-trained AI model, wherein the text output is a response to one or more prompt words. Execution of instruction 300 causes AI accelerator 296, host processor 282, and / or computing system 280 to repeat the warping and repair operations to obtain multiple generated views, which can be fused into a 360° panoramic view.

[0067] The computing system 280 is therefore considered to enhance performance to at least this extent, namely, by using a second pre-trained AI model to guide the restoration operation to produce higher-quality images and more coherent scene layouts. Furthermore, the ability to start with the output of camera 304, synthesized images, perspective sketches, and text input makes method 60 more scalable and suitable for a wider range of applications. In fact, realistic and diverse scenes can be used to create 3D point clouds and immersive videos for applications such as video games, VR, and 3D environments.

[0068] Figure 6 A semiconductor device 350 (e.g., a chip, die, package) is illustrated. The illustrated device 350 includes one or more substrates 352 (e.g., silicon, sapphire, gallium arsenide) and logic 354 (e.g., transistor arrays and other integrated circuit / IC components) coupled to the substrates 352. In an embodiment, the logic 354 implements the method 60 already discussed. Figure 4 One or more aspects of ).

[0069] Logic 354 can be at least partially implemented in configurable hardware or functionally fixed hardware. In one example, logic 354 includes a transistor channel region disposed (e.g., embedded) within substrate 352. Therefore, the interface between logic 354 and substrate 352 may not be a steep face. Logic 354 can also be considered as an epitaxial layer grown on the initial wafer of substrate 352.

[0070] Figure 7 The illustration depicts a processor core 400 according to one embodiment. The processor core 400 can be the core of any type of processor, such as a microprocessor, embedded processor, digital signal processor (DSP), network processor, or other device that executes code. Although Figure 7 The diagram only shows one processor core 400; the processing element may alternatively include more than one. Figure 7 The processor core 400 is illustrated in the figure. The processor core 400 may be a single-threaded core or, at least in one embodiment, the processor core 400 may be multi-threaded, i.e., each of its cores may include more than one hardware thread context (or "logical processor").

[0071] Figure 7 The diagram also illustrates memory 470 coupled to processor core 400. Memory 470 can be any of various types of memory (including various layers of memory hierarchy) known to those skilled in the art or otherwise available. Memory 470 may include one or more instructions of code 413 executed by processor core 400, wherein code 413 may implement the method 60 already discussed. Figure 4 The processor core 400 executes the sequence of program instructions indicated by code 413. Each instruction can enter the front-end section 410 and be processed by one or more decoders 420. The decoder 420 can generate micro-operations as its output, such as fixed-width micro-operations in a predefined format, or it can generate other instructions, micro-instructions, or control signals that reflect the original code instructions. The front-end section 410 illustrated also includes register renaming logic 425 and scheduling logic 430, which typically allocate resources and queue operations corresponding to the translated instructions for execution.

[0072] Processor core 400 is shown to include execution logic 450 having a set of execution units 455-1 to 455-N. Some embodiments may include several execution units dedicated to a specific function or set of functions. Other embodiments may include only one execution unit or a single execution unit capable of performing a specific function. The execution logic 450 shown performs operations specified by code instructions.

[0073] After completing the operations specified by the execution code instructions, the backend logic 460 retrieves the instructions of code 413. In one embodiment, processor core 400 allows out-of-order execution but requires instructions to be submitted in order. Submission logic 465 can take various forms well known to those skilled in the art (e.g., reordering buffers or similar forms). In this way, during the execution of code 413, processor core 400 is transformed, at least in the following ways: the output generated by the decoder, the hardware registers and tables utilized by register renaming logic 425, and any registers (not shown) modified by execution logic 450.

[0074] Although Figure 7 Not shown, but the processing element may include other on-chip elements having processor core 400. For example, the processing element may include memory control logic as well as processor core 400. The processing element may include I / O control logic and / or may include I / O control logic integrated with memory control logic. The processing element may also include one or more caches.

[0075] Now for reference Figure 8 The diagram shown is a block diagram of an embodiment of the computing system 1000 according to an embodiment. Figure 8 The system shown is a multiprocessor system 1000, which includes a first processing element 1070 and a second processing element 1080. Although two processing elements 1070 and 1080 are shown, it should be understood that embodiments of system 1000 may also include only one such processing element.

[0076] System 1000 is illustrated as a point-to-point interconnected system, wherein the first processing element 1070 and the second processing element 1080 are coupled via point-to-point interconnect 1050. It should be understood that... Figure 8 Any or all interconnections shown in the diagram can be implemented as a multipoint bus instead of a point-to-point interconnection.

[0077] like Figure 8 As shown, each of processing elements 1070 and 1080 can be a multi-core processor, comprising a first processor core and a second processor core (i.e., processor cores 1074a and 1074b and processor cores 1084a and 1084b). These cores 1074a, 1074b, 1084a, and 1084b can be configured to operate in a manner similar to the combinations described above. Figure 7 The method discussed executes the instruction code.

[0078] Each processing element 1070, 1080 may include at least one shared cache 1896a, 1896b. Shared caches 1896a, 1896b may store data (e.g., instructions) used by one or more components of the processor (e.g., cores 1074a, 1074b and 1084a, 1084b). For example, shared caches 1896a, 1896b may locally cache data stored in memories 1032, 1034 for faster access by processor components. In one or more embodiments, shared caches 1896a, 1896b may include one or more intermediate caches, such as level 2 (L2), level 3 (L3), level 4 (L4), or other levels of cache, last-level cache (LLC), and / or combinations thereof.

[0079] Although only two processor elements 1070 and 1080 are shown, it should be understood that the scope of the embodiments is not limited thereto. In other embodiments, one or more additional processing elements may be present in a given processor. Alternatively, one or more of the processing elements 1070 and 1080 may be other elements besides a processor, such as an accelerator or a field-programmable gate array (FPGA). For example, additional processing elements may include an additional processor identical to the first processor 1070, an additional processor heterogeneous or asymmetric to the first processor 1070, an accelerator (e.g., a graphics accelerator or digital signal processing (DSP) unit), a FPGA, or any other processing element. There may be various differences between processing elements 1070 and 1080 in a range of quality metrics, including architecture, microarchitecture, thermal characteristics, power consumption characteristics, etc. These differences may manifest in practice as asymmetry and heterogeneity between processing elements 1070 and 1080. For at least one embodiment, the various processing elements 1070 and 1080 may be located in the same die package.

[0080] The first processing element 1070 may further include a memory controller (MC) 1072 and point-to-point (PP) interfaces 1076 and 1078. Similarly, the second processing element 1080 may include an MC 1082 and PP interfaces 1086 and 1088. Figure 8 As shown, MCs 1072 and 1082 couple the processor to the corresponding memories, namely memories 1032 and 1034, which may be components of the main memory locally connected to the respective processor. Although MCs 1072 and MC 1082 are illustrated as being integrated into processing elements 1070 and 1080, in alternative embodiments, the MC logic may be separate logic outside of processing elements 1070 and 1080, rather than logic integrated therein.

[0081] The first processing element 1070 and the second processing element 1080 can be coupled to the I / O subsystem 1090 via PP interconnects 1076 and 1086, respectively. For example... Figure 8 As shown, the I / O subsystem 1090 includes PP interconnects 1094 and 1098. Furthermore, the I / O subsystem 1090 includes an interface 1092 for coupling the I / O subsystem 1090 to a high-performance graphics engine 1038. In one embodiment, a bus 1049 may be used to couple the graphics engine 1038 to the I / O subsystem 1090. Alternatively, point-to-point interconnects may couple these components.

[0082] Conversely, the I / O subsystem 1090 can be coupled to the first bus 1016 via interface 1096. In one embodiment, the first bus 1016 may be a Peripheral Component Interconnect (PCI) bus, or a bus such as a high-speed PCI bus or other third-generation I / O interconnect bus, but the scope of the embodiment is not limited thereto.

[0083] like Figure 8 As shown, various I / O devices 1014 (e.g., biometric scanners, speakers, cameras, sensors) can be coupled to a first bus 1016 and a bus bridge 1018 that couples the first bus 1016 to a second bus 1020. In one implementation, the second bus 1020 may be a low pin count (LPC) bus. Various devices can be coupled to the second bus 1020, including, for example, a keyboard / mouse 1012, a communication device 1026, and a data storage unit 1019 (such as a disk drive or other mass storage device), which in one embodiment may contain code 1030. The illustrated code 1030 can implement the method 60 already discussed. Figure 4 In addition, the audio I / O 1024 can be coupled to the second bus 1020, and the battery 1010 can power the computing system 1000.

[0084] Note: Other embodiments have been considered. For example, the system could implement a multipoint bus or other similar communication topology, instead of... Figure 8 A peer-to-peer architecture. Furthermore... Figure 8 The components can be used alternatively compared to Figure 8 The division is based on the number of more or fewer integrated chips shown.

[0085] In one example, the techniques described in this paper were integrated into the Intel GAUDI AI Accelerator platform to enable AI capabilities for multimodal panoramic image generation. The techniques described in this paper can also be incorporated into the AI ​​Personal Computer (PC) Accelerator Program, which connects independent hardware vendors (IHVs) and independent software vendors (ISVs) with Intel resources, including AI toolchains, training, collaborative engineering, software optimization, hardware, design resources, technical expertise, joint marketing, and sales opportunities.

[0086] Additional notes and examples:

[0087] Example 1 includes

[0088] A performance-enhanced computing system includes a network controller, a processor coupled to the network controller, and a memory coupled to the processor, the memory including a plurality of executable program instructions that, when executed by the processor, cause the processor to perform a warping operation on an input image to obtain a warped view and a mask, wherein the mask indicates missing regions in the warped view associated with the warping operation, a repair operation is performed on the missing regions by a first pre-trained artificial intelligence (AI) model based on text output from a second pre-trained AI model, wherein the text output is a response to one or more prompt words, and the warping and repair operations are repeated to obtain a plurality of generated views.

[0089] Example 2 includes the computing system of Example 1, wherein when multiple executable program instructions are executed, the processor also causes the processor to merge multiple generated views into a 360° panoramic view.

[0090] Example 3 includes the computing system of Example 2, wherein when multiple executable program instructions are executed, the processor also generates one or more of a video or a 3D point cloud based on a 360° panoramic view.

[0091] Example 4 includes a computing system of any one of Examples 1 to 3, wherein when a plurality of executable program instructions are executed, the processor also generates one or more cue words at least in part based on a description of an input image from a third pre-trained AI model, wherein the one or more cue words are used to apply distinctions between scene layout regions, and wherein the one or more cue words include negative cue words and positive cue words.

[0092] Example 5 includes a computing system of any of Examples 1 to 3, wherein when multiple executable program instructions are executed, the processor also applies super-resolution to the output of the repair operation.

[0093] Example 6 includes at least one computer-readable storage medium containing a plurality of executable program instructions that, when executed by a computing system, cause the computing system to perform a warping operation on an input image to obtain a warped view and a mask, wherein the mask indicates missing regions in the warped view associated with the warping operation, a first pre-trained artificial intelligence (AI) model performs a repair operation on the missing regions based on text output from a second pre-trained AI model, wherein the text output is a response to one or more prompt words, and the warping and repair operations are repeated to obtain a plurality of generated views.

[0094] Example 7 includes at least one computer-readable storage medium of Example 6, wherein when a plurality of executable program instructions are executed, the computing system further causes the computing system to merge a plurality of generated views into a 360° panoramic view.

[0095] Example 8 includes at least one computer-readable storage medium of Example 7, wherein when a plurality of executable program instructions are executed, a computing system further causes a video or a three-dimensional point cloud to be generated based on a 360° panoramic view.

[0096] Example 9 includes at least one computer-readable storage medium of Example 6, wherein when a plurality of executable program instructions are executed, a computing system is also caused to generate one or more cue words at least in part based on a description of an input image from a third pre-trained AI model, wherein the one or more cue words are used to apply distinctions between scene layout regions, and wherein the one or more cue words include negative cue words and positive cue words.

[0097] Example 10 includes at least one computer-readable storage medium of Example 6, wherein when a plurality of executable program instructions are executed, the computing system also applies super-resolution to the output of the repair operation.

[0098] Example 11 includes at least one computer-readable storage medium of any of Examples 6 to 10, wherein when a plurality of executable program instructions are executed, a computing system is also caused to generate an input image based on a pre-trained diffusion model and one or more of text input, sketch input, or synthetic image.

[0099] Example 12 includes at least one computer-readable storage medium of any of Examples 6 to 10, wherein an input image is associated with a camera output.

[0100] Example 13 includes a semiconductor device comprising one or more substrates; and logic coupled to the one or more substrates, wherein the logic is at least partially implemented in one or more configurable or functionally fixed hardware, the logic being used to perform a warping operation on an input image to obtain a warped view and a mask, wherein the mask indicates missing regions in the warped view associated with the warping operation, a repair operation is performed on the missing regions by a first pre-trained artificial intelligence (AI) model based on text output from a second pre-trained AI model, wherein the text output is a response to one or more prompt words, and the warping and repair operations are repeated to obtain multiple generated views.

[0101] Example 14 includes the semiconductor device of Example 13, wherein logic is also used to merge multiple generated views into a 360° panoramic view.

[0102] Example 15 includes the semiconductor device of Example 14, wherein logic is further configured to generate one or more of a video or a 3D point cloud based on a 360° panoramic view.

[0103] Example 16 includes the semiconductor device of Example 13, wherein logic is used to generate one or more cue words based at least in part on a description of an input image from a third pre-trained AI model, wherein the one or more cue words are used to apply distinctions between scene layout regions, and wherein the one or more cue words include negative cue words and positive cue words.

[0104] Example 17 includes the semiconductor device of Example 13, wherein logic is also used to apply super-resolution to the output of the repair operation.

[0105] Example 18 includes a semiconductor device of any of Examples 13 to 17, wherein logic is further configured to generate an input image based on a pre-trained diffusion model and one or more of a text input, a sketch input, or a synthetic image.

[0106] Example 19 includes a semiconductor device of any of Examples 13 to 17, wherein an input image is associated with a camera output.

[0107] Example 20 includes a semiconductor device of any of Examples 13 to 19, wherein logic coupled to one or more substrates includes transistor regions disposed within one or more substrates.

[0108] Example 21 includes a method for an operationally enhanced computing system, comprising performing a warp operation on an input image to obtain a warped view and a mask, wherein the mask indicates missing regions in the warped view associated with the warp operation, performing a repair operation on the missing regions by a first pre-trained artificial intelligence (AI) model based on text output from a second pre-trained AI model, wherein the text output is a response to one or more cue words, and repeating the warp and repair operations to obtain multiple generated views.

[0109] Example 22 includes an apparatus that contains means for performing the method of Example 21.

[0110] While example dimensions / models / values / ranges may have been given, the embodiments are not limited thereto. As manufacturing technologies (e.g., photolithography) mature, smaller devices are expected to be manufactured. Furthermore, to simplify the description and discussion, and to avoid obscuring certain aspects of the embodiments, well-known power / ground connections for IC chips and other components may or may not be shown in the figures. Additionally, to avoid obscuring the embodiments, arrangements may be shown in block diagram form, taking into account the fact that the details of implementations of such block diagram arrangements are highly dependent on the computing system in which the embodiments will be implemented; that is, these details should be entirely within the scope of those skilled in the art. In the context of setting forth specific details (e.g., circuitry) to describe exemplary embodiments, those skilled in the art should understand that the embodiments can be practiced without these specific details or with changes to them. Therefore, this description should be considered illustrative rather than restrictive.

[0111] The term “coupling” is used herein to refer to any type of direct or indirect relationship between the components under discussion and may be applied to electrical, mechanical, fluid, optical, electromagnetic, electromechanical, or other connections. Furthermore, the terms “first,” “second,” etc., are used herein only to facilitate discussion and, unless otherwise stated, do not carry a specific temporal or chronological meaning.

[0112] As used in this application and claims, a list of items connected by the term "one or more of..." can refer to any combination of the listed terms. For example, the expression "one or more of A, B, or C" may mean A; B; C; A and B; A and C; B and C; or A, B, and C.

[0113] Those skilled in the art will understand from the foregoing description that the broad techniques of the embodiments can be implemented in various forms. Therefore, while embodiments have been described in conjunction with specific examples, the true scope of the embodiments should not be so limited, as other modifications will become apparent to those skilled in the art upon studying the drawings, specification, and appended claims.

Claims

1. A performance-enhanced computing system, comprising: Network controller; A processor coupled to the network controller; as well as A memory coupled to the processor, the memory including a plurality of executable program instructions, which, when executed by the processor, cause the processor to: A warp operation is performed on the input image to obtain a warped view and a mask, wherein the mask indicates missing regions in the warped view associated with the warp operation. The missing region is repaired by a first pre-trained artificial intelligence (AI) model based on text output from a second pre-trained AI model, wherein the text output is a response to one or more prompt words, and Repeat the deformation and repair operations to obtain multiple generated views.

2. The computing system of claim 1, wherein when the plurality of executable program instructions are executed, the processor further causes the processor to merge the plurality of generated views into a 360° panoramic view.

3. The computing system of claim 2, wherein when the plurality of executable program instructions are executed, the processor is further caused to generate one or more of a video or a three-dimensional point cloud based on the 360° panoramic view.

4. The computing system according to any one of claims 1 to 3, wherein when the plurality of executable program instructions are executed, the processor is further caused to generate the one or more cue words at least in part based on a description of the input image from a third pre-trained AI model, wherein the one or more cue words are used to apply distinctions between scene layout regions, and wherein the one or more cue words include negative cue words and positive cue words.

5. The computing system according to any one of claims 1 to 3, wherein when the plurality of executable program instructions are executed, the processor further applies super-resolution to the output of the repair operation.

6. At least one computer-readable storage medium containing a plurality of executable program instructions, which, when executed by a computing system, cause the computing system to: A warp operation is performed on the input image to obtain a warped view and a mask, wherein the mask indicates missing regions in the warped view associated with the warp operation. The missing region is repaired by a first pre-trained artificial intelligence (AI) model based on text output from a second pre-trained AI model, wherein the text output is a response to one or more prompt words, and Repeat the deformation and repair operations to obtain multiple generated views.

7. The at least one computer-readable storage medium of claim 6, wherein when the plurality of executable program instructions are executed, the computing system further causes the plurality of generated views to merge into a 360° panoramic view.

8. The at least one computer-readable storage medium of claim 7, wherein when the plurality of executable program instructions are executed, the computing system further causes the computing system to generate one or more of a video or a three-dimensional point cloud based on the 360° panoramic view.

9. The at least one computer-readable storage medium of claim 6, wherein when the plurality of executable program instructions are executed, the computing system is further caused to generate the one or more prompt words at least in part based on a description of the input image from a third pre-trained AI model, wherein the one or more prompt words are used to apply distinctions between scene layout regions, and wherein the one or more prompt words include negative prompt words and positive prompt words.

10. The at least one computer-readable storage medium of claim 6, wherein when the plurality of executable program instructions are executed, the computing system further applies super-resolution to the output of the repair operation.

11. The at least one computer-readable storage medium according to any one of claims 6 to 10, wherein when the plurality of executable program instructions are executed, the computing system is further caused to generate the input image based on a pre-trained diffusion model and one or more of text input, sketch input, or synthetic image.

12. The at least one computer-readable storage medium according to any one of claims 6 to 10, wherein the input image is associated with a camera output.

13. A semiconductor device comprising: One or more substrates; as well as Logic coupled to the one or more substrates, wherein the logic is at least partially implemented in one or more configurable or functionally fixed hardware, the logic being used to: A warp operation is performed on the input image to obtain a warped view and a mask, wherein the mask indicates a missing region in the warped view associated with the warp operation; The missing region is repaired by a first pre-trained artificial intelligence (AI) model based on text output from a second pre-trained AI model, wherein the text output is a response to one or more prompt words, and Repeat the deformation and repair operations to obtain multiple generated views.

14. The semiconductor device of claim 13, wherein the logic is further configured to fuse the plurality of generated views into a 360° panoramic view.

15. The semiconductor device of claim 14, wherein the logic is further configured to generate one or more of a video or a three-dimensional point cloud based on the 360° panoramic view.

16. The semiconductor device of claim 13, wherein the logic is configured to generate the one or more cue words at least in part based on a description of the input image from a third pre-trained AI model, wherein the one or more cue words are configured to apply distinctions between scene layout regions, and wherein the one or more cue words include negative cue words and positive cue words.

17. The semiconductor device of claim 13, wherein the logic is further configured to apply super-resolution to the output of the repair operation.

18. The semiconductor device according to any one of claims 13 to 17, wherein the logic is further configured to generate the input image based on a pre-trained diffusion model and one or more of text input, sketch input, or synthetic image.

19. The semiconductor device according to any one of claims 13 to 17, wherein the input image is associated with a camera output.

20. The semiconductor device according to any one of claims 13 to 17, wherein the logic coupled to the one or more substrates comprises a transistor region disposed within the one or more substrates.

Images