An image extension method, system, device and medium

Abstract

The application relates to an image expansion method, system, device and medium, and belongs to the technical field of image processing, wherein the method comprises the following steps: extracting a structure line in a to-be-expanded image, and determining a vanishing point of the structure line in an expansion area according to a slope of the structure line; determining an extension line of the structure line in the expansion area according to the vanishing point; and expanding the to-be-expanded image according to the extension line through a trained diffusion model to obtain an expanded image. The application can improve the line connection accuracy and the correctness of the perspective field continuation during image expansion.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image expansion method, system, device and medium. Background Technology

[0002] Image outpainting is a technique that uses intelligent algorithms to generate new content around a given image, thereby enlarging the image size or supplementing missing parts. This technology aims to ensure that the expanded image maintains visual coherence and plausibility, seamlessly integrating with the original image to provide users with more complete and richer visual information.

[0003] Currently, image augmentation techniques based on diffusion probability models, such as the StableDiffusion model, have become the mainstream technology in the field of image augmentation. Existing image augmentation techniques can generally be divided into two categories: one is texture synthesis methods based on traditional computer vision, which analyze the texture features of an image and generate similar textures in the augmented area to achieve image augmentation; the other is mask redrawing methods based on general large models, which use general large models to redraw the masked parts of the image to complete image augmentation.

[0004] However, existing image expansion technologies have significant shortcomings in specific fields such as architectural design, engineering drawing, and professional renderings. General-purpose models (such as SD 1.5 / XL) lack the necessary knowledge and understanding of building codes, perspective logic, and engineering lines, making it difficult to meet the needs of these specialized fields when expanding images. For example, when expanding a half-section of a building, a general-purpose model may fail to accurately grasp the building's structural features, incorrectly extending a "beam-column structure" into "trees" or "irregular geometric shapes," creating a "structural illusion." This results in an expanded image that does not meet architectural design requirements and cannot be applied to real-world professional scenarios. Summary of the Invention

[0005] In view of this, it is necessary to provide an image extension method, system, device, and medium to solve the problem of structural illusion that easily occurs when using a general model to extend images in the prior art.

[0006] To address the aforementioned problems, in a first aspect, the present invention provides an image augmentation method, comprising: Extract the structural lines from the image to be expanded, and determine the vanishing point of the structural lines in the expanded region based on the slope of the structural lines; Based on the vanishing point, determine the extension line of the structural line in the extended region; The image to be expanded is expanded using the trained diffusion model based on the extension line to obtain the expanded image.

[0007] In one possible implementation, extracting structural lines from the image to be expanded includes: Determine the first edge image of the adjacent expansion region in the image to be expanded; Extract the structural lines pointing to the extended region from the first edge image.

[0008] In one possible implementation, the step of expanding the image to be expanded according to the extension line using a trained diffusion model to obtain an expanded image includes: Determine the weight of the extension line; The image to be expanded is expanded using a trained diffusion model based on the extension lines and their weights to obtain an expanded image.

[0009] In one possible implementation, the step of expanding the image to be expanded based on the image of the extension line using a trained diffusion model to obtain an expanded image includes: Determine the second edge image of the image to be expanded that is adjacent to the expanded region; The spectrum is obtained by performing a Fourier transform on the second edge image; The target frequency component characterizing the texture details of the second edge image is extracted from the spectrogram, and the target frequency component is injected into Gaussian noise in the form of additive noise to obtain the initial noise; The trained diffusion model is used to expand the image to be expanded based on the extension line and the initial noise, resulting in an expanded image.

[0010] In one possible implementation, the step of expanding the image to be expanded using a trained diffusion model based on the extension line and the initial noise to obtain an expanded image includes: A latent space feature map is generated based on the initial noise, the image to be expanded, and the expanded region; Using the trained diffusion model, the latent space feature map is denoised in multiple steps according to the extension line, and after each denoising step, the overlapping area of ​​the image to be expanded and the expanded area in the latent space feature map is backfilled. The latent space feature map after multi-step denoising is decoded to obtain the extended image.

[0011] In one possible implementation, the step of expanding the image to be expanded according to the extension line using a trained diffusion model to obtain an expanded image includes: The first expansion prompt word for the image to be expanded is generated using a visual language model; Determine the style identifier of the image to be expanded; The first extended prompt word is combined with the style identifier to obtain the second extended prompt word; The image to be expanded is expanded using the trained diffusion model based on the second expansion prompt and the extension line to obtain an expanded image; wherein the diffusion model is trained using sample images of a specific style marked with style identifiers.

[0012] In one possible implementation, the image to be expanded contains a building, and the sample image includes architectural design drawings, construction drawings, and renderings.

[0013] Secondly, the present invention also provides an image expansion system, comprising: The vanishing point determination module is used to extract the structural lines in the image to be expanded and determine the vanishing point of the structural lines in the expansion region based on the slope of the structural lines. An extension line determination module is used to determine the extension line of the structural line in the extended region based on the vanishing point; The image expansion module is used to expand the image to be expanded according to the extension line using a trained diffusion model to obtain an expanded image.

[0014] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein the memory is used to store a program; the processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the image expansion method described in any of the preceding claims.

[0015] Fourthly, the present invention also provides a computer-readable storage medium for storing a computer-readable program, wherein the program or instructions, when executed by a processor, are capable of implementing the steps in any of the above-described image expansion methods.

[0016] The beneficial effects of this invention are: Before expanding the image using a diffusion model, this invention first extracts the structural lines in the image to be expanded. Based on the slope of the structural lines, the vanishing point of the structural lines in the expansion region is determined. Based on the vanishing point, the extension line of the structural lines in the expansion region is determined. Then, the image including the extension line is input into the diffusion model, guiding the diffusion model to expand the image based on the extension line. This can improve the accuracy of line continuity during image expansion and avoid the problem of bending or breaking of lines of objects in the image during expansion. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating an embodiment of the image expansion method provided by the present invention; Figure 2 For the present invention Figure 1 A flowchart illustrating an embodiment of S103; Figure 3 For the present invention Figure 1 A flowchart illustrating another embodiment of S103; Figure 4 This is a schematic diagram of the complete process of an image expansion method provided by the present invention; Figure 5 This is a flowchart illustrating another embodiment of an image expansion method provided by the present invention; Figure 6 A schematic diagram of the structure of an embodiment of the image extension system provided by the present invention; Figure 7 A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] In the description of the embodiments of this invention, unless otherwise stated, "a plurality of" means two or more. The terms "first," "second," etc., used in the embodiments of this invention are used to distinguish similar objects, and are not used to describe a specific order or sequence, nor to indicate or imply their relative importance or implicitly specify the number of indicated technical features. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, and the number of objects is not limited; for example, a first object can be one or more.

[0021] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0022] Reference Figure 1 The diagram illustrates a flowchart of an embodiment of the image augmentation method provided by the present invention, the method comprising: S101, extract the structure lines in the image to be expanded, and determine the vanishing point of the structure lines in the expanded region based on the slope of the structure lines.

[0023] The Deep Hough Transform (DHT) or Line Segment Detector (LSD) can be used to extract structural lines, such as floor slab lines and window ridge lines, from the image to be expanded. Then, the slope of the structural lines is calculated, and the vanishing point of the structural lines in the expanded region is obtained by fitting the slope.

[0024] An extended area refers to a newly generated region of the image outside the image to be extended, used to extend the image content. This region can be specified by the user. For example, if the resolution of the image to be extended is 1024×1024, and the user specifies an extension of 512 pixels to the right, the canvas will be expanded to 1536×1024, and the extended area will be the 512×1024 area on the right side of the canvas.

[0025] S102, Based on the vanishing point, determine the extension line of the structure line in the extended region.

[0026] Connecting the structure lines to the vanishing points yields extensions of the structure lines in the extended region. This step is equivalent to drawing a "virtual structural skeleton" in the extended region. This "virtual structural skeleton" serves as input to the diffusion model in the next step, constraining subsequent pixels generated by the diffusion model to adhere to these extensions. This ensures the continuity of the generated extended lines with the original image lines and guarantees the correctness of the perspective field of the newly generated object.

[0027] S103 uses a trained diffusion model to expand the image to be expanded based on the extension line, thus obtaining the expanded image.

[0028] The general diffusion model can be fine-tuned and trained beforehand to better suit the image expansion task. Then, in actual use, the image including the extension line and the encoded image to be expanded can be input into the ControlNet and the backbone network (U-Net) of the trained diffusion model, respectively, so that the diffusion model can expand the image to be expanded and obtain the final expanded image.

[0029] The image expansion method provided in this embodiment can be applied to an image expansion software system, which can run on a terminal device. The terminal device can be a tablet computer, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), mobile phone, etc. This embodiment does not impose any restrictions on the specific type of terminal device.

[0030] In summary, this embodiment first extracts the structural lines in the image to be expanded before expanding it using the diffusion model. The vanishing point of the structural lines in the expansion region is determined based on the slope of the structural lines. Based on the vanishing point, the extension line of the structural lines in the expansion region is determined. Then, the image of the extension line is input into the diffusion model, guiding the diffusion model to expand the image to be expanded based on the extension line. This can improve the accuracy of line continuity during image expansion and avoid the problem of bending or breaking of lines of objects in the image during expansion.

[0031] In some embodiments of the present invention, the step of extracting structural lines from the image to be expanded includes: Determine the first edge image of the adjacent expansion region in the image to be expanded; Extract the structural lines pointing to the extended region from the first edge image.

[0032] This embodiment extracts structural lines only from the first edge image, which narrows the extraction range and reduces the extraction of unnecessary structural lines (i.e., structural lines that will not extend into the extended region). Furthermore, extracting only structural lines pointing towards the extended region from the first edge image further reduces the extraction of unnecessary structural lines.

[0033] In some embodiments of the present invention, S103 includes: Determine the weight of the extension line; The trained diffusion model expands the image to be expanded according to the extension lines and their weights, resulting in an expanded image.

[0034] To address the issue of inconsistent image quality between old and new areas, and to ensure that expanded areas possess the same quality as the original areas from the initial generation stage... Figure 1 Consistent texture graininess, as described in some embodiments of the present invention, such as Figure 2 As shown, S103 includes: S201, determine the second edge image of the adjacent expansion region in the image to be expanded.

[0035] The second edge image and the first edge region image can be the same size or different; this embodiment does not impose specific restrictions on this.

[0036] S202, Perform Fourier transform on the second edge image to obtain the spectrum.

[0037] S203, extract the target frequency component representing the texture details of the second edge image from the spectrogram, and inject the target frequency component into Gaussian noise in the form of additive noise to obtain the initial noise.

[0038] First, Gaussian noise is generated, and then the target frequency component is added to the Gaussian noise as additional noise to obtain the initial noise.

[0039] S204 uses a trained diffusion model to expand the image to be expanded based on the extension line and initial noise, thus obtaining the expanded image.

[0040] This embodiment introduces the frequency characteristics of the original image edges into the initial noise, which makes the connection between the old and new regions in the expanded image more unified and harmonious, and the texture granularity consistent.

[0041] In some embodiments of the present invention, S204 includes: Generate a latent space feature map based on the initial noise, the image to be expanded, and the expanded region; Using a trained diffusion model, the latent space feature map is denoised in multiple steps based on the extension line. After each denoising step, the overlapping area between the image to be expanded and the expanded area in the latent space feature map is re-painted. The latent space feature map after multi-step denoising is decoded to obtain the extended image.

[0042] Specifically, the process begins by encoding the image to be expanded and the expanded region using an encoder, such as a Variational Autoencoder (VAE), to generate an initial latent space feature map. This map is then noise-added based on the initial noise, resulting in a noisy latent space feature map. This noisy latent space feature map is then used as input to a diffusion model, which performs multi-step denoising. After each denoising step, latent space backfilling is performed on the overlapping areas (the seams between the two regions) of the image to be expanded and the expanded region in the latent space feature map, until the denoising process is complete. Finally, the output of the diffusion model is decoded using the VAE to obtain the final expanded image. In this expanded image, the glass curtain wall texture and floor lines of the original image extend perfectly into the new region, and the direction of light and shadow is completely consistent with the original image.

[0043] Latent space backfilling refers to a process where, during local redrawing (such as face retouching, object replacement, or image expansion), the predicted results are not directly used to cover the entire area. Instead, the predicted results are mixed with the latent features of the original image at a certain ratio (controlled by the mask and alpha value), and the mixed result is used as the starting point for the next step of denoising. Its expression is:

[0044] In the formula, t represents the number of denoising steps; This represents the prediction results of U-Net; This represents the latent vector of the original image after noise has been added; Represents the mixing coefficient, which is between 0 and 1. When, it means that the model prediction results are used for full coverage. When, it means that the original image area remains unchanged.

[0045] In summary, this embodiment further ensures pixel-level fusion of old and new areas through latent space backfilling technology.

[0046] To address the design style drift problem that occurs during image expansion, in some embodiments of the present invention, such as... Figure 3 As shown, S103 includes: S301, Generate the first expansion prompt word for the image to be expanded using a visual language model.

[0047] S302, Determine the style identifier for the image to be expanded.

[0048] S303, combine the first extended prompt word with the style identifier to obtain the second extended prompt word.

[0049] S304, using a trained diffusion model, expands the image to be expanded based on the second expansion prompt and the extension line to obtain the expanded image; wherein, the diffusion model is trained on sample images of a specific style marked with style identifiers.

[0050] This embodiment, through fine-tuning the diffusion model, enables the model to learn the stylistic features of the image represented by the style identifier (such as specific color temperature, line thickness, unique material, etc.). Then, by adding the style identifier to the prompt, the diffusion model can be instructed to expand the image according to the corresponding style, ensuring that the style of the new and old regions is consistent, and avoiding the copyright or style mixing risks that may be caused by directly using external models.

[0051] In some embodiments of the present invention, S103 includes: The trained diffusion model expands the image to be expanded based on the second expansion prompt, the extension line, and the initial noise, thus obtaining the expanded image.

[0052] The generation of initial noise and the generation of the second extended prompt word are as described in the above embodiment. This embodiment combines the second extended prompt word, the extension line, and the initial noise to expand the image to be expanded.

[0053] In some embodiments of the present invention, the image to be expanded includes buildings, and the sample image includes architectural design drawings, construction drawings, and renderings.

[0054] A building is a structure or place constructed of building materials, which has a certain space and is used by people for production, living or other activities, such as houses and bridges.

[0055] When expanding strong structural images containing buildings, structural illusions are more likely to occur. Therefore, this embodiment can train the diffusion model for expanding such images and optimize the expansion process as described above.

[0056] Reference Figure 4 This diagram illustrates the complete process of an image augmentation method provided by the present invention. The core of this embodiment lies in "private model fine-tuning + structure line prior guidance + frequency domain consistency constraint". The specific process includes: S1, Construct a dedicated diffusion model.

[0057] Based on open-source modeling (such as Stable Diffusion XL), full fine-tuning or training with high-rank LoRA (Low-Rank Adaptation) is performed using high-quality architectural drawings, renderings, and construction drawings datasets. "Style identifiers" (token embeddings) are embedded in the model, for example...<CITIC_Style> This makes the model highly sensitive to unique design elements.

[0058] S2, based on the prediction of structural lines in the perspective field.

[0059] Based on the vanishing point, construct the perspective field. Based on the perspective field, generate a "virtual structural skeleton" in the extended region.

[0060] S3, constructs latent space noise that is mixed in the frequency domain.

[0061] Perform a Fourier transform (FFT) on the edge regions of the original image to extract their high-frequency components (texture details). Inject these high-frequency components as additive noise into the initial Gaussian noise of the extended region.

[0062] S4, denoising generation based on private semantic alignment.

[0063] Multimodal cue word construction: Utilizing a visual language model to identify the content of the original image, and combining it with...<CITIC_Style> Identifier, generate Prompt.

[0064] Structural constraint denoising: The "virtual structural skeleton" generated by S2 is used as the input condition (Conditioning) of the diffusion model ControlNet, and the U-Net denoising process of the diffusion model is guided in parallel with the text Prompt.

[0065] Overlapping area consistency verification: At each step of denoising (Time step t), latent re-painting is performed on the overlapping area between the original image and the expanded image.

[0066] This embodiment has the following beneficial effects: 1. Improved structural accuracy: When expanding images of strong structures such as high-rise buildings and bridges, the accuracy of line continuity is significantly improved compared to general models, avoiding the "crooked building" phenomenon.

[0067] 2. Seamless integration: Through frequency domain noise and latent space backfilling technology, the splicing gaps that are perceptible to the human eye are eliminated.

[0068] 3. Copyright and style security: Ensure that the extended content conforms to the design style required by users, and avoid the risk of copyright or style mixing that may result from using external models.

[0069] Reference Figure 5 The diagram illustrates a flowchart of another embodiment of an image augmentation method provided by the present invention, the flowchart of which includes: 1. Input and Preprocessing: The system receives the original image. The resolution is 1024×1024. The user specified an extension of 512 pixels to the right.

[0070] The system expands the canvas to 1536×1024 and fills the expanded area with pure black.

[0071] 2. Prior extraction of structural lines (S2): Use the trained LSD model for recognition Horizontal lines within 256 pixels of the right edge (such as interlayer waistlines).

[0072] Calculate the slopes of these lines and fit the vanishing point on the right. .

[0073] according to An extended white line drawing (Skeleton Map) is created within the black extended area to serve as a structural guide diagram.

[0074] 3. Noise initialization (S3): A 64×1024 slice is cropped from the right edge of the original image, and an FFT transform is performed to obtain the amplitude spectrum. .

[0075] Generate standard Gaussian noise .

[0076] right Perform frequency domain filtering to approximate its amplitude spectrum. The final initial noise is obtained. .

[0077] 4. Model Inference (S4): Load the private model weights of the large model CITIC_Diffusion_v2.ckpt.

[0078] Input suggestions: "High-rise office building, glass curtain wall, photorealistic"<CITIC_Style> ".

[0079] Simultaneously input the Skeleton Map obtained from S2 into the ControlNet module (with a weight set to 0.8).

[0080] Perform 50 steps of DDIM sampling. After each denoising step, force the Latent of the original image area to be replaced back to the original Latent (after adding noise), retaining only the soft transition blending of 16 pixels at the boundary.

[0081] 5. Output: The final image is obtained through VAE decoding. At this point, the glass curtain wall texture and floor lines of the original image extend perfectly into the new area, and the direction of light and shadow is completely consistent with the original image.

[0082] Reference Figure 6 The diagram illustrates a structural schematic of an embodiment of the image extension system provided by the present invention. The system 600 includes: The vanishing point determination module 601 is used to extract the structure lines in the image to be expanded and determine the vanishing point of the structure lines in the expansion area based on the slope of the structure lines. Extension line determination module 602 is used to determine the extension line of the structure line in the extended region based on the vanishing point; The image expansion module 603 is used to expand the image to be expanded according to the extension line using a trained diffusion model to obtain an expanded image.

[0083] It should be noted that the implementation principles or processes of the above modules can be referred to the aforementioned embodiments of the image expansion method, and will not be elaborated here.

[0084] Reference Figure 7 The present invention illustrates an electronic device 700. The electronic device 700 includes a processor 701, a memory 702, and a display 703. Figure 7 Only some components of the electronic device 700 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0085] In some embodiments, processor 701 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 702 or process data, such as the image expansion method of the present invention.

[0086] In some embodiments, processor 701 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 701 may be local or remote. In some embodiments, processor 701 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, internal cloud, multi-cloud, etc., or any combination thereof.

[0087] In some embodiments, memory 702 may be an internal storage unit of electronic device 700, such as a hard disk or memory of electronic device 700. In other embodiments, memory 702 may also be an external storage device of electronic device 700, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 700.

[0088] Furthermore, the memory 702 may include both internal storage units of the electronic device 700 and external storage devices. The memory 702 is used to store application software and various types of data installed on the electronic device 700.

[0089] In some embodiments, display 703 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 703 is used to display information from electronic device 700 and to display a visual user interface. Components 701-703 of electronic device 700 communicate with each other via a system bus.

[0090] In one embodiment, when processor 701 executes the image expansion program in memory 702, the following steps can be performed: Extract the structure lines in the image to be expanded, and determine the vanishing point of the structure lines in the expanded region based on the slope of the structure lines; Based on the vanishing point, determine the extension line of the structural line in the extended region; The trained diffusion model is used to expand the image to be expanded according to the extension line, thus obtaining the expanded image.

[0091] It should be understood that when the processor 701 executes the image expansion program in the memory 702, in addition to the functions mentioned above, it can also perform other functions, as detailed in the description of the corresponding method embodiments above.

[0092] Furthermore, this embodiment of the invention does not specifically limit the type of electronic device 700 mentioned. Electronic device 700 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the invention, electronic device 700 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0093] In one embodiment, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by the processor, implements the steps of any of the image expansion methods described above.

[0094] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0095] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. An image expansion method, characterized in that, include: Extract the structural lines from the image to be expanded, and determine the vanishing point of the structural lines in the expanded region based on the slope of the structural lines; Based on the vanishing point, determine the extension line of the structural line in the extended region; The image to be expanded is expanded using the trained diffusion model based on the extension line to obtain the expanded image.

2. The image expansion method according to claim 1, characterized in that, The extraction of structural lines from the image to be expanded includes: Determine the first edge image of the adjacent expansion region in the image to be expanded; Extract the structural lines pointing to the extended region from the first edge image.

3. The image expansion method according to claim 1, characterized in that, The process of expanding the image to be expanded using a trained diffusion model based on the extension line to obtain an expanded image includes: Determine the weight of the extension line; The image to be expanded is expanded using a trained diffusion model based on the extension lines and their weights to obtain an expanded image.

4. The image expansion method according to claim 1, characterized in that, The process of expanding the image to be expanded using a trained diffusion model based on the image of the extension line to obtain an expanded image includes: Determine the second edge image of the image to be expanded that is adjacent to the expanded region; The spectrum is obtained by performing a Fourier transform on the second edge image; The target frequency component characterizing the texture details of the second edge image is extracted from the spectrogram, and the target frequency component is injected into Gaussian noise in the form of additive noise to obtain the initial noise; The trained diffusion model is used to expand the image to be expanded based on the extension line and the initial noise, resulting in an expanded image.

5. The image expansion method according to claim 4, characterized in that, The process of expanding the image to be expanded using a trained diffusion model based on the extension line and the initial noise to obtain an expanded image includes: A latent space feature map is generated based on the initial noise, the image to be expanded, and the expanded region; Using the trained diffusion model, the latent space feature map is denoised in multiple steps according to the extension line, and after each denoising step, the overlapping area of ​​the image to be expanded and the expanded area in the latent space feature map is backfilled. The latent space feature map after multi-step denoising is decoded to obtain the extended image.

6. The image expansion method according to claim 1, characterized in that, The process of expanding the image to be expanded using a trained diffusion model based on the extension line to obtain an expanded image includes: The first expansion prompt word for the image to be expanded is generated using a visual language model; Determine the style identifier of the image to be expanded; The first extended prompt word is combined with the style identifier to obtain the second extended prompt word; The image to be expanded is expanded using the trained diffusion model based on the second expansion prompt and the extension line to obtain an expanded image; wherein the diffusion model is trained using sample images of a specific style marked with style identifiers.

7. The image expansion method according to claim 5, characterized in that, The image to be expanded contains buildings, and the sample image includes architectural design drawings, construction drawings, and renderings.

8. An image augmentation system, characterized in that, include: The vanishing point determination module is used to extract the structural lines in the image to be expanded and determine the vanishing point of the structural lines in the expansion region based on the slope of the structural lines. An extension line determination module is used to determine the extension line of the structural line in the extended region based on the vanishing point; The image expansion module is used to expand the image to be expanded according to the extension line using a trained diffusion model to obtain an expanded image.

9. An electronic device, characterized in that, Including memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps of the image expansion method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions that, when executed by a processor, can implement the steps of the image expansion method according to any one of claims 1 to 7.

Images