Diffusion model-based image stitching rectangularization method, system, device and medium
By using optical flow estimation based on a diffusion model and image-to-image diffusion model repair, the problem of non-rectangular boundaries in image stitching was solved, improving image quality and visual effects.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UNIV OF ELECTRONICS SCI & TECH OF CHINA
- Filing Date
- 2024-02-02
- Publication Date
- 2026-05-29
Smart Images
Figure CN118014898B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computational graphics and computer vision, and in particular to a method, system, device, and medium for rectangularizing stitched images based on a diffusion model. Specifically, it relates to using a diffusion model in a deep neural network to rectangularize the boundaries of a stitched image. Background Technology
[0002] Image stitching is a technique in which multiple overlapping images of a scene are stitched together to generate an image with a wide field of view (FOV) and high resolution. These methods typically employ homography transformation for global alignment or mesh deformation for local alignment of overlapping areas. Because of the different shooting perspectives, image stitching often results in non-rectangular boundaries, which can lead to unattractive images and affect downstream image processing; therefore, processing of these boundaries is usually necessary.
[0003] To address the issue of non-rectangular boundaries, current solutions include:
[0004] Cropping: After taking a rectangular area that completely contains the image content, cropping the content outside the area will result in a reduction of the image content;
[0005] Filling: Using existing generative diffusion models to generate content like blank areas in the stitched image results in low relevance between the generated content and the actual content;
[0006] Distortion: By detecting specific features in an image, a distorted grid is estimated and the image is transformed. In this way, significant image structures can be preserved, while less important areas are stretched or squeezed to achieve rectangular boundary shapes. However, this method may distort nonlinear features and cause artifacts in the generated image.
[0007] Existing distortion-based methods all rely on mesh transformations, which often leads to unpleasant misalignment and distortion at mesh boundaries. Furthermore, simply transforming the image based on the mesh can result in irreparable detail blurring in areas of significant motion. To overcome these problems and obtain better rectangularized images, this invention provides a method, system, device, and medium for rectangularizing stitched images based on a diffusion model. This method employs a diffusion model to perform finer-grained transformations on the image by estimating optical flow, and then uses an image-to-image diffusion model to restore details in the transformed image. Summary of the Invention
[0008] This invention provides a method, system, device, and medium for rectangularizing stitched images based on a diffusion model. It performs finer-grained transformations on images by estimating optical flow and then uses an image-to-image diffusion model to restore details in the transformed images.
[0009] This invention is achieved through the following technical solution: a method for rectangularizing stitched images based on a diffusion model, comprising the following steps:
[0010] Step S1: Collect a dataset, which includes a test set and a training set;
[0011] Step S2: Establish a diffusion model, which includes a motion diffusion model module and a content diffusion model module;
[0012] Step S3: Use the loss function and the training set to train the motion diffusion model module and the content diffusion model module in the diffusion model to generate a trained diffusion model;
[0013] Step S4: Collect the stitched images that need to be corrected and input them into the trained diffusion model. Then, sequentially enter the motion diffusion model module and the content diffusion model module to generate the repaired rectangular image.
[0014] To better implement the present invention, step S1 further includes:
[0015] The training set includes stitched images, masks, and real rectangular images, while the test set includes stitched images and masks.
[0016] To better realize the present invention, step S2 further includes:
[0017] The motion diffusion model module and the content diffusion model module are connected sequentially from front to back.
[0018] To better implement the present invention, the method for training the motion diffusion model module in step S3 further includes:
[0019] The training set provides a stitched image and a motion stream. Gaussian noise related to a random time t is added to obtain a noisy optical flow at time t. The goal is to generate a motion stream that is the same as the given motion stream and to make the image obtained after applying the motion stream to the stitched image consistent with the real image.
[0020] During training, the inputs are a stitched image, a mask, and a real optical flow field obtained through a grid in the dataset. The stitched image and mask are used as conditions to sample the predicted optical flow field from Gaussian noise, with the expectation that the stitched image can be restored into a normal rectangular image after being processed by the optical flow field predicted from the data.
[0021] To better implement the present invention, the method for training the content diffusion model module in step S3 further includes:
[0022] Given a rectangularly corrected intermediate state image and a real target image, the diffusion model is trained to predict the real image at a given time step by adding noise to the rectangularly corrected intermediate state image at random time steps. The goal is for the generated real image to be consistent with the given real image.
[0023] The confidence matrix is obtained through the intermediate results generated by the motion diffusion model module. The image and mask are used as conditions in the sampling process, and the output is a rectangular image with restored details.
[0024] To better realize the present invention, further, 6. the method for generating the model after entering the motion diffusion model module in step S4 includes:
[0025] An intermediate state image with rectangular correction is generated by motion flow, along with a confidence matrix of the same size as the image, where each pixel has a value of [0,1].
[0026] To better implement this invention, the method further includes the following steps: the method for generating the content diffusion model after entering the content diffusion model module in step S4 includes:
[0027] The intermediate state image that has undergone rectangular correction and the confidence matrix generated by the motion diffusion model module are used as conditions to input into the diffusion model. The randomly generated Gaussian noise is gradually denoised through U-net to finally obtain the target image.
[0028] This invention also provides a system for reclamping stitched images based on a diffusion model, comprising a dataset acquisition unit, a model building unit, a model training unit, and an image restoration generation unit, wherein:
[0029] A dataset acquisition unit is used to acquire datasets, which include test sets and training sets.
[0030] A model building unit is used to build a diffusion model, which includes a motion diffusion model module and a content diffusion model module.
[0031] The model training unit is used to train the motion diffusion model module and the content diffusion model module in the diffusion model using the loss function and the training set, and to generate a trained diffusion model.
[0032] The image repair generation unit is used to collect the stitched images that need to be corrected and input them into the trained diffusion model. The images then enter the motion diffusion model module and the content diffusion model module in sequence to generate the repaired rectangular image.
[0033] The present invention also provides an electronic device comprising a processor and a memory; the processor includes the diffusion-based image rectangularization system described in the second aspect above.
[0034] The present invention also provides a computer-readable storage medium comprising instructions that, when executed on an electronic device described in the third aspect, cause the electronic device to perform the method described in the first aspect.
[0035] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0036] (1) This invention provides a method, system, device and medium for rectangularization of stitched images based on diffusion model. It introduces a novel diffusion-based learning framework, which combines a motion diffusion model to generate a motion field, effectively transitioning the irregular boundaries of stitched images to the intermediate state of rectangularization correction. Then, a content diffusion model is used to further repair the image details.
[0037] (2) This invention provides a method, system, device and medium for rectangularizing spliced images based on a diffusion model. The proposed innovative method ensures the accuracy of geometric boundaries and overall visual appeal, and surpasses the performance of previous methods in both quantitative and qualitative aspects. Attached Figure Description
[0038] The present invention will be further described in conjunction with the following drawings and embodiments. All inventive concepts of the present invention should be considered as disclosed content and within the scope of protection of the present invention.
[0039] Figure 1 A schematic diagram illustrating a diffusion-based method and system for rectangularizing stitched images, as well as a conventionally used distortion-based method in a device or medium, provided in this application embodiment.
[0040] Figure 2 A schematic diagram illustrating the reasoning process of a diffusion model in a method, system, device, and medium for rectangularizing stitched images based on a diffusion model, provided in an embodiment of this application.
[0041] Figure 3 A schematic diagram illustrating the training process of a diffusion model in a method, system, device, and medium for rectangularizing stitched images based on a diffusion model, as provided in this application embodiment;
[0042] Figure 4 This document presents an example of a method and system for rectangularizing stitched images based on a diffusion model, along with an illustration of the effect of the diffusion model in the medium. Detailed Implementation
[0043] To more clearly illustrate the technical solutions of the embodiments of the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments, and therefore should not be regarded as a limitation on the scope of protection. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0044] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "set up," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0045] Example 1:
[0046] This embodiment presents a method, system, device, and medium for rectangularizing stitched images based on a diffusion model, such as... Figure 1 The diagram shown is a schematic of the previously used distortion-based method.
[0047] In the data acquisition and processing of this invention, publicly available industry datasets are used. The datasets include a test set and a training set. The training set includes: a stitched image S, a mask M, and a real rectangular image gt. The test set includes: the stitched image S and the mask M.
[0048] Example 2:
[0049] This embodiment is a further optimization based on Embodiment 1, such as... Figure 2 As shown, this invention specifically includes two modules: a motion diffusion model module and a content diffusion model module. Wherein M... S I S This is the stitched image and the mask (marking which parts of the image are blank; black indicates that there is no image content at the corresponding pixel position). The output of MDM is represented by θ1 and θ2, which correspond to the neural network diffusion models of MDM and CDM, respectively. The related content will be explained later.
[0050] The other parts of this embodiment are the same as those in Embodiment 1, so they will not be described again.
[0051] Example 3:
[0052] This embodiment is a further optimization based on the above embodiment 1 or 2, such as... Figures 2-3 As shown, the motion diffusion model module: This module's inference process accepts a stitched image (four channels in total: the red, green, and blue channels of the image, and a mask marking the actual position of the image) as input to the motion diffusion model module, and gradually removes randomly generated Gaussian noise. The module used for denoising is U-net (short for "U-shaped network," a deep learning diffusion model commonly used in image segmentation tasks. It combines a Convolutional Neural Network (CNN) and an encoder-decoder architecture, exhibiting good performance). Finally, the original image pixel position corresponding to each pixel of the target image is obtained. A rectangularized corrected intermediate state image and a confidence matrix of the same size as the target image are generated through motion flow, with each pixel taking a value of [0,1].
[0053] Content Diffusion Model Module: This module's inference process accepts the rectangularized corrected intermediate state image and the confidence matrix generated by the motion diffusion model module, using them as conditions input into the diffusion model. Randomly generated Gaussian noise is progressively denoised using U-net, finally yielding the target image. This module is responsible for restoring the details of the rectangularized corrected intermediate state image to obtain the final target image.
[0054] The other parts of this embodiment are the same as any one of the embodiments 1-2 above, so they will not be described again.
[0055] Example 4:
[0056] This embodiment is a further optimization based on any one of embodiments 1-3 above, such as... Figure 3 As shown, the training of the motion diffusion model module aims to enable the diffusion model to generate corresponding optical flow fields from the stitched image and its corresponding mask, thereby obtaining a rectangular image using the optical flow field. The training process is as follows: given the stitched image and the motion flow, Gaussian noise related to a random time t is added to obtain the noisy optical flow at time t; the goal is that the generated motion flow is the same as the given motion flow, and the image obtained after applying the motion flow to the stitched image is consistent with the real image. During the training process, the stitched image S, the mask M, and the real optical flow field F obtained through the mesh in the dataset are input. S and M are used as conditions, and the predicted optical flow field F′ is sampled from the Gaussian noise, with the expectation that the stitched image S can be restored into a normal rectangular image after being processed by F′. The loss function of the motion diffusion model module training process consists of two parts:
[0057]
[0058]
[0059] in The final optical flow is obtained from the diffusion model output, where x0 is the optical flow generated from the pre-generated mesh; I R These are real photos. See above.
[0060] The final loss function of the motion diffusion model module is a combination of the above two parts:
[0061]
[0062] Before entering the motion diffusion model module, the confidence matrix will be calculated using the results obtained from the motion diffusion model module. The calculation process is as follows:
[0063] Where ω0 is a given parameter, M0 is the normalized pixel movement distance (the L2 norm of each pixel value in the optical flow), and max is the maximum value of each pixel in the two matrices. See above.
[0064] like Figure 3 As shown, the training of the motion diffusion model module aims to correct detail distortions in images generated by the module, such as subtle white edges, curved straight objects, and discontinuous lines in continuous objects. The training process involves providing a rectangularly corrected intermediate image and a real target image. Noise is added to the rectangularly corrected intermediate image at random time steps to train the diffusion model to predict the real image at a given time step, with the goal of generating a real image consistent with the given real image. A confidence matrix is obtained from the intermediate results generated by the motion diffusion model module. In the sampling process, S and M are used as conditions, and the output is the rectangle image x′0 after detail restoration. The loss function for this process is l. cdm =||x′0-x0|| 2 .
[0065] Combining the results of the two modules, the final repaired rectangular image is:
[0066] The other parts of this embodiment are the same as any one of the embodiments 1-3 above, so they will not be described again.
[0067] Example 5:
[0068] This embodiment is a further optimization based on any one of embodiments 1-4 above, such as... Figure 2As shown, in the inference phase of the diffusion model, a stitched image I to be corrected and its corresponding mask M are input. The motion diffusion model module performs noise reduction on a time step sequence t∈{1, 2, ..., T} in descending order to obtain the optical flow field F. F is then used to modify the stitched image to obtain a rectangular image and a confidence matrix. I′ and W are then fed into the content diffusion model module and denoised using the same technique described above to obtain the final result I. result .
[0069] The other parts of this embodiment are the same as any one of the embodiments 1-4 above, so they will not be described again.
[0070] Example 6:
[0071] This embodiment is a further optimization based on any one of embodiments 1-5 above, such as... Figure 4 The diagram shows the effect of this invention. The first column is the input stitched image, the second column is the actual result, the third column is the result obtained by previous best work, and the fourth column is the result obtained in this study. The right-hand images (difference images) of the third and fourth columns show the difference between the image obtained by this method and the actual image, with brighter areas showing greater differences. It can be seen that the method proposed in this paper, compared to the methods proposed by previous best work, can obtain an image that is closer to the original image (the difference image is darker).
[0072] The present invention also provides a diffusion model-based image recombination system that matches the method, including a dataset acquisition unit, a model building unit, a model training unit, and an image restoration generation unit.
[0073] The present invention also provides an electronic device, which includes a processor and a memory; the processor includes the above-described diffusion model-based image rectangularization system.
[0074] Example 7:
[0075] The present invention also provides a computer-readable storage medium comprising instructions; when the instructions are executed on the electronic device described in the above embodiments, the electronic device causes the electronic device to perform the methods described in the above embodiments. Optionally, the computer-readable storage medium may be a memory.
[0076] The processor involved in the embodiments of this application can be a chip. For example, it can be a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a system-on-chip (SoC), a central processing unit (CPU), a network processor (NP), a digital signal processing circuit (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.
[0077] The memory involved in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0078] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0079] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0080] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0081] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms.
[0082] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located on one device or distributed across multiple devices. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0083] In addition, the functional modules in the various embodiments of this application can be integrated into one device, or each module can exist physically separately, or two or more modules can be integrated into one device.
[0084] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).
[0085] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications or equivalent changes made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.
Claims
1. A method for rectangularizing stitched images based on a diffusion model, characterized in that, Includes the following steps: Step S1: Collect a dataset, which includes a test set and a training set; Step S2: Establish a diffusion model, which includes a motion diffusion model module and a content diffusion model module; Step S3: Use the loss function and the training set to train the motion diffusion model module and the content diffusion model module in the diffusion model to generate a trained diffusion model; The method for training the motion diffusion model module includes: the training set is given by stitched images and motion streams, with random time intervals added. The relevant Gaussian noise is obtained at time The goal is to generate a motion flow that is identical to a given motion flow and to apply the motion flow to the stitched images so that the resulting image is consistent with the real image. During training, the stitched image, mask, and real optical flow field obtained through the grid in the dataset are input. The stitched image and mask are used as conditions to sample the predicted optical flow field from Gaussian noise, with the expectation that the stitched image can be restored into a normal rectangular image after being subjected to the optical flow field predicted by the data. The method for training the content diffusion model module includes: providing a rectangularly corrected intermediate state image and a real target image; adding noise to the rectangularly corrected intermediate state image at random time steps; training the diffusion model to predict the real image at a given time step; with the goal of generating a real image that is consistent with the given real image. The confidence matrix is obtained through the intermediate results generated by the motion diffusion model module. The image and mask are used as conditions in the sampling process, and the output is a rectangular image after the details are repaired. Step S4: Collect the stitched images that need to be corrected and input them into the trained diffusion model. Then, sequentially enter the motion diffusion model module and the content diffusion model module to generate the repaired rectangular image.
2. The method for rectangularizing stitched images based on a diffusion model according to claim 1, characterized in that, Step S1 includes: The training set includes stitched images, masks, and real rectangular images, while the test set includes stitched images and masks.
3. The method for rectangularizing stitched images based on a diffusion model according to claim 1, characterized in that, Step S2 includes: The motion diffusion model module and the content diffusion model module are connected sequentially from front to back.
4. The method for rectangularizing stitched images based on a diffusion model according to claim 1, characterized in that, The method for generating the model after entering the motion diffusion model module in step S4 includes: The intermediate state image with rectangular correction is generated by the motion flow, along with a confidence matrix of the same size as the image, where each pixel has a value of [0, 1].
5. The method for rectangularizing stitched images based on a diffusion model according to claim 1, characterized in that, The method for generating the content diffusion model module in step S4 includes: The intermediate state image that has undergone rectangular correction and the confidence matrix generated by the motion diffusion model module are used as conditions to input into the diffusion model. The randomly generated Gaussian noise is gradually denoised through U-net to finally obtain the target image.
6. A system for rectangularizing stitched images based on a diffusion model, characterized in that, It includes a dataset acquisition unit, a model building unit, a model training unit, and an image restoration generation unit, among which: A dataset acquisition unit is used to acquire datasets, which include test sets and training sets. A model building unit is used to establish a diffusion model, which includes a motion diffusion model module and a content diffusion model module. The model training unit is used to train the motion diffusion model module and the content diffusion model module in the diffusion model using the loss function and the training set, and to generate a trained diffusion model. The method for training the motion diffusion model module includes: the training set is given by stitched images and motion streams, with random time intervals added. The relevant Gaussian noise is obtained at time The goal is to generate a motion flow that is identical to a given motion flow and to apply the motion flow to the stitched images so that the resulting image is consistent with the real image. During training, the stitched image, mask, and real optical flow field obtained through the grid in the dataset are input. The stitched image and mask are used as conditions to sample the predicted optical flow field from Gaussian noise, with the expectation that the stitched image can be restored into a normal rectangular image after being subjected to the optical flow field predicted by the data. The method for training the content diffusion model module includes: providing a rectangularly corrected intermediate state image and a real target image; adding noise to the rectangularly corrected intermediate state image at random time steps; training the diffusion model to predict the real image at a given time step; with the goal of generating a real image that is consistent with the given real image. The confidence matrix is obtained through the intermediate results generated by the motion diffusion model module. The image and mask are used as conditions in the sampling process, and the output is a rectangular image after the details are repaired. The image repair generation unit is used to collect the stitched images that need to be corrected and input them into the trained diffusion model. The images then enter the motion diffusion model module and the content diffusion model module in sequence to generate the repaired rectangular image.
7. An electronic device, characterized in that, It includes a processor and a memory; the processor is used to run the image rectification system based on the diffusion model as described in claim 6.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on the electronic device as claimed in claim 7, cause the electronic device to perform the method as claimed in any one of claims 1-5.