Picture re-rectangularization method and system based on priori knowledge, equipment and medium
By using the prior knowledge of the pre-trained diffusion model, the image heavy rectangular diffusion model is reconstructed, which solves the problem of irregular boundaries of the stitching image, and achieves a more efficient and robust image heavy rectangular effect.
Patent Information
- Application Number
- CN202510085415.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-20
- Publication Date
- 2025-05-02
AI Technical Summary
The prior art is difficult to effectively solve the problem of irregular boundary of stitching images. Traditional methods such as cropping, generative models filling missing areas or grid transformation have problems such as boundary damage, inconsistent content or high computing resource consumption.
By using the prior knowledge in the pre-trained diffusion model, the picture is reconstructed, and a two-stage training strategy and a specific loss function are used, combining the variational autoencoder and UNet structure to achieve the picture is reconstructed.
It achieves less training volume and shorter inference time, achieves better results and generalization performance, significantly improving the performance and robustness of image re-rectangularization.
Smart Images
Figure CN119919288A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of computational graphics and computer vision technology, and in particular to a method and system, device, and medium for re-rectangularizing an image based on prior knowledge, wherein a spliced image is re-rectangularized by utilizing prior knowledge in a pre-trained diffusion model. Background Art
[0002] By merging multiple partially overlapping images, we can get a mosaic image, but it often has irregular boundaries. Image rectangularization is the process of converting a mosaic image with irregular boundaries into a rectangle with regular boundaries.
[0003] To solve this problem, the simplest method is cropping, but the cost is to reduce the field of view of the image. Another method is to use a generative model to fill the gaps and fill the missing boundary areas with newly generated content, but this method will introduce new content, destroy the consistency of the data, and limit its practical use. Some researchers have tried to use mesh-warping and content-aware cutting (Seam-carving) technology to solve irregular borders, but such operations can only maintain linear structures and destroy all nonlinear content.
[0004] After that, some people tried to use deep learning to solve this problem, such as using deep models to estimate more accurate meshes, or using diffusion models to generate pixel-by-pixel optical flows. However, these methods either leave obvious traces of splicing at the mesh boundaries, or rely on deep models trained from scratch, which requires a lot of computing power. In addition, due to the lack of relevant training data and the complexity of deep models, these methods generally lack robustness and generalization capabilities.
[0005] With the development of generative artificial intelligence, a large number of pre-trained models have emerged in the market. The present invention provides a method and system, device, and medium for image re-rectangularization based on prior knowledge, which attempts to use the prior knowledge (Priors) in the existing pre-trained model to complete the image re-rectangularization task, redirecting the large amount of information read during the pre-training process to the re-rectangularization task, so as to achieve better results and generalization performance with less training and shorter inference time. Summary of the invention
[0006] The present invention provides a method and system, a device and a medium for re-rectangularizing an image based on prior knowledge, and re-rectangularizes a spliced image by utilizing the prior knowledge in a pre-trained diffusion model.
[0007] The present invention is implemented by the following technical solution: A method for re-rectangularizing an image based on prior knowledge, comprising the following steps:
[0008] Step S1, adjusting the number of layers on the input layer and the output layer of the variational autoencoder in the backbone structure of the stable diffusion model, and reconstructing the image re-rectangular diffusion model;
[0009] Step S2, performing two-stage training on the image re-rectangularization diffusion model, and outputting the trained image re-rectangularization diffusion model;
[0010] Step S3, using the trained image re-rectangularization diffusion model to complete the image re-rectangularization task, and verifying the trained image re-rectangularization diffusion model.
[0011] In order to better implement the present invention, further, step S1 includes:
[0012] The input layer is rewritten to 12 channels, and the output layer is rewritten to 4 channels.
[0013] In order to better implement the present invention, further, the image re-rectangularization diffusion model in step S1 includes an encoder of a variational autoencoder, UNet, and a decoder of a variational autoencoder.
[0014] In order to better implement the present invention, further, the optical flow estimator is transformed from the UNet in the backbone structure of the stable diffusion model;
[0015] The variational autoencoder is transformed from the variational autoencoder in the backbone structure of the stable diffusion model.
[0016] In order to better implement the present invention, further, the training process of the image re-rectangularization diffusion model follows a two-stage strategy;
[0017] In the first stage, the UNet of the image re-rectangularization diffusion model is trained independently, and the variational autoencoder is frozen;
[0018] In the second stage, the UNet and variational autoencoder of the image re-rectangularization diffusion model are jointly refined;
[0019] The following four loss functions are used to supervise the training process:
[0020]
[0021] Where F represents the optical flow, f represents the normalized optical flow, f′ represents the normalized optical flow concatenated with the all-one matrix, and I S represents the spliced graph, I R represents a rectangular image, M represents a mask, and v θ Represents a vgg-16 model for semantic feature extraction; represents the diffusion model reconstruction loss function; represents the conditional loss function, represents the mask loss function; represents the perceptual loss function; Z(f′) represents the features of the optical flow after normalization and concatenation of the all-one matrix after being encoded by the variational autoencoder; It indicates the clean features of the optical flow of the zeroth step estimated by UNet based on the noise features of the tth step after normalization and splicing of the all-one matrix and encoded by the variational autoencoder; warp indicates the use of the pixel-by-pixel mapping relationship expressed by the optical flow to transform the image; It means that the optical flow in pixel space is obtained by UNet based on the noise characteristics of the tth step, which is normalized and concatenated with the all-one matrix, and then encoded by the variational autoencoder, and then decoded by the decoder of the variational autoencoder; || and its subscript 2 mean taking the bi-norm of the expression between two symbols.
[0022] In order to better implement the present invention, further, the specific method for completing the image re-rectangularization task in step S3 includes:
[0023] The mask and the spliced image are input into the trained image re-rectangularization diffusion model. After three steps of VAE encoding, DDIM diffusion process, and VAE decoding, the estimated optical flow is obtained. The optical flow is warped onto the spliced image to obtain the re-rectangularized image to complete the image re-rectangularization task.
[0024] In order to better implement the present invention, further, the method for verifying the trained image re-rectangularization diffusion model in step S3 includes a sampling step disaster method and an adaptive integration strategy method.
[0025] The present invention also provides a system for re-rectangularizing an image based on prior knowledge, comprising a model building unit, a model training unit and a model reasoning verification unit, wherein:
[0026] A model building unit, used to adjust the number of layers on the input layer and the output layer of the variational autoencoder in the backbone structure of the stable diffusion model, and rebuild the image re-rectangular diffusion model;
[0027] A model training unit, used for performing two-stage training on the image re-rectangularization diffusion model and outputting a trained image re-rectangularization diffusion model;
[0028] The model reasoning and verification unit is used to use the trained image re-rectangularization diffusion model to complete the image re-rectangularization task and verify the trained image re-rectangularization diffusion model.
[0029] The present invention also provides an electronic device, which includes a processor and a memory; the processor includes the image re-rectangularization system based on prior knowledge recorded in the second aspect above.
[0030] The present invention also provides a computer-readable storage medium, which includes instructions; when the instructions are executed on the electronic device described in the third aspect, the electronic device executes the method described in the first aspect.
[0031] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0032] (1) The present invention provides a method, system, device, and medium for image re-rectangularization based on prior knowledge. We propose an image re-rectangularization framework based on prior knowledge of a diffusion model, named StableRec. We propose the concept of Sampling Steps Disaster (SSD), which provides theoretical support for the phenomenon that increasing the number of inference steps in certain situations leads to worse results, and thereby achieves further inference.
[0033] (2) The present invention provides a method, system, device, and medium for re-rectangularizing an image based on prior knowledge, and proposes an adaptive ensemble strategy (AES) to aggregate the results of multiple inferences into a more consistent improved result. A large number of experiments have shown that StableRec achieves the most advanced performance and generalization capabilities in public benchmark tests. Compared with the previous diffusion model method, the inference speed of StableRec is more than 200 times faster. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The present invention is further described in conjunction with the following drawings and embodiments, and all the concepts and innovations of the present invention should be regarded as the disclosed contents and the protection scope of the present invention.
[0035] Figure 1 A schematic diagram of a method for re-rectangularizing an image based on prior knowledge and a method flow diagram in a system, a device, and a medium provided in an embodiment of the present application;
[0036] Figure 2 A schematic diagram of a model training process in a method for re-rectangularizing an image based on prior knowledge and a system, device, and medium provided in an embodiment of the present application;
[0037] Figure 3 A schematic diagram of a process of performing sampling step disaster in a method for re-rectangularizing an image based on prior knowledge and a system, device, and medium provided in an embodiment of the present application;
[0038] Figure 4A schematic diagram of sampling proof in a method for re-rectangularizing an image based on prior knowledge and a system, device, and medium provided in an embodiment of the present application;
[0039] Figure 5 A schematic diagram of a diffusion model in a method for re-rectangularizing an image based on prior knowledge and a system, device, and medium provided in an embodiment of the present application;
[0040] Figure 6 A schematic diagram of test results in a method for re-rectangularizing an image based on prior knowledge and a system, device, and medium provided in an embodiment of the present application;
[0041] Figure 7 Schematic diagram B of a method for re-rectangularizing an image based on prior knowledge and a system, device, and medium test results provided in an embodiment of the present application. DETAILED DESCRIPTION
[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be understood that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, and therefore should not be regarded as limiting the scope of protection. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technical personnel in this field without making creative work are within the scope of protection of the present invention.
[0056] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "disposed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0057] Embodiment 1:
[0058] The present embodiment provides a method, system, device, and medium for re-rectifying an image based on prior knowledge, such as Figure 1As shown in the figure, we proposed a framework for image re-rectangularization based on prior knowledge of the diffusion model, named StableRec; we proposed the concept of Sampling Steps Disaster (SSD), which provides theoretical support for the phenomenon that increasing the number of inference steps in certain situations leads to worse results, and thus achieves further inference. The process of reconstructing the image re-rectangularization diffusion model and training it is called the inference process. The mask and the spliced image are input into the model, and the estimated optical flow is obtained after three steps of VAE encoding, DDIM diffusion process, and VAE decoding. The re-rectangularized image is obtained by warping the optical flow onto the spliced image.
[0059] Embodiment 2:
[0060] This embodiment is further optimized on the basis of Embodiment 1. In order to use the pre-trained parameters of the Stable Diffusion model, our model will inherit its backbone structure (the "backbone structure" is VAE (Variational Autoencoder) and UNet. A VAE includes an encoder and a decoder. The structure of our model is: VAE.encoder-unet-vae.decoder). VAE consists of and only consists of one encoder and one decoder, and only the number of layers is adjusted on the input and output layers to adapt to the data structure required in the re-rectangularization task.
[0061] Specifically, the optical flow pseudo-label in the input is spliced with an all-one channel to form homogeneous coordinates; the mask and the spliced image are spliced together in the channel direction as conditional input. The homogeneous optical flow, mask and spliced image have a total of 12 channels in the feature space, of which the homogeneous optical flow feature occupies 4 channels, so the input layer of UNet is rewritten to 12 channels and the output is changed to 4 channels.
[0062] We inherit the VAE (Variational Autoencoder) and UNet module structures in SD and build our model on this basis.
[0063] Our goal is to transform UNet into an optical flow estimator and VAE into a detail restorer. The function of the optical flow estimator is to estimate the optical flow between the stitching image and the desired rectangular image. After this optical flow is warped on the stitching image, the desired rectangular image can be obtained. The detail restorer means that since there is a lot of prior knowledge in VAE, the image obtained after decoding+warping should have correct details and reasonable geometry.
[0064] By simply adjusting the number of layers in the input and output layers of the variational autoencoder of the stable diffusion model, UNet can be transformed into an optical flow estimator and VAE into a detail restorer. In the input layer, in order to retain the prior knowledge as much as possible, the weights in the newly added layer inherit the weights of the input layer in the original pre-training at the time of initialization, and are finally divided by 3 (because the number of input channels increases from 4 to 12 layers to triple the original) to ensure that the prior knowledge during pre-training is retained as much as possible.
[0065] In order to normalize the optical flow to the original range of the VAE input, the input hyperparameter γ (manually set to estimate a "maximum absolute value in the flow data") is first used to approximately match the maximum absolute value in the flow data; then the optical flow and an all-one matrix are concatenated together to construct homogeneous coordinates, which are also adapted to the 3 input channels of the pre-trained VAE encoder.
[0066] The original input range of VAE is -1 to 1. The optical flow needs to be divided by the "maximum value in the flow data" and then concatenated into an all-one matrix to fit the original input range of VAE. Since the maximum absolute value in the flow data is roughly estimated by the naked eye, there may be a small number of values outside the range after the division, but it does not affect the reasoning effect. Most values fall between -1 and 1.
[0067] For UNet, we use the spliced image and mask as conditional input. The conditional diffusion model is a common way to guide the diffusion model to generate specific content. The role of the condition is to guide the denoiser (Unet) to gradually denoise the noisy data and obtain a clean image (optical flow). The noisy optical flow will be gradually denoised under the guidance of the condition.
[0068] The other parts of this embodiment are the same as those of Embodiment 1, and thus will not be described in detail.
[0069] Embodiment 3:
[0070] This embodiment is further optimized on the basis of the above embodiment 1 or 2. Figure 2 The following is a schematic diagram of our model training process.
[0071] Our training process follows a two-stage strategy. In the first stage, UNet is trained independently while VAE is frozen. In the second stage, UNet and VAE are jointly refined.
[0072] We use the following four loss functions to supervise the training process:
[0073]
[0074] Where F represents the optical flow, f represents the normalized optical flow, f′ represents the normalized optical flow concatenated with the all-one matrix, and I S represents the spliced graph, I R represents a rectangular image, M represents a mask, and v θ Represents a vgg-16 model for semantic feature extraction; represents the diffusion model reconstruction loss function; represents the conditional loss function, represents the mask loss function; represents the perceptual loss function; z(f′) represents the features of the optical flow after normalization and concatenation of the all-one matrix after being encoded by the variational autoencoder; It indicates the clean features of the optical flow of the zeroth step estimated by UNet based on the noise features of the tth step after normalization and splicing of the all-one matrix and encoded by the variational autoencoder; warp indicates the use of the pixel-by-pixel mapping relationship expressed by the optical flow to transform the image; It means the optical flow in pixel space obtained by UNet based on the noise characteristics of the tth step, after the optical flow of the normalized and concatenated all-one matrix of the zeroth step is estimated by the variational autoencoder, and then decoded by the decoder of the variational autoencoder. || represents the || symbol on the left side of each formula, the || symbol on the right side, and the subscript 2 in the lower right corner, which together mean "take the two norms of the expression between the two || symbols".
[0075] The other parts of this embodiment are the same as any of the above embodiments 1-2, so they are not repeated here.
[0076] Embodiment 4:
[0077] This embodiment is further optimized based on any one of the above embodiments 1-3. Figure 3 As shown, it is a schematic diagram of our process of Sampling Steps Disaster. Sampling Steps Disaster is a proof that "increasing the number of sampling steps will bring wrong results". Our invention sets the number of sampling steps to 1 to solve this problem.
[0078] If a loss other than the diffusion reconstruction loss is used, such as a conditional loss (such as the photometric loss and mask loss in our framework) to train the diffusion model, it will lead to performance degradation during multi-step reasoning, which is a seemingly counterintuitive phenomenon. We propose the sampling step curse as follows Figure 3As shown in the figure, to explain this phenomenon, the dotted arrows in the figure indicate the supervision of the diffusion reconstruction loss and the conditional loss, pointing to the distribution of the pseudo optical flow and the real optical flow respectively. The solid arrow is the joint effect of the two, pointing to the learned distribution. During the inference process, according to the model prediction value of each step, the noise scheduler calculates the input for the next sampling. The input calculated in this way is not completely consistent with the distribution of the noisy optical flow in the forward diffusion, so directly using it as the input for the next step will produce errors, which will accumulate as the number of sampling steps increases.
[0079] Therefore, under the premise of conditional loss, one-step sampling is proved to be the best strategy, as follows Figure 4 shown.
[0080] The other parts of this embodiment are the same as any one of the above embodiments 1-3, so they are not repeated here.
[0081] Embodiment 5:
[0082] This embodiment further optimizes any one of the above embodiments 1-4. The Adaptive Ensemble Strategy uses a DDIM sampler, and only needs to set the sampling step of the sampler to 1. The sampling step disaster proves that using a sampling step greater than 1 will cause errors during reasoning, so the sampling step is set to 1.
[0083] like Figure 5 As shown in , the generative nature of the diffusion model may lead to inconsistent results. In addition, the rectangularization process involves transforming the image using a motion field, which makes it difficult to handle small boundaries. These two problems can be effectively alleviated by aggregating different outputs. edge Represents a fixed mask, where the values near the edge are 0 and the values elsewhere are 1; in our example, “near the edge” means “within 20 pixels of the edge” (of course 20 can be replaced by other numbers). warp The mask M obtained after the mask of the spliced image is warped by the estimated optical flow aes By M edge and M warp The element-wise inner product is finally used in the adaptive integration strategy. Specifically, corresponding to M aes The area with a median value of 0 will use the minimum filter during integration, while the area with a median value of 1 will use the median or mean filter.
[0084] If the minimum filter is used directly, the white edges will not be selected in the minimum set because their pixel values are the largest. However, using this filter will blur the edges and affect image quality. In order to preserve edge details, we apply the same optical flow transformation to the mask, and the non-border areas detected by the transformed mask will be applied with a median filter. Such an adaptive minimum median filter will allow us to aggregate different outputs into a more unified and higher quality image. We name this strategy the Adaptive Aggregation Strategy (AES).
[0085] like Figure 6 and Figure 7 The following are the test results we provide on the open source datasets DIR-D and APAP-Conssite. It can be seen that in DIR-D, the parts pointed out by arrows of other methods are distorted or twisted, such as twisted car doors, deformed hairstyles, bent parasol poles, etc. In comparison, our method reduces or eliminates these distortions. In APAP-Conssite, other methods may leave irregular boundaries to varying degrees, while our method has better performance on the boundaries.
[0086] "Inference" refers to the process of giving a model an input and the model giving an output. The curse of sampling steps is a theoretical proof that an increase in the number of sampling steps during inference will lead to errors in the results, making it correct and efficient to use one-step sampling during inference for our model; the adaptive integration strategy is a method of synthesizing multiple slightly different results into a more robust result after multiple inferences on the same input get different results (our model is a generative model, and multiple inferences on the same input will get multiple slightly different results).
[0087] The other parts of this embodiment are the same as any one of the above embodiments 1-4, so they are not repeated here.
[0088] Embodiment 6:
[0089] The present invention also provides a system for re-rectangularizing an image based on prior knowledge matching the method.
[0090] The present invention also provides an electronic device, which includes a processor and a memory; the processor includes the above-mentioned image re-rectangularization system based on prior knowledge.
[0091] Embodiment 7:
[0092] The present invention also provides a computer-readable storage medium, which includes instructions; when the instructions are executed on the electronic device described in the above embodiment, the electronic device executes the method described in the above embodiment. Optionally, the computer-readable storage medium may be a memory.
[0093] The processor involved in the embodiments of the present application may be a chip. For example, it may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.
[0094] The memory involved in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0095] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0096] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0097] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0098] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.
[0099] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one device or distributed on multiple devices. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.
[0100] In addition, each functional module in each embodiment of the present application may be integrated into one device, or each module may exist physically separately, or two or more modules may be integrated into one device.
[0101] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing a computer program instruction on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or may contain one or more servers, data centers and other data storage devices that can be integrated with a medium. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).
[0102] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the present invention.
Claims
1. A method for re-rectangularizing an image based on prior knowledge, characterized in that: The following steps are involved: Step S1, adjusting the number of layers on the input layer and the output layer of the variational autoencoder in the backbone structure of the stable diffusion model, and reconstructing the image re-rectangular diffusion model; Step S2, performing two-stage training on the image re-rectangularization diffusion model, and outputting the trained image re-rectangularization diffusion model; Step S3, using the trained image re-rectangularization diffusion model to complete the image re-rectangularization task, and verifying the trained image re-rectangularization diffusion model.
2. The method for re-rectangularizing an image based on prior knowledge according to claim 1, characterized in that: The step S1 comprises: The input layer is rewritten to 12 channels, and the output layer is rewritten to 4 channels.
3. The method for re-rectangularizing an image based on prior knowledge according to claim 1, characterized in that: The image re-rectangularization diffusion model in step S1 includes an encoder of a variational autoencoder, UNet, and a decoder of a variational autoencoder.
4. The method for re-rectangularizing an image based on prior knowledge according to claim 1, characterized in that: include: The optical flow estimator is transformed from UNet in the backbone structure of the stable diffusion model; The variational autoencoder is transformed from the variational autoencoder in the backbone structure of the stable diffusion model.
5. The method for re-rectangularizing an image based on prior knowledge according to claim 1, characterized in that: The training process of the image re-rectangularization diffusion model follows a two-stage strategy; In the first stage, the UNet of the image re-rectangularization diffusion model is trained independently, and the variational autoencoder is frozen; In the second stage, the UNet and variational autoencoder of the image re-rectangularization diffusion model are jointly refined; The following four loss functions are used to supervise the training process: Where F represents the optical flow, f represents the normalized optical flow, f′ represents the normalized optical flow concatenated with the all-one matrix, and I S represents the spliced graph, I R represents a rectangular image, M represents a mask, and v θ Represents a vgg-16 model for semantic feature extraction; l diff represents the diffusion model reconstruction loss function; l pl represents the conditional loss function, l mask represents the mask loss function; l pct represents the perceptual loss function; z (f′) It represents the features of the optical flow after normalization and concatenation of the all-one matrix after being encoded by the variational autoencoder; It indicates the clean features of the optical flow of the zeroth step estimated by UNet based on the noise features of the tth step after normalization and splicing of the all-one matrix and encoded by the variational autoencoder; warp indicates the use of the pixel-by-pixel mapping relationship expressed by the optical flow to transform the image; It means the optical flow in pixel space obtained by UNet based on the noise characteristics of the tth step, after the normalization of the optical flow of the zeroth step and the concatenation of the all-one matrix, after the clean optical flow features encoded by the variational autoencoder, and then decoded by the decoder of the variational autoencoder. || and its subscript 2 mean taking the bi-norm of the expression between two symbols.
6. The method for re-rectangularizing an image based on prior knowledge according to claim 1, characterized in that: The specific method for completing the image re-rectangularization task in step S3 includes: The mask and the spliced image are input into the trained image re-rectangularization diffusion model. After three steps of VAE encoding, DDIM diffusion process, and VAE decoding, the estimated optical flow is obtained. The optical flow is warped onto the spliced image to obtain the re-rectangularized image to complete the image re-rectangularization task.
7. The method for re-rectangularizing an image based on prior knowledge according to claim 1, characterized in that: The method for verifying the trained image re-rectangularization diffusion model in step S3 includes a sampling step disaster method and an adaptive integration strategy method.
8. A system for re-rectangularizing an image based on prior knowledge, characterized in that: It includes a model building unit, a model training unit and a model reasoning verification unit, wherein: A model building unit, used to adjust the number of layers on the input layer and the output layer of the variational autoencoder in the backbone structure of the stable diffusion model, and rebuild the image re-rectangular diffusion model; A model training unit, used for performing two-stage training on the image re-rectangularization diffusion model and outputting a trained image re-rectangularization diffusion model; The model reasoning and verification unit is used to use the trained image re-rectangularization diffusion model to complete the image re-rectangularization task and verify the trained image re-rectangularization diffusion model.
9. An electronic device, characterized in that: It comprises a processor and a memory; the processor is used to run the image re-rectangularization system based on prior knowledge as described in claim 8.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes instructions; when the instructions are executed on the electronic device as claimed in claim 9, the electronic device is caused to execute the method as claimed in any one of claims 1 to 7.