Two-stage image rectangularization method and system based on deep unwrapping and warping network

Abstract

The invention discloses a two-stage image rectangularization method and a two-stage image rectangularization system based on a deep unwrapping and warping network, and belongs to the technical field of image processing. The grid predictor and the deformer are used for rectangularization deformation, and the decorator is used for rectangularization alignment; two loss functions are adopted to train a grid predictor and a decorator of the deep expansion distortion network respectively; and finally, target image rectangularization is carried out based on the trained deep expansion and distortion network. According to the method, the problem of boundary dislocation of a traditional rectangularization method is solved through the two stages of rectangularization deformation and rectangularization alignment, the visual quality and practicability of the image are improved, key information in the original image is protected, and unnecessary information loss is avoided.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically to a two-stage image rectangularization method and system based on a deep unfolding warping network. Background Art

[0002] Image stitching technology is widely used in many fields, including but not limited to panoramic imaging, remote sensing, infrared imaging, and robot motion planning. Existing image stitching methods mainly rely on feature detection and matching to achieve image alignment and fusion, which includes traditional algorithms and feature extraction techniques based on deep learning. However, in practice, due to the projection transformation process of pixel redistribution and the potential impact of camera movement, the stitched images often have irregular and visually poor edges. Such irregular boundaries not only affect the aesthetics of the image, but also limit the application experience of the image in scenes such as viewing, publishing, and reading. More importantly, almost all current downstream tasks, especially those involving scene understanding, such as image classification, object detection, and semantic segmentation, are trained based on rectangular images. For such tasks, processing stitched images with irregular boundaries may cause the algorithm to be unstable near these boundaries, thereby affecting the overall performance. Therefore, developing effective solutions to improve or eliminate irregular boundaries in stitched images has become the key to improving image quality and the accuracy of downstream tasks.

[0003] In order to obtain a more complete and harmonious rectangular image visually, image rectangularization technology has gradually become a research hotspot. According to different processing strategies and their impact on the original image content, image rectangularization methods can be roughly divided into the following three categories: 1) Methods based on rectangular frame cropping: This is the most direct and simple way to form a rectangular image by cropping non-rectangular areas. Although it is relatively easy to implement, this method will inevitably reduce the receptive field of the image and cause some valuable information to be lost. 2) Technology based on image restoration: Using advanced image restoration algorithms, the blank areas in the image are filled with reasonable textures or features to achieve rectangularization. This method can creatively supplement the missing parts without reducing the original information, but it also means introducing some data that did not originally exist. 3) Content-preserving rectangularization methods: This type of method attempts to achieve the effect of rectangularization by redistributing useful pixels to blank areas by locally distorting or geometrically transforming the image without reducing or increasing the original image content. Although slight deformation may be introduced in this process, the authenticity and integrity of the image can be better maintained overall.

[0004] Although traditional image rectangularization technology has alleviated the problem of irregular boundaries after image stitching to a certain extent, it still faces many challenges and limitations in practical applications: 1) Information loss: Using a simple cropping method to process image boundaries is easy to operate, but it will inevitably cause the loss of useful information. For application scenarios that require the retention of all details, such as medical image analysis or precision engineering measurement, this method is obviously inappropriate. 2) Loss of authenticity: Although image restoration technology can effectively fill in boundary gaps and restore the continuity of the image, the content it generates may not match the actual situation, resulting in the authenticity of the image being affected. This inaccurate information may have serious consequences in certain professional fields, such as historical relics restoration or legal evidence collection. 3) Inefficiency: Content-preserving rectangularization methods aim to minimize information loss and ensure the integrity and accuracy of image content. However, such methods usually require complex calculations and long processing time, which increases the user's operational burden and is not suitable for real-time or large-scale data processing needs. 4) Insufficient adaptability: Traditional methods show obvious limitations in dealing with nonlinear scenes, especially when facing complex backgrounds or significant changes in perspective. The instability of its performance and the difficulty in ensuring the consistency of its output results pose a major obstacle in applications that require high reliability and accuracy. Summary of the invention

[0005] In view of this, the present invention provides a two-stage image rectangularization method and system based on a deep unfolding warp network that solves at least some of the above-mentioned technical problems. The method performs image rectangularization based on a deep unfolding warp network, and the deep unfolding warp network integrates a grid predictor and a deformer to optimize the grid estimation in an unfolded manner; it also integrates a decorator to further optimize the image to ensure that the final output is a complete rectangular image; it not only solves the problems of boundary dislocation and content distortion existing in traditional image rectangularization technology, but also provides a new solution for the field of image processing through an efficient two-stage processing flow.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] In a first aspect, the present invention provides a two-stage image rectangularization method based on a deep unfolding warping network, the method comprising the following steps:

[0008] S1. Constructing a deep unfolding and warping network, wherein the deep unfolding and warping network includes: a grid predictor and a deformer for rectangular deformation, and a decorator for rectangular alignment;

[0009] S2. Using two loss functions to respectively train the grid predictor and decorator of the deep unfolding warp network;

[0010] S3. Rectangularize the target image based on the trained deep unfolding warping network.

[0011] Furthermore, in S1, the rectangular deformation is completed by a grid predictor and a deformer; wherein:

[0012] The rectangular target is expressed as:

[0013]

[0014]

[0015] In the formula, m represents the predicted initial grid position, and the final output is X represents the stitched image, Y represents the image background, represents the transformation operator, △ represents the interpolation algorithm, and φ(m) is the prior term constraining the mesh deformation offset;

[0016] For step size s, s = 1, ..., S, S represents the number of steps to be expanded; the grid predictor is a network used for regression, and its role is:

[0017]

[0018] In the formula, represents the grid predictor, Δm s represents the grid offset relative to the target grid at step s, X s-1 represents the distortion result of the input spliced ​​image in step s-1, M s-1 represents the synchronous warping result of the input stitched image mask in step s-1, M represents the initial mask of the input stitched image, indicating the range of the valid area of ​​the input stitched image;

[0019] The role of the deformer is to:

[0020]

[0021] Where, X s represents the distortion result of the input spliced ​​image X0 in the sth step, M s is the intermediate result of M0, which represents the initial mask indicating the valid pixel positions, represents the deformer, X0 represents the initial input spliced ​​image, m s-1 It represents the grid position predicted in the s-1th step. After the number of steps S, the final output is (X S ,M S ).

[0022] Furthermore, in S1, the decorator includes a partial convolution head and a super-resolution backbone, and the decorator is used to align the final output of the deformer in a rectangular manner, and its realization objectives are as follows:

[0023]

[0024] Where θ represents the decorator parameter, ⊙ represents element-wise multiplication, Represents a decorator.

[0025] Furthermore, for each pixel position on the target grid, the decorator obtains its corresponding position in the input coordinate space by applying the homography transformation matrix.

[0026] Furthermore, in S2, two loss functions are used to respectively train the grid predictor and decorator of the deep unfolding warp network, where:

[0027] The loss function for training the grid predictor is:

[0028]

[0029] Where α1, α2, α3 are weights of different losses, L1 is the sum of the absolute values ​​between the calculation results of the intermediate expansion step and the target result, is the perceptual loss between the calculation result of the intermediate expansion step and the target result, It is the cumulative sum of the absolute values ​​between the result mask and the all-one matrix in the calculation of the intermediate expansion step; is a regularization term;

[0030] The loss function for training the decorator is:

[0031]

[0032] In the formula, β is a weighting factor, X D It is the final result of the decorator output.

[0033] In a second aspect, the present invention further provides a two-stage image rectangularization system based on a deep unfolding warp network, which is applied to the above-mentioned two-stage image rectangularization method based on a deep unfolding warp network to perform two-stage image rectangularization based on a deep unfolding warp network, and the system includes:

[0034] A network construction module, used to construct a deep unfolding and warping network, wherein the deep unfolding and warping network includes: a grid predictor and a deformer for rectangular deformation, and a decorator for rectangular alignment;

[0035] A network training module, using two loss functions to respectively train a grid predictor and a decorator of the deep unfolding warp network;

[0036] The network application module performs rectangularization of the target image based on the trained deep unfolding and distortion network.

[0037] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned two-stage image rectangularization method based on a deep unfolding warping network.

[0038] In a fourth aspect, the present invention also provides a storage medium having stored thereon one or more programs readable by a computing device, wherein the one or more programs include instructions which, when executed by the computing device, enable the computing device to perform a two-stage image rectangularization method based on a deep unfolding warping network as described above.

[0039] Compared with the prior art, the present invention has at least the following beneficial technical effects:

[0040] The present invention provides a two-stage image rectangularization method and system based on a deep unfolding warp network. Image rectangularization is performed based on the deep unfolding warp network. The deep unfolding warp network integrates a grid predictor and a deformer to optimize the grid estimation in an unfolded manner. A decorator is also integrated to further optimize the image to ensure that the final output is a complete rectangular image. It not only solves the problems of boundary dislocation and content distortion existing in traditional image rectangularization technology, but also provides a new solution for the field of image processing through an efficient two-stage processing flow. It is beneficial to improve the visual quality and practicality of the image, facilitates the protection of key information in the original image, and helps to avoid unnecessary information loss.

[0041] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0042] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0045] Figure 1 A schematic flow chart of a two-stage image rectangularization method based on a deep unfolding warping network provided in an embodiment of the present invention.

[0046] Figure 2 A schematic diagram comparing the two-stage rectangularization strategy provided by an embodiment of the present invention and the traditional one-stage rectangularization strategy.

[0047] Figure 3 A schematic diagram of the principle framework of two-stage image rectangularization provided by an embodiment of the present invention.

[0048] Figure 4 A schematic diagram of the specific designs of the grid predictor, deformer and decorator provided in an embodiment of the present invention.

[0049] Figure 5 A schematic diagram of a comparison result between the method of the present invention and other rectangularization methods provided in an embodiment of the present invention.

[0050] Figure 6 Another schematic diagram of comparison results between the method of the present invention and other rectangularization methods provided in an embodiment of the present invention.

[0051] Figure 7 A schematic diagram of the function of the decorator provided in an embodiment of the present invention.

[0052] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0054] In the description of the present invention, it should be noted that: in some processes described in the specification and drawings of this application, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be performed in the order in which they appear in this document or may be performed in parallel. In addition, various serial numbers are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0055] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Embodiment 1:

[0057] See also Figure 1 As shown, an embodiment of the present invention provides a two-stage image rectangularization method based on a deep unfolding warping network, and the method mainly includes:

[0058] S1. Construct a deep unfolding and warping network, which includes a grid predictor and a deformer for rectangular deformation, and a decorator for rectangular alignment;

[0059] S2, using two loss functions to train the grid predictor and decorator of the deep unfolding warp network respectively;

[0060] S3. Rectangularize the target image based on the trained deep unfolding warping network.

[0061] Combine the following Figure 2-Figure 7 As shown, the working principle and specific implementation methods of the method of the present invention are introduced:

[0062] The present invention proposes a two-stage image rectangularization method based on a deep unfolding warp network, which solves the boundary misalignment problem of the traditional rectangularization method through the two stages of rectangularization deformation and rectangularization alignment, improves the visual quality and practicality of the image, protects the key information in the original image, and avoids unnecessary information loss.

[0063] In this embodiment, the method of the present invention mainly reduces distortion and boundary holes caused by traditional rectangularization methods from two key aspects, retains the original content to the maximum extent, and realizes high-quality image rectangularization. In the rectangularization deformation stage, the ideal rectangular solution is approached by grid estimation and deformation technology. The core of this process is to accurately estimate the initial grid distribution, especially when dealing with the irregular boundaries that actually exist in the stitched image. By optimizing the distribution of the grid, the content of the original image can be retained to a large extent, while reducing the obvious distortion and pixel loss caused by deformation. However, although the efforts in this stage can significantly improve the overall structure of the image, some small holes cannot be completely avoided due to the possible errors in grid estimation and the misalignment between the input image and the grid boundary. In addition, during the deformation process, the interpolation operation may also cause a decrease in image quality. In order to solve the above problems, the present invention introduces a rectangularization alignment stage. The main task of this stage is to further optimize the image quality by enhancing the pixel value. Through the alignment stage, reliable context information can be obtained from adjacent areas to fill the small holes left over from the previous stage and repair the image degradation caused by interpolation. Through a carefully designed alignment algorithm, it can be ensured that the image obtains a smoother and more natural visual effect while maintaining the original content.

[0064] The specific design points of the present invention include the following process:

[0065] 1. Rectangular deformation:

[0066] In the embodiment of the present invention, the rectangular deformation process is first formulated. Given an input spliced ​​image X and its corresponding background Y, the rectangularization target can be expressed as:

[0067]

[0068] Among them, m represents the predicted initial grid position, and the final output is represents the transformation operator from the target rigid coordinate space to the coordinate space X for each pixel based on the local mesh. △ represents the interpolation algorithm, such as bilinear interpolation or nearest neighbor interpolation. φ(m) is the prior term that constrains the mesh deformation offset.

[0069] In the design of the rectangular deformation stage of the present invention, it is expanded into a series of steps to better estimate m:

[0070]

[0071]

[0072]

[0073] Among them, s is a step indicator, S represents the number of steps to be expanded, which can be taken as 3 here.

[0074] Considering that the deviations of the intermediate results obtained by deformation may accumulate, the deformation is performed at each step using the original input X0. In addition, the strategy of optimizing the grid offset Δm relative to the target grid is adopted instead of directly predicting the initial grid position. The process becomes:

[0075]

[0076] m s+1 =m s +Δm s ,

[0077]

[0078]

[0079] Among them, m0 is used as the target grid and the predicted variable becomes Δm.

[0080] The present invention uses a grid predictor and a deformer w to complete the above rectangular deformation process. For the step size s, s = 1, ..., S, the role of the grid predictor is:

[0081]

[0082] In the formula, represents the grid predictor, Δm s represents the grid offset relative to the target grid at step s, X s-1 represents the distortion result of the input spliced ​​image in step s-1, M s-1 represents the synchronous warping result of the input stitched image mask in step s-1, M represents the initial mask of the input stitched image, indicating the range of the valid area of ​​the input stitched image;

[0083] The purpose of the deformer is to achieve:

[0084]

[0085] Where, X s represents the distortion result of the input spliced ​​image X0 in the sth step, M s is the intermediate result of M0, which represents the initial mask indicating the valid pixel positions, represents the deformer, X0 represents the initial input spliced ​​image, m s-1 represents the grid position predicted at step s-1. Since the current step takes into account the estimated deviation of the previous step, there is a performance correction in each step. After step S, the output is (X S ,M S ). The network at this stage of the invention unfolds the optimization of rectangularization into alternating steps, sharing the grid predictor in each step Parameters.

[0086] 2. Rectangular alignment:

[0087] In the embodiment of the present invention, three groups (X s ,M s ),s=1,2,3,where X S In theory, it represents the intermediate image after good deformation processing, while M S The corresponding mask is . It is worth noting that in this process, tiny blank areas will appear near the boundary rows or columns, which inspires the design idea of ​​the rectangularization alignment stage of the present invention. Although this challenge is similar to the image restoration task in nature, the present invention specifically emphasizes that it is forbidden to use synthetic patterns or textures to fill these gaps, because doing so will deviate from the core concept of rectangularization. Therefore, the present invention proposes a decorator with strictly restricted generation capabilities. To achieve rectangular alignment, its main function is to accurately fill the above-mentioned tiny missing parts based on the principle of local connection. In addition, considering that the degradation of image quality is almost inevitable when performing deformation operations, especially in terms of the selection and performance of interpolation algorithms. In view of this, the decorator of the present invention not only needs to have the above-mentioned precise filling ability, but also should play a certain image enhancement role when necessary to restore or improve the image quality damaged by deformation. The goals to be achieved are as follows:

[0088]

[0089] Among them, θ represents the parameter of the decorator, ⊙ represents element-wise multiplication, Represents a decorator.

[0090] 3. Deep unfolding twisted network design:

[0091] In the embodiment of the present invention, the entire deep unfolding warp network includes three parts: a grid predictor and Deformers Used in the rectangular deformation stage, decorator It is used in the rectangular alignment stage, and its specific design structure is as follows Figure 4 shown.

[0092] (1) Grid Predictor : For the grid predictor , which is a network for regression:

[0093]

[0094] in, h and w are the number of grid cells in the height direction and width direction, respectively. Since the grid in the target rigid grid is represented by four corner points, there are (h+1)×(w+1) grid points in the grid, where h=12 and w=16.

[0095] (2) Deformer For deformers For , it is a fixed operator that performs distortion and does not require training. Given the four corner points (x1, y1),…,(x4, y4) of any grid in the target grid, and their corresponding paired corner points (u1, v1),…,(u4, v4) on the prediction grid, the homography transformation matrix can be solved by the following formula:

[0096]

[0097] Then, for each pixel position on the target grid, its corresponding position in the input coordinate space can be obtained by applying the homography transformation matrix.

[0098] (3) Decorator :Decorator of the present invention The invention comprises a partial convolution head and a super-resolution backbone. Based on the requirement of rectangular alignment, the model characteristics of image restoration and super-resolution are combined. The partial convolution head is limited to providing reliable pixels with high confidence rather than generating additional features. Specifically, the invention superimposes a partial convolution on the first layer:

[0099]

[0100] Among them, X' S ,M' S In X S ,M S The sliding window of W0 is the weight of the partial convolution kernel. Note that only the mask is used here and the gradient is not updated forward when the network is trained. This design can avoid the false dependency caused by empty pixels when extracting deep features. Then, the output X p Divide into two partsX p1 , X p2 , and obtain the final features:

[0101]

[0102] Among them, Sigmoid refers to the Sigmoid activation function, and ELU represents the Exponential Linear Unit activation function.

[0103] 4. Two-stage loss function design:

[0104] The present invention involves two stages, namely rectangular deformation and rectangular alignment, and therefore two loss functions are used to train the network in the present invention.

[0105] (1) Rectangular deformation stage. In this stage, the grid predictor is supervised in all the unfolding steps. The total loss function consists of four parts:

[0106]

[0107] Among them, α1, α2, α3 are the weights of different losses, which are set to 5e-6, 1.0, and 1.0 respectively. and is the common L1 loss function, It is the calculation result X of the intermediate expansion step s The cumulative sum of the absolute values ​​between (s=1,…,S) and the target result Y. It is the calculation result mask M of the intermediate expansion step calculation sThe cumulative sum of the absolute values ​​between (s=1,…,S) and the all-one matrix. Yes X s The perceptual loss between (s=1,…,S) and the target result Y mainly extracts features through a VGG network to calculate the absolute value sum between the features. It is a regularization term that corresponds to the φ(Δm) term in the rectangular deformation objective, which limits excessive deformation of the mesh.

[0108] (2) Rectangularization and Alignment Stage. Since the decorator must pay attention to small blank areas, we adopt a weighted loss function, namely

[0109]

[0110] Among them, β is a weighting factor, which is set to 6.0 here, and X D Represents the final result of the decorator output.

[0111] V. Experimental verification:

[0112] In the embodiment of the present invention, in order to ensure the efficiency and accuracy of the experiment, the present invention selects the following configuration of the computer environment for testing and verification: equipped with Intel Core i7-9700K processor and NVIDIA Tesla V100 graphics processor, the system frequency is 3.6GHz, the system memory is 64GB, and the operating system is Ubuntu 20.04. The implementation of the present invention relies on the PyTorch deep learning framework. Next, the DIR-D dataset will be used as the original data source to introduce the verification process of the present invention in detail: the grid predictor and the decorator are jointly trained on the public DIR-D dataset through a specially designed loss function, thereby generating an algorithm instance that can be used for actual deployment. The DIR-D dataset is currently the only publicly available image rectangularization dataset, containing 5839 training images and 519 test samples, and the resolution of all images is 384×512. This dataset provides rich resources for the training and verification of the algorithm, ensuring the robustness of the model in terms of diversity and complexity. For the two stages of rectangularization deformation and rectangularization alignment, the present invention adopts the same poly learning strategy, whose factor is 0.96, and the initial learning rate is the same, both 1e-4. Adam was selected as the optimizer for all networks, with a weight decay of 1e-5. For the rectangular deformation stage, the training batch size was 4 and the training step was 200. For the rectangular alignment stage, the training batch size was the same, but a training step of 100 was set. For the decorator, the number of residual modules was 16 and the number of basic feature channels was 64.

[0113] like Figure 5 and Figure 6As shown in FIG. 1 , the rectangularization quality of the image is improved by using the method of the present invention. Figure 5 As shown in part (b), due to the irregular stitching boundary, cropping the original image using the maximum inner rectangular box will cause the original image to suffer a loss of almost half of its viewing angle. Figure 5 Part (c) shows the result of image filling, where there are many repeated mountains and beaches, as indicated by the red arrows. These obvious additions of textures are unreliable and contradict the textures presented in the original image, which may be harmful for high-security situations. Figure 5 As shown in part (f), the present invention can effectively reduce the boundary distortion problem commonly seen in traditional methods, and can maintain the naturalness and continuity of the image, especially when processing irregular boundaries.

[0114] 2) In order to quantitatively demonstrate the superiority of the method described in the present invention, three common image quality evaluation indicators are used for evaluation on the DIR-D dataset: PNSR (peak signal-to-noise ratio), SSIM (structural similarity) and LPIPS (image similarity metric). The results are shown in the following table.

[0115] Table 1 Index results of different algorithms on the DIR-D dataset

[0116] method PNSR↑ SSIM↑ LPIPS↓ Input image benchmark 11.30 0.3245 41.76 How people's method 16.22 0.4173 28.46 The method of Nie et al. 21.12 0.6954 15.57 Method of the present invention 25.03 0.7509 14.86

[0117] 3) Compared with the single-stage rectangularization method, the two-stage method of the present invention has a significant quality improvement effect, such as Figure 7 In the rectangularization and alignment stage, a decorator based on local connections is used to fill small holes without introducing false patterns or textures, thus ensuring the authenticity and accuracy of the image.

[0118] From the description of the above embodiments, those skilled in the art can know that the embodiments of the present invention provide a two-stage image rectangularization method based on a deep unfolding warp network. The present invention uses a deep unfolding warp network to unfold the rectangularization into rectangularization deformation and rectangularization alignment stages respectively. The deep learning model automatically learns and adjusts the nonlinear deformation in the image, restores the structural features of the original image more accurately, reduces or even avoids the loss of useful information, and is particularly suitable for advanced application scenarios that require retaining all details. The method has the following advantages:

[0119] 1) This method is based on a deep unfolding warp network and integrates a mesh predictor and a deformer to optimize the mesh estimation in an unfolding manner.

[0120] 2) After conventional rectangularization, there are still subtle boundary problems. The present invention designs an additional rectangularization alignment solution - decorator to further optimize the image and ensure that the final output is a complete rectangular image.

[0121] 3) A large number of experimental results have verified the effectiveness and superiority of the present invention, proving that it has surpassed the existing state-of-the-art methods in many indicators. The present invention not only solves the problems of boundary misalignment and content distortion existing in traditional image rectangularization technology, but also provides a new solution for the field of image processing through an efficient two-stage processing flow.

[0122] Embodiment 2:

[0123] Furthermore, an embodiment of the present invention further provides a two-stage image rectangularization system based on a deep unfolding warp network, which is applied to a two-stage image rectangularization method based on a deep unfolding warp network described in the above embodiment 1, and performs a two-stage image rectangularization based on a deep unfolding warp network, and the system includes:

[0124] A network construction module, used to construct a deep unfolding and warping network, wherein the deep unfolding and warping network includes: a grid predictor and a deformer for rectangular deformation, and a decorator for rectangular alignment;

[0125] A network training module, using two loss functions to respectively train a grid predictor and a decorator of the deep unfolding warp network;

[0126] The network application module performs rectangularization of the target image based on the trained deep unfolding and distortion network.

[0127] An embodiment of the present invention provides a two-stage image rectangularization system based on a deep unfolding warping network, and its working principle and technical effects are the same as those of the aforementioned method embodiment. For the sake of brief description, for parts not mentioned in this embodiment, reference can be made to the corresponding contents in the aforementioned method embodiment, which will not be repeated here.

[0128] Example 3

[0129] Reference Figure 8 As shown, an embodiment of the present invention also provides an electronic device for performing two-stage image rectangularization based on a deep unfolding warp network. The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10.

[0130] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing programs or modules stored in the memory 11, and calling data stored in the memory 11.

[0131] Embodiment 4:

[0132] An embodiment of the present invention also provides a storage medium on which one or more programs readable by a computing device are stored, and the one or more programs include instructions, which, when executed by the computing device, enable the computing device to execute a two-stage image rectangularization method based on a deep unfolding warp network in Example 1.

[0133] Examples of computer-readable storage media include: read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, non-volatile memory, CD-ROM, DVD-ROM, Blu-ray or optical disk storage, hard disk drive (HDD), solid-state drive (SSD), card-type memory (such as a multimedia card, a secure digital (SD) card, or an extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state disk, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be executed in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files, and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0134] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products, etc. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several distinct components, and by means of a suitably programmed computer.

[0136] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0137] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A two-stage image rectangularization method based on a deep unfolding warping network, characterized in that: The method comprises the following steps: S1. Constructing a deep unfolding and warping network, wherein the deep unfolding and warping network includes: a grid predictor and a deformer for rectangular deformation, and a decorator for rectangular alignment; S2. Using two loss functions to respectively train the grid predictor and decorator of the deep unfolding warp network; S3. Rectangularize the target image based on the trained deep unfolding warping network.

2. According to claim 1, a two-stage image rectangularization method based on deep unfolding warping network is characterized in that: In S1, the rectangular deformation is completed by the grid predictor and the deformer; wherein: The rectangularized target is expressed as: In the formula, m represents the predicted initial grid position, and the final output is X represents the stitched image, Y represents the image background, represents the transformation operator, △ represents the interpolation algorithm, and φ(m) is the prior term constraining the mesh deformation offset; For step size s, s = 1, ..., S, S represents the number of steps to be expanded; the grid predictor is a network used for regression, and its role is: In the formula, represents the grid predictor, Δm s represents the grid offset relative to the target grid at step s, X s-1 represents the distortion result of the input spliced ​​image in step s-1, M s-1 represents the synchronous warping result of the input stitched image mask in step s-1, M represents the initial mask of the input stitched image, indicating the range of the valid area of ​​the input stitched image; The role of the deformer is to: Where, X s represents the distortion result of the input spliced ​​image X0 in the sth step, M s is the intermediate result of M0, M0 represents the initial mask indicating the valid pixel positions, represents the deformer, X0 represents the initial input spliced ​​image, m s-1 It represents the grid position predicted in the s-1th step. After the number of steps S, the final output is (X S ,M S ).

3. The two-stage image rectangularization method based on deep unfolding warping network according to claim 2 is characterized in that: In S1, the decorator includes a partial convolution head and a super-resolution backbone. The decorator is used to align the final output of the deformer in a rectangular manner. The goals of the decorator are as follows: Where θ represents the decorator parameter, ⊙ represents element-wise multiplication, Represents a decorator.

4. The two-stage image rectangularization method based on deep unfolding warping network according to claim 3 is characterized in that: For each pixel position on the target grid, the decorator obtains its corresponding position in the input coordinate space by applying the homography transformation matrix.

5. The two-stage image rectangularization method based on deep unfolding warping network according to claim 3 is characterized in that: In S2, two loss functions are used to train the grid predictor and decorator of the deep unfolding warp network respectively, where: The loss function for training the grid predictor is: Where a1, a2, α3 are the weights of different losses. It is the cumulative sum of the absolute values ​​between the calculation results of the intermediate expansion steps and the target results. is the perceptual loss between the calculation result of the intermediate expansion step and the target result, It is the cumulative sum of the absolute values ​​between the result mask and the all-one matrix in the calculation of the intermediate expansion step; is a regularization term; The loss function for training the decorator is: In the formula, β is a weighting factor, X D It is the final result of the decorator output.

6. A two-stage image rectangularization system based on a deep unwrapping warp network, characterized in that: When applied, a two-stage image rectangularization method based on a deep unfolding warping network according to any one of claims 1 to 5 is executed to perform a two-stage image rectangularization based on a deep unfolding warping network, and the system comprises: A network construction module, used to construct a deep unfolding and warping network, wherein the deep unfolding and warping network includes: a grid predictor and a deformer for rectangular deformation, and a decorator for rectangular alignment; A network training module, using two loss functions to respectively train a grid predictor and a decorator of the deep unfolding warp network; The network application module performs rectangularization of the target image based on the trained deep unfolding and distortion network.

7. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement a two-stage image rectangularization method based on a deep unfolding warp network as described in any one of claims 1 to 5.

8. A storage medium having stored thereon one or more programs readable by a computing device, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform a two-stage image rectangularization method based on a deep unfolding warp network as described in any one of claims 1 to 5.

Images