Two-stage image rectangularization method and system based on deep unfolding warping network

Abstract

The application discloses a two-stage image rectangularization method and system based on a deep unfolding warping network, and belongs to the technical field of image processing. In the application, a deep unfolding warping network is constructed, the deep unfolding warping network comprises a grid predictor and a transformer for rectangularization deformation, and a decorator for rectangularization alignment; two loss functions are used to train the grid predictor and the decorator of the deep unfolding warping network respectively; and finally, target image rectangularization is performed based on the trained deep unfolding warping network. The application solves the misalignment problem of the boundary of the traditional rectangularization method through two stages of rectangularization deformation and rectangularization alignment, improves the visual quality and practicability of the image, protects the key information in the original image, and avoids unnecessary information loss.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and more particularly to a two-stage image rectangularization method and system based on a deep unfolding warping network. BACKGROUND

[0002] Image stitching technology is widely used in many fields, including but not limited to panoramic imaging, remote sensing, infrared imaging, and robot motion planning. Existing image stitching methods mainly rely on feature detection and matching to achieve image alignment and fusion, which includes traditional algorithms and deep learning-based feature extraction techniques. However, in actual operation, due to the projection transformation process of pixel redistribution and potential camera movement, the stitched image often has irregular and visually poor edges. This irregular boundary not only affects the aesthetics of the image, but also limits the application experience of the image in viewing, publishing, and reading scenarios. More importantly, almost all current downstream tasks, especially those involving scene understanding, such as image classification, object detection, and semantic segmentation, are trained based on rectangular images. For these tasks, processing stitched images containing irregular boundaries may cause the algorithm to perform unstable near these boundaries, thereby affecting overall performance. Therefore, developing effective solutions to improve or eliminate irregular boundaries in stitched images is crucial to improving image quality and downstream task accuracy.

[0003] In order to obtain a visually more complete and harmonious rectangular image, image rectangularization technology has gradually become a research hotspot. According to different processing strategies and their impact on the original image content, image rectangularization methods can be roughly divided into the following three categories: 1) Rectangular frame cropping-based methods: This is the most direct and simple way to form a rectangular image by cropping non-rectangular areas. Although relatively easy to implement, this method inevitably reduces the receptive field of the image, resulting in the loss of some valuable information. 2) Image inpainting-based techniques: Using advanced image inpainting algorithms, reasonable textures or features are filled into the blank areas of the image, thus achieving rectangularization. This method can creatively supplement the missing parts without reducing the original information, but also means introducing some data that did not originally exist. 3) Content-preserving rectangularization methods: This type of method attempts to redistribute useful pixels to blank areas through local warping or geometric transformation without reducing or increasing the original image content, thus achieving rectangularization. Although this process may introduce slight distortion, it generally maintains the authenticity and integrity of the image.

[0004] Traditional image rectangularization techniques have alleviated the problem of irregular boundaries to some extent after image stitching, but still face many challenges and limitations in practical applications: 1) Information loss: Using simple cropping methods to handle image boundaries, although simple to operate, inevitably causes the loss of useful information. For application scenarios that require the preservation of all details, such as medical image analysis or precision engineering measurement, this method is obviously not suitable. 2) Reality impairment: Image inpainting techniques can effectively fill in the boundary gaps and restore the continuity of the image, but the generated content may not match the actual situation, resulting in the reality of the image being affected. This inaccurate information may have serious consequences in some professional fields, such as historical artifact restoration or legal evidence collection. 3) Low efficiency: Content-preserving rectangularization methods aim to minimize information loss and ensure the integrity and accuracy of image content. However, such methods usually require complex calculations and longer processing times, increasing the user's operational burden and are not suitable for real-time or large-scale data processing requirements. 4) Lack of adaptability: Traditional methods have obvious limitations in handling nonlinear scenarios, especially when faced with complex backgrounds or significant changes in perspective. The instability of their performance and the consistency of their output results make it difficult to guarantee the high reliability and accuracy required in applications. SUMMARY

[0005] Therefore, the present application provides a two-stage image rectangularization method and system based on a deep unwrapping warping network, which solves the problems of boundary misplacement and content distortion in traditional image rectangularization techniques and provides a new solution for the image processing field through an efficient two-stage processing process.

[0006] To achieve the above-mentioned purpose, the technical scheme adopted by the present application is as follows:

[0007] In a first aspect, the present application provides a two-stage image rectangularization method based on a deep unwrapping warping network, which includes the following steps:

[0008] S1, constructing a deep unwrapping warping network, the deep unwrapping warping network comprising: a grid predictor and a deformer for rectangularization deformation, and a decorator for rectangularization alignment;

[0009] S2, training the grid predictor and the decorator of the deep unwrapping warping network using two loss functions respectively;

[0010] S3, performing target image rectangularization based on the deep unfolding warping network trained.

[0011] Further, in the S1, the rectangularization deformation is completed by a grid predictor and a deformer; wherein:

[0012] The rectangularized target is represented as:

[0013]

[0014]

[0015] wherein m represents the predicted initial grid position, and the final output is X represents the stitched image, and Y represents the image background, represents a transformation operator, △ represents an interpolation algorithm, and φ(m) is a prior term for constraining the grid deformation offset;

[0016] For a step size s, s = 1, …, S, S represents the number of unfolding steps; the grid predictor is a network for regression, which functions as:

[0017]

[0018] wherein represents the grid predictor, and Δm s represents the grid offset related to the target grid at the s-th step, X s-1 represents the warping result of the input stitched image at the s-1-th step, M s-1 represents the synchronous warping result of the input stitched image mask at the s-1-th step, M represents the initial mask of the input stitched image, and indicates the range of the valid area of the input stitched image;

[0019] The deformer functions as:

[0020]

[0021] wherein X s represents the warping result of the input stitched image X0 at the s-th step, M s is an intermediate result of M0, and M0 represents an initial mask indicating the valid pixel position, represents the deformer, X0 represents the initial input stitched image, and m s-1 represents the predicted grid position at the s-1-th step, and after the step size S, the final output is (X S , M S ).

[0022] Further, in the S1, the decorator includes a partial convolution head and a super-resolution backbone, and the decorator is used to perform rectangularization alignment on the final output of the deformer, which achieves the target as follows:

[0023]

[0024] wherein, θ represents the parameters of the decorator, and represents the element-wise multiplication, represents the decorator.

[0025] Further, for the position of each pixel on the target mesh, the decorator obtains its corresponding position in the input coordinate space by applying the homography transformation matrix.

[0026] Further, in the S2, two loss functions are used to train the mesh predictor and the decorator of the deep unfolding warping network, wherein:

[0027] The loss function for training the mesh predictor is:

[0028]

[0029] wherein, α1, α2, α3 are the weights of different losses, L1 is the cumulative sum of the absolute values between the calculation results of the intermediate unfolding step and the target results, is the perceptual loss between the calculation results of the intermediate unfolding step and the target results, is the cumulative sum of the absolute values between the mask of the calculation results of the intermediate unfolding step and the all-one matrix; is a regularization term;

[0030] The loss function for training the decorator is:

[0031]

[0032] wherein, β is a weighting factor, X D is the final result output by the decorator.

[0033] In a second aspect, the present application further provides a two-stage image rectangularization system based on a deep unfolding warping network, which is applied to the two-stage image rectangularization method based on a deep unfolding warping network described above to perform two-stage image rectangularization based on a deep unfolding warping network. The system comprises:

[0034] a network construction module, configured to construct a deep unfolding warping network, wherein the deep unfolding warping network comprises a mesh predictor and a deformer for rectangularization deformation, and a decorator for rectangularization alignment;

[0035] a network training module, configured to train the mesh predictor and the decorator of the deep unfolding warping network by using two loss functions;

[0036] a network application module, configured to perform target image rectangularization based on the trained deep unfolding warping network.

[0037] In a third aspect, an electronic device is provided, comprising a processor and a memory, the memory storing machine executable instructions executable by the processor, and the processor executes the machine executable instructions to implement the two-stage image rectangularization method based on the depth-unfolded warping network.

[0038] In a fourth aspect, a storage medium is provided, storing one or more programs readable by a computing device, the one or more programs comprising instructions that, when executed by the computing device, cause the computing device to perform the two-stage image rectangularization method based on the depth-unfolded warping network.

[0039] Compared with the prior art, the present application has at least the following beneficial technical effects:

[0040] The present application provides a two-stage image rectangularization method and system based on a depth-unfolded warping network. The depth-unfolded warping network integrates a grid predictor and a deformer to optimize grid estimation in an unfolded manner. It also integrates a decorator to further optimize the image and ensure that the final output is a complete rectangular image. Not only does it solve the problems of boundary misplacement and content distortion in traditional image rectangularization techniques, but it also provides a new solution for the image processing field through an efficient two-stage processing flow. It is beneficial to improve the visual quality and practicality of images, protect key information in original images, and help avoid unnecessary information loss.

[0041] Other features and advantages of the present application will be further described in the following description, and in part will become apparent to those skilled in the art from the following description, or will be learned by practice of the present application. The objects and other advantages of the present application will be realized and achieved by the structure particularly pointed out in the written description and the accompanying drawings.

[0042] The technical solutions of the present application will be further described in detail below with the help of the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other accompanying drawings without creative labor based on these accompanying drawings.

[0044] The accompanying drawings are used to provide a further understanding of the present application, and constitute a part of the specification, together with the embodiments of the present application, to explain the present application, and do not constitute a limitation of the present application.

[0045] FIG. 1 A two-stage image rectangularization method based on deep unfolding warping network provided by the embodiment of the present application is shown in the flowchart.

[0046] FIG. 2 A comparison diagram of the two-stage rectangularization strategy provided by the embodiment of the present application and the traditional one-stage rectangularization strategy is shown.

[0047] FIG. 3 A principle framework diagram of the two-stage image rectangularization provided by the embodiment of the present application is shown.

[0048] FIG. 4 A specific design diagram of the grid predictor, the deformer and the decorator provided by the embodiment of the present application is shown.

[0049] FIG. 5 A comparison result diagram of the method of the present application and other rectangularization methods is shown.

[0050] FIG. 6 Another comparison result diagram of the method of the present application and other rectangularization methods is shown.

[0051] FIG. 7 An action diagram of the decorator provided by the embodiment of the present application is shown.

[0052] FIG. 8 An electronic device structure diagram provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments of the present application.

[0054] In the description of the present application, it should be noted that in some processes described in the specification and drawings, a plurality of operations appear in a specific order, but it should be clearly understood that these operations can be executed or in parallel without the order appearing in the text. In addition, various serial numbers are only for description purposes and cannot be understood as indicating or implying relative importance.

[0055] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0056] Embodiment 1:

[0057] Referring to FIG. 1 As shown in the figure, the embodiment of the application provides a two-stage image rectangularization method based on a deep unfolding warping network, which mainly includes:

[0058] S1, a deep unfolding warping network is constructed, which includes a grid predictor and a deformer for rectangularization deformation, and a decorator for rectangularization alignment;

[0059] S2, the grid predictor and the decorator of the deep unfolding warping network are trained respectively by using two loss functions;

[0060] S3, target image rectangularization is performed based on the trained deep unfolding warping network.

[0061] The working principle and specific implementation mode of the method of the application will be introduced below FIGS. 2-7 As shown in the figure, the working principle and specific implementation mode of the method of the application will be introduced below

[0062] The application provides a two-stage image rectangularization method based on a deep unfolding warping network, which solves the misalignment problem of the boundary of the traditional rectangularization method through two stages of rectangularization deformation and rectangularization alignment, improves the visual quality and practicability of the image, protects the key information in the original image, and avoids unnecessary information loss.

[0063] In this embodiment, the method of the application mainly reduces the distortion and the boundary hole caused by the traditional rectangularization method from two key aspects, maximally retains the original content, and simultaneously realizes high-quality image rectangularization. In the rectangularization deformation stage, grid estimation and deformation technology are used to approach the ideal rectangular solution. The core of this process lies in accurately estimating the initial grid distribution, especially when dealing with irregular boundaries that actually exist in the stitched image. By optimizing the distribution of the grid, the content of the original image can be largely retained, while the obvious distortion and pixel loss caused by deformation are reduced. However, although the efforts of this stage can significantly improve the overall structure of the image, due to the errors in grid estimation and the misalignment between the input image and the grid boundary, some small holes cannot be completely avoided. In addition, during the deformation process, the interpolation operation may also cause the degradation of image quality. In order to solve the above problems, the application introduces a rectangularization alignment stage. The main task of this stage is to further optimize the image quality by enhancing the pixel value. Through the alignment stage, reliable context information can be obtained from adjacent regions to fill in the small holes left by the previous stage and repair the image degradation caused by interpolation. Through the carefully designed alignment algorithm, the image can be ensured to maintain the original content while obtaining a more smooth and natural visual effect.

[0064] The specific design points of the application include the following processes:

[0065] I. Rectification deformation:

[0066] In the embodiments of the present application, the rectification deformation process is first formalized. Given an input stitched image X and its corresponding background Y, the rectification goal can be represented as:

[0067]

[0068] where m represents the predicted initial mesh position, and the final output is represents the transformation operator from the target rigid coordinate space to the coordinate space X based on the local mesh for each pixel. △ represents an interpolation algorithm, such as bilinear interpolation or nearest neighbor interpolation. φ(m) is a prior term that constrains the mesh deformation offset.

[0069] In the design of the rectification deformation stage of the present application, it is unfolded into a series of steps to better estimate m:

[0070]

[0071]

[0072]

[0073] where s is a step indicator, and S represents the number of unfolded steps, which can be taken as 3 here.

[0074] Considering that the deviation of the intermediate results obtained by deformation may accumulate, the original input X0 is used to perform deformation in each step. In addition, the strategy of optimizing the mesh offset related to the target mesh Δm is adopted instead of directly predicting the initial mesh position, and the process becomes:

[0075]

[0076] m s+1 =m s +Δm s ,

[0077]

[0078]

[0079] where m0 is the target mesh, and the predicted variable becomes Δm.

[0080] The present application completes the unfolding process of the aforementioned rectification deformation by a mesh predictor and a deformer w. For step s, s = 1, …, S, the role of the mesh predictor is:

[0081]

[0082] wherein, denotes the mesh predictor, Δm s denotes the mesh offset related to the target mesh at step s, X s-1 denotes the warping result of the input mosaic image at step s-1, M s-1 denotes the synchronized warping result of the input mosaic image mask at step s-1, M denotes the initial mask of the input mosaic image, indicating the range of the valid region of the input mosaic image;

[0083] While the warper is to achieve:

[0084]

[0085] wherein, X s denotes the warping result of the input mosaic image X0at step s, M s is the intermediate result of M0, M0denotes the initial mask indicating the valid pixel positions, denotes the warper, X0denotes the initial input mosaic image, m s-1 denotes the predicted mesh position at step s-1. Since the estimation bias of the previous step is considered in the current step, there is a performance correction in each step, and after step S, the output is (X S , M S ). The network of this stage of the present application unfolds the rectangularization optimization into alternating steps, sharing the parameters of the mesh predictor in each step.

[0086] II. Rectangularization alignment:

[0087] In the embodiment of the present application, through the aforementioned rectangularization warping processing stage, three groups of (X s , M s ) s = 1, 2, 3 have been obtained, wherein X S represents the well-warping processed intermediate image in theory, and M S is its corresponding mask. It is worth noting that in this process, a small white area will appear near the boundary row or column, which inspires the design idea of the rectangularization alignment stage of the present application. Although this challenge is similar in nature to the image inpainting task, the present application emphasizes that it is prohibited to fill these gaps with synthetic patterns or textures, because doing so will deviate from the core idea of rectangularization. Therefore, the present application proposes a decorator To achieve the rectangularization alignment, the main function thereof is to accurately fill the above-mentioned tiny missing parts based on the principle of local connection. In addition, considering that the degradation of image quality is almost inevitable when performing the deformation operation, especially in the selection and performance of the interpolation algorithm. In view of this, the decorator of the present application not only needs to have the above-mentioned accurate filling capability, but also should play a certain image enhancement role when necessary to restore or improve the image quality damaged due to deformation. The decorator The implementation target of the decorator is as follows:

[0088]

[0089] Wherein, θ represents the parameter of the decorator, and represents the element-level multiplication, The decorator is represented as:

[0090] III. Deeply expanded warping network design:

[0091] In the embodiment of the present application, the entire deeply expanded warping network includes three parts: a grid predictor and a deformer for the rectangularization deformation stage, and a decorator for the rectangularization alignment stage, and the specific design structure thereof is as shown in FIG. 4 .

[0092] (1) Grid predictor : For the grid predictor , it is a network for regression:

[0093]

[0094] Wherein, And h and w are the number of grid lattices along the height direction and the width direction, respectively. Since the grid in the target rigid grid is represented by four corner points, there are (h+1)×(w+1) grid points in the grid, and here h=12, w=16.

[0095] (2) Deformer : For the deformer , it is a fixed operator for performing warping and does not need to be trained. Given the four corner points (x1, y1), …, (x4, y4) of any one grid in the target grid, and their corresponding pairs of corner points (u1, v1), …, (u4, v4) on the predicted grid, the homographic transformation matrix thereof can be solved by the following formula:

[0096]

[0097] Then, for the position of each pixel on the target grid, its corresponding position in the input coordinate space can be obtained by applying the homographic transformation matrix.

[0098] (3) Decorator : The decorator of the present application includes a partial convolution head and a super-resolution backbone. Based on the requirement of rectangular alignment, combined with the model characteristics of image inpainting and super-resolution. The partial convolution head is limited to provide reliable pixels with high confidence, rather than generating additional features. Specifically, the present application superimposes a partial convolution on the first layer:

[0099]

[0100] where X' S , M' S are the input and output of the partial convolution head, respectively. X S , M S are the input and output of the sliding window, respectively, and W0 is the weight of the partial convolution kernel. Note that only the mask is used here, and the gradient is not updated forward when training the network. With this design, it can avoid the false dependence caused by empty pixels when extracting deep features. Then, the output X p is divided into two parts X p1 , X p2 , and the final feature is obtained:

[0101]

[0102] where Sigmoid refers to the Sigmoid activation function, and ELU represents the Exponential Linear Unit (ELU) activation function.

[0103] Four, two-stage loss function design:

[0104] In the present application, two stages of rectangularization deformation and rectangularization alignment are involved, so two loss functions are used to train the network in the present application.

[0105] (1) Rectangularization deformation stage. In this stage, the grid predictor is supervised in all the unfolding steps. The total loss function is composed of four parts:

[0106]

[0107] where α1, α2, α3 are the weights of different losses, which are set to 5e-6, 1.0, 1.0, respectively. and are common L1 loss functions, is the absolute value of the cumulative sum of the calculation results X s (s = 1, …, S) of the intermediate unfolding steps and the target results Y. is the absolute value of the cumulative sum of the calculation results mask M san absolute value sum between (s = 1, …, S) and an all-one matrix. is X s a perceptual loss between (s = 1, …, S) and the target result Y, which is mainly calculated by an absolute value sum between features extracted by a VGG network. is a regularization term echoing the term of in the rectangularization deformation target, which limits the excessive deformation of the mesh.

[0108] (2) Rectangularization alignment stage. Since the decorator must pay attention to small blank areas, we use a weighted loss function, i.e.

[0109]

[0110] where β is a weighting factor, which is set to 6.0 here, X D represents the final result output by the decorator.

[0111] Five, experimental verification:

[0112] In the embodiments of the present application, in order to ensure the efficiency and accuracy of the experiment, the computer environment configured as follows is selected by the present application for testing and verification: equipped with Intel Core i7-9700K processor and NVIDIA Tesla V100 graphics processor, system frequency is 3.6GHz, system memory is 64GB, operating system is Ubuntu 20.04. The implementation of the present application depends on the PyTorch deep learning framework, and next, taking the DIR-D dataset as the original data source, the verification process of the present application will be introduced in detail: the grid predictor and the decorator are jointly trained on the public DIR-D dataset through a specially designed loss function, thereby generating an algorithm instance that can be used for actual deployment. The DIR-D dataset is the only publicly available image rectangularization dataset, containing 5839 training images and 519 test samples, and the resolution of all images is 384x512. This dataset provides rich resources for the training and verification of the algorithm, ensuring the robustness of the model in terms of diversity and complexity. For the two stages of rectangularization deformation and rectangularization alignment, the present application adopts the same poly learning strategy, the factor is 0.96, and the initial learning rate is the same, which is 1e-4. Adam is selected as the optimizer for the network, and the weight decay is 1e-5. For the rectangularization deformation stage, the training batch size is 4, and the training step is 200. For the rectangularization alignment stage, the training batch is the same, but 100 training steps are set. For the decorator, the number of residual modules therein is 16, and the basic feature channel number is 64.

[0113] As FIG. 5 and FIG. 6As shown, the quality of the rectangularization of the image is improved by using the method of the present application. As FIG. 5 As shown in part (b), due to the irregular stitching boundary, the original image suffers from almost half of the view angle loss when cropped by the largest inner rectangle. FIG. 5 Part (c) shows the result of image padding, in which there are many repeated mountains and beaches, as indicated by the red arrows in the figure. The obvious increase in these textures is unreliable and contrary to the textures exhibited by the original image, which can be harmful in high-security situations. As FIG. 5 As shown in part (f), the present application can effectively reduce the boundary distortion problem commonly seen in traditional methods. In particular, when dealing with irregular boundaries, the naturalness and continuity of the image can be maintained.

[0114] 2) In order to quantitatively demonstrate the superiority of the method of the present application, three commonly used image quality evaluation indicators were used on the DIR-D dataset: PNSR (peak signal-to-noise ratio), SSIM (structural similarity), and LPIPS (image similarity measure). The results are shown in the following table.

[0115] Table 1 Index results of different algorithms on DIR-D dataset

[0116] Method PNSR↑ SSIM↑ LPIPS↓ Input image reference 11.30 0.3245 41.76 Method of Whom 16.22 0.4173 28.46 Method of Nie et al. 21.12 0.6954 15.57 Method of the present invention 25.03 0.7509 14.86

[0117] 3) The two-stage method of the present application has a remarkable quality improvement effect compared to the single-stage rectangularization method, as FIG. 7 shown. In the rectangularization alignment stage, by using a local connection-based decorator, small holes are filled without introducing false patterns or textures, ensuring the authenticity and accuracy of the image.

[0118] From the description of the above embodiments, those skilled in the art can know that: the embodiments of the present application provide a two-stage image rectangularization method based on a deep unfolding warping network, the present application uses a deep unfolding warping network to unfold the rectangularization into a rectangularization deformation and a rectangularization alignment stage, automatically learns and adjusts the nonlinear deformation in the image through a deep learning model, more accurately restores the structural features of the original image, reduces or even avoids the loss of useful information, and is especially suitable for high-level application scenarios that require all details to be preserved. The method has the following advantages:

[0119] 1) The method is based on a deep unfolding warping network and integrates a mesh predictor and a deformer to optimize the mesh estimation in an unfolding manner.

[0120] 2) After conventional rectangularization processing, there are still subtle boundary problems, and the present application designs an additional rectangularization alignment scheme - a decorator to further optimize the image and ensure that the final output is a complete rectangular image.

[0121] 3) A large number of experimental results verify the effectiveness and superiority of the present application, which proves that it surpasses the existing most advanced method in many indicators. The present application not only solves the problems of boundary misplacement and content distortion in traditional image rectangularization technology, but also provides a new solution for the image processing field through an efficient two-stage processing flow.

[0122] Embodiment 2

[0123] Further, the embodiment of the present application also provides a two-stage image rectangularization system based on a deep unfolding warping network, which is applied to the two-stage image rectangularization method based on the deep unfolding warping network described in the above embodiment 1 to perform two-stage image rectangularization based on the deep unfolding warping network. The system comprises:

[0124] a network construction module, configured to construct a deep unfolding warping network, wherein the deep unfolding warping network comprises a grid predictor and a deformer for rectangularization deformation, and a decorator for rectangularization alignment;

[0125] a network training module, configured to train the grid predictor and the decorator of the deep unfolding warping network respectively by using two loss functions;

[0126] a network application module, configured to perform target image rectangularization based on the trained deep unfolding warping network.

[0127] The two-stage image rectangularization system based on the deep unfolding warping network provided by the embodiment of the present application has the same working principle, technical effects and the above-mentioned method embodiments. For brief description, the part not mentioned in this embodiment can refer to the corresponding content in the above-mentioned method embodiments, which will not be repeated here.

[0128] Embodiment 3

[0129] Referring to FIG. 8 The embodiment of the present application also provides an electronic device for performing two-stage image rectangularization based on a deep unfolding warping network. The electronic device can comprise a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and can further comprise a computer program stored in the memory 11 and executable on the processor 10.

[0130] The processor 10 can be composed of integrated circuits in some embodiments, for example, can be composed of a single packaged integrated circuit, or can be composed of multiple packaged integrated circuits with the same function or different functions, including one or more combinations of central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control core of the electronic device, connects various components of the entire electronic device through various interfaces and lines, and executes programs or modules stored in the memory 11 and calls data stored in the memory 11 to perform various functions and process data of the electronic device.

[0131] Embodiment 4:

[0132] The embodiment of the present application also provides a storage medium having one or more programs readable by a computing device stored thereon, the one or more programs including instructions that, when executed by the computing device, cause the computing device to perform the two-stage image rectangularization method based on the depth-unfolded warping network in embodiment 1.

[0133] Examples of the computer-readable storage medium herein include read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, DVD-ROM, Blu-ray or optical disc memory, hard disk drive (HDD), solid state drive (SSD), card memory (such as a multimedia card, a secure digital (SD) card, or an extreme digital (XD) card), magnetic tape, a floppy disk, a magneto-optical data storage device, an optical data storage device, a hard disk, a solid state disk, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the computer-readable storage medium described above can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc., and in addition, in one example, the computer program and any associated data, data files, and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0134] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0135] It should be noted that the word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding the

[0136] The various embodiments in the specification are described in progressive manner, each embodiment focusing on differences from other embodiments, and the same or similar parts between embodiments can be mutually referred to.

[0137] The above description of disclosed embodiments provides enabling concepts for implementing or using the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A two-stage image rectangularization method based on deep unwrapping warping network, characterized in that: The method comprises the following steps: S1. Constructing a deep unfolding and warping network, wherein the deep unfolding and warping network includes a grid predictor and a deformer for rectangular deformation, and a decorator for rectangular alignment; S2. Using two loss functions to train the grid predictor and decorator of the deep unfolding warp network respectively; S3, rectangularizing the target image based on the trained deep unfolding warping network; In S1, the rectangular deformation is completed by the grid predictor and the deformer; wherein: The rectangular target is expressed as: Where m represents the predicted initial grid position, and the final output is X represents the stitched image, Y represents the image background, represents the transformation operator, △ represents the interpolation algorithm, and φ(m) is the prior term that constrains the mesh deformation offset; For step size s, s = 1, ..., S, S represents the number of steps to be expanded; the grid predictor is a network used for regression, and its role is: Where, represents the grid predictor, Δm s represents the grid offset relative to the target grid at step s, X s-1 represents the distortion result of the input spliced ​​image in step s-1, M s-1 represents the synchronous warping result of the input stitching image mask in step s-1, M represents the initial mask of the input stitching image, and indicates the range of the valid area of ​​the input stitching image; The purpose of the deformer is to: Where, X s Indicates the distortion result of the input spliced ​​image X0 in step s, M s is the intermediate result of M0, which represents the initial mask indicating the valid pixel positions, represents the deformer, X0 represents the initial input spliced ​​image, m s-1 Represents the grid position predicted in the s-1th step. After the number of steps S, the final output is (X S ,M S ).

2. A two-stage image rectangularization method based on a deep unwrapping warping network according to claim 1, characterized in that: In S1, the decorator includes a partial convolution head and a super-resolution backbone. The decorator is used to rectangularize and align the final output of the deformer. Its implementation goals are as follows: Where θ represents the decorator parameter, ⊙ represents element-wise multiplication, Represents a decorator.

3. The two-stage image rectangularization method based on deep unwrapping warping network according to claim 2 is characterized in that: For each pixel position on the target grid, the decorator obtains its corresponding position in the input coordinate space by applying the homography transformation matrix.

4. The two-stage image rectangularization method based on deep unwrapping warping network according to claim 2 is characterized in that: In S2, two loss functions are used to train the grid predictor and decorator of the deep unfolding warp network respectively, where: The loss function for training the grid predictor is: Where α1, α2, α3 are the weights of different losses, It is the cumulative sum of the absolute values ​​between the calculation results of the intermediate expansion steps and the target result. is the perceptual loss between the calculation result of the intermediate expansion step and the target result, It is the cumulative sum of the absolute values ​​between the result mask and the all-one matrix in the intermediate expansion step; is a regularization term; The loss function for training the decorator is: Where β is a weighting factor, X D It is the final result of the decorator output.

5. A two-stage image rectangularization system based on a deep unwrapping warping network, characterized by: When applied, a two-stage image rectangularization method based on a deep expansion warping network according to any one of claims 1 to 4 is executed to perform two-stage image rectangularization based on a deep expansion warping network, and the system includes: A network construction module for constructing a deep unfolding and warping network, wherein the deep unfolding and warping network includes a grid predictor and a deformer for rectangular deformation, and a decorator for rectangular alignment; A network training module, which uses two loss functions to train the grid predictor and decorator of the deep unfolding warp network respectively; The network application module performs rectangularization of the target image based on the trained deep unfolding and distortion network.

6. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement a two-stage image rectangularization method based on a deep unfolding warping network as described in any one of claims 1 to 4.

7. A storage medium having stored thereon one or more programs readable by a computing device, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform a two-stage image rectangularization method based on a deep unfolding warp network according to any one of claims 1 to 4.

Images