Template-guided deformed image alignment correction and mask generation method

Abstract

The invention relates to the technical field of image processing, in particular to a template-guided deformed image alignment correction and mask generation method, and the method comprises the steps: obtaining a template image and a corresponding deformed photographed image, and carrying out the size normalization processing. And then inputting the normalized image into a preset neural network, and synchronously generating a prediction mask image and a prediction offset matrix through feature extraction, feature fusion and up-sampling processing. And resampling the deformation photographing image based on the prediction offset matrix to obtain a preliminary alignment image. And after binarization processing is carried out on the predicted mask image, pixel-level multiplication is carried out on the predicted mask image and the preliminary alignment image, and finally a result image which is in pixel-level alignment with the template image and a binary mask which accurately represents an effective area are generated. According to the method, end-to-end deformation image correction is realized, a complete image and a local image can be processed, and the problem of alignment deviation caused by lack of template guidance and global reference in a traditional method is effectively solved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a template-guided method for aligning and correcting deformed images and generating masks. Background Technology

[0002] In fields such as document digitization, industrial quality inspection, and medical imaging, image correction and alignment are crucial for achieving standardized processing and analysis. Taking document image processing as an example, users often need to correct handheld photos of deformed documents (such as images of bent or perspective-distorted paper) into flat, directly comparable, or archived standard images. However, existing technologies have significant limitations in handling such problems:

[0003] On the one hand, most traditional image correction methods rely on the features of a single image (such as edge detection and perspective transformation) and lack template guidance mechanisms. For example, perspective correction algorithms based on homography matrices can only handle global affine deformations. When there are nonlinear deformations such as wrinkles or bends in the local area of ​​the document, the corrected image and the local area of ​​the template (such as a standard document scan) are prone to scaling inconsistencies or positional shifts, resulting in pixel-level alignment failure.

[0004] On the other hand, existing solutions have high requirements for the integrity of the input image, usually requiring users to take images containing the entire document page. When the input is a partial document image (such as only taking a picture of a part of the document), traditional algorithms lack a global structural reference and cannot accurately infer the spatial location of the missing area, resulting in inaccurate global coordinate mapping between the corrected image and the template, and even content distortion.

[0005] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this invention is to provide a template-guided method for aligning and correcting deformed images and generating masks. This method aims to solve the technical problems of traditional image correction methods, which lack a template-guided mechanism and have high requirements for the integrity of the input image, and are prone to pixel-level alignment failure and global coordinate mapping inaccuracy when processing locally nonlinear deformed images.

[0007] To achieve the above objectives, the present invention provides a template-guided method for aligning and correcting deformed images and generating masks, the method comprising:

[0008] Obtain the template image and the corresponding deformed photographic image;

[0009] The template image and the deformed photograph image are subjected to size normalization processing to obtain a normalized image;

[0010] The normalized image is input into a preset neural network, and through feature extraction, feature fusion and upsampling processing, a prediction mask image and a prediction offset matrix are generated.

[0011] The deformed photographic image is resampled based on the predicted offset matrix to generate a preliminary aligned image;

[0012] The predicted mask image is binarized to obtain a binary mask image;

[0013] The binary mask image is multiplied pixel-wise with the preliminary aligned image to generate the final aligned image and the effective region mask.

[0014] Optionally, the step of normalizing the size of the template image and the deformed photographic image to obtain a normalized image includes:

[0015] The template image and the deformed photograph image are scaled to a preset uniform size using bilinear interpolation;

[0016] During the scaling process, an edge-filling strategy is adopted to fill image areas that are smaller than the target size with black pixels to obtain a normalized image, ensuring that the normalized image retains the main content of the original image and is of consistent size.

[0017] Optionally, the step of inputting the normalized image into a preset neural network and generating a prediction mask image and a prediction offset matrix through feature extraction, feature fusion, and upsampling processing includes:

[0018] A preset neural network is used to perform convolution, pooling and activation function processing on the template image part and the deformed photograph image part in the normalized image to obtain the template feature map and the photograph feature map.

[0019] The template feature map and the photographed feature map are concatenated along the channel dimension to generate a fused feature map;

[0020] The fused feature map is downsampled sequentially to extract deep features, and then upsampled to restore detailed features, outputting a three-channel result;

[0021] The three-channel results are separated, with the first channel serving as the prediction mask image, and the second and third channels being normalized and used as the width and height offsets, respectively, which together constitute the prediction offset matrix.

[0022] Optionally, the preset neural network includes a U-Net network structure, and the step of sequentially performing downsampling on the fused feature map to extract deep features, and then performing upsampling to recover detailed features, includes:

[0023] The U-Net network structure is used as the backbone network. In the downsampling stage, the feature map resolution is gradually reduced through convolutional layers and pooling layers, and in the upsampling stage, the feature map resolution is gradually increased through transposed convolutional layers and skip connections.

[0024] The Swin-Transformer module is integrated into the downsampling and upsampling stages of the U-Net network to enhance global feature modeling capabilities.

[0025] Optionally, the step of resampling the deformed photographic image based on the predicted offset matrix to generate a preliminary aligned image includes:

[0026] The sampling grid coordinates are generated based on the width offset and the height offset;

[0027] The deformation photograph image is resampled using bilinear interpolation using the sampling grid coordinates to obtain the preliminary aligned image.

[0028] Optionally, the step of binarizing the predicted mask image to obtain a binary mask image includes:

[0029] A preset threshold is applied to binarize and determine the value of each pixel in the predicted mask image;

[0030] If the pixel value is greater than the preset threshold, it is marked as 1 to indicate a valid region; otherwise, it is marked as 0 to indicate an invalid region, and the binary mask image is generated.

[0031] Optionally, the size of the predicted mask image and the predicted offset matrix is ​​1 / 4 of the size of the input image;

[0032] The step of multiplying the binary mask image and the preliminary aligned image pixel-level to generate the final aligned image and the effective region mask includes:

[0033] Perform a pixel-level multiplication operation between the binary mask image and the preliminary aligned image, so that the pixel values ​​of the invalid areas in the final aligned image are set to 0, while the content of the valid areas is retained, thereby obtaining the final aligned image;

[0034] The binary mask image of the invalid region is used as the mask of the valid region.

[0035] Optionally, separating the three-channel results, wherein the first channel is used as the prediction mask image, and the second and third channels are normalized and used as offsets in the width and height directions, respectively, includes:

[0036] Apply a normalization function to the pixel values ​​of the second and third channels to map the offsets to the range of [-1, 1], and obtain the normalized width and height offsets;

[0037] The normalization function is: norm(x) = [2 × (x - min)] / (max - min) - 1, where min and max are the minimum and maximum pixel values ​​in the channel, respectively.

[0038] Furthermore, to achieve the above objectives, the present invention also provides a template-guided deformed image alignment correction and mask generation device, the device comprising: a memory, a processor, and a template-guided deformed image alignment correction and mask generation program stored in the memory and executable on the processor, the template-guided deformed image alignment correction and mask generation program being configured to implement the steps of the template-guided deformed image alignment correction and mask generation method as described above.

[0039] In addition, to achieve the above objectives, the present invention also provides a medium storing a template-guided deformed image alignment correction and mask generation program, wherein when the template-guided deformed image alignment correction and mask generation program is executed by a processor, it implements the steps of the template-guided deformed image alignment correction and mask generation method as described above.

[0040] This invention provides a template-guided method for aligning and correcting deformed images and generating masks. By introducing prior structural information from the template image and combining it with the feature fusion capabilities of a neural network, the method achieves global semantic alignment and local detail matching between the deformed image and the template. This solves the scaling inconsistencies and positional offsets caused by the lack of reference in traditional single-image correction methods, ensuring a one-to-one correspondence between the pixel-level spatial coordinates of the aligned image and the template. Based on a Unet and Swin-Transformer backbone network, through multi-level feature extraction (downsampling to capture global structure, upsampling to restore local details) and long-distance dependency modeling (Transformer module), it can effectively handle nonlinear deformations such as wrinkles and bends, while also supporting input of local document images and mask generation via templates. The global structure guides the inference of the spatial location of missing regions, avoiding content distortion caused by incomplete input in traditional methods. The end-to-end network jointly outputs the predicted offset matrix and the mask image. By multiplying the binarized mask and the aligned image at the pixel level, invalid content such as background noise and occluded areas are accurately removed, retaining only the valid regions that match the template (the "1" marked regions in the mask). This solves the problem of ambiguous definition of valid regions caused by the separation of mask generation and alignment tasks in traditional methods, improving the reliability of subsequent analysis. Normalization processing unifies the input size. The neural network completes feature extraction, deformation field prediction and mask generation simultaneously in one forward propagation, avoiding the cumbersome process and computational redundancy of traditional multi-stage methods (such as alignment before segmentation), significantly improving processing efficiency and making it suitable for scenarios with high real-time requirements. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating an embodiment of the template-guided deformation image alignment correction and mask generation method of the present invention;

[0042] Figure 2 This is a schematic diagram of a specific scenario of an embodiment of the template-guided deformation image alignment correction and mask generation method of the present invention.

[0043] Figure 3 This is an image correction effect diagram of an embodiment of the template-guided deformed image alignment correction and mask generation method of the present invention;

[0044] Figure 4 This is a Unet network structure diagram of an embodiment of the template-guided deformation image alignment correction and mask generation method of the present invention.

[0045] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0046] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0047] Reference Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the template-guided deformation image alignment correction and mask generation method of the present invention, which presents an embodiment of the template-guided deformation image alignment correction and mask generation method of the present invention.

[0048] In one embodiment, the template-guided deformed image alignment correction and mask generation method includes:

[0049] In one embodiment, a template-guided method for deformed image alignment correction and mask generation is provided, the method comprising:

[0050] Step S100: Obtain the template image and the corresponding deformation photograph image.

[0051] The template image can be a reference image representing the standard form of the target document, containing complete structural layout and semantic information. It provides prior structural information as a global reference benchmark for deformed image alignment, guiding nonlinear deformation recovery and spatial mapping inference. It is understood that the template image can be generated by a high-precision scanning device or retrieved from a standard document library, and is typically a distortion-free, clearly aligned image. In this embodiment, the template image can be one or more of the following, including but not limited to standard scanned images, electronically rendered images, and composite template images. The deformed photographic image can be a document image with geometric deformation taken by the user, possibly containing bending, wrinkles, perspective distortion, or partial occlusion. It is used as input for processing and needs to be corrected to achieve spatial alignment with the template image and extract the effective content area. It is understood that the deformed photographic image can be acquired by a mobile device camera in an uncontrolled environment, and the imaging process is affected by the shooting angle, lighting, and the physical state of the paper. For example, the deformed photographic image can be one or more of the following, including but not limited to perspective distortion images, partially bent images, and partially occluded images.

[0052] Obtaining the template image and the corresponding deformed image can be achieved by reading a pair of images from a storage medium or communication interface, one being the standard template and the other the deformed image to be corrected. Furthermore, obtaining the template image and the corresponding deformed image can be achieved by loading the image pair from a specified path on the local file system or by receiving uploaded image data packets in real time via an API interface. This fulfills the basic steps for initiating the processing flow, ensuring the system has the necessary input data.

[0053] Step S200: Perform size normalization processing on the template image and the deformed photographed image to obtain a normalized image.

[0054] The normalized image can be a template image or a deformed photograph image that has been resized and standardized to a preset resolution and scale. This is used to eliminate feature scale bias caused by inconsistent resolution of the original images, thereby improving the consistency of model input and generalization ability. Understandably, the normalized image can be obtained using interpolation algorithms such as bilinear or bicubic interpolation to scale the original image to the fixed size required for network input.

[0055] Size normalization is performed on the template image and the deformed image to obtain a normalized image. This can be achieved by independently scaling both images to a uniform size, such as 256×256, while maintaining the aspect ratio, or by cropping and padding. Furthermore, size normalization of the template image and the deformed image can be achieved by using a scaling strategy that maintains the aspect ratio with edge zero-padding, or by forcibly stretching to a fixed size. This ensures a uniform input scale and avoids the impact of resolution differences on the stability of feature extraction.

[0056] In step S300, the normalized image is input into a preset neural network, and a prediction mask image and a prediction offset matrix are generated through feature extraction, feature fusion and upsampling processing.

[0057] The pre-defined neural network can be a deep learning model architecture used to jointly predict the offset matrix and the mask image, enabling multi-task joint inference and simultaneously completing feature extraction, deformation modeling, and mask generation. It is understood that the pre-defined neural network can be constructed based on a Unet encoder-decoder structure and a Swin-Transformer module, learning the deformation field and effective region distribution patterns through a large number of paired samples during the training phase. In this embodiment, the pre-defined neural network can include, but is not limited to, one or more of the following: a Unet-Swin hybrid encoder, a Transformer enhanced decoder, and a multi-scale fusion network. The pre-defined neural network works in conjunction with the normalized image: receiving the normalized image as input; and works in conjunction with the predicted offset matrix and the predicted mask image: outputting two key intermediate results.

[0058] Feature extraction is the process of capturing multi-level visual features from a normalized image to construct a feature space for subsequent matching and fusion, supporting global structure recognition and local detail perception. Understandably, feature extraction can be achieved by downsampling through convolutional layers and Transformer blocks to extract feature representations at different abstract levels, such as edges, textures, and semantics.

[0059] Feature fusion can be an operation that interactively integrates the corresponding hierarchical features of a template image and a deformed photographic image. This strengthens the modeling of the correspondence between the template and the input, improving the spatial inference capability for missing regions. Understandably, feature fusion can be achieved by using cross-attention mechanisms or feature concatenation operations at various levels of the encoder to establish cross-image semantic associations.

[0060] Upsampling can be a process of gradually restoring spatial resolution during the network decoding stage. It is used to restore fine deformation fields and accurate mask boundaries, ensuring that the output has pixel-level precision. Understandably, upsampling can be achieved by combining transposed convolutions or interpolation with skip connections, fusing high-level semantics and low-level detail features to reconstruct a high-resolution output.

[0061] The predicted offset matrix can be a two-dimensional vector field describing the displacement required for each pixel from the deformed image to the template image coordinate system. It guides the resampling process, achieving non-linear alignment of the deformed image to the template space. Understandably, the predicted offset matrix can be obtained directly from the output of a pre-defined neural network decoder, with each position containing offsets in both the horizontal and vertical directions. For example, the predicted offset matrix can be one or more of the following, including but not limited to dense optical flow offset fields, sparse control point offset maps, and piecewise affine parameter fields.

[0062] The predictive mask image can be a grayscale image reflecting the probability distribution of valid content regions in the deformed image. It serves as a basis for subsequent binarization, defining which regions should be retained in the final aligned image. Understandably, the predictive mask image can be obtained by a pre-defined neural network, with values ​​ranging from 0 to 1, representing the probability that each pixel belongs to a valid region.

[0063] Inputting a normalized image into a pre-defined neural network, and generating a prediction mask image and prediction offset matrix through feature extraction, feature fusion, and upsampling, can be achieved by feeding the normalized image as a tensor into the front end of the neural network, triggering the forward propagation process. Furthermore, inputting the normalized image into the pre-defined neural network can be achieved by inputting multiple pairs of images simultaneously in a single-channel or three-channel stacked manner, or by using batch processing, thereby activating the feature extraction and fusion mechanisms within the network. Generating the prediction mask image and prediction offset matrix through feature extraction, feature fusion, and upsampling can be achieved by the network encoder extracting multi-scale features, and the decoder gradually generating two output branches through fusion and upsampling: one output offset field and one output mask probability map. Furthermore, generating the prediction mask image and prediction offset matrix through feature extraction, feature fusion, and upsampling can be achieved by using a shared encoder + dual decoder head structure to output two types of results respectively, or by introducing an attention gating mechanism during the decoding process to optimize the fusion weights, thereby achieving end-to-end joint prediction and ensuring consistency in alignment and segmentation tasks.

[0064] Step S400: Resample the deformed photographed image based on the predicted offset matrix to generate a preliminary aligned image.

[0065] The predicted offset matrix can be used to guide the resampling process, enabling non-linear alignment of the deformed image to the template space.

[0066] Resampling can be a spatial transformation operation that recalculates the pixel value distribution based on the predicted offset matrix, used to generate a preliminary correction result that is spatially aligned with the template image. Understandably, resampling can be achieved using bilinear or nearest-neighbor interpolation methods, filling the original image pixels to new coordinate positions based on the inverse mapping.

[0067] The initial aligned image can be an intermediate image obtained after resampling, which has corrected geometric deformation but still contains background or noise. It is used as one of the inputs for pixel-level multiplication to generate the final clean aligned image. Understandably, the initial aligned image can be obtained by applying a resampling operation to the original deformed photographic image.

[0068] Resampling deformed images based on the predicted offset matrix to generate a preliminary aligned image can be achieved by rearranging the pixels of the original deformed image to a new coordinate grid using the spatial mapping relationship defined by the predicted offset matrix. Furthermore, resampling deformed images based on the predicted offset matrix can be achieved by using forward mapping combined with interpolation to fill blank areas or by using reverse mapping to prevent pixel overlap and holes, thereby achieving the technical effect of geometric correction from deformation space to template space.

[0069] Step S500: Binarize the predicted mask image to obtain a binary mask image.

[0070] The predicted mask image serves as the basis for subsequent binarization, defining which regions should be retained in the final aligned image. Binarization transforms the predicted mask image with continuous values ​​into a binary image containing only 0s and 1s, clearly distinguishing valid and invalid regions and creating sharp mask boundaries. It's understood that the binarization process can be achieved by setting a threshold, such as 0.5, where values ​​above the threshold are set to 1 (foreground) and values ​​below are set to 0 (background). The binary mask image can be a mask image containing only 0 and 1 pixel values, identifying the ultimately retained content regions, used for pixel-level multiplication, and masking irrelevant background and occluded areas. It's understood that the binary mask image can be obtained by performing binarization on the predicted mask image.

[0071] Binarizing the predicted mask image yields a binary mask image by setting a fixed or adaptive threshold, converting grayscale values ​​into binary labels. Furthermore, binarization of the predicted mask image can be achieved by using a globally fixed threshold, such as 0.5, for segmentation, or by applying a local adaptive thresholding algorithm to handle uneven lighting conditions. This results in the generation of clear boundaries, facilitating subsequent precise region selection.

[0072] Step S600: Multiply the binary mask image and the preliminary aligned image pixel by pixel to generate the final aligned image and the effective region mask.

[0073] In this process, the binary mask image can be used to participate in pixel-level multiplication, masking irrelevant backgrounds and occluded areas. The preliminary aligned image can be used as one of the inputs for pixel-level multiplication. Pixel-level multiplication is an operation that multiplies the binary mask image and the preliminary aligned image pixel by pixel, used to remove invalid areas and output a clean, valid content image. It can be understood that pixel-level multiplication can be achieved by performing a multiplication operation on each pixel in the same spatial coordinates: if the mask value is 1, the original value is retained; if it is 0, it is set to zero.

[0074] The final aligned image can be a standardized output image that is perfectly aligned with the template image after removing background interference. It can be used for archiving, comparison, or downstream analysis tasks. Understandably, the final aligned image can be generated through pixel-level multiplication operations, resulting from the combined action of the initial aligned image and the binary mask image. The effective region mask can be a binary image recording the location information of the ultimately retained region. It has the same spatial structure as the final aligned image and is used to assist subsequent processing systems in identifying the effective content range, supporting region selection or feature extraction. Understandably, the effective region mask can be generated synchronously with the final aligned image, and its data originates from the binary mask image.

[0075] Multiplying the binary mask image and the initial aligned image pixel-by-pixel to generate the final aligned image and the effective region mask can be performed by executing the multiplication operation pixel by pixel in the same spatial coordinates, outputting the cropped aligned image and the corresponding mask. Furthermore, the pixel-by-pixel multiplication of the binary mask image and the initial aligned image can be performed by traversing pixels line by line in CPU memory or efficiently executed on the GPU using parallel tensor operations, thereby achieving the technical effect of content cleansing and outputting standardized results that can be used for analysis.

[0076] Taking the digital archiving of medical reports as an example, the template-guided deformation image alignment correction and mask generation method in this embodiment can be used when medical staff take photos of patients' paper examination reports with their mobile phones. Due to the slight curling of the paper and the fact that only part of the content is captured, traditional OCR systems have difficulty accurately recognizing the images. This solution uses a standard hospital report template as the template image. After inputting this partially deformed photo, the system infers the complete page structure through template guidance, automatically corrects the curvature, fills in the spatial position of missing areas, outputs a flat image aligned with the template, and accurately crops out the table and text areas, excluding parts obscured by fingers, thereby ensuring the accuracy of subsequent text extraction and structured parsing.

[0077] For example, in the scenario of on-site verification of industrial drawings, the template-guided deformation image alignment correction and mask generation method of this embodiment can be used when engineers photograph local areas of old drawings during equipment maintenance. These drawings may contain aging wrinkles and perspective tilt. The system uses the original standard drawing exported from CAD as a template, combines it with the local blurred image, utilizes neural networks to capture the correspondence of subtle lines, predicts complex nonlinear deformation fields, completes pixel-level alignment, and generates an effective area mask to filter stains and shadows. This allows the comparison software to accurately identify modification marks or assembly deviations, improving on-site maintenance efficiency.

[0078] In one embodiment, the template image and the deformed photograph image are subjected to size normalization processing to obtain a normalized image, including:

[0079] The template image and the deformed photograph image are scaled to a preset uniform size using bilinear interpolation.

[0080] Bilinear interpolation can be an image resampling method based on a weighted average of four nearest-neighbor pixels. It can be used to achieve a smooth transition during image scaling, reducing jaggedness and information loss introduced by geometric deformation. Furthermore, the operating principle of bilinear interpolation can be explained in context: a 2×2 neighborhood of source pixels is selected around the target pixel coordinates, and bilinear weighted interpolation is performed based on their distance to calculate the output value. For example, bilinear interpolation can include, but is not limited to, one or more of unidirectional linear interpolation, bidirectional linear interpolation, and multi-channel synchronous interpolation. The preset uniform size can be a fixed input resolution required by the neural network model, such as 256×256 or 512×512, which can be used to ensure that all input images have the same spatial dimension, meeting the requirements of deep learning models for tensor shape consistency. Furthermore, the method of obtaining the preset uniform size can be explained in context: it is determined by the network structure design, usually determined during the training phase and enforced during inference.

[0081] In this embodiment, the template image and the deformed photograph image are scaled to a preset uniform size using bilinear interpolation. This can be achieved by applying the bilinear interpolation algorithm to each of the two images separately, adjusting their dimensions to the size required for the network input. Furthermore, this operation can be implemented by (e.g., scaling proportionally first and then cropping to the target size, or directly stretching to the target size without maintaining the aspect ratio), thereby achieving image scale standardization and reducing feature deviations caused by resolution differences.

[0082] During the scaling process, an edge-filling strategy is adopted to fill image areas that are smaller than the target size with black pixels to obtain a normalized image.

[0083] Edge padding can be a method that adds extra pixels to the image boundaries to achieve a specified resolution when the original image, after scaling, is still smaller than the target size. This can be used to maintain the overall size consistency of the normalized image and avoid content loss due to cropping. Furthermore, the operating principle of edge padding can be explained in context: it adds rows or columns of pixels in the four directions (top, bottom, left, and right) of the image, with the filling position calculated based on the difference between the target size and the current size. In a specific embodiment, edge padding can be implemented in conjunction with zero padding: specifically through zero padding.

[0084] Zero-padding can be an operation that fills blank areas of an image with pixels having a value of 0. It can be used to augment size without altering the original content, while making the filled area easily identifiable as invalid background in the network. Furthermore, the operating principle of zero-padding can be explained in context: the channel value of the newly added pixel is set to 0 and integrated into the boundary position of the original image matrix. For example, zero-padding can include one or more of the following: top-left padding, center-symmetric padding, and uniform padding around the edges. Black pixels can be pixels with a color channel value of (0, 0, 0), appearing as pure black in grayscale or color images. They can be used as a visual representation of zero-padding, clearly identifying areas without actual image content. Furthermore, the method of obtaining black pixels can be explained in context: in digital image representation, they are generated by setting the R, G, and B channel values ​​to the minimum value of 0.

[0085] The main content of the original image can be the effective region containing the main semantic information, such as key visual elements like document text and charts, which can provide a valid data foundation for subsequent feature extraction and template matching. Furthermore, the acquisition method of the main content of the original image can be explained in context: it is formed by focusing on the target object during the shooting process, and should be preserved as completely as possible during normalization processing.

[0086] In this embodiment, an edge-filling strategy is employed during the scaling process. Image areas smaller than the target size are filled with black pixels to obtain a normalized image. This can be achieved by determining whether the scaled image reaches the preset size; if there are missing areas, black pixels are added around them until the size matches. Furthermore, this operation can be implemented by (e.g., placing the original image in the center of the filled image, or aligning it to the upper left corner and then expanding the filling towards the lower right), thereby ensuring that all normalized images have completely consistent spatial dimensions while avoiding the introduction of non-realistic content that could interfere with semantic understanding.

[0087] In one embodiment, a normalized image is input into a preset neural network, and a prediction mask image and a prediction offset matrix are generated through feature extraction, feature fusion, and upsampling processing, including:

[0088] By performing convolution, pooling, and activation function processing on the template image portion and the deformed photographed image portion of the normalized image using a preset neural network, template feature maps and photographed feature maps are obtained.

[0089] The template image portion can be a data region identified as the template image in the normalized image, used as a standard reference input in the independent feature extraction process. It is understood that the template image portion can be obtained by distinguishing it from other images through channel stacking. Furthermore, the template image portion can work in conjunction with the deformed image portion to ensure an independent processing path when entering the network. The deformed image portion can be a data region identified as the deformed image to be corrected in the normalized image, used to provide the original content information to be corrected. It is understood that the deformed image portion can be obtained by organizing it alongside the template image. Furthermore, the deformed image portion can include, but is not limited to, one or more of the following: slight distortion feature maps, local occlusion feature maps, nonlinear bending feature maps, etc.

[0090] Convolution operations can be mathematical operations that calculate weighted sums by sliding convolution kernels across image space, used to capture local correlations in an image. It's understandable that convolution operations can be obtained by extracting local spatial features through multiple learnable filters. Furthermore, convolution operations can be combined with pooling operations to reduce computational burden through downsampling. Pooling operations can be aggregation operations that downsample local regions, used to enhance feature translation invariance. It's understandable that pooling operations can be obtained using max pooling or average pooling. Furthermore, pooling operations can be combined with activation function processing to assist in deep feature extraction. Activation function processing can be an operation that applies nonlinear transformations to neuron outputs, used to improve the network's ability to fit complex mappings. It's understandable that activation function processing can be obtained by introducing nonlinearity using functions such as ReLU and Sigmoid. Furthermore, activation function processing can be combined with convolution operations to break the limitations of linear combinations.

[0091] Template feature maps are feature representations generated by a pre-defined neural network through convolution, pooling, and activation function processing on a normalized template image portion. They are used to preserve the semantic structural features of the template image. It is understood that template feature maps can be obtained by progressively extracting structural information such as edges, corners, and textures through multi-layer convolutional neural networks. Furthermore, template feature maps can include low-level edge feature maps, mid-level texture feature maps, and high-level semantic layout feature maps. Image feature maps are feature representations generated by a pre-defined neural network through convolution, pooling, and activation function processing on a normalized deformed image portion. They are used to reflect the content structure of the deformed image. It is understood that image feature maps can be obtained by independently extracting visual patterns from the input image using the same preprocessing path as template feature maps. Furthermore, image feature maps can include slight distortion feature maps, local occlusion feature maps, and non-linear curvature feature maps.

[0092] By performing convolution, pooling, and activation functions on the template image portion and the deformed photographic image portion of the normalized image using a pre-defined neural network, template feature maps and photographic feature maps are obtained. This can be achieved by feeding the two images into a shared or independent sequence of convolutional layers, performing convolution calculations, pooling dimensionality reduction, and nonlinear activation sequentially to generate their respective multi-level feature maps. Furthermore, this operation can be implemented by using a weight-sharing dual-branch structure to process the two images separately, thereby extracting their independent structural features, preserving the differences between the two inputs, and providing a foundation for subsequent fusion.

[0093] The template feature map and the photographed feature map are concatenated along the channel dimension to generate a fused feature map.

[0094] In this context, the channel dimension can be a dimension axis in a feature tensor representing different feature types or sources, used to organize multi-channel features. Understandably, the channel dimension is typically obtained after the height and width in deep learning frameworks. Furthermore, the channel dimension can support parallel representation and operations on multi-source or multi-type features. Concatenation is an operation that joins two tensors along a certain dimension to form a higher-dimensional tensor, used to achieve feature fusion. Understandably, concatenation can be obtained by performing a concatenation operation along the channel dimension. Furthermore, concatenation can preserve the integrity of the original information.

[0095] A fused feature map can be a joint feature tensor formed by concatenating a template feature map and an image feature map along the channel dimension, used to construct semantic associations across images. Understandably, the fused feature map can be obtained by merging two feature maps along the channel axis at the same spatial resolution. Furthermore, the fused feature map can work in conjunction with the template feature map and the image feature map, relying on their outputs as input; and it can work in conjunction with downsampling, serving as the starting data basis for downsampling operations.

[0096] The template feature map and the captured feature map are concatenated along the channel dimension to generate a fused feature map. This can be achieved by merging two tensors along the channel axis on feature maps at the same level, forming a new feature map with double the number of channels. Furthermore, this operation can be implemented by performing cross-image concatenation at each encoding level, thereby establishing an explicit correspondence between the template and the input and promoting semantic alignment.

[0097] The fused feature map is downsampled sequentially to extract deep features, and then upsampled to restore detailed features, outputting a three-channel result.

[0098] Downsampling can be a process of progressively reducing the spatial resolution of feature maps to extract deep semantic features. Understandably, downsampling can be achieved by increasing the convolution stride or compressing the spatial size through pooling operations. Furthermore, downsampling can expand the receptive field. Deep features can be high-order abstract feature representations obtained after multiple downsampling layers, used to support the recognition and modeling of overall deformation trends. Understandably, deep features can be generated deep within the network encoder. Furthermore, deep features can reflect the overall semantics of the image rather than its detailed texture. Upsampling can be a process of progressively restoring the spatial resolution of feature maps to reconstruct fine structures. Understandably, upsampling can be achieved by expanding the spatial size through transposed convolutions, interpolation, etc. Furthermore, upsampling can be combined with skip connections to supplement details.

[0099] Detail features can be low-level feature information with high spatial resolution, used to assist in recovering fine details such as edges and text. Understandably, detail features can be obtained from shallow layers of the network or passed to the decoder via skip connections. Furthermore, detail features can collaborate with deep features to achieve the fusion of global and local features. The three-channel result can be a three-dimensional tensor output from the network's decoding end, containing the prediction results of three independent encoding tasks. Understandably, the three-channel result can be obtained by upsampling back to the input size and then generating a single tensor. Furthermore, the three-channel result can include mask branch channels, horizontal offset branch channels, vertical offset branch channels, etc.

[0100] The fused feature map is downsampled sequentially to extract deep features, then upsampled to restore detailed features, outputting a three-channel result. This can be achieved by first compressing the spatial dimensions through multi-level convolution and pooling to obtain high-level semantics, then gradually restoring the resolution through transposed convolution and interpolation, finally outputting a three-channel tensor. Furthermore, this operation can achieve detail restoration by combining skip connections with a U-Net architecture, thus balancing global structural understanding with local deformation restoration and supporting high-quality output.

[0101] The three-channel results are separated, with the first channel serving as the prediction mask image, and the second and third channels being normalized and used as the width and height offsets, respectively, which together form the prediction offset matrix.

[0102] The first channel can be a specific output channel in the three-channel result that undertakes the mask prediction task, used directly as the data source for the predicted mask image. It can be understood that the first channel can be obtained by specifying a channel in the final output layer to specifically output the probability value of a pixel belonging to the valid region. Furthermore, the first channel can use a sigmoid function constraint to ensure its probabilistic nature. The second channel can be an output channel in the three-channel result that encodes horizontal offset information, used to form the X component of the predicted offset matrix after normalization. It can be understood that the second channel can be obtained by setting an independent channel at the end of the network to regress the horizontal displacement of each pixel. Furthermore, the second channel can use learnable normalization parameters to dynamically adjust the offset scale.

[0103] The third channel can be the output channel encoding the vertical offset information in the three-channel result, used to form the Y component of the predicted offset matrix after normalization. It can be understood that the third channel can be obtained by setting an independent channel at the end of the network to regress the vertical displacement of each pixel. Furthermore, the third channel can be obtained using a learnable normalized image, which can be obtained by normalizing the second and third channels; positive values ​​indicate a right shift, and negative values ​​indicate a left shift. Furthermore, the width offset can constitute the horizontal coordinate component of the predicted offset matrix, guiding the horizontal position adjustment during the resampling process. The height offset can represent the distance a pixel needs to move from the deformed image to the template image on the vertical axis; it can be obtained by normalizing the third channel, with positive values ​​indicating a downward shift and negative values ​​indicating an upward shift. Furthermore, the height offset can constitute the vertical coordinate component of the predicted offset matrix, guiding the vertical position adjustment during the resampling process.

[0104] The three-channel results are separated, with the first channel serving as the prediction mask image, and the second and third channels, after normalization, serving as the width and height offsets, respectively. Together, they form the prediction offset matrix. This can be achieved by splitting the three-channel tensor output by the network into three independent outputs. Furthermore, this operation can be achieved by dynamically adjusting the offset scale using learnable normalization parameters, thereby enabling decoupled output across multiple tasks and ensuring that the physical meaning of the mask and the offset field is clear and numerically controllable.

[0105] In one embodiment, the preset neural network includes a U-Net network structure, which sequentially performs downsampling on the fused feature map to extract deep features, and then performs upsampling to recover detailed features. This includes: using the U-Net network structure as the backbone network, wherein the downsampling stage gradually reduces the feature map resolution through convolutional layers and pooling layers, and the upsampling stage gradually increases the feature map resolution through transposed convolutional layers and skip connections; and integrating a Swin-Transformer module in the downsampling and upsampling stages of the U-Net network to enhance the global feature modeling capability.

[0106] The U-Net network structure can be a convolutional neural network with a symmetric encoder-decoder architecture, which can be used to support multi-scale feature extraction and reconstruction. Furthermore, the U-Net network structure can construct an encoder by stacking convolutional layers and pooling layers, and construct a decoder by stacking transposed convolutional layers and skip connections, thereby achieving end-to-end feature mapping. Understandably, the U-Net network structure, as the backbone network, can receive fused feature maps as initial input and provide embedding positions for the Swin-Transformer module to enhance global modeling. The backbone network can be the neural network skeleton responsible for core feature extraction and transformation tasks, and can be instantiated from the U-Net network structure, thereby uniformly scheduling sub-modules such as downsampling, upsampling, and Swin-Transformer integration.

[0107] Convolutional layers can be network layers that perform local weighted summation operations in image space using learnable filters, and can be used to extract low-level visual features such as local textures and edges. Pooling layers can be network layers that perform aggregation operations on local neighborhoods to reduce spatial dimensions, and can be implemented using max pooling or average pooling. The downsampling stage can be a process in the U-Net encoder that gradually reduces the resolution of feature maps, and can be composed of alternating stacks of convolutional layers and pooling layers to form a multi-scale feature pyramid. Furthermore, the downsampling stage can capture deep semantics and overall deformation trends by halving the output resolution and doubling the number of channels at each level. Deep features can be feature representations with low spatial resolution and high channel count generated at the end of the downsampling stage, which can characterize the overall layout of the document, perspective tendencies, and macroscopic deformation patterns.

[0108] Transposed convolutional layers can be learnable layers that perform inverse convolution operations to increase the spatial size of feature maps, and can be used with convolutional kernels to learn upsampling weights. Skip connections can be a connection method that directly passes the feature map of a layer in the downsampling stage to the corresponding layer in the upsampling stage, which can be implemented using channel concatenation or addition operations. The upsampling stage can be the process of gradually restoring the resolution of feature maps in the U-Net decoder, and can combine the high-resolution features introduced by upsampling of transposed convolutional layers and skip connections to reconstruct fine geometric structures. Detail features can be feature representations with high spatial resolution and rich structural information recovered at the end of the upsampling stage, which can support pixel-level offset prediction and precise mask boundary localization.

[0109] The Swin-Transformer module can be a variant of the Transformer based on a sliding window partitioning mechanism, which can be used to model long-range spatial dependencies. Furthermore, the Swin-Transformer module can divide the feature map into non-overlapping local windows, compute self-attention within each window, and achieve cross-window information interaction through a shifting window mechanism. It is understood that the Swin-Transformer module can include one or more of the following: fixed-window Swin modules, dynamic-window Swin modules, hierarchical-window Swin modules, etc., thereby enhancing the network's ability to understand the global semantic structure.

[0110] The U-Net network structure is adopted as the backbone network. In the downsampling stage, the feature map resolution is gradually reduced through convolutional layers and pooling layers. This can be achieved by stacking convolutional and pooling layers sequentially to build an encoder, halving the output resolution and doubling the number of channels at each stage. Furthermore, this operation can be achieved by replacing pooling layers with convolutions with a stride of 2, or by adding batch normalization and activation functions after each stage to enhance stability, thereby constructing a multi-scale feature pyramid to provide structural support for subsequent global deformation understanding.

[0111] The upsampling stage progressively increases the feature map resolution through transposed convolutional layers and skip connections. This can be achieved by using transposed convolutions for upsampling in the network decoder and then concatenating the downsampled feature maps from the corresponding layers by channel before inputting them into the next decoding layer. Furthermore, this operation can be further refined by using bilinear interpolation instead of transposed convolutions for upsampling, followed by ordinary convolutional correction, or by introducing a gating mechanism in the skip connections to regulate the information flow. This approach restores spatial dimensions while injecting original details, ensuring the accuracy of local deformation reconstruction.

[0112] Integrating the Swin-Transformer module into the downsampling and upsampling stages of the U-Net network can be achieved by inserting the Swin module after each downsampling convolutional layer and before pooling, and after each upsampling transposed convolutional layer and after skip connection fusion. Furthermore, this operation can balance efficiency and performance by deploying the Swin module only in the deep downsampling path and the top upsampling path, or by using a lightweight Swin-T version to reduce computational overhead. This allows for the introduction of long-range dependency modeling while maintaining the local inductive bias of the CNN, enhancing the cross-regional semantic alignment capability between the template and the input.

[0113] In one embodiment, the deformed photographic image is resampled based on the predicted offset matrix to generate a preliminary aligned image, including:

[0114] The sampling grid coordinates are generated based on the offsets in the width and height directions;

[0115] The deformed photographic image is resampled using bilinear interpolation using the sampling grid coordinates to obtain a preliminary aligned image.

[0116] The sampling grid coordinates can be a two-dimensional coordinate matrix describing the sampling position of each pixel in the target image within the source image. Furthermore, the acquisition method of the sampling grid coordinates can be explained in context: it is generated by superimposing standard grid coordinates on the width and height offsets in the predicted offset matrix, representing the floating-point coordinates that need to be read from the deformed image for each output position. In this embodiment, the sampling grid coordinates serve as the spatial mapping basis for resampling, and their purpose is to convert the predicted deformation field into executable pixel positioning instructions. For example, the sampling grid coordinates can include, but are not limited to, one or more of normalized grid coordinates, pixel coordinate grids, and relative offset grid coordinates.

[0117] Generating sampling grid coordinates based on width and height offsets can be achieved by adding standard unit grid coordinates element-wise to the width and height offsets respectively, generating a floating-point sampling position matrix. Furthermore, this operation can be implemented by: mapping the offsets back to pixel coordinates after superimposing them in normalized coordinate space; or by directly superimposing the offsets in pixel coordinate space and clipping them to the effective range, thereby achieving the technical effect of transforming an abstract deformation field into executable spatial mapping instructions.

[0118] Bilinear interpolation can be considered an image resampling algorithm that calculates pixel values ​​at floating-point coordinates based on a weighted average of the four nearest integer coordinates. Further, the operating principle of bilinear interpolation can be explained in context: for each floating-point position in the sampling grid coordinates, its four nearest integer pixels (top-left, top-right, bottom-left, and bottom-right) are located, and the output value is obtained by summing these pixels in inverse proportion to their distance. In this embodiment, bilinear interpolation works in conjunction with the sampling grid coordinates: relying on them to provide floating-point sampling positions; and with the deformed image: using it as the source image for pixel intensity retrieval. Resampling can be an operation that redistributes the pixels of the original image to a new coordinate system based on spatial mapping relationships. Further, the resampling method involves using the sampling grid coordinates as instructions to call the bilinear interpolation engine to perform pixel-by-pixel intensity reassignment on the deformed image. Its purpose is to complete the geometric transformation from deformed space to template space, serving as the actual execution step for predicting the offset matrix.

[0119] Bilinear interpolation resampling is performed on the deformed photographic image using sampling grid coordinates to obtain a preliminary aligned image. This can be achieved by performing bilinear interpolation on the deformed photographic image using the sampling grid coordinates as an index, outputting a new image with the same size as the template image. Furthermore, this operation can be implemented either in parallel on the GPU using PyTorch's `grid_sample` function, or on the CPU by iterating through the coordinates and calling the `OpenCVremap` function. This achieves sub-pixel-level spatial distortion restoration, preserving texture details while achieving pixel-level geometric alignment.

[0120] In one embodiment, the predicted mask image is binarized to obtain a binary mask image, including:

[0121] A preset threshold is applied to binarize each pixel value of the predicted mask image.

[0122] The preset threshold can be a critical value used to determine the category of pixels in the predicted mask image, and can be used to control the sensitivity of the binarization process. Furthermore, the preset threshold can be a fixed value determined offline based on segmentation performance (such as IoU, Dice coefficient) on the validation set, and used during the inference phase. It is understood that the preset threshold, as a classification boundary, is directly related to the performance of the validation data after model training. In this embodiment, the preset threshold can be one or more of the following, including but not limited to a global static threshold, a dynamic reference value based on data distribution statistics, and a multi-region adaptive threshold. The pixel value can be the numerical value at each location in the predicted mask image, and can be used as the input for binarization processing. Furthermore, the pixel value can be the value output by the last activation function (such as Sigmoid) of a preset neural network, and its acquisition method is directly related to the confidence distribution of the network output.

[0123] Furthermore, applying a preset threshold to binarize each pixel value of the predicted mask image can be done by iterating through all pixels of the predicted mask image, comparing each pixel value with the preset threshold, and determining whether it exceeds the threshold. For example, this operation can be performed pixel-by-pixel on the CPU, or by performing logical judgments in tensor batch parallelism on the GPU. The technical effect of this operation is to establish unified classification rules, providing a decision-making basis for subsequent labeling.

[0124] If the pixel value is greater than the preset threshold, it is marked as 1 to indicate a valid area; otherwise, it is marked as 0 to indicate an invalid area, and a binary mask image is generated.

[0125] The valid region can be the part of the deformed image that corresponds to the semantic structure of the template and should be retained in the final output. It can be used to identify information areas such as document text, tables, and graphics that need to be retained. The invalid region can be the part of the image that does not belong to the main content of the target document. It can be used to identify parts that should be removed in the alignment result. Furthermore, the distinction between valid and invalid regions is based on the comparison result of pixel values ​​and preset thresholds, which are represented by 1 and 0 respectively in the mask.

[0126] Furthermore, if a pixel value is greater than a preset threshold, it is marked as 1 to indicate a valid region; otherwise, it is marked as 0 to indicate an invalid region. This can be achieved by assigning binary labels based on the comparison results: assigning a value of 1 when the condition is met and 0 when it is not, thus constructing a binary spatial distribution map. For example, this operation can directly generate a binary matrix through hard thresholding, or by introducing a smooth transition function to simulate soft switching before truncation. The technical effect of this operation is to realize the transformation from a blurred probability map to a clear spatial mask, enhancing the clarity of region boundaries.

[0127] Furthermore, generating a binary mask image can involve arranging all marked pixels in their original spatial order into a new image matrix, forming a single-channel image containing only 0s and 1s. In a specific embodiment, this operation can construct the output image structure through memory copying or save it as a standard image format by calling an image library interface (such as OpenCV). The technical advantage of this operation is that it outputs a standard mask that can be used for pixel-level multiplication, supporting subsequent content filtering operations. Taking mobile acquisition of bank documents as an example, the template-guided deformation image alignment correction and mask generation method in this embodiment can be as follows: A customer takes a picture of a local area of ​​a loan application form with their mobile phone, and the image has finger occlusion and paper edge curvature. The system uses a standard application form template as a guide, predicts the mask probability map through a neural network, and performs binarization processing using a preset threshold of 0.5. High-confidence areas are marked as valid content, and finger-covered areas and low-response background areas are automatically excluded, ensuring that the final aligned image contains only complete field information, improving OCR recognition accuracy and automated review efficiency.

[0128] In one embodiment, the size of the prediction mask image and the prediction offset matrix is ​​1 / 4 of the input image size; multiplying the binary mask image with the preliminary alignment image at the pixel level to generate the final alignment image and the effective region mask includes: performing a pixel-level multiplication operation on the binary mask image and the preliminary alignment image, such that the pixel values ​​of the invalid regions in the final alignment image are set to 0, while the content of the effective regions is retained, thereby obtaining the final alignment image; and using the binary mask image of the invalid regions as the effective region mask.

[0129] The input image size can be the spatial resolution of the template image and the deformed photograph image after normalization, typically expressed in pixels for width and height. Understandably, the input image size can be determined by preset network input requirements, with all original images adjusted to this fixed size through size normalization. Furthermore, the input image size serves as a computational benchmark to determine the spatial scale relationships of feature maps within the network, particularly the relative resolution of the predicted output.

[0130] Pixel-level multiplication is an image processing operation that performs multiplication one pixel at a time. Essentially, pixel-level multiplication multiplies the pixel values ​​of the initially aligned image with the pixel values ​​of the binary mask image at the same coordinates. For example, pixel-level multiplication can be performed in parallel across the entire image using GPU tensor operations, or it can be performed on the CPU using multi-threaded block-by-block calculations. Furthermore, pixel-level multiplication can achieve spatially precise content preservation or suppression: when the mask value is 1, the original pixel is preserved; when it is 0, 0 is output, thus clearing invalid region information.

[0131] Invalid regions are areas in the final aligned image that do not belong to the valid document content. Understandably, invalid regions can be identified by pixel locations in the binary mask image that have a value of 0, corresponding to background, occlusion, distortion, or unstructured matching parts. Furthermore, invalid regions indicate image areas that need to be masked; they are set to zero during pixel-level multiplication operations to prevent interference with subsequent analysis.

[0132] The valid region can be any content area in the final aligned image that matches the semantic structure of the template and has analytical value. Understandably, the valid region can be identified by pixel locations with a value of 1 in the binary mask image, typically corresponding to areas containing text, tables, or other key elements. Furthermore, the valid region indicates the range of information that should be retained, ensuring that the output image contains only valid document content aligned with the template.

[0133] The binary mask image and the initial aligned image are multiplied at the pixel level. This sets the pixel values ​​of invalid regions in the final aligned image to 0, while retaining the content of valid regions, thus obtaining the final aligned image. This can be achieved by synchronously reading pixel by pixel, multiplying the binary mask image and the initial aligned image at the same coordinate position to generate a new image data matrix. Furthermore, this operation can be implemented in parallel using GPU tensor operations to perform the multiplication operation on the entire image, or by using multi-threaded block traversal of pixels on the CPU to perform the calculation. This allows for precise removal of non-target regions from the initial aligned image, outputting a clean image containing only valid content, thus improving the input quality for downstream tasks.

[0134] Using a binary mask image of invalid regions as a mask for valid regions can be achieved by directly reusing a pre-binarized mask image as the output, where regions with a value of 1 are marked as valid regions and regions with a value of 0 are marked as invalid regions. Furthermore, this operation can be implemented by adding a format conversion operation before the output to adapt to different system interface requirements, or by performing morphological closure processing on the mask image to eliminate small holes. This provides region validity annotations that strictly correspond to the final aligned image, supporting subsequent modules for region cropping, feature extraction, or quality assessment.

[0135] In one embodiment, the three-channel results are separated, where the first channel serves as the predicted mask image, and the second and third channels are normalized and used as offsets in the width and height directions, respectively, including:

[0136] Apply a normalization function to the pixel values ​​of the second and third channels to map the offsets to the range of [-1, 1], and obtain the normalized offsets in the width and height directions.

[0137] The normalization function can be a deterministic mathematical transformation that maps the pixel values ​​of the second or third channel to the interval [-1, 1]. Furthermore, the normalization function's operating principle can be explained in context, employing a linear stretching plus center translation formula, where min and max are the extreme values ​​of all pixels within the current channel. In this embodiment, the normalization function works in conjunction with the second and third channels, affecting their pixel values ​​and directly generating the normalized width and height offsets.

[0138] The normalized width offset can be a horizontal displacement within the range of [-1, 1] after being processed by a normalization function. Furthermore, the method for obtaining the normalized width offset can be explained in context: applying the normalization function to each pixel in the second channel, the output is the relative horizontal offset of that pixel in the normalized image coordinate system. In this embodiment, the normalized width offset, as the X component of the prediction offset matrix, can be directly parsed into the horizontal component of the source image sampling coordinates by the resampling operation.

[0139] The normalized height offset can be a vertical displacement within the range of [-1, 1] after processing by a normalization function. Furthermore, the method for obtaining the normalized height offset can be explained in context: applying the normalization function to each pixel in the third channel outputs the relative vertical offset of that pixel in the normalized image coordinate system. In this embodiment, the normalized height offset, as the Y component of the prediction offset matrix, can be directly parsed into the vertical component of the source image sampling coordinates by the resampling operation.

[0140] `min` can be the minimum value of all pixel values ​​in the second or third channel. Furthermore, the method of obtaining `min` can be explained in the context, involving performing global extreme value statistics on the corresponding channel before normalization. In this embodiment, `min` participates in the normalization function calculation, determining the lower bound of the linear mapping. `max` can be the maximum value of all pixel values ​​in the second or third channel. Furthermore, the method of obtaining `max` can be explained in the context, involving performing global extreme value statistics on the corresponding channel before normalization. In this embodiment, `max` participates in the normalization function calculation, determining the upper bound of the linear mapping.

[0141] The range [-1, 1] can be the standard numerical range of the normalized offset. Furthermore, the acquisition method of the [-1, 1] range can be explained in context; it is guaranteed by the mathematical structure of the normalization function and is a fixed output domain after linear transformation. In this embodiment, the [-1, 1] range corresponds to the normalized device coordinates (NDC) of the normalized image plane, consistent with the semantics of the graphics resampling interface.

[0142] Furthermore, a normalization function is applied to the pixel values ​​of the second and third channels, mapping the offsets to the range [-1, 1] to obtain the normalized width and height offsets. This can be achieved by calculating the min and max values ​​of the second and third channels respectively, and then substituting them into the normalization function to calculate the new values ​​pixel by pixel. Furthermore, this operation can be further improved by using intra-batch channel extrema for normalization during the training phase and single-sample channel extrema for normalization during the inference phase. This achieves the technical effect of giving the offsets a unified coordinate semantics, improving resampling accuracy and cross-sample robustness.

[0143] The normalization function norm(x) = [2 × (x - min)] / (max - min) - 1 is used to transform the channel pixels. This can be done by substituting the min and max values ​​of the channel for each pixel x in a specified channel and performing scalar calculations according to the formula. Furthermore, this operation can be implemented on the CPU using a loop-based traversal approach, or by performing batch operations on the GPU using vectorized tensors. This achieves adaptive linear normalization, preserves the relative deformation structure within the channel, and avoids detail suppression caused by global parameters.

[0144] Furthermore, to achieve the above objectives, the present invention also provides a template-guided deformed image alignment correction and mask generation device, the device comprising: a memory, a processor, and a template-guided deformed image alignment correction and mask generation program stored in the memory and executable on the processor, the template-guided deformed image alignment correction and mask generation program being configured to implement the steps of the template-guided deformed image alignment correction and mask generation method as described above.

[0145] In addition, to achieve the above objectives, the present invention also provides a medium storing a template-guided deformed image alignment correction and mask generation program, wherein when the template-guided deformed image alignment correction and mask generation program is executed by a processor, it implements the steps of the template-guided deformed image alignment correction and mask generation method as described above.

[0146] Other embodiments or specific implementations of the template-guided deformation image alignment correction and mask generation device described in this invention can be referred to the above-described method embodiments, and will not be repeated here.

[0147] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A template-guided method for aligning and correcting deformed images and generating masks, characterized in that, The method includes: Obtain the template image and the corresponding deformed photographic image; The template image and the deformed photograph image are subjected to size normalization processing to obtain a normalized image; The normalized image is input into a preset neural network, and through feature extraction, feature fusion and upsampling processing, a prediction mask image and a prediction offset matrix are generated. The deformed photographic image is resampled based on the predicted offset matrix to generate a preliminary aligned image; The predicted mask image is binarized to obtain a binary mask image; The binary mask image is multiplied pixel-wise with the preliminary aligned image to generate the final aligned image and the effective region mask.

2. The template-guided deformation image alignment correction and mask generation method as described in claim 1, characterized in that, The step of normalizing the size of the template image and the deformed photograph image to obtain a normalized image includes: The template image and the deformed photograph image are scaled to a preset uniform size using bilinear interpolation; During the scaling process, an edge-filling strategy is adopted to fill image areas that are smaller than the target size with black pixels to obtain a normalized image, ensuring that the normalized image retains the main content of the original image and is of consistent size.

3. The template-guided deformation image alignment correction and mask generation method as described in claim 1, characterized in that, The step of inputting the normalized image into a preset neural network and generating a prediction mask image and a prediction offset matrix through feature extraction, feature fusion, and upsampling processing includes: A preset neural network is used to perform convolution, pooling and activation function processing on the template image part and the deformed photograph image part in the normalized image to obtain the template feature map and the photograph feature map. The template feature map and the photographed feature map are concatenated along the channel dimension to generate a fused feature map; The fused feature map is downsampled sequentially to extract deep features, and then upsampled to restore detailed features, outputting a three-channel result; The three-channel results are separated, with the first channel serving as the prediction mask image, and the second and third channels being normalized and used as the width and height offsets, respectively, which together constitute the prediction offset matrix.

4. The template-guided deformation image alignment correction and mask generation method as described in claim 3, characterized in that, The preset neural network includes a U-Net network structure. The step of sequentially downsampling the fused feature map to extract deep features and then upsampling it to recover detailed features includes: The U-Net network structure is used as the backbone network. In the downsampling stage, the feature map resolution is gradually reduced through convolutional layers and pooling layers, and in the upsampling stage, the feature map resolution is gradually increased through transposed convolutional layers and skip connections. The Swin-Transformer module is integrated into the downsampling and upsampling stages of the U-Net network to enhance global feature modeling capabilities.

5. The template-guided deformation image alignment correction and mask generation method as described in claim 3, characterized in that, The step of resampling the deformed photographic image based on the predicted offset matrix to generate a preliminary aligned image includes: The sampling grid coordinates are generated based on the width offset and the height offset; The deformation photograph image is resampled using bilinear interpolation using the sampling grid coordinates to obtain the preliminary aligned image.

6. The template-guided deformation image alignment correction and mask generation method as described in claim 1, characterized in that, The step of binarizing the predicted mask image to obtain a binary mask image includes: A preset threshold is applied to binarize and determine the value of each pixel in the predicted mask image; If the pixel value is greater than the preset threshold, it is marked as 1 to indicate a valid region; otherwise, it is marked as 0 to indicate an invalid region, and the binary mask image is generated.

7. The template-guided deformation image alignment correction and mask generation method as described in claim 1, characterized in that, The size of the predicted mask image and the predicted offset matrix is ​​1 / 4 of the size of the input image; The step of multiplying the binary mask image and the preliminary aligned image pixel-level to generate the final aligned image and the effective region mask includes: Perform a pixel-level multiplication operation between the binary mask image and the preliminary aligned image, so that the pixel values ​​of the invalid areas in the final aligned image are set to 0, while the content of the valid areas is retained, thereby obtaining the final aligned image; The binary mask image of the invalid region is used as the mask of the valid region.

8. The template-guided deformation image alignment correction and mask generation method as described in claim 3, characterized in that, The separation of the three-channel results, wherein the first channel serves as the prediction mask image, and the second and third channels are normalized and used as offsets in the width and height directions, respectively, includes: Apply a normalization function to the pixel values ​​of the second and third channels to map the offsets to the range of [-1, 1], and obtain the normalized width and height offsets; The normalization function is: norm(x) = [2 × (x - min)] / (max - min) - 1, where min and max are the minimum and maximum pixel values ​​in the channel, respectively.

9. A template-guided deformed image alignment correction and mask generation device, characterized in that, The device includes: a memory, a processor, and a template-guided deformation image alignment correction and mask generation program stored in the memory and executable on the processor, the template-guided deformation image alignment correction and mask generation program being configured to implement the steps of the template-guided deformation image alignment correction and mask generation method as described in any one of claims 1 to 8.

10. A medium, characterized in that, The medium stores a template-guided deformation image alignment correction and mask generation program, which, when executed by a processor, implements the steps of the template-guided deformation image alignment correction and mask generation method as described in any one of claims 1 to 8.

Images