Double-branch semi-supervised wide-angle image correction system and method based on Swin-Transform

Abstract

The invention provides a double-branch semi-supervised wide-angle image correction system and method based on Swin-Transform, which are applied to the technical field of computer vision, a distorted image is converted into three image blocks with different sizes through a feature extraction backbone network module, and features are extracted from the three image blocks for superposition to form a fusion feature map; an output feature map is generated through a self-adaptive fusion double-branch coding and decoding network module, a new position after each pixel point is displaced according to the respective offset is reconstructed through image reconstruction, a corrected image is generated, data annotation does not need to be carried out, and therefore the problem that the learning effect of a model is affected by inconsistent data annotation is solved. Therefore, according to the scheme, the efficiency of local feature extraction and polymorphic object recognition is improved, a semi-supervised training strategy is adopted, an agent segmentation task is combined, labeled data and unlabeled data are effectively combined, and the training effect is remarkably optimized.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a dual-branch semi-supervised wide-angle image correction system and method based on Swin-Transformer. Background Art

[0002] With the advancement of smartphone photography technology, the market demand for ultra-wide-angle lenses is growing. However, wide-angle lens shooting often causes perspective distortion, resulting in serious stretching of objects in the image, especially portraits and background areas.

[0003] At present, deep learning methods in the field of image correction can be divided into two main categories: model-based methods and model-free methods. Model-based methods focus on predicting the parameters of a given distortion model, while model-free methods focus on directly learning the deformation field or generating the corrected image. A major advantage of the model-free method is that it does not directly involve the specific parameters of the distortion, allowing multiple distortion models to be processed in the same framework. Although the model-free method can handle multiple distortion models, it has deficiencies in global feature extraction, multi-scale fusion, and retaining corrected image features. In addition, the scarcity of real samples in wide-angle datasets, the inconsistency of data annotation quality, and the limitations of existing models in processing facial details and deep semantic changes have all put forward higher requirements for image correction technology.

[0004] Therefore, there is a lack of an effective solution in the existing technology to address the shortcomings of deep learning image correction models in global feature extraction, multi-scale fusion, and retaining corrected image features, the scarcity of real samples in wide-angle datasets, the inconsistent quality of data annotation, and the limitations of existing models in processing facial details and deep semantic changes. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, this application provides a dual-branch semi-supervised wide-angle image correction system and method based on Swin-Transformer, which is applied to the field of computer vision technology. On the basis of the existing model, it combines Swin-Transformer and adaptive fusion dual-branch encoding and decoding network modules to improve the efficiency of local feature extraction and multi-modal object recognition. In terms of learning strategy, semi-supervised learning is adopted to significantly optimize the training effect.

[0006] In the first aspect, the present application proposes a dual-branch semi-supervised wide-angle image correction system based on Swin-Transformer, which at least includes:

[0007] Distorted image input module: used to obtain a distorted image and input the distorted image into a feature extraction backbone network module;

[0008] The feature extraction backbone network module is used to convert the distorted image into three image blocks of different sizes, and extract feature superposition from the three image blocks of different sizes to form a fused feature map;

[0009] Adaptive fusion dual-branch codec network module: at least includes an encoder and a decoder. In a semi-supervised training strategy, the encoder performs a deformable convolution operation on the fused feature map to obtain an output feature map; the decoder uses a proxy segmentation task to perform optical flow estimation on the output feature map, and the optical flow estimation is used to predict the offset of each pixel in the output feature map, and reconstruct each pixel at a new position after being displaced by its own offset through image reconstruction to generate a corrected image;

[0010] The corrected image output module is used to output the corrected image.

[0011] The present application proposes a dual-branch semi-supervised wide-angle image correction system based on Swin-Transformer, which at least includes a distorted image input module, a feature extraction backbone network module, an adaptively fused dual-branch encoding and decoding network module, and a corrected image output module. Through the distorted image input module, a distorted wide-angle image is obtained and provided to a subsequent processing module. The distorted image is converted into three image blocks of different sizes through the feature extraction backbone network module, and features are extracted and superimposed therefrom to form a fused feature map, so as to better capture multi-scale features; the encoder in the adaptively fused dual-branch encoding and decoding network module performs a deformable convolution operation on the fused feature map, which can flexibly adjust the features of each pixel to generate an output feature map. The decoder estimates the optical flow of the output feature map through the proxy segmentation task, predicts the offset of each pixel, and reconstructs each pixel point according to the new position after the displacement of the respective offset through image reconstruction to generate a corrected image, without the need for data annotation, thereby solving the problem that inconsistent data annotation affects the learning effect of the model. The corrected image output module finally outputs the corrected image. Therefore, based on the existing model, this solution combines Swin-Transformer and adaptive fusion dual-branch encoding and decoding network modules to improve the efficiency of local feature extraction and multi-modal object recognition. In terms of learning strategy, a semi-supervised training strategy is adopted, combined with the proxy segmentation task, which effectively combines labeled and unlabeled data and significantly optimizes the training effect.

[0012] Further, the semi-supervised training strategy includes at least a proxy segmentation task, direction and range consistency, and consistency regularization;

[0013] The proxy segmentation task is used to perform optical flow estimation on the output feature map and predict the offset of each pixel in the output feature map;

[0014] The direction and range consistency is used to check whether the offset of each pixel in the output feature map is within a preset standard range, and to obtain pixels whose offset does not exceed the preset standard range to form a cluster of subtle movements;

[0015] Consistency regularization is used to introduce a loss amount and adjust the offset loss term of each pixel during image reconstruction to make the generated corrected image more accurate.

[0016] This application proposes a dual-branch semi-supervised wide-angle image correction system based on Swin-Transformer. The semi-supervised training strategy is used to train the model throughout the entire correction process of the distorted image. Through the proxy segmentation task and consistency mechanism, it effectively combines labeled and unlabeled data, reduces overfitting through consistency regularization, and solves the problem of inaccurate corrected images caused by inconsistent data annotations.

[0017] Furthermore, the feature extraction backbone network module uses VIT as the baseline network, adopts the Swin-Transformer network to replace the Transformer network, and converts the distorted image into three image blocks of different sizes, wherein the sizes of the three image blocks are Patch=8*8, Patch=16*16, and Patch=32*32, respectively.

[0018] This application proposes a dual-branch semi-supervised wide-angle image correction system based on Swin-Transformer, which optimizes the feature extraction process by replacing the traditional Transformer network with the Swin-Transformer network. Swin-Transformer can better process image blocks of different scales and improve the ability to capture multi-scale features. Converting distorted images into three image blocks of different sizes, 88, 1616, and 32*32, helps to extract image features more comprehensively and provide richer information for subsequent correction.

[0019] Furthermore, the formula for extracting features from three image blocks of different sizes by the feature extraction backbone network module is:

[0020]

[0021] Among them, x class is the number of distorted image segmentation categories; It is a two-dimensional image sequence data that expands the input image according to the block size p (p = 8, 16, 32). N represents the number of features, and the parameter x is the input of three images of different sizes; L is the sequence data dimension, L = HW / p 2 , where HW represents the image dimension; E and E posRepresent block coding and position coding matrices respectively; Represents the output sequence data of the current layer, and the subscript p represents the different sizes of the blocks; Represents the output sequence data of the previous layer, Represents the output sequence data of the next layer; Represents the sequence data after encoding at the current layer, represents the sequence data after encoding in the next layer; W-MSA represents the window multi-head self-attention layer; MLP represents the multi-layer perceptron, SW-MSA represents the shifted window multi-head self-attention layer, LN represents the normalization layer, Represents the sequence data after conversion into three image blocks of different sizes.

[0022] This application proposes a dual-branch semi-supervised wide-angle image correction system based on Swin-Transformer. The feature extraction backbone network module extracts features from three image blocks of different sizes and uses multi-head self-attention layers and multi-layer perceptrons for feature encoding. These technical features improve the global feature extraction capability of images by multi-scale processing and encoding of images. In addition, through block coding and position coding matrices, the details and structural information of the image can be better preserved, thereby effectively solving the deficiencies in multi-scale fusion and global feature extraction.

[0023] Furthermore, the formula for forming the fusion feature map by the feature extraction backbone network module is:

[0024]

[0025] Among them, Reshape means array change, Conv means convolution, BN means sample normalization, Add(Upsample) means bilinear interpolation upsampling, y Out It is the feature matrix after the fusion of multiple block-encoded ViST networks, that is, the fused feature map.

[0026] Furthermore, the adaptive fusion dual-branch encoding and decoding network module uses U-HRNet as the baseline network, and in the semi-supervised training strategy, the encoder performs a deformable convolution operation on the fusion feature map, including:

[0027] Use the original convolution kernel to convolve the fused feature map to obtain the initial output feature map y Out Each position p0 in is as follows: Among them, K is the conventional convolution kernel, p n is the offset position, ω is the initial weight;

[0028] The initial output feature map is convolved again to obtain the deformation offset Δp of each pixel in the deformable convolution. n, thus obtaining the output feature map after correcting the offset:

[0029]

[0030] Furthermore, the encoder is also used to calculate the offset pixel value of each pixel point using a bilinear interpolation method: Among them, x(p) represents the pixel value at any position of the feature map, where p = p0 + p n +Δp n ; x(q) represents the pixel value of the neighboring pixel point; BI represents the kernel function of bilinear interpolation.

[0031] Furthermore, in the direction and range consistency, the value of each pixel in the output feature map is (-δ, δ), the preset standard range is an interval F of (-∞, -δ]∪[δ, +∞), and the pixel points whose offset does not exceed the interval F are grouped into clusters of subtle movements, and the expression of the cluster is:

[0032] L DRC =[L ce (S1',S1")+L ce (S2',S2')]

[0033] Where S(i, j) represents the mask or optical flow map for segmenting the output feature map; F(i, j) is the pixel position in the mask or optical flow map after segmentation, and L ce represents the cross entropy loss, L DRC Indicates scope consistency.

[0034] Furthermore, in the consistency regularization, the loss introduced is: L CR =[L mL (F1',F2')+λ2L ms (F1',F2')], where L CR represents consistency regularization, F1' and F2' are the optical flow maps outputted from the two branches of the output feature map after entering the twin network STU-HRNet, L ms is the consistency regularization loss, L mL is the loss of the main task, and λ2 is the weight coefficient.

[0035] In the second aspect, the present application proposes a dual-branch semi-supervised wide-angle image correction method based on Swin-Transformer, which is applied to any of the above-mentioned dual-branch semi-supervised wide-angle image correction systems based on Swin-Transformer, and comprises the steps of:

[0036] S1: Obtain a distorted image, and input the distorted image into a feature extraction backbone network module;

[0037] S2: converting the distorted image into three image blocks of different sizes, and extracting features from the three image blocks of different sizes and superimposing them to form a fused feature map;

[0038] S3: In the semi-supervised training strategy, the encoder performs a deformable convolution operation on the fused feature map to obtain an output feature map; the decoder uses a proxy segmentation task to perform optical flow estimation on the output feature map, the optical flow estimation is used to predict the offset of each pixel in the output feature map, and reconstructs each pixel at a new position after being displaced by its own offset through image reconstruction to generate a corrected image;

[0039] S4: Outputting the corrected image.

[0040] Beneficial effect: The present application proposes a dual-branch semi-supervised wide-angle image correction system and method based on Swin-Transformer, which at least includes a distorted image input module, a feature extraction backbone network module, an adaptive fusion dual-branch codec network module and a correction image output module. Through the distorted image input module, the distorted wide-angle image is obtained and provided to the subsequent processing module. The distorted image is converted into three image blocks of different sizes (Patch=8*8, Patch=16*16, Patch=32*32) through the feature extraction backbone network module, and features are extracted and superimposed to form a fused feature map to better capture multi-scale features; the encoder in the adaptive fusion dual-branch codec network module performs a deformable convolution operation on the fused feature map, which can flexibly adjust the features of each pixel to generate an output feature map. The decoder estimates the optical flow of the output feature map through the proxy segmentation task, predicts the offset of each pixel, and reconstructs each pixel point according to the new position after the displacement of the respective offset through image reconstruction to generate a corrected image without data annotation, thereby solving the problem that inconsistent data annotation affects the learning effect of the model. The corrected image output module finally outputs the corrected image. Therefore, based on the existing model, this solution combines the Swin-Transformer and the adaptive fusion dual-branch encoding and decoding network module to improve the efficiency of local feature extraction and multi-modal object recognition. In terms of learning strategy, a semi-supervised training strategy is adopted, combined with the proxy segmentation task, which effectively combines labeled and unlabeled data and significantly optimizes the training effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flowchart of a dual-branch semi-supervised wide-angle image correction method based on Swin-Transformer proposed in this application.

[0042] Figure 2 This is a network structure diagram of a dual-branch semi-supervised wide-angle image correction system based on Swin-Transformer proposed in this application.

[0043] Figure 3 This is the network structure diagram of the feature extraction backbone network module proposed in this application.

[0044] Figure 4 This is a network structure diagram of the adaptive fusion dual-branch encoding and decoding network module proposed in this application.

[0045] Figure 5 This application proposes the operating results of a dual-branch semi-supervised wide-angle image correction system and method based on Swin-Transformer. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application usually described and marked in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0047] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first, second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0048] The following disclosure provides many different implementations or examples for achieving the purpose of the present invention, which solve the deficiencies of the deep learning image correction model in the prior art in global feature extraction, multi-scale fusion, and retaining corrected image features, as well as the scarcity of real samples in wide-angle datasets, inconsistent data annotation quality, and limitations of existing models in processing facial details and deep semantic changes.

[0049] Please refer to Figures 1 to 5 , this application proposes a dual-branch semi-supervised wide-angle image correction system based on Swin-Transformer, which at least includes:

[0050] Distorted image input module: used to obtain the distorted image and input the distorted image into the feature extraction backbone network module;

[0051] Feature extraction backbone network module: used to convert the distorted image into three image blocks of different sizes, and extract feature superposition from the three image blocks of different sizes to form a fused feature map;

[0052] Adaptive fusion dual-branch codec network module: includes at least an encoder and a decoder. In the semi-supervised training strategy, the encoder performs a deformable convolution operation on the fused feature map to obtain an output feature map; the decoder uses a proxy segmentation task to perform optical flow estimation on the output feature map. The optical flow estimation is used to predict the offset of each pixel in the output feature map, and reconstructs each pixel at a new position after being displaced by its own offset through image reconstruction to generate a corrected image.

[0053] Corrected image output module: used to output the corrected image.

[0054] Please refer to Figure 2 , Figure 2 The backbone network in the figure is the feature extraction backbone network module, and the correction result is the corrected image. The feature extraction backbone network module adopts the Swin-Transformer structure to construct the MFViST network, which converts the input distorted image into three image blocks of different sizes, namely 8x8, 16x16 and 32x32 pixels. For each size of image block, the system uses a multi-head self-attention mechanism for feature extraction, and then performs feature fusion through a multi-layer perceptron. This multi-scale feature extraction method can capture the local details and global structural information of the image, effectively improving the comprehensiveness and accuracy of feature extraction.

[0055] The adaptive fusion dual-branch codec network module includes an encoder and a decoder. The encoder uses a deformable convolutional network, which first applies a 3x3 regular convolution to the fused feature map to obtain the initial output feature map. Then, another 3x3 convolution is used to predict the offset of each pixel, with the range limited to [-1,1]. Finally, the fused feature map is sampled according to the predicted offset to generate the final output feature map. This deformable convolution operation can adaptively adjust the receptive field and better handle nonlinear distortion in wide-angle images.

[0056] The decoder part adopts the U-Net structure, which restores the image resolution of the output feature map through multiple upsampling and skip connections. At each decoding layer, the optical flow of the feature map is estimated using a proxy segmentation task. Specifically, a two-dimensional offset vector is predicted for each pixel, indicating the new position of the pixel in the rectified image. Finally, the rectified image is reconstructed using the bilinear interpolation method based on the predicted offset.

[0057] In the semi-supervised training strategy, this system introduces proxy segmentation tasks, direction and range consistency, and consistency regularization. The proxy segmentation task is used to estimate the optical flow of the output feature map and predict the offset of each pixel. Direction and range consistency is used to check whether the offset of each pixel in the feature map is within the preset standard range, and obtain pixels whose offset does not exceed the preset standard range to form clusters of subtle movements. Consistency regularization introduces a loss amount to adjust the offset loss term of each pixel during image reconstruction, thereby improving the accuracy of the generated corrected image.

[0058] The system uses a small number of labeled real wide-angle images to train a large number of unlabeled distorted images. For labeled data, the reconstruction loss and optical flow estimation loss are directly calculated. For unlabeled data, a consistency regularization term is introduced to require the model to produce consistent correction results for two different enhanced versions of the same input image. This strategy can make full use of limited labeled data while improving the generalization ability of the model through unlabeled data.

[0059] In practical applications, the system can flexibly adjust the size and number of image blocks. For example, in addition to 8x8, 16x16, and 32x32 image blocks, the size of the image blocks can be increased or decreased according to specific needs. In addition, the parameters of the deformable convolution can also be fine-tuned according to different types of distortion to obtain the best correction effect.

[0060] The modules of the system are closely related and interactive. The distorted image input module provides raw data for the feature extraction backbone network module. The output of the feature extraction backbone network module directly affects the performance of the adaptive fusion dual-branch codec network module. The encoder and decoder work together to realize the whole process from feature extraction to pixel offset prediction. Finally, the corrected image output module presents the processed results to the user.

[0061] Compared with the prior art, the solution of this application uses Swin-Transformer as the feature extraction backbone network, which can better capture long-distance dependencies, which is crucial for processing large-scale distortion in wide-angle images. Through multi-scale feature extraction, the system can focus on local details and global structures at the same time, improving the accuracy and robustness of correction. The introduction of deformable convolution enables the model to dynamically adjust the receptive field according to the input content, which is more suitable for processing non-uniform distortion in wide-angle images than fixed convolution kernels. Experiments show that this adaptive mechanism can significantly improve the correction effect, especially in areas with severe distortion such as image edges. The semi-supervised training strategy effectively solves the problem of scarcity of real samples in wide-angle datasets. By utilizing a large amount of unlabeled data, the model can learn a wider range of distortion patterns and improve the generalization ability in actual scenes. The introduction of consistency regularization further enhances the adaptability of the model to different types of distortion. The proxy segmentation task and optical flow estimation method enable the system to accurately predict the displacement of each pixel, which is more flexible than the global parameter estimation method and can better handle local nonlinear distortion. Through pixel reconstruction, the system can retain the details and semantic information of the original image to the greatest extent, avoiding the blur and distortion problems common in traditional methods.

[0062] In general, the dual-branch semi-supervised wide-angle image correction system based on Swin-Transformer proposed in this application has achieved significant performance improvements in global feature extraction, multi-scale fusion, and retention of corrected image features through innovative network structure and training strategies, providing an effective new method for solving the problem of wide-angle image distortion. This method can not only handle complex distortion situations, but also maintain good performance when samples are scarce, providing a better image correction experience for wide-angle photography enthusiasts.

[0063] Furthermore, the semi-supervised training strategy includes at least a proxy segmentation task, direction and range consistency, and consistency regularization;

[0064] The proxy segmentation task is used to estimate the optical flow of the output feature map and predict the offset of each pixel in the output feature map;

[0065] Direction and range consistency is used to check whether the offset of each pixel in the output feature map is within the preset standard range, and to obtain pixels whose offset does not exceed the preset standard range to form a cluster of subtle movements;

[0066] Consistency regularization is used to introduce a loss amount and adjust the offset loss term of each pixel during image reconstruction to make the generated corrected image more accurate.

[0067] Among them, this technical solution achieves more accurate wide-angle image correction in a semi-supervised training environment by combining proxy segmentation tasks, direction and range consistency, and consistency regularization. This method can not only effectively deal with the problem of insufficient labeled data, but also maintain the local consistency and global accuracy of the corrected image. Compared with traditional fully supervised methods, this solution has significant advantages in data efficiency and generalization ability, providing an innovative solution for the field of wide-angle image correction.

[0068] In a specific implementation, the proxy segmentation task can be implemented in a variety of ways. One possible implementation is to use a convolutional neural network with a U-Net structure, which can perform pixel-by-pixel segmentation on the output feature map to predict the offset of each pixel. Another implementation method can use a fully convolutional network (FCN) to directly predict the offset on the output feature map. Both methods can effectively estimate optical flow. This solution prefers to use the U-Net structure, so that the corrected image may have more advantages in retaining detail information. The direction and range consistency mechanism can be implemented by setting a threshold to determine whether the pixel offset is within a preset standard range. For example, a specific threshold δ∈N can be introduced as a standard for judging whether the offset is within a predetermined (-∞, -δ] or [δ, +∞) interval, and these two intervals represent displacements in the negative or positive direction, respectively. At the same time, those pixels whose offsets fall within this interval are classified as clusters representing subtle movements. Consistency regularization can be achieved by introducing additional loss functions. For example, the L1 or L2 norm can be used as a regularization term to penalize the pixel offset. Specifically, the regularization coefficient λ can be set in the range of 0.01 to 0.1, and the optimal value can be determined by cross-validation. In this way, image distortion caused by over-correction can be avoided while maintaining the correction effect.

[0069] Compared with the traditional fully supervised method mentioned in the background technology, the technical solution of this application solves the problem that the traditional method usually requires a large amount of labeled data, which is often difficult to obtain or costly in practical applications. The semi-supervised training strategy adopted in this application can improve the model performance by effectively utilizing unlabeled data under the condition of limited labeled data. This not only improves the efficiency of data utilization, but also enhances the generalization ability of the model, enabling it to better adapt to different types of wide-angle distortion.

[0070] Furthermore, the feature extraction backbone network module uses VIT as the baseline network, adopts the Swin-Transformer network to replace the Transformer network, and converts the distorted image into three image blocks of different sizes, where the sizes of the three image blocks are Patch=8*8, Patch=16*16, and Patch=32*32, respectively.

[0071] Among them, the Swin-Transformer network is used to replace the traditional Transformer network, which has stronger local feature extraction capabilities and higher computational efficiency; the feature extraction backbone network module converts the image into image blocks of different sizes, which can realize multi-scale feature extraction, which is conducive to capturing local details and global structural information of the image; the conversion into three image block settings of Patch = 8*8, Patch = 16*16, and Patch = 32*32 can achieve a good balance between retaining details and reducing the amount of calculation. In this way, the feature extraction backbone network module can extract rich multi-scale features from the distorted image and provide effective feature representation for subsequent image correction tasks. This method of multi-scale feature extraction and fusion can effectively improve the accuracy and robustness of wide-angle image correction, especially when dealing with complex scenes and images with different degrees of distortion.

[0072] Furthermore, the formula for extracting features from three image blocks of different sizes by the feature extraction backbone network module is:

[0073]

[0074] Among them, x class is the number of distorted image segmentation categories; is the two-dimensional image sequence data obtained by expanding the input image by block size p (p = 8, 16, 32), N represents the number of features, and parameter x is the input of three images of different sizes; L is the sequence data dimension, L = HW / p2, where HW represents the image dimension; E and E pos Represent block coding and position coding matrices respectively; Represents the output sequence data of the current layer, and the subscript p represents the different sizes of the blocks; Represents the output sequence data of the previous layer, Represents the output sequence data of the next layer; Represents the sequence data after encoding at the current layer, represents the sequence data after encoding in the next layer; W-MSA represents the window multi-head self-attention layer; MLP represents the multi-layer perceptron, SW-MSA represents the shifted window multi-head self-attention layer, LN represents the normalization layer, Represents the sequence data after conversion into three image blocks of different sizes.

[0075] Please refer to Figure 3, wherein the present application proposes a formula for extracting features from three image blocks of different sizes by the feature extraction backbone network module, and uses the Swin-Transformer structure to process image blocks of different sizes, which can capture long-distance dependencies better than traditional convolutional neural networks. At the same time, by processing image blocks of multiple scales, it can better adapt to the degree of distortion in different areas of wide-angle images. This method can handle local details well while maintaining the global consistency of the image, thereby improving the quality and accuracy of wide-angle image correction. The specific implementation method is as follows:

[0076] First, the input distorted image is segmented into three image patches of different sizes: 8x8, 16x16, and 32x32 pixels. This multi-scale segmentation can capture the characteristics of the image at different scales and help to fully understand the image content.

[0077] For each size of image block, this application uses block coding and position coding for initialization. Block coding can use linear projection or convolution operations to convert image blocks into fixed-dimensional feature vectors. Position coding can use sine-cosine coding or learnable position embedding to add position information to each image block. This encoding method can retain the spatial information of the image block, which is beneficial for subsequent feature extraction.

[0078] Next, this application uses a multi-head self-attention layer (MSA) and a multi-layer perceptron (MLP) to iteratively process the encoded sequence data. The MSA layer can capture the long-distance dependencies between image blocks, while the MLP layer further extracts nonlinear features. Through multiple iterations, the model can gradually extract more abstract and meaningful feature representations.

[0079] It is worth noting that the feature extraction method of the present application is closely integrated with other modules in the previous method. For example, the adaptive fusion dual-branch codec network module can directly use the multi-scale features extracted here to perform more refined image correction. This combination not only improves the efficiency of feature extraction, but also enhances the correction ability of the entire system.

[0080] In practical applications, the feature extraction method of the present application can be adjusted according to specific needs. For example, different combinations of image block sizes can be tried, such as 4x4, 8x8, 16x16, or 16x16, 32x32, 64x64, to accommodate images of different resolutions. In addition, the number and dimensions of MSA and MLP layers can also be adjusted according to computing resources and performance requirements.

[0081] Furthermore, the formula for forming the fusion feature map by the feature extraction backbone network module is:

[0082]

[0083] Among them, Reshape means array change, Conv means convolution, BN means sample normalization, Add(Upsample) means bilinear interpolation upsampling, y Out It is the feature matrix after the fusion of multiple block-encoded ViST networks, that is, the fused feature map.

[0084] In practical applications, by fusing feature matrices of different block codes, this method effectively retains multi-scale information from details to the global level.

[0085] Furthermore, the adaptive fusion dual-branch encoder-decoder network module uses U-HRNet as the baseline network. In the semi-supervised training strategy, the encoder performs deformable convolution operations on the fused feature map including:

[0086] Use the original convolution kernel to convolve the fused feature map to obtain the initial output feature map y Out Each position p0 in is as follows: Among them, K is the conventional convolution kernel, p n is the offset position, ω is the initial weight; the initial output feature map is convolved again to obtain the deformation offset Δp of each pixel in the deformable convolution n , thus obtaining the output feature map after correcting the offset:

[0087]

[0088] Please refer to Figure 4 , the encoding and decoding network is an adaptively fused dual-branch encoding and decoding network module. This technical solution uses a deformable convolution operation to process the fused feature map, which is an adaptive convolution method. First, the fused feature map is preliminarily convolved using a conventional convolution kernel to obtain an initial output feature map. Then, this initial output feature map is convolved again to calculate the deformation offset of each pixel. Finally, these offsets are applied to the original convolution operation to obtain the corrected output feature map. The core advantage of this method is that it can adaptively adjust the sampling position of the convolution kernel so that the convolution operation can better adapt to the geometric deformation in the image. This adaptive ability is particularly important for tasks such as wide-angle image correction that involve complex geometric transformations. It can more accurately capture the local deformation features in the image, thereby improving the accuracy and effect of the correction.

[0089] In a specific implementation, assuming that the input fused feature map is of size 128x128x64, we first perform a preliminary convolution on it using a 3x3 regular convolution kernel to obtain the initial output feature map y. Then, we convolve y with another 3x3 convolution kernel to obtain an offset map of size 128x128x18 (corresponding to the x and y offsets of 9 sampling points). Next, we apply these offsets to the original 3x3 convolution operation. Specifically, for each position p0 in the output feature map, calculate:

[0090] Where K is a 3x3 sampling grid, ω is the initial weight, x is the input feature map, Δp n is the deformation offset of the corresponding sampling point. This operation is implemented by bilinear interpolation to ensure that the gradient can be back-propagated correctly.

[0091] In this way, our network can dynamically adjust the sampling position of the convolution kernel according to the input content, so as to better adapt to the complex geometric deformation in wide-angle images. For example, when dealing with severe distortion at the edge of the image, the network may learn a larger offset to capture information in a wider range; while when dealing with smaller distortion in the center of the image, the network may learn a smaller offset to keep the integrity of the local structure.

[0092] Compared with the prior art, the technical solution of the present application has the following advantages: First, compared with the method of fixed convolution kernel, the deformable convolution operation of the present application can more flexibly adapt to different degrees of geometric deformation, thereby improving the correction effect. Second, compared with the method of directly predicting the entire deformation field, the method of the present application performs correction in a local adaptive manner, which can capture the global structure while maintaining local details. Finally, the method of the present application is implemented on the basis of U-HR Net, which can perform adaptive processing while maintaining high-resolution features.

[0093] Furthermore, the encoder is used to calculate the offset pixel value of each pixel using bilinear interpolation: Among them, x(p) represents the pixel value at any position of the feature map, where p = p0 + p n +Δp n ; x(q) represents the pixel value of the four neighboring pixels at the position x(p); BI represents the kernel function of bilinear interpolation.

[0094] Among them, the technical feature uses bilinear interpolation to calculate the pixel value of each pixel after offset. This method can accurately calculate the pixel value after offset while maintaining the continuity of the image, thereby improving the quality and accuracy of the corrected image. Specifically, the method realizes the accurate calculation of pixel values ​​through the following steps: 1. Determine the four neighboring pixels of the pixel to be calculated. 2. Use the bilinear interpolation kernel function to calculate the weight between the pixel to be calculated and the four neighboring pixels. 3. According to the calculated weight, the values ​​of the four neighboring pixels are weighted and summed to obtain the pixel value after offset. The advantages of this technical solution are: improving the accuracy of pixel value calculation, avoiding the error that may be caused by simple rounding; maintaining the continuity of the image, reducing the aliasing or distortion that may occur during the correction process; applicable to various complex wide-angle image distortion situations, with strong versatility. Through this precise pixel value calculation method, combined with deformable convolution and semi-supervised training strategies, the system can better handle the distortion problem in wide-angle images, especially in dealing with facial details and deep semantic changes. This not only improves the quality of the corrected image, but also enhances the system's adaptability to different types of wide-angle image distortion.

[0095] Furthermore, in the direction and range consistency, the value of each pixel in the output feature map is (-δ, δ), and the preset standard range is the interval F of (-∞, -δ]∪[δ, +∞). The pixels whose offset does not exceed the interval F are grouped into clusters of subtle movements. The cluster expression is:

[0096]

[0097] Where S(i,j) represents the mask or optical flow map of the segmentation output feature map; (i,j) is the pixel position in the mask or optical flow map after segmentation, and L ce represents the cross entropy loss.

[0098] Among them, this technical feature effectively identifies and screens out pixel offsets that meet the requirements by setting a preset standard range and a subtle motion cluster. Specifically: the preset standard range F provides a reasonable threshold range for the pixel offset by setting the interval of (-∞, -δ]∪[δ, +∞). This helps to filter out excessive offsets that may be caused by noise or errors. The subtle motion cluster M clusters the pixels whose offsets are within the preset range, effectively clustering the pixels that meet the requirements. This clustering method is helpful for subsequent processing and analysis. The mask or optical flow map S(i,j) is obtained by segmenting the output feature map, providing specific offset information for each pixel. This representation method is intuitive and easy to handle. In this way, the system can more accurately identify and process the pixels that need to be corrected, thereby improving the accuracy of wide-angle image correction. At the same time, this method is also conducive to reducing unnecessary calculations and improving the efficiency of the algorithm. Among them, the generation of the mask or optical flow map S(i,j) can be done in a variety of ways. One method is to use a convolutional neural network to generate a mask directly from the output feature map; another method is to use an optical flow estimation algorithm, such as FlowNet or PW C-Net, to generate an optical flow map of the pixel. These two methods have their own advantages and can be selected according to specific needs.

[0099] Furthermore, in the consistency regularization, the loss introduced is: L CR =[L mL (F1',F2')+λ2L ms (F1',F2')], where F'1 and F'2 are the optical flow maps output after the output feature maps enter the twin network STU-HRNet respectively.

[0100] In practical applications, by introducing consistency regularization, this technical solution can more effectively utilize limited labeled data in a semi-supervised learning framework, while improving the model's generalization ability for unlabeled data. This method not only improves the accuracy of the corrected image, but also enhances the stability and reliability of the model when dealing with various wide-angle image distortions.

[0101] Among them, the consistency regularization loss can be implemented in a variety of ways. First, the structure of the twin network STU-HRNet can be adjusted according to specific needs. For example, network layers of different depths can be used, or different convolution kernel sizes can be used to capture features of different scales. Secondly, the way the loss is calculated can also change. In addition to using the L2 norm, you can also consider using the L1 norm or other distance metrics to calculate the difference between F′1 and F′2.

[0102] This consistency regularization forms a complete semi-supervised learning framework with the previously mentioned proxy segmentation task and direction and range consistency. The proxy segmentation task provides a preliminary estimate of the unlabeled data, the direction and range consistency ensures the rationality of the pixel offset, and the consistency regularization further ensures the stability of the model output. These three components work together to significantly improve the effect of semi-supervised wide-angle image correction.

[0103] In practical applications, L ms is the consistency regularization loss, L ms The weight of can be adjusted according to the specific situation. For example, a weight coefficient λ2 can be set between 0.1 and 1.0, so that the final loss function is: L CR =[L mL (F1',F2')+λ2L ms (F1',F2')], where L mL is the loss of the main task. By adjusting the value of λ2, the importance of the main task and consistency regularization can be balanced.

[0104] As can be seen from the above, in order to solve the problems of scarcity of real samples in wide-angle datasets, inconsistent data annotation quality, and limitations of existing models in processing facial details and deep semantic changes. The present invention discloses a dual-branch UNet semi-supervised wide-angle image rectification algorithm (STU-HRNet) based on Swin-Transformer. Based on the existing model architecture, this method adopts the Swin-Transformer module instead of the Transformer module to construct the network structure of MFViST, which is specially designed to capture the subtle global and local information of the image. In addition, combined with the dual-branch UNet network, the method also incorporates a multi-scale fusion module composed of multiple deformable convolutional layers, further improving the model's ability to recognize and extract features of objects of different shapes and structures. In terms of training strategy, a semi-supervised learning method is adopted to reduce the burden and cost of preparing high-quality manually annotated datasets. This semi-supervised learning strategy includes two key components: direction and range consistency (DRC) and consistency regularization (CR), which effectively combines labeled and unlabeled data while avoiding overfitting in training to enhance the learning efficiency and performance of the model. At the same time, the algorithm also introduces a proxy segmentation task, which not only participates in the training process of the model, but also further promotes the adaptability and accuracy of the model for wide-angle image correction tasks by establishing the association between image segmentation results, corrected optical flow maps and labeled true values.

[0105] In order to better demonstrate the advantages of the present invention, three quantitative evaluation indicators are used to improve the objectivity and reliability of the evaluation. Specifically, linearity (StraightAcc) is used as an indicator to measure the linearity of image lines; shape consistency (ShapeAcc) is used to evaluate the fidelity of the shape of objects in the image; and conformal accuracy (ConformalAcc) is used as an indicator to evaluate the accuracy of the conformal properties of the image. These three indicators can be used to quantify the effect of image processing from different aspects, providing a more objective and comprehensive evaluation method for image correction and other processing tasks.

[0106] (1) Linearity (StraightAcc): In order to accurately maintain the linear structure in the image without being affected by distortion, the key is to estimate the most suitable parameters.

[0107]

[0108] Where α2 is the distance between the two endpoints of any given line segment based on the corrected image, and α1 is the perpendicular distance between the midpoint of the given line segment and the straight line connecting the two projected endpoints of the given line segment. If the deformation of the line segment is low, the straightness metric of the given line segment is close to 1, otherwise it is 0.

[0109] (2) Shape Consistency (ShapeAcc): In order to accurately evaluate the facial similarity between the rectified image and the reference image, a method based on facial landmark marking is adopted.

[0110]

[0111] Where n is the number of fixed sampling points in each face. and are the different key points of the corresponding face in the corrected image and the labeled image; θi represents the angle between the straight line between the key point of the corrected image and the midpoint of the face and the straight line between the key point of the labeled image and the midpoint of the face.

[0112] (3) Conformal Accuracy (ConformalAcc): In order to accurately evaluate the geometric shape preservation around the salient region, four sampling points are first selected around the region.

[0113]

[0114] where β n Represents the distance between the salient point and the projected sampling point. If the four values ​​from β1 to β4 are close to each other and the value is closer to 1, it indicates that the local morphology around the salient area is less distorted; otherwise, the distortion is greater.

[0115] In order to evaluate the superiority of the STU-HRNet correction method, the State-Of-The-Art (currently the best model) ResNeXt in the public correction method of wide-angle images was selected as the baseline model for comparison. The data sets were selected from the GooglePortraits data set and the WideAnglePortraits data set:

[0116] The experimental results are shown in the table:

[0117]

[0118] Compared with the ResNeXt algorithm, on the GooglePortraits dataset, the straightness (StraightAcc) of the present invention is improved by 0.1873, the shape consistency (ShapeAcc) is improved by 0.0784, and the conformal accuracy (ConformalAcc) is improved by 0.1328. At the same time, in the test results on the WideAnglePortraits dataset, the straightness (StraightAcc) is improved by 0.1392, the shape consistency (ShapeAcc) is improved by 0.0743, and the conformal accuracy (ConformalAcc) is improved by 0.0856. These objective data reflect the superiority of the present invention in processing wide-angle image distortion, especially in maintaining key indicators such as straightness, shape consistency and conformal accuracy.

[0119] Figure 5 The experimental results of the method for correcting distorted images are shown. The semi-supervised training strategy allows the model to use more distorted images for training, and under the action of Swin-Transformer, it makes full use of global and local information to adaptively fuse the corrected image features, thereby achieving better correction results.

[0120] In the second aspect, the present application proposes a dual-branch semi-supervised wide-angle image correction method based on Swin-Transformer, which is applied to any of the above-mentioned dual-branch semi-supervised wide-angle image correction systems based on Swin-Transformer, and includes the steps of:

[0121] S1: Obtain the distorted image and input the distorted image into the feature extraction backbone network module;

[0122] S2: convert the distorted image into three image blocks of different sizes, extract features from the three image blocks of different sizes and superimpose them to form a fused feature map;

[0123] S3: In the semi-supervised training strategy, the encoder performs a deformable convolution operation on the fused feature map to obtain an output feature map; the decoder uses a proxy segmentation task to perform optical flow estimation on the output feature map. The optical flow estimation is used to predict the offset of each pixel in the output feature map, and reconstruct each pixel at its new position after being displaced by its respective offset through image reconstruction to generate a corrected image;

[0124] S4: Output the corrected image.

[0125] Among them, this method is applied to any of the above-mentioned dual-branch semi-supervised wide-angle image correction systems based on Swin-Transformer, including feature extraction backbone network, semi-supervised training strategy, deformable convolution operation, proxy segmentation task and optical flow estimation and image reconstruction. These features work together to achieve effective correction of wide-angle images, solve problems such as global feature extraction, multi-scale fusion, and retaining corrected image features, while alleviating the problems of scarcity of real samples and inconsistent data annotation quality, and improving the model's ability to handle complex geometric deformations.

[0126] The feature extraction backbone network module of the present application can be implemented in a variety of ways. For example, the Swin-Transformer network can be used to replace the traditional Transformer network, and the distorted image can be converted into three image blocks of different sizes, and the block size can be set to 8x8, 16x16 and 32x32 pixels. This multi-scale feature extraction method can capture different levels of information of the image, thereby improving the comprehensiveness and accuracy of feature extraction.

[0127] Semi-supervised training strategies can include proxy segmentation tasks, direction and range consistency, and consistency regularization. The proxy segmentation task is used to estimate the optical flow of the output feature map and predict the offset of each pixel. Direction and range consistency is used to check whether the offset of each pixel in the feature map is within the preset standard range and form clusters of subtle movements. Consistency regularization adjusts the offset loss term of each pixel by introducing a loss to improve the accuracy of the corrected image.

[0128] The deformable convolution operation is performed in the encoder, which processes the fused feature map and enhances the model's adaptability to complex geometric deformations. The specific implementation can include a two-step convolution operation: first, the fused feature map is convolved with the original convolution kernel to obtain the initial output feature map; then the initial output feature map is convolved again to obtain the deformation offset of each pixel, thereby obtaining the output feature map after the offset is corrected.

[0129] The decoder uses a proxy segmentation task to perform optical flow estimation and predict the offset of each pixel in the output feature map. This method helps to accurately locate the area that needs to be corrected and improve the correction effect. The results of optical flow estimation are used in the subsequent image reconstruction process.

[0130] Image reconstruction is the last key step, which reconstructs the pixels according to the predicted offset to generate the corrected image. This process can use the bilinear interpolation method to calculate the pixel value of each pixel after the offset, ensuring the smoothness and continuity of the reconstructed image.

[0131] There are close connections and interactions between these features and steps. The feature extraction backbone network provides rich multi-scale feature information for subsequent processing. The semi-supervised training strategy runs through the entire process, effectively utilizing limited labeled data and a large amount of unlabeled data. The deformable convolution operation and optical flow estimation work together to improve the model's ability to handle complex geometric deformations. Finally, the image reconstruction step combines the processing results of the previous steps to generate the final corrected image.

[0132] When solving the problem of wide-angle image correction, the method of the present application first obtains multi-scale feature information through the feature extraction backbone network module, effectively solving the problems of global feature extraction and multi-scale fusion. Then, the application of the semi-supervised training strategy alleviates the problems of scarcity of real samples and inconsistent data annotation quality, and improves the generalization ability of the model. The combination of deformable convolution operations and optical flow estimation enhances the model's ability to handle complex geometric deformations, especially in dealing with facial details and deep semantic changes. Finally, through the precise image reconstruction process, the important features of the corrected image are retained.

[0133] Compared with the existing technology, the method of the present application introduces Swin-Transformer and multi-scale feature extraction to improve the comprehensiveness and accuracy of feature extraction; adopts a semi-supervised training strategy to effectively solve the problems of data scarcity and inconsistent annotation quality; combines deformable convolution and optical flow estimation to enhance the model's adaptability to complex geometric deformations; and ensures the high quality of the corrected image through a sophisticated image reconstruction process.

[0134] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0135] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0137] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0138] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of protection of the present application.

Claims

1. A dual-branch semi-supervised wide-angle image correction system based on Swin-Transformer, characterized in that: At least: Distorted image input module: used to obtain a distorted image and input the distorted image into a feature extraction backbone network module; The feature extraction backbone network module is used to convert the distorted image into three image blocks of different sizes, and extract feature superposition from the three image blocks of different sizes to form a fused feature map; Adaptive fusion dual-branch codec network module: at least includes an encoder and a decoder. In a semi-supervised training strategy, the encoder performs a deformable convolution operation on the fused feature map to obtain an output feature map; the decoder uses a proxy segmentation task to perform optical flow estimation on the output feature map, and the optical flow estimation is used to predict the offset of each pixel in the output feature map, and reconstruct each pixel at a new position after being displaced by its own offset through image reconstruction to generate a corrected image; Corrected image output module: used for outputting the corrected image.

2. According to claim 1, a dual-branch semi-supervised wide-angle image correction system based on Swin-Transformer is characterized in that: The semi-supervised training strategy includes at least a proxy segmentation task, direction and range consistency, and consistency regularization; The proxy segmentation task is used to perform optical flow estimation on the output feature map and predict the offset of each pixel in the output feature map; The direction and range consistency is used to check whether the offset of each pixel in the output feature map is within a preset standard range, and to obtain pixels whose offset does not exceed the preset standard range to form a cluster of subtle movements; Consistency regularization is used to introduce a loss amount and adjust the offset loss term of each pixel during image reconstruction to make the generated corrected image more accurate.

3. The dual-branch semi-supervised wide-angle image correction system based on Swin-Transformer according to claim 2, characterized in that: The feature extraction backbone network module uses VIT as the baseline network, adopts the Swin-Transformer network to replace the Transformer network, and converts the distorted image into three image blocks of different sizes, wherein the sizes of the three image blocks are Patch=8*8, Patch=16*16, and Patch=32*32 respectively.

4. The dual-branch semi-supervised wide-angle image correction system based on Swin-Transformer according to claim 3, characterized in that: The formula for extracting features from three image blocks of different sizes by the feature extraction backbone network module is: Among them, x class is the number of distorted image segmentation categories; It is a two-dimensional image sequence data that expands the input image according to the block size p (p = 8, 16, 32). N represents the number of features, and the parameter x is the input of three images of different sizes; L is the sequence data dimension, L = HW / p 2 , where HW represents the image dimension; E and E pos Represent block coding and position coding matrices respectively; Represents the output sequence data of the current layer, and the subscript p represents the different sizes of the blocks; Represents the output sequence data of the previous layer, Represents the output sequence data of the next layer; Represents the sequence data after encoding at the current layer, represents the sequence data after encoding in the next layer; W-MSA represents the window multi-head self-attention layer; MLP represents the multi-layer perceptron, SW-MSA represents the shifted window multi-head self-attention layer, LN represents the normalization layer, Represents the sequence data after conversion into three image blocks of different sizes.

5. According to claim 4, a dual-branch semi-supervised wide-angle image correction system based on Swin-Transformer is characterized in that: The formula for forming the fusion feature map by the feature extraction backbone network module is: Among them, Reshape means array change, Conv means convolution, BN means sample normalization, Add(Upsample) means bilinear interpolation upsampling, y Out It is the feature matrix after the fusion of multiple block-encoded ViST networks, that is, the fused feature map.

6. A dual-branch semi-supervised wide-angle image correction system based on Swin-Transformer according to claim 2, characterized in that: The adaptive fusion dual-branch codec network module uses U-HRNet as the baseline network. In the semi-supervised training strategy, the encoder performs a deformable convolution operation on the fusion feature map, including: Use the original convolution kernel to convolve the fused feature map to obtain the initial output feature map y Out Each position p0 in is as follows: Among them, K is the conventional convolution kernel, p n is the offset position, ω is the initial weight; The initial output feature map is convolved again to obtain the deformation offset Δp of each pixel in the deformable convolution. n , thus obtaining the output feature map after correcting the offset:

7. A dual-branch semi-supervised wide-angle image correction system based on Swin-Transformer according to claim 6, characterized in that: The encoder is also used to calculate the offset pixel value of each pixel using bilinear interpolation: Among them, x(p) represents the pixel value at any position of the fused feature map, where p = p0 + p n +Δp n ; x(q) represents the pixel value of the four neighboring pixels at the position x(p); BI represents the kernel function of bilinear interpolation.

8. The dual-branch semi-supervised wide-angle image correction system based on Swin-Transformer according to claim 7, characterized in that: In the direction and range consistency, the value of each pixel in the output feature map is (-δ, δ), the preset standard range is an interval F of (-∞, -δ]∪[δ, +∞), and the pixel points whose offset does not exceed the interval F are grouped into clusters of subtle movements. The expression of the cluster is: L DRC =[L ce (S1',S1”)+L ce (S2',S2”)] Where S(i, j) represents the mask or optical flow map for segmenting the output feature map; F(i, j) is the pixel position in the mask or optical flow map after segmentation, and L ce represents the cross entropy loss, L DRC Indicates scope consistency.

9. The dual-branch semi-supervised wide-angle image correction system based on Swin-Transformer according to claim 8, characterized in that: In the consistency regularization, the loss introduced is: L CR =[L mL (F1',F2')+λ2L ms (F1',F2')], where L CR represents consistency regularization, F1' and F2' are the optical flow maps outputted from the two branches of the output feature map STU-HRNet, and L ms is the consistency regularization loss, L mL is the loss of the main task, and λ2 is the weight coefficient.

10. A dual-branch semi-supervised wide-angle image correction method based on Swin-Transformer, characterized in that: The method is applied to the dual-branch semi-supervised wide-angle image correction system based on Swin-Transformer as described in any one of claims 1 to 7, comprising the steps of: S1: Obtain a distorted image, and input the distorted image into a feature extraction backbone network module; S2: converting the distorted image into three image blocks of different sizes, and extracting features from the three image blocks of different sizes and superimposing them to form a fused feature map; S3: In the semi-supervised training strategy, the encoder performs a deformable convolution operation on the fused feature map to obtain an output feature map; the decoder uses a proxy segmentation task to perform optical flow estimation on the output feature map, the optical flow estimation is used to predict the offset of each pixel in the output feature map, and reconstructs each pixel at a new position after being displaced by its own offset through image reconstruction to generate a corrected image; S4: Outputting the corrected image.

Images