Automatic fisheye image distortion correction method based on global and local feature fusion
An automatic distortion correction method for fisheye images, which integrates global and local features, utilizes deep neural networks to extract and weightedly fuse features, thus solving the problem of low accuracy in distortion parameter estimation in traditional methods and generating high-quality distortion-corrected images.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NAT UNIV OF DEFENSE TECH
- Filing Date
- 2025-12-18
- Publication Date
- 2026-05-08
AI Technical Summary
Traditional fisheye image distortion correction methods mainly focus on long arc features, which result in slow iterative solution speed and unstable performance. They also fail to effectively utilize the rich semantic category information of straight line features, leading to low accuracy in distortion parameter estimation.
An automatic distortion correction model for fisheye images based on the fusion of global and local features is constructed. Global and local features are extracted through a deep neural network and weighted fusion is performed using an attention mechanism to improve the accuracy of distortion parameter estimation.
It improves the accuracy of distortion parameter estimation in fisheye images, enhances distortion correction effects, and generates high-quality distortion-corrected images.
Smart Images
Figure CN121998876A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to automatic distortion correction technology for fisheye images, specifically to an automatic distortion correction method for fisheye images based on the fusion of global and local features. Background Technology
[0002] Fisheye image distortion correction technology aims to estimate distortion parameters from an input fisheye image and transform the image based on the estimation results to remove distortion and obtain a distortion-corrected image. In practical fisheye image distortion correction, image regions corresponding to semantic categories rich in linear features (such as buildings and roads) often contain more structural deformation information than regions corresponding to other categories, making them more meaningful for fisheye image distortion parameter estimation and correction. Traditional fisheye image distortion correction methods mainly focus on long arc features in fisheye images. However, due to limitations in the extraction accuracy of long arc objects, the iterative solution process is slow, and the distortion correction effect is unstable. Furthermore, traditional methods do not focus on specific semantic categories rich in linear features, failing to utilize this information to further improve the accuracy of fisheye distortion parameter estimation. To address these issues, this patent proposes an automatic fisheye image distortion correction method based on global and local feature fusion. This method employs a deep neural network model to fully mine the global and local features of the fisheye image and performs weighted fusion of these features, thereby further improving the accuracy of fisheye distortion parameter estimation and enhancing the distortion correction effect. Summary of the Invention
[0003] In view of this, it is necessary to propose an automatic distortion correction method for fisheye images based on the fusion of global and local features to address the aforementioned technical problems. This method constructs a simulation dataset for automatic distortion correction of fisheye images, proposes an automatic distortion correction model for fisheye images based on the fusion of global and local features, and trains the proposed model using the dataset. The trained model is then tested on input fisheye images to generate distortion parameters and distortion-corrected images.
[0004] The automatic distortion correction method for fisheye images based on the fusion of global and local features mainly includes the following steps: Step 1: Construct a simulation dataset for automatic fisheye image distortion correction. Based on existing semantic segmentation datasets, fisheye distortion parameters are set, and the normal, distortion-free images and semantic segmentation results in the semantic segmentation dataset are transformed according to the fisheye distortion model to generate simulated fisheye images and corresponding semantic segmentation results. Each sample in the simulation dataset for automatic fisheye image distortion correction includes a fisheye image, the corresponding semantic segmentation result, and distortion parameters. The dataset is divided into three parts: a training set, a validation set, and a test set. Existing semantic segmentation datasets include the ADE20K dataset, etc. The distortion parameters of the fisheye image include the coordinates of the center of the fisheye region and the distortion polynomial coefficients. The sample size ratio of the training set, validation set, and test set is 6:2:2.
[0005] Step 2: Construct an automatic distortion correction model for fisheye images based on the fusion of global and local features. This model mainly includes: a global feature extraction module, a local region information extraction module, an attention-based weighted fusion module, a distortion parameter estimation module, and a distortion correction module. The global feature extraction module aims to extract global features from the input fisheye image, grasping the overall distortion degree and distortion parameter information of the entire input image. This module can be implemented using a pre-trained deep neural network VGG16, selecting the output of intermediate convolutional layers as global features. The local region information extraction module performs semantic segmentation on the input fisheye image and encodes specific regions rich in linear information from the semantic segmentation results, obtaining local region features while maintaining the width and height of these features consistent with the global features. These specific semantic regions rich in linear features include areas such as buildings and roads. The global feature extraction module and the local region information extraction module are connected to the attention-based weighted fusion module. Global and local features are concatenated and weighted using an attention mechanism, allowing the attention-based weighted fusion module to focus on features in specific regions, further improving the estimation accuracy of fisheye distortion parameters. The distortion parameter estimation module generates fisheye distortion parameter results, and the distortion correction module then obtains the final fisheye distortion-corrected image.
[0006] Step 3: Iteratively train the proposed model using the dataset constructed in Step 1 to obtain a trained automatic distortion correction model for fisheye images based on the fusion of global and local features. The training process of the proposed model can be completed using deep learning frameworks such as PyTorch, Tensorflow, or MXNet.
[0007] Step 4: Using the trained fisheye image automatic distortion correction model based on global and local feature fusion, perform fusion feature extraction and distortion parameter estimation on the test fisheye image, and finally generate the distortion-corrected image.
[0008] The beneficial effects of this invention are as follows: Compared with traditional fisheye image distortion correction algorithms, the method proposed in this invention uses a deep neural network model to fully mine the global and local features of fisheye images and performs weighted fusion on these features. This can more effectively utilize the semantic categories with rich linear features and the structural deformation information they contain, thereby further improving the accuracy of fisheye distortion parameter estimation and enhancing the distortion correction effect. Attached Figure Description
[0009] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is an architecture diagram of an automatic distortion correction model for fisheye images based on the fusion of global and local features. Detailed Implementation
[0010] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0011] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0012] like Figure 1 As shown in the figure, the network framework of the fisheye image automatic distortion correction method based on global and local feature fusion provided in this embodiment has the following main implementation steps: This invention proposes an automatic distortion correction method for fisheye images based on the fusion of global and local features. The method constructs a simulation dataset for automatic distortion correction of fisheye images, proposes an automatic distortion correction model for fisheye images based on the fusion of global and local features, and trains the proposed model using the dataset. The trained model is then tested on input fisheye images to generate distortion parameters and a distortion-corrected image.
[0013] The automatic distortion correction method for fisheye images based on the fusion of global and local features mainly includes the following steps: Step 1: Construct a simulation dataset for automatic fisheye image distortion correction. Based on existing semantic segmentation datasets, fisheye distortion parameters are set, and the normal, distortion-free images and semantic segmentation results in the semantic segmentation dataset are transformed according to the fisheye distortion model to generate simulated fisheye images and corresponding semantic segmentation results. Each sample in the simulation dataset for automatic fisheye image distortion correction includes a fisheye image, the corresponding semantic segmentation result, and distortion parameters. The dataset is divided into three parts: a training set, a validation set, and a test set. Existing semantic segmentation datasets include the ADE20K dataset, etc. The distortion parameters of the fisheye image include the coordinates of the center of the fisheye region and the distortion polynomial coefficients. The sample size ratio of the training set, validation set, and test set is 6:2:2.
[0014] Step 2: Construct an automatic distortion correction model for fisheye images based on the fusion of global and local features. The model structure is as follows: Figure 2 As shown, the model mainly includes: a global feature extraction module, a local region information extraction module, an attention mechanism weighted fusion module, a distortion parameter estimation module, and a distortion correction module. The global feature extraction module aims to extract global features from the input fisheye image, grasping the overall distortion degree and distortion parameter information of the entire input image. This module can be implemented using a pre-trained deep neural network VGG16, selecting the output of intermediate convolutional layers as global features. The local region information extraction module performs semantic segmentation on the input fisheye image and selects specific regions rich in linear information from the semantic segmentation results for encoding, obtaining local region features while maintaining the width and height of these features consistent with the global features. These specific semantic regions rich in linear features include areas such as buildings and roads. The global feature extraction module and the local region information extraction module are connected to the attention mechanism weighted fusion module, concatenating the global and local region features and using an attention mechanism for weighted fusion. This allows the attention mechanism weighted fusion module to focus on features in specific regions, further improving the estimation accuracy of fisheye distortion parameters. The distortion parameter estimation module generates fisheye distortion parameter results, and the distortion correction module then obtains the final fisheye distortion-corrected image.
[0015] Step 3: Iteratively train the proposed model using the dataset constructed in Step 1 to obtain a trained automatic distortion correction model for fisheye images based on the fusion of global and local features. The training process of the proposed model can be completed using deep learning frameworks such as PyTorch, Tensorflow, or MXNet. During training, the optimizer is Adam, and the network is built and trained under the PyTorch deep learning development framework with an initial learning rate set to...
[0016] Step 4: Using a pre-trained fisheye image automatic distortion correction model based on global and local feature fusion, the test fisheye image is subjected to fusion feature extraction and distortion parameter estimation, ultimately generating a distortion-corrected image. The proposed method can effectively extract and fuse global and local features, and on this basis, perform distortion parameter estimation to generate a high-quality distortion-corrected image.
[0017] Specifically, the global feature extraction module uses the output of intermediate convolutional layers of a pre-trained deep neural network (e.g., VGG16, ResNet50, etc.) as the global feature representation, denoted as [symbol missing]. ,in These represent the height, width, and number of channels of the feature map, respectively; the local region information extraction module uses a semantic segmentation network (such as DeepLabv3 or a Transformer-based segmentation network) to process the input fisheye image. Perform pixel-level category prediction to obtain the semantic mask. Based on semantic masks, specific regions rich in linear information are selected, and feature encoding is performed on these selected regions to extract local features. The design combines spatial and channel attention modules for global features. and local features Weighted fusion is performed, and the weighted fusion formula is defined as follows: ; in, As a feature of fusion, For learnable weight parameters, satisfying ; The attention weight map is calculated as follows: ; in, This represents the Sigmoid activation function. This represents element-wise multiplication. This indicates a combination of spatial and channel attention module operations; The distortion parameter estimation module will fuse features. The distortion parameter vector is output by mapping to the fisheye distortion parameter space through a series of convolutional and fully connected layers. ,in Indicates the first The distortion model parameters are defined as follows: The distortion model is a simplified radial distortion model, and the sampling formula is: ; in: The angle between the viewing angle and the optical axis, This corresponds to the pixel radius after distortion; The distortion correction module utilizes the estimated distortion parameters The input fisheye image is corrected using inverse distortion mapping to restore a distortion-free image. .
[0018] Furthermore, attention mechanisms Designed to incorporate channel attention Spatial attention The product of: ; in, This represents the global average pooling function, and its output size. , The weight matrix in channel attention. The convolution kernel parameters represent spatial attention. , , These represent convolution, matrix multiplication, and element-wise multiplication, respectively.
[0019] Weight parameters in the model Iterative updates are performed using soft constraints, guided by the following regularization loss: ; in, To balance the weights of the regularization terms and ensure the stability and sparsity of the fused weights.
[0020] Specifically, the local region information extraction module incorporates a dynamic selection mechanism for local regions, based on a local region straightness evaluation function. Measured region The line segment characteristics of the inner edge, ,in, For the region The set of all detected edge segments within, Evaluate the straightness of a single line segment e, selecting only those that meet the threshold. Feature encoding is performed on the region to ensure the effectiveness of local features.
[0021] In this embodiment, the automatic distortion correction model for fisheye images incorporates a regression loss based on distortion parameters. and image reprojection error loss Define the joint loss function: ; Among them, the distortion parameter regression loss , For prediction parameters, For the true labeled parameters, the image reprojection error loss is calculated by comparing the key points of the corrected image with the true points before distortion.
[0022] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered to be within the scope of this specification. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
Claims
1. An automatic distortion correction method for fisheye images based on global and local feature fusion mainly includes the following steps: Step 1: Construct a simulation dataset for automatic distortion correction of fisheye images. Each sample in the simulation dataset includes a fisheye image, the corresponding semantic segmentation result, and distortion parameters. Step 2: Construct an automatic distortion correction model for fisheye images based on the fusion of global and local features. The model includes: a global feature extraction module, a local region information extraction module, an attention mechanism weighted fusion module, a distortion parameter estimation module, and a distortion correction module. Step 3: Use the dataset constructed in Step 1 to iteratively train the fisheye image automatic distortion correction model to obtain the trained fisheye image automatic distortion correction model based on the fusion of global and local features. Step 4: Using the trained fisheye image automatic distortion correction model based on global and local feature fusion, perform fusion feature extraction and distortion parameter estimation on the test fisheye image, and finally generate the distortion-corrected image; The global feature extraction module aims to extract global features from the input fisheye image, grasping the overall distortion degree and distortion parameter information of the entire input image. The local region information extraction module performs semantic segmentation on the input fisheye image and selects specific regions rich in linear information from the semantic segmentation results for encoding to obtain local region features. The global feature extraction module and the local region information extraction module are connected to the attention mechanism weighted fusion module, enabling the attention mechanism weighted fusion module to focus on features in specific regions. The distortion parameter estimation module generates fisheye distortion parameter results, and the distortion correction module obtains the final fisheye distortion-corrected image.
2. The automatic distortion correction method for fisheye images based on global and local feature fusion according to claim 1, characterized in that: The global feature extraction module uses the output of the intermediate convolutional layers of a pre-trained deep neural network as the global feature representation, denoted as [symbol missing]. ,in These represent the height, width, and number of channels of the feature map, respectively; the local region information extraction module uses a semantic segmentation network to process the input fisheye image. Perform pixel-level category prediction to obtain the semantic mask. Based on semantic masks, specific regions rich in linear information are selected, and feature encoding is performed on these selected regions to extract local features. The design combines spatial and channel attention modules for global features. and local features Weighted fusion is performed, and the weighted fusion formula is defined as follows: ； in, As a feature of fusion, For learnable weight parameters, satisfying ; The attention weight map is calculated as follows: ； in, This represents the Sigmoid activation function. This represents element-wise multiplication. This indicates a combination of spatial and channel attention module operations; The distortion parameter estimation module will fuse features. The distortion parameter vector is output by mapping to the fisheye distortion parameter space through a series of convolutional and fully connected layers. ,in Indicates the first The distortion model parameters are defined as follows: The distortion model is a simplified radial distortion model, and the sampling formula is: ； in, The angle between the viewing angle and the optical axis, This corresponds to the pixel radius after distortion; The distortion correction module utilizes the estimated distortion parameters The input fisheye image is corrected using inverse distortion mapping to restore a distortion-free image. .
3. The automatic distortion correction method for fisheye images based on global and local feature fusion according to claim 2, characterized in that: Attention mechanism Designed to incorporate channel attention Spatial attention The product of: ； in, This represents the global average pooling function, and its output size. , The weight matrix in channel attention. The convolution kernel parameters represent spatial attention. These represent convolution, matrix multiplication, and element-wise multiplication, respectively.
4. The automatic distortion correction method for fisheye images based on global and local feature fusion according to claim 2, characterized in that: Weight parameters in the model Iterative updates are performed using soft constraints, guided by the following regularization loss: ； in, To balance the weights of the regularization terms and ensure the stability and sparsity of the fused weights.
5. The automatic distortion correction method for fisheye images based on global and local feature fusion according to claim 2, characterized in that: The local region information extraction module incorporates a dynamic selection mechanism for local regions, based on a local region straightness evaluation function. Measured region The line segment characteristics of the inner edge, ,in, For the region The set of all detected edge segments within, Evaluate the straightness of a single line segment e, selecting only those that meet the threshold. Feature encoding is performed on the region to ensure the effectiveness of local features.
6. The automatic distortion correction method for fisheye images based on global and local feature fusion according to claim 2, characterized in that: Automatic distortion correction model for fisheye images incorporates regression loss based on distortion parameters. and image reprojection error loss Define the joint loss function: ； Among them, the distortion parameter regression loss , For prediction parameters, For the true labeled parameters, the image reprojection error loss is calculated by comparing the key points of the corrected image with the true points before distortion. , and These are the weight parameters.
7. The automatic distortion correction method for fisheye images based on global and local feature fusion according to claim 1, characterized in that: The existing semantic segmentation dataset in step one includes the ADE20K dataset. The distortion parameters of the fisheye image include the center coordinates of the fisheye region and the distortion polynomial coefficients. The ratio of the number of samples in the training set, validation set, and test set is 6:2:
2.
8. The automatic distortion correction method for fisheye images based on global and local feature fusion according to claim 1, characterized in that: The training process of the model proposed in step three is completed using the PyTorch, Tensorflow, or MXNet deep learning framework.