An end-to-end geometric correction method for pointer instrument images based on dual branches
By regressing perspective and rotation parameters separately through a dual-branch network, and combining circular mask and differentiable interpolation sampling, the perspective distortion and planar rotation coupling problems of pointer instrument images are solved, achieving high-accuracy end-to-end geometric correction, which is suitable for instrument reading processing in complex environments.
Patent Information
- Application Number
- CN202611088142.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-22
- Publication Date
- 2026-11-10
AI Technical Summary
In the existing technology, perspective distortion and planar rotation of pointer instrument images are coupled with each other, artificial features are easily affected by complex environments, background boundaries cause deviations in direction judgment, and the accumulation of errors in multi-stage processing leads to a decrease in the accuracy of instrument readings.
An end-to-end geometric correction method for pointer instrument images based on a dual-branch approach is adopted. Image features are extracted through a shared feature extraction network, perspective parameter regression branch and rotation parameter regression branch are set in parallel, a circular mask is used to reduce background interference, differentiable interpolation sampling is used for correction, and a joint loss function is combined for training.
Independent regression of perspective and rotation parameters was achieved, background interference was suppressed, error accumulation was reduced, the accuracy of instrument readings and generalization ability in complex environments were improved, and standardized images were output for easy subsequent processing.
Smart Images

Figure CN123071202A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine vision and industrial image processing technology, specifically to an end-to-end geometric correction method for pointer instrument images based on a dual-branch architecture. Background Technology
[0002] Pointer-type meters are widely used in fields such as power, petroleum, chemical, machinery manufacturing, rail transportation, and industrial automation. With the development of intelligent inspection technology, using fixed cameras, mobile terminals, inspection robots, or drones to collect meter images and automatically identify meter readings through machine vision technology has become an important technical means to replace manual meter reading.
[0003] In actual image acquisition, because the camera's optical axis is usually not always perpendicular to the instrument panel, the acquired pointer instrument images are prone to both perspective distortion and planar rotation. Perspective distortion causes a circular dial to appear as an ellipse or other projected shape in the image, resulting in non-linear changes in scale spacing and pointer angle; planar rotation causes the instrument's zero-scale position, digital direction, and unit text direction to deviate from the preset standard direction. These geometric distortions reduce the accuracy of pointer segmentation, scale positioning, pointer angle calculation, and instrument reading mapping.
[0004] Existing geometric correction methods mainly include circle or ellipse detection methods based on Hough transform, contour fitting methods, corner or local feature matching methods, template matching methods, and camera calibration parameter-based methods. These methods typically rely on clear, intact, and stable dial edges, scales, or local features. Manually designed features are prone to failure when there are reflections, raindrops, blur, dim light, stains, or complex backgrounds in the instrument image.
[0005] Existing deep learning-based instrument image processing methods typically treat instrument detection, dial segmentation, perspective correction, and rotation correction as independent processing stages. On the one hand, multi-stage processing can cause errors from previous stages to propagate to subsequent stages; on the other hand, some methods only predict transformation parameters through neural networks without utilizing the structural and pixel information of the corrected image for end-to-end constraints. Furthermore, some methods use a single parameter branch to simultaneously output different types of geometric parameters, which can easily lead to mutual interference between perspective and rotation parameters.
[0006] In addition, conventional neural networks may infer the rotation direction using non-dial features such as rectangular boundaries, filled regions, or background textures of the input image, resulting in lower loss on the training data, but producing larger correction errors when the background changes in the real environment.
[0007] Therefore, it is necessary to provide a geometric correction method that can utilize the prior knowledge of the circular structure of the pointer instrument, learn perspective transformation and planar rotation separately, and implement end-to-end joint supervision through the correction image. Summary of the Invention
[0008] (a) Technical problems to be solved
[0009] To address the shortcomings of existing technologies, this invention provides an end-to-end geometric correction method for pointer instrument images based on a dual-branch architecture. This method solves the problems of perspective distortion and planar rotation coupling, the susceptibility of artificial features to interference from complex environments, orientation deviations caused by background boundaries, and the accumulation of errors in multi-stage processing in existing technologies.
[0010] (II) Technical Solution
[0011] To achieve the above objectives, the present invention provides the following technical solution: an end-to-end geometric correction method for pointer instrument images based on dual branches, comprising: acquiring an image to be corrected containing a pointer instrument dial, normalizing the size of the image to be corrected, generating a circular mask according to the center position and effective radius of the pointer instrument dial, and performing pixel-by-pixel operations on the image to be corrected and the circular mask to obtain a mask input image;
[0012] The mask input image is input into a pre-trained geometric correction network. The shared image features of the mask input image are extracted through the shared feature extraction network of the geometric correction network. The shared image features are then aggregated globally to obtain a global feature vector that simultaneously represents the perspective deformation and orientation information of the dial.
[0013] The global feature vector is input into the perspective parameter regression branch and the rotation parameter regression branch, which are set in parallel. The perspective parameter regression branch outputs homography correction parameters for mapping the mask input image to a standard viewpoint, and the rotation parameter regression branch outputs rotation correction parameters for adjusting the pointer instrument dial to a preset standard direction.
[0014] A homography correction matrix is constructed based on the homography correction parameters, and a first inverse sampling grid pointing from the perspective correction image coordinates to the mask input image coordinates is established based on the homography correction matrix. The mask input image is then sampled using the first inverse sampling grid to obtain a perspective correction intermediate image.
[0015] Based on the rotation correction parameters, a second inverse sampling grid is established, pointing from the coordinates of the target correction image to the coordinates of the perspective correction intermediate image. The perspective correction intermediate image is then sampled using the second inverse sampling grid to obtain a standardized pointer instrument image that simultaneously completes perspective correction and planar rotation correction.
[0016] Preferably, the shared feature extraction network includes multiple convolutional feature extraction units and an adaptive global average pooling layer connected in sequence; the perspective parameter regression branch and the rotation parameter regression branch share the shared feature extraction network, the perspective parameter regression branch includes a first fully connected layer, a first nonlinear activation layer and a first output layer, and the rotation parameter regression branch includes a second fully connected layer, a second nonlinear activation layer and a second output layer.
[0017] Preferably, the perspective parameter regression branch outputs eight homography correction parameters, and the homography correction matrix is constructed according to the following formula; the output layer of the perspective parameter regression branch is initialized to output parameters corresponding to the unit homography matrix, and the output layer of the rotation parameter regression branch is initialized to output zero rotation parameters.
[0018] Preferably, the effective area in the circular mask is determined by the center position of the dial and the effective radius of the dial. The circular mask has a continuously changing transition area at the boundary between the effective area and the ineffective area. When training the geometric correction network, the circular mask is used to weight the image reconstruction error between the standardized pointer instrument image and the corresponding standard forward image to reduce the influence of the background pixels outside the dial on the network training.
[0019] Preferably, the training samples of the geometric correction network are generated in the following manner: a standard positive pointer instrument image is acquired; random perspective transformation and random planar rotation transformation are sequentially applied to the standard positive pointer instrument image; and at least one image degradation processing is applied, including brightness change, contrast change, color change, blur, noise, reflection, raindrop simulation, and image compression, to obtain a distorted input image; the inverse transformation parameters of the random perspective transformation are recorded as homography correction parameter labels, and the inverse rotation parameters of the random planar rotation transformation are recorded as rotation correction parameter labels.
[0020] Preferably, the geometric correction network is trained using a joint loss function, which includes at least: a mask image reconstruction loss determined based on the standardized pointer instrument image and the standard positive pointer instrument image; a perspective parameter supervision loss determined based on the predicted homography correction parameters and the homography correction parameter labels; and a rotation parameter supervision loss determined based on the predicted rotation correction parameters and the rotation correction parameter labels. The mask image reconstruction loss is backpropagated through the first inverse sampling grid and the second inverse sampling grid to the shared feature extraction network, the perspective parameter regression branch, and the rotation parameter regression branch.
[0021] Preferably, the perspective parameter supervision loss further includes corner reprojection loss, which is determined based on the positional differences of multiple reference points in the preset dial area after transformation by the predicted homography correction matrix and the true homography correction matrix; when the rotation correction parameter covers the angular period boundary, the rotation parameter supervision loss is determined based on the shortest period angular distance between the predicted angle and the true angle.
[0022] (III) Beneficial Effects
[0023] Compared with the prior art, the present invention provides an end-to-end geometric correction method for pointer instrument images based on a dual-branch architecture, which has the following advantages:
[0024] Reduce task interference between perspective and rotation parameters.
[0025] By using a shared feature extraction network and a parallel bi-branch regression structure, targeted parameter regression is performed on perspective distortion and rotation direction while sharing the visual features of the dial.
[0026] Suppress incorrect associations caused by non-dial backgrounds.
[0027] Circular masks enable the network to focus on effective content such as dial outlines, scales, numbers, text, and pointers, while reducing interference from rectangular boundaries, filled areas, and background textures on rotation parameter prediction.
[0028] Achieve end-to-end optimization.
[0029] The first and second inverse sampling grids use differentiable interpolation, and the reconstruction loss of the final corrected image can simultaneously constrain the perspective branch, rotation branch, and shared feature extraction network.
[0030] Improve the physical interpretability of geometric parameters.
[0031] The perspective branch outputs a homography matrix with clear projection geometry, while the rotation branch outputs rotation parameters with clear directional significance, facilitating result detection, anomaly judgment, and downstream system calls.
[0032] Improve the generalization ability in complex industrial environments.
[0033] By applying degradation effects such as brightness changes, color changes, blurring, reflections, raindrops, and image compression to the training images, the network reduces its dependence on fixed colors and imaging conditions.
[0034] This facilitates subsequent instrument readings.
[0035] The output image simultaneously performs perspective and rotation correction, providing a unified standardized image for pointer segmentation, scale positioning, pointer angle calculation, and reading mapping. Attached Figure Description
[0036] Figure 1 Dual-branch pointer instrument image correction model structure;
[0037] Figure 2 Data preprocessing steps;
[0038] Figure 3 Partial training set samples;
[0039] Figure 4 Compare the correction effect of the experimental group on the test samples;
[0040] Figure 5 Comparison of pointer instrument image calibration results. Detailed Implementation
[0041] To better understand the purpose, structure, and function of this invention, the following description is provided in conjunction with the appendix. Figure 1-5 The present invention provides a more detailed description of an end-to-end geometric correction method for pointer instrument images based on a dual-branch structure.
[0042] 1. Method Framework
[0043] The overall architecture of the model is as follows Figure 1 As shown, it comprises three core components: a feature extraction backbone network, a two-branch regression head, and a sequential geometric correction module. This correction method mainly consists of four core parts:
[0044] (1) Data preprocessing and augmentation module, used to synthesize training data with perspective and rotation distortion from standard forward images and construct geometric correspondence;
[0045] (2) Feature extraction backbone network, used to encode high-level semantic and geometric features from distorted input images;
[0046] (3) Two-branch regression head, which regresses perspective transformation parameters and rotation angles based on the extracted features;
[0047] (4) Sequential geometric correction module: The input image is subjected to differentiable perspective and rotation transformations using the prediction parameters to generate the final correction result. The entire process is trained end-to-end in a supervised learning manner.
[0048] 1.1 Data Preprocessing and Synthesis
[0049] To train the model, a large number of pairs of distorted input images are required. ) and positive target image ( Since paired data is difficult to obtain in real-world scenarios, this paper employs a method of randomly synthesizing distorted data from standard forward instrument images. For each standard forward image in the dataset... ,like Figure 2As shown, the specific steps are as follows:
[0050] At the four corners of the image Apply random perturbation to generate the perturbed target corner point. Calculate using these two sets of points To perspective distortion image homography matrix It must meet the following requirements:
[0051]
[0052] in, It is a 3x3 matrix. This transformation simulates projection distortion caused by non-frontal shooting. The network needs to learn its inverse transformation. The first 8 parameters are used as one of the targets for perspective correction.
[0053] Image after perspective transformation Apply a Rotation angle of uniform sampling within the range The final input image is obtained. This operation simulates the instrument's arbitrary orientation within a plane.
[0054] To prevent the network from using irrelevant background features such as rectangular borders of images for cheating corrections, and A centralized circular mask is used to black out the four corner areas, forcing the model to focus on the geometric and content features of the dashboard itself.
[0055] Only for the input image Random brightness and contrast fluctuations are applied to improve the model's robustness to changes in lighting conditions. Finally, each training sample is represented as a quadruple. ,in yes The 8-dimensional vectorized representation of the model. The model uses distorted images... As input, the geometric parameters required for correction are predicted through an end-to-end differentiable geometric transformation network, and perspective correction and rotation transformation are applied sequentially to finally output a fully corrected positive instrument image.
[0056] 1.2 Feature Extraction Backbone Network
[0057] We employ a ResNet-18 pre-trained on ImageNet as the backbone network, leveraging its powerful spatial feature extraction capabilities. We remove the final global average pooling layer and fully connected layers, retaining the convolutional layers as feature encoders. For an input size of... The image is processed by a backbone network to obtain feature maps. .in For the number of channels, This represents the spatial dimensions of the feature map. This feature map encodes high-level semantic information and spatial geometric features of the input image. To further extract the global representation, an adaptive average pooling layer compresses the feature map into a global feature vector. This serves as the input to the subsequent regression head. This design ensures the model's adaptability to different input sizes.
[0058] 1.2.1 Two-branch regression head
[0059] The model employs a parallel dual-branch structure, branching from the global feature vectors respectively. The regression perspective transformation parameters and rotation angles are used. Both branches consist of two fully connected layers, with nonlinearity introduced through the ReLU activation function.
[0060] 1.2.2 Transspectral Correction Head
[0061] The perspective correction head maps the 512-dimensional feature vector f into an 8-dimensional parameter vector. The corresponding homography matrix The homography matrix consists of 8 elements (the last element is always 1). The homography matrix is defined as:
[0062]
[0063] in These are the 8-dimensional parameters predicted by the network. The homography matrix describes the geometric mapping from the input distorted image space to the standard forward image space. To ensure the stability of network training, the weights of the last fully connected layer are set to zero during initialization, and the biases are set to the parameter values corresponding to the unit transformation. This makes the initial output of the network approximate the identity transformation.
[0064] 1.2.3 Rotating Correction Head
[0065] The rotation correction head starts from the 512-dimensional feature vector f and outputs a scalar. This represents the rotation angle (in degrees) of the input image relative to the plane of the front view. A positive value indicates counter-clockwise rotation, and a negative value indicates clockwise rotation. The angle range is... The last fully connected layer in this branch is initialized with zero weights and zero biases, so that the network's initial prediction has a zero rotation angle.
[0066] 1.2.4 Sequential Geometric Correction Process
[0067] The calibration process employs a differentiable spatial transformation network to ensure end-to-end trainability of the entire model. Geometric calibration consists of two sequential steps: first, perspective correction is performed to eliminate projection distortion, followed by rotational transformation to adjust the instrument to a positive orientation.
[0068] (1) Perspective correction
[0069] Using the predicted 8-dimensional parameters Reconstructed homography matrix Subsequently, a differentiable perspective transformation was achieved by constructing a standardized sampling grid and applying matrix transformations, based on... For the input image Perform inverse perspective transformation to obtain a preliminarily corrected image. This is used to eliminate perspective distortion.
[0070] (2) Rotation correction
[0071] Building upon perspective correction, further correction is made for in-plane rotation. The predicted angles are then... Take the negative Construct the corresponding 2D rotational affine transformation matrix :
[0072]
[0073] in The value is in radians.
[0074] Again, through differentiable bilinear sampling, for Apply this rotational transformation to construct the rotated sampling mesh:
[0075]
[0076] in The standard grid is expanded to homogeneous coordinates and then multiplied by the rotation matrix. Bilinear sampling is used from... According to Sampling is performed to obtain the final corrected image. .
[0077] The sequential geometric correction design aligns with physical intuition: first, perspective projection distortion caused by non-frontal shooting is eliminated, then the instrument's rotation within the image plane is corrected. The entire transformation process is fully differentiable, and the gradient can be obtained from the output image through bilinear sampling. Backpropagation to geometric parameters and This enables end-to-end supervised learning.
[0078] This design enables the model to handle both perspective distortion and planar rotation geometric distortions simultaneously, providing standardized, forward-aligned instrument image inputs for subsequent pointer detection and reading tasks.
[0079] 1.3 Loss Function and Training Strategy
[0080] The goal of model training is to make the final output image As close as possible to the target's forward image We use L1 loss as the main loss function because it promotes sharper textures in image reconstruction tasks and is less sensitive to outliers.
[0081]
[0082] Where N is the total number of pixels.
[0083] In the early stages of training, to guide the network to converge quickly to a meaningful geometric parameter space, an auxiliary parameter supervision loss is introduced. The predicted homography parameters are calculated separately. and rotation angle Compared with the actual parameters used in data synthesis , Mean square error between:
[0084]
[0085]
[0086] The total loss function is the weighted sum of the two:
[0087]
[0088] In this experiment, In subsequent training, the weights can be gradually reduced. This allows the model to focus more on the final image reconstruction quality.
[0089] The model was trained using the Adam optimizer, with an initial learning rate set to... Weight decay is employed to prevent overfitting. Standard data partitioning is used during training, and the loss is monitored on the validation set to select the best model.
[0090] 2. Experimental Design and Results Analysis
[0091] To verify the effectiveness of the unified calibration model method for pointer-type instruments, a series of ablation and comparative experiments were conducted. A total of 970 original, perfect frontal images were collected through actual industrial sites, laboratory photography, and online data collection. Through a series of controllable data augmentation and degradation transformations, paired training samples with varying degrees of geometric distortion and image degradation were generated.
[0092] The data synthesis process consists of three stages: basic enhancement, degradation synthesis, and dataset partitioning. First, each original perfect image is slightly enhanced, including small-scale scaling (98%-102%), translation (±2%), rotation (±3°), and subtle brightness and contrast adjustments, to increase the diversity of the base images and generate approximately 1,000 enhanced perfect images as a baseline for subsequent processing.
[0093] Then, a controlled geometric and image degradation transformation is applied to each base perfect image to generate a corresponding distorted image. The degradation process simulates various types of distortion that may occur in real-world scenes.
[0094] Geometric distortion: including slight tilt (rotation within ±5°) and perspective distortion (random corner perturbation of 1%-3%), simulating projection distortion caused by non-frontal shooting.
[0095] Image degradation includes Gaussian blur (3×3 or 5×5 kernel), motion blur (simulating camera shake), Gaussian noise (variance 0.0005-0.005), JPEG compression artifacts (quality parameter 40-80), and random brightness and contrast variations.
[0096] Environmental factors: Simulate a rainy day environment, add a slight raindrop effect (8%-20% transparency), and slightly darken the overall image.
[0097] To construct a difficulty gradient, we divided the synthetic data into two levels based on degradation intensity: 60% of the samples were configured with mild degradation, and 40% were configured with moderate degradation. Moderate degradation increased the distortion amplitude and noise intensity, providing the model with more challenging training samples.
[0098] We generated approximately 3000 pairs of training samples (input distorted image and target perfect image) from 970 original perfect images. To ensure the reliability of the evaluation, the data was divided into training, validation, and test sets in a 7:1.5:1.5 ratio. The partitioning was performed on a per-original perfect-image basis to ensure that different enhanced and degraded versions of the same original image did not appear in different partitions, preventing data leakage. A portion of the training set is shown below. Figure 3 As shown.
[0099] All comparative experiments used the same Adam optimizer with an initial learning rate of 1e-4, a batch size of 16, and 100 training epochs. A fixed random seed was used to ensure reproducibility of the results. All models were trained and tested on a single NVIDIA RTX 3090 GPU.
[0100] To comprehensively evaluate the performance of the calibration model, this paper adopts both image quality and geometric accuracy metrics. For image quality, peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are selected; for geometric accuracy, mean absolute error of rotation angle (MAE) and mean square error of homography parameter (MSE) are selected.
[0101] (1) Peak signal-to-noise ratio (PSNR)
[0102] PSNR measures the pixel-level difference between the reconstructed image and the reference image, defined as follows:
[0103]
[0104] in, The maximum possible value of the image pixels (in this paper, the image is normalized to). Therefore MSE stands for Mean Squared Error. A higher PSNR value indicates less image distortion.
[0105] (2) Structural Similarity Index (SSIM)
[0106] SSIM evaluates image similarity based on three aspects: brightness, contrast, and structure. The calculation formula is as follows:
[0107]
[0108] in, , Images and The mean, , Standard deviation For covariance, , It is the stability constant. The range of values for SSIM is... The closer the value is to 1, the more similar the two images are.
[0109] (3) Mean absolute error (MAE) of rotation angle
[0110] This metric measures the predicted rotation angle. From a realistic perspective The deviation between them is defined as
[0111]
[0112] in The total number of test samples, with angles in degrees. The smaller this value, the more accurate the model's correction for rotation.
[0113] (4) Mean square error (MSE) of homography parameter
[0114] The eight free parameters (the first eight elements) of the homography matrix describe the perspective transformation, and their predicted values... Compared with the true value The mean square error between them is defined as
[0115]
[0116] This indicator reflects the accuracy of perspective correction; the smaller the value, the closer the predicted homography parameter is to the true value.
[0117] 2.1 Ablation Experiment
[0118] To quantitatively evaluate the effectiveness of each key component in the proposed unified calibration model, this paper designs and details ablation experiments. The specific ablation experimental group setup is shown in Table 1:
[0119] Table 1 Ablation Experiment Group Settings
[0120] Table1Settingsofablationexperimentgroups
[0121]
[0122] The experimental results are shown in Table 2. The complete model achieved the best results in PSNR, SSIM, angular error, and homography error, which proves the necessity of simultaneously using perspective and rotation branches, combining L1 loss with parameter supervision, and data augmentation. Baseline is a simple network that directly regresses images. Its PSNR (22.32dB) is similar to that of the partial ablation experimental group, but its SSIM (0.5183) is significantly lower, indicating that although it can roughly reconstruct pixel values, its structural similarity is poor and it lacks geometric correction capabilities.
[0123] Table 2 Ablation Experiment Results
[0124] Table2Resultsofablationexperiments
[0125]
[0126] Experiment A1, with the rotation branch removed, retained a small homography error (5.48), but PSNR and SSIM decreased compared to the complete model, indicating that the perspective branch can accurately estimate perspective transformations but cannot compensate for rotational distortion. Experiment A2, with the perspective branch removed, achieved an angular error of 7.14°, but the homography error was as high as 39.51, and both PSNR and SSIM were low, indicating that although the rotation branch can learn rotation, it cannot cope with perspective distortion.
[0127] The B1 experiment using only L1 loss achieved homography error (5.42) comparable to the full model, but the angle error increased to 7.71°. PSNR and SSIM were between the full model and A1 and A2, indicating that the image reconstruction loss is sufficient to drive the network to learn some geometric parameters, but the lack of direct supervision will limit the angle accuracy.
[0128] The angle error of the B2 experiment using only parameter supervision was 8.14°, while the homography error was close to 40°, and the PSNR and SSIM were the worst among all ablation groups. This indicates that without image loss constraints, parameter prediction is difficult to guarantee image reconstruction quality at the same time.
[0129] In the data augmentation ablation experiment, the geometric error of the C1 experiment increased significantly after removing the circular mask (angle 72.37°, homography 40.45), and the PSNR dropped to 20.99dB, proving that the mask can effectively prevent the model from cheating by using edge features and is a necessary prerequisite for geometric learning.
[0130] The C2 experiment with color jitter removal achieved a PSNR of 23.17 dB and an SSIM of 0.7809, both of which are relatively high among the ablation groups. However, its angle error (52.46°) and homography error (39.51) are still large, indicating that although color jitter does not directly affect pixel reconstruction, it is crucial for the accurate learning of geometric parameters.
[0131] In summary, the complete model achieved optimal results across all metrics, fully validating the necessity of the joint design of components such as perspective branching, rotation branching, parameter supervision, and data augmentation.
[0132] 2.2 Comparative Experiment
[0133] We compared the model's performance under different architectures, adjusted loss functions, and data augmentation replacements. The specific experimental design is shown in Table 3.
[0134] Table 3 Comparison of experimental group settings
[0135] Table3Settingsofcomparisonexperimentgroups
[0136]
[0137] The experimental results are shown in Table 4. In experiment D1, the correction order was adjusted to rotation followed by perspective. The PSNR was 22.14 dB and the SSIM was 0.6457. Although the homography error (5.01) was comparable to that of the complete model, the angular error was as high as 93.45°, indicating that the prediction of the rotation branch had basically failed. This shows that the correction order has a significant impact on angle learning, and rotating before perspective may hinder the propagation of the rotation gradient.
[0138] Table 4 Comparison of experimental results
[0139] Table4Resultsofcomparisonexperiments
[0140]
[0141] Experiments E1 and E2 adjusted the weights and combinations of the loss function. E1 increased the parameter supervision weights to 0.5, while E2 introduced SSIM loss on top of L1 and parameter supervision. The results of the two experiments were very close: PSNR approximately 21.50 dB, SSIM approximately 0.607, angular error approximately 3.45°, and homography error approximately 4.94. Compared to the complete model, the image quality decreased, but the angular accuracy improved slightly. This indicates that increasing the parameter supervision weights or introducing SSIM loss did not bring significant benefits, and the current loss combination is close to optimal.
[0142] In the F1 experiment, replacing the circular mask with a square mask resulted in a PSNR of 18.45 dB, an SSIM of 0.5310, and an angular error of 87.71°, making it the worst-quality image among all experiments. This indicates that the blurred edges of the circular mask have a positive effect on geometric learning, while the sharp edges of the square mask interfere with the model. In the F2 experiment, replacing color jitter with Gaussian noise achieved a PSNR of 25.78 dB, the highest except for the complete model. However, its angular error (87.72°) and homography error (5.43) were similar to F1, and geometric parameter prediction also failed. This suggests that while Gaussian noise helps with image reconstruction, it does not help with geometric parameter learning and may even destroy the geometric features of the input image. Overall, the complete model remains the best across all metrics, further validating the rationality of its design.
[0143] Figure 4The calibration results of each comparative experimental group on the test samples are shown. The complete model accurately predicted the instrument's rotation angle (6.0°), and the calibrated instrument scale was clearly legible. Experiment D1 changed the calibration order to rotation followed by perspective, predicting an angle of -14.7°, which deviated significantly from the true angle, and the intermediate result (AfterRotation) showed geometric distortion. Experiments E1 and E2 used parameter supervision with λ=0.5 and introduced SSIM loss, respectively, predicting angles of 1.2° and -10.9°. E1 had higher angle accuracy, but both showed some perspective distortion in their intermediate calibration results. Experiment F1 used a square mask instead of a circular mask, and the model could hardly learn the geometric parameters, predicting a constant angle of -0.4°. The sharp edges of the input mask also interfered with feature extraction. Experiment F2 replaced color jitter with Gaussian noise, also obtaining a fixed angle output of -0.4°, indicating that noise enhancement failed to provide effective geometric constraints. The above visualization results are consistent with the quantitative indicators in Table 2, further verifying the necessity of each component in geometric calibration.
[0144] 2.3 Comparison Experiment of Pointer Instrument Image Correction
[0145] To verify the performance of the proposed method in a real complex environment, we selected the method based on Faster R-CNN and feature matching proposed by Liu et al.
[15] , denoted as Method A. The method based on YOLOX and semantic segmentation proposed by Hou et al.
[14] , denoted as Method B, was used for comparison. Method A first detects the position of the instrument by Faster R-CNN, then extracts image features by Harris corner points, performs feature matching by combining normalized cross-correlation and random sampling consistency algorithms, calculates the homography matrix based on the matching points, corrects the tilted elliptical dial to a circular image by perspective transformation, and finally identifies the pointer reading by Hough transformation. Method B uses YOLOX network to detect the dial area, segments the main scale point, pointer and dial center with the help of attention U-Net, fits an ellipse based on the scale point and determines its focus and vertex, draws a circle with half the major axis of the ellipse as the radius, maps the ellipse to a circle by perspective transformation, obtains the corrected front dial, and then calculates the final reading by the improved angle method. The comparative experiments were conducted under six different image quality conditions, denoted as a to f, covering interference factors such as rotation, tilt, raindrops, reflections, blurring, and changes in illumination, as shown in Table 5. The robustness and accuracy of each method in real-world environments were evaluated by comparing their correction effects under different conditions.
[0146] Table 5. Experimental group settings for pointer instrument image calibration comparison
[0147] Table5Settingsofpointermeterimagecorrectioncomparisongroups
[0148]
[0149] like Figure 5 As shown, the experimental results indicate that all three methods can achieve good correction results under slightly disturbed sample (a). As the degree of disturbance increases, methods A and B gradually reveal their shortcomings.
[0150] In sample (b), method A's tilt correction effect is mediocre. This is because the feature matching method is highly dependent on image texture, and severe rotation will destroy the repeatability of feature points, resulting in inaccurate homography matrix calculation. Method B's rotation correction effect is poor, possibly because the semantic segmentation network deviates in locating dial elements when tilt and rotation coexist, affecting subsequent angle calculations.
[0151] In sample (c), method A's rotation correction is poor, as the local texture changes caused by raindrops interfere with the extraction and matching of feature points. Method B's tilt correction has a certain degree of distortion, and the rotation correction is not perfect either. This is because raindrop occlusion causes the attention U-Net to segment the scale points incompletely, thus affecting the accuracy of ellipse fitting.
[0152] In sample (d), method A failed to complete both tilt and rotation corrections. The reflective areas caused numerous mismatches in feature point matching, making it difficult for the RANSAC algorithm to select the correct model. Method B's rotation correction also failed, as reflections interfered with the segmentation of the dial center point and scale.
[0153] In sample (e), rotation correction was not completed for either method. The blurred image significantly reduced the number of feature points for method A, increasing the difficulty of matching; for method B, the blurring resulted in unclear segmentation boundaries and misalignment of the pointer and scale.
[0154] Sample (f) represents the most severe interference. Method A's rotation correction not only failed but also had the opposite effect, further amplifying the rotation angle. This is likely due to the combined effects of severe raindrops and poor lighting, causing feature matching to completely fail and the estimated homography matrix to be entirely incorrect. Method B exhibits a magnified dial, resulting in the loss of some dial details, and rotation correction also fails. This is because semantic segmentation under extreme conditions produces incomplete dial regions, and the fitted ellipse cannot reflect the true dial shape. In contrast, the unified correction model proposed in this paper effectively completes tilt and rotation correction under all test conditions. This method employs an end-to-end learning approach, directly training the network on a large amount of synthetic data to predict geometric transformation parameters. It does not rely on specific feature points or dial elements in the image, thus maintaining stable correction capabilities even when encountering interference such as raindrops, reflections, and blur. The corrected image exhibits high quality and complete details, demonstrating better robustness and accuracy.
[0155] 3. Conclusion
[0156] This paper proposes a unified deep learning model combining perspective and rotation correction for image geometric correction tasks. Based on the ResNet18 backbone, the model predicts homography parameters and rotation angles through a dual-branch structure and utilizes a differentiable spatial transformation module to achieve end-to-end image correction. To verify the effectiveness of each module, ablation and comparative experiments were designed. Experimental results show that the complete model achieves optimal performance in both image reconstruction quality (PSNR 26.98 dB, SSIM 0.9284) and geometric parameter accuracy (angle error 3.22°, homography error 5.42). Ablation studies confirm that the perspective branch, rotation branch, circular mask, color jitter, and parameter-supervised loss are all indispensable components; the absence of any one of them leads to a significant performance degradation. Comparative experiments further demonstrate the impact of the correction order on angle learning and show that the current combination of loss functions is close to optimal, while replacing mask shape and noise type cannot simultaneously achieve both geometric accuracy and image quality. In summary, the proposed model can effectively correct complex geometric distortions and achieves superior performance while maintaining a low parameter count of 11.44M and a real-time inference speed of 2.29ms, providing a reliable solution for image preprocessing in practical applications.
[0157] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.
Claims
1. A two-branch-based end-to-end geometric correction method for pointer instrument images, characterized in that, include: A calibration image containing the dial of an analog instrument is obtained, the size of the calibration image is normalized, and a circular mask is generated according to the center position and effective radius of the dial. The calibration image and the circular mask are then processed pixel by pixel to obtain the mask input image. The mask input image is input into a pre-trained geometric correction network. The shared image features of the mask input image are extracted through the shared feature extraction network of the geometric correction network. The shared image features are then aggregated globally to obtain a global feature vector that simultaneously represents the perspective deformation and orientation information of the dial. The global feature vector is input into the perspective parameter regression branch and the rotation parameter regression branch, which are set in parallel. The perspective parameter regression branch outputs homography correction parameters for mapping the mask input image to a standard viewpoint, and the rotation parameter regression branch outputs rotation correction parameters for adjusting the pointer instrument dial to a preset standard direction. A homography correction matrix is constructed based on the homography correction parameters, and a first inverse sampling grid pointing from the perspective correction image coordinates to the mask input image coordinates is established based on the homography correction matrix. The mask input image is then sampled using the first inverse sampling grid to obtain a perspective correction intermediate image. Based on the rotation correction parameters, a second inverse sampling grid is established, pointing from the coordinates of the target correction image to the coordinates of the perspective correction intermediate image. The perspective correction intermediate image is then sampled using the second inverse sampling grid to obtain a standardized pointer instrument image that simultaneously completes perspective correction and planar rotation correction.
2. The end-to-end geometric correction method for pointer instrument images based on dual branches according to claim 1, characterized in that, The shared feature extraction network includes multiple convolutional feature extraction units and an adaptive global average pooling layer connected in sequence; the perspective parameter regression branch and the rotation parameter regression branch share the shared feature extraction network, the perspective parameter regression branch includes a first fully connected layer, a first nonlinear activation layer and a first output layer, and the rotation parameter regression branch includes a second fully connected layer, a second nonlinear activation layer and a second output layer.
3. The end-to-end geometric correction method for pointer instrument images based on dual branches according to claim 1, characterized in that, The perspective parameter regression branch outputs eight homography correction parameters and constructs the homography correction matrix according to the following formula; the output layer of the perspective parameter regression branch is initialized to output the parameters corresponding to the unit homography matrix, and the output layer of the rotation parameter regression branch is initialized to output zero rotation parameters.
4. The end-to-end geometric correction method for pointer instrument images based on dual branches according to claim 1, characterized in that, The effective area in the circular mask is determined by the center position of the dial and the effective radius of the dial. The circular mask sets a continuously changing transition area at the boundary between the effective area and the ineffective area. When training the geometric correction network, the circular mask is used to weight the image reconstruction error between the standardized pointer instrument image and the corresponding standard forward image to reduce the influence of the background pixels outside the dial on the network training.
5. The end-to-end geometric correction method for pointer instrument images based on dual branches according to claim 1, characterized in that, The training samples of the geometric correction network are generated as follows: a standard positive pointer instrument image is acquired, and random perspective transformation and random planar rotation transformation are sequentially applied to the standard positive pointer instrument image. At least one image degradation processing is applied, including brightness change, contrast change, color change, blur, noise, reflection, raindrop simulation, and image compression, to obtain a distorted input image. The inverse transformation parameters of the random perspective transformation are recorded as homography correction parameter labels, and the inverse rotation parameters of the random planar rotation transformation are recorded as rotation correction parameter labels.
6. The end-to-end geometric correction method for pointer instrument images based on dual branches according to claim 5, characterized in that, The geometric correction network is trained using a joint loss function, which includes at least: a mask image reconstruction loss determined based on the standardized pointer instrument image and the standard positive pointer instrument image; a perspective parameter supervision loss determined based on the predicted homography correction parameters and the homography correction parameter labels; and a rotation parameter supervision loss determined based on the predicted rotation correction parameters and the rotation correction parameter labels. The mask image reconstruction loss is backpropagated through the first inverse sampling grid and the second inverse sampling grid to the shared feature extraction network, the perspective parameter regression branch, and the rotation parameter regression branch.
7. The end-to-end geometric correction method for pointer instrument images based on dual branches according to claim 6, characterized in that, The perspective parameter supervision loss also includes corner reprojection loss, which is determined based on the positional differences of multiple reference points in the preset dial area after transformation by the predicted homography correction matrix and the true homography correction matrix; when the rotation correction parameter covers the angular period boundary, the rotation parameter supervision loss is determined based on the shortest period angular distance between the predicted angle and the true angle.
