A sub-pixel light spot displacement measurement method based on a twin network

Abstract

The application discloses a sub-pixel light spot displacement measurement method based on a twin network, and realizes sub-pixel level high-precision, strong generalization ability, light weight, and large, medium and small displacement compatible light spot displacement measurement through light spot accurate extraction, online data generation, light weight network design and two-stage cascade architecture, and can be widely applied to scenes such as precise optical measurement, structure deformation monitoring and micro-displacement detection.

Description

Technical Field

[0001] This application relates to the fields of image processing and precision measurement technology, and in particular to a subpixel spot displacement measurement method based on twin networks. Background Technology

[0002] In fields such as precision optical measurement, microscopic imaging, and structural deformation monitoring, spot displacement detection is a core method for achieving micro-displacement measurement. Traditional spot displacement measurement methods mainly include the centroid method, edge detection method, and template matching method. These methods can achieve a certain degree of accuracy in displacement detection under ideal imaging conditions, but they have significant shortcomings in practical applications.

[0003] 1) Traditional methods rely on manually designed features, which are sensitive to environmental factors such as changes in lighting, noise interference, and background clutter. They have poor robustness and are difficult to maintain stable accuracy under complex working conditions.

[0004] 2) The high cost and difficulty of obtaining high-precision labeled samples required for displacement measurement, coupled with the limited amount of measured data, result in insufficient generalization ability of the model and a tendency to overfit.

[0005] 3) Existing deep learning methods mostly use a single network architecture for shift regression, which has a large number of parameters and high computational complexity, making it difficult to deploy on memory-constrained embedded devices.

[0006] 4) A single network cannot simultaneously achieve robustness against large displacements and subpixel accuracy against small displacements. It is prone to positioning errors in large displacement scenarios and cannot meet the high-precision measurement requirements in small displacement scenarios.

[0007] In summary, existing spot displacement measurement technologies cannot simultaneously meet the practical application requirements of high precision, high robustness, lightweight design, and compatibility with large, medium, and small displacements. Therefore, a novel sub-pixel-level spot displacement detection method is urgently needed to solve the above problems. Summary of the Invention

[0008] The technical problem to be solved by this application is to provide a sub-pixel spot displacement detection method that can overcome the problems of low measurement accuracy, weak anti-interference ability, difficulty in obtaining labeled data, easy overfitting of models, and inability to balance large displacement robustness and small displacement accuracy in existing spot displacement detection technologies, and achieve high precision, lightweight and highly robustness.

[0009] This application provides a sub-pixel spot displacement measurement method based on Siamese networks, comprising the following steps:

[0010] S10, Spot Energy Extraction: A combination of gamma adaptive enhancement, maximum inter-class variance threshold segmentation, and soft mask weighting is used to automatically locate the spot region and extract the effective spot energy.

[0011] S20, online data generation and augmentation, uses an online sample generation strategy that includes random cropping, random placement, random flipping and random displacement to dynamically construct sample pairs during training;

[0012] S30, Construct a lightweight basic network architecture with lightweight convolutional layers, regression branches and global average pooling layers as the core. The lightweight basic network adopts a weight-sharing Siamese network structure.

[0013] S40, a spatial attention module is added to the lightweight basic network architecture to construct a coarse-tuning model, and a fine-tuning model is constructed by combining the three-channel Sobel gradient input with the lightweight basic network architecture; the coarse-tuning model is used to achieve fast coarse localization of the light spot over a large area, and the image pre-alignment of the light spot region is completed based on the coarse localization result, and sub-pixel residual correction is performed through the fine-tuning model.

[0014] According to some embodiments, the spot energy extraction in step S10 includes the following steps:

[0015] S101, Adaptive gamma enhancement is performed on the input original spot grayscale image;

[0016] S102, perform maximum inter-class variance adaptive threshold segmentation on the gamma-enhanced image to obtain a binarized image; extract the maximum connected region through contour retrieval, filter out noise and interference regions, and generate a binary mask containing only the target spot to complete the accurate localization of the spot region.

[0017] S103, the mask is dilated by a preset kernel size to expand the spot area, the pixels outside the dilated area are taken as pure background pixels, and the background brightness value is calculated by median filtering to avoid the interference of abnormal pixels on the background estimation.

[0018] S104, the expanded mask is subjected to Gaussian blurring with a preset kernel size to obtain a soft mask, which makes the energy transition at the edge of the spot smooth and improves the energy stability during the displacement process;

[0019] S105: Subtract the background value from the original image, and then multiply it pixel by pixel with the soft mask to obtain the pure spot energy distribution, thus completing the accurate extraction of spot energy.

[0020] According to some embodiments, step S101, which involves adaptive gamma enhancement of the input original spot grayscale image, includes:

[0021] The image is divided into foreground and background using the maximum inter-class variance method, and the average pixel value of the foreground is calculated. ;

[0022] Calculate the adaptive gamma threshold ,in The maximum pixel value of the image. The preset adjustment coefficient;

[0023] Based on the local mean filtering result with a preset kernel size, maintain the image pixel size greater than 100%. To reduce the brightness of the background, a gamma transform with a gamma value > 1 is applied to other areas to suppress the background brightness.

[0024] According to some embodiments, the online data generation and expansion in step S20 includes the following steps:

[0025] S201, Dataset partitioning: The single spot video is extracted frame by frame to obtain multiple frames of original spot images. These are then divided into training source images and validation source images according to a preset ratio. After subsequent online generation, the source images can be expanded into image pairs to form the final training set and validation set. The training set is used for updating model parameters, and the validation set is used to monitor the model convergence status. The validation set uses a fixed random seed.

[0026] S202, Randomly cut and paste: Based on the center position of the light spot, the radius of the light spot, and the maximum displacement reserved boundary range, randomly select the cutting starting point and size to ensure that the light spot is intact and not cut off; paste the cut light spot area onto the canvas so that the light spots are randomly distributed within the picture;

[0027] S203, random flipping, performs random flipping enhancement on the spot region, including horizontal flipping, vertical flipping and diagonal flipping, to prevent the spot from occupying too much of a corner of the cropping block and to enrich the dataset;

[0028] S204, random displacement, adopts a block displacement sampling strategy, divides the displacement of the coarse-tuned model and the fine-tuned model into multiple displacement intervals, and samples the displacement of each interval according to a preset ratio; the displacement in the x and y directions is sampled independently and randomly assigned positive and negative directions, so that the model covers the full range of displacement scenarios;

[0029] S205, Spot Interpolation Displacement, simulates sub-pixel displacement based on spot energy distribution, keeps the total spot energy constant, performs displacement operation on the effective energy region, uses bicubic interpolation to ensure displacement accuracy, generates the displacement image, and forms sample pairs.

[0030] According to some embodiments, step S40, which uses the coarse adjustment model to achieve fast coarse localization of the light spot over a large area and completes image pre-alignment of the light spot region based on the coarse localization result, includes:

[0031] S401, input the reference spot image and the spot image to be detected into the coarse adjustment model at the same time to obtain the global coarse displacement;

[0032] S402, the global coarse displacement is inversely normalized to obtain the inversely normalized coarse displacement;

[0033] S403, the pre-aligned image is obtained by performing sub-pixel interpolation displacement alignment on the spot of the reference spot image using inverse normalization coarse displacement.

[0034] Step S40, which involves performing sub-pixel residual correction using the refined model, includes:

[0035] S404, input the pre-aligned image and the image of the spot to be detected into the refinement model to obtain the sub-pixel residual displacement;

[0036] S405, the inverse normalized coarse displacement is added to the sub-pixel residual displacement to obtain the final high-precision displacement.

[0037] According to some embodiments, the coarse-tuning model adopts a weight-shared Siamese network structure, including two identical Siamese encoders with shared weights, a spatial attention mask module and a feature pixel-wise weighting module, as well as a pixel-wise mask fusion module, a feature difference operation module, a regression head module and a global average pooling module.

[0038] The encoder is based on standard convolution and constructed using multi-layer depthwise separable convolution. The inputs of the two encoders are a reference spot image and a spot image to be detected, respectively. The outputs of the two encoders are each divided into two paths: one path serves as the backbone feature, and the other path enters the spatial attention mask module. The spatial attention mask module includes ordinary convolution and a sigmoid activation function. The spatial attention mask module compresses multi-channel features into a single channel through ordinary convolution and generates a spatial weight mask through the sigmoid activation function. Both spatial weight masks are fed into the pixel-wise mask fusion module for pixel-wise fusion, and then fed into the pixel-wise feature weighting module for pixel-wise weighting with the backbone feature. The weighted features are then fed into the feature difference operation module for feature difference operation to obtain differential features. The differential features are fed into the regression head module, and after multi-layer convolution, batch normalization, and linear rectified activation processing, they are fed into the global average pooling module. After global average pooling processing, global coarse displacements in the x and y directions are output.

[0039] According to some embodiments, the depthwise separable convolution includes depthwise convolution and pointwise convolution, wherein the number of input channels of the depthwise convolution is the same as the number of output channels, and the number of input channels of the pointwise convolution is equal to the number of output channels of the depthwise convolution, wherein the number of output channels of the pointwise convolution is independently configured by each layer of depthwise separable convolution.

[0040] According to some embodiments, the subpixel interpolation displacement alignment in step S403 includes performing subpixel shift on the spot of the reference spot image based on the global coarse displacement output by the coarse adjustment model, reusing the spot mask and dilation processing obtained during spot extraction, and using bicubic interpolation to perform high-precision alignment of the spot to obtain a pre-aligned image.

[0041] According to some embodiments, the input of the refinement model is a three-channel feature tensor, which is composed of a single-channel grayscale spot image, a Sobel horizontal gradient map, and a Sobel vertical gradient map stacked together.

[0042] The refined model employs a high-resolution, lightweight convolutional network, comprising two encoders with identical structures and shared weights, a feature difference operation module, a difference aggregation module, a regression head module, and a global average pooling module.

[0043] The encoder is constructed by sequentially connecting multiple ordinary convolution, batch normalization, and linear rectified activation modules with leakage.

[0044] The inputs of the two encoders are the three-channel feature tensor of the pre-aligned image and the three-channel feature tensor of the spot image to be detected, respectively. The outputs of the two encoders are sent to the feature difference operation module. The output of the feature difference operation module is sequentially output as sub-pixel residual displacement through the difference aggregation module, the regression head module, and the global average pooling module.

[0045] According to some embodiments, the step of adding the inverse normalized coarse displacement to the sub-pixel residual displacement to obtain the final high-precision displacement includes: inverse normalizing the global coarse displacement output by the coarse-tuning model and adding it component by component to the sub-pixel residual displacement output by the fine-tuning model to obtain the final displacements in the x and y directions. and , Where n is the preset displacement normalization coefficient of the coarse-tuned model, corresponding to the displacement mapping interval during the training phase of the coarse-tuned model. and These are the global coarse displacements in the x and y directions output by the coarse-tuned model, respectively. and These are the subpixel residual displacements in the x and y directions output by the refined model, respectively.

[0046] The beneficial effects of this application are as follows:

[0047] The technical solution of this application achieves spot displacement measurement with sub-pixel-level high precision, strong generalization ability, lightweight design, and compatibility with large, medium and small displacements through precise spot extraction, online data generation, lightweight network design and two-level cascade architecture. It can be widely used in precision optical measurement, structural deformation monitoring, micro-displacement detection and other scenarios. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 The overall flowchart of the coarse adjustment, pre-alignment, and fine-tuning two-stage cascaded spot displacement measurement according to the example embodiment is shown.

[0050] Figure 2 A coarse-tuned model network structure diagram of a weight-sharing twin network structure according to an example embodiment is shown.

[0051] Figure 3 A refined model network structure diagram of a high-resolution lightweight convolutional network structure according to an example embodiment is shown.

[0052] Figure 4 The diagram illustrates the training loss versus validation loss curves during the coarse-tuned model training process according to an example embodiment.

[0053] Figure 5 The training loss and validation loss curves are shown during the training process of the refined model according to the example embodiment.

[0054] Figure 6 The graph shows the results of continuous spot displacement detection in a label-free scene using a cascaded model according to an example embodiment. Detailed Implementation

[0055] The embodiments of this application will now be described in detail with reference to the accompanying drawings. It should be understood that the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0056] Those skilled in the art should understand that the following specific embodiments or implementation methods are a series of optimized configurations listed in this application to further explain the specific application content. These configuration methods can be combined or used in conjunction with each other, unless this application explicitly states that some or a specific embodiment or implementation method cannot be associated with or used in conjunction with other embodiments or implementation methods. Furthermore, the following specific embodiments or implementation methods are only considered as optimized configurations and are not intended to limit the scope of protection of this application.

[0057] Example 1:

[0058] A subpixel spot displacement measurement method based on Siamese networks includes the following steps:

[0059] Step 1, Precise extraction of light spot energy:

[0060] By combining gamma adaptive enhancement, maximum inter-class variance threshold segmentation, and soft mask weighting, the spot region is automatically located and the effective spot energy is extracted, thus suppressing background interference and improving the accuracy of displacement simulation.

[0061] Step 2, Online Data Generation and Expansion:

[0062] By employing an online sample generation strategy that includes random cropping, random placement, random flipping, and random displacement, sample pairs are dynamically constructed during training to enrich the sample distribution, effectively alleviate overfitting, and improve the model's generalization ability.

[0063] Step 3, Build a lightweight basic network architecture:

[0064] We construct a lightweight basic network architecture with lightweight convolutional layers, regression branches, and global average pooling layers as the core. It relies on high-resolution feature learning to ensure feature representation capabilities. The overall parameter count is low and the computation speed is fast. It serves as the common underlying structure for both coarse-tuning and fine-tuning models.

[0065] Step 4: Construct a two-level cascaded architecture of coarse adjustment, pre-alignment, and fine-tuning:

[0066] Based on the aforementioned lightweight basic network architecture, and tailored to different task requirements, a coarse-tuning, pre-alignment, and fine-tuning serial cascaded architecture is constructed:

[0067] A spatial attention module is added to the basic architecture to build a coarse adjustment model, which enables fast coarse localization of the light spot over a large area. Based on the coarse localization results, the image of the light spot region is pre-aligned, and the residual displacement is compressed to the sub-pixel range.

[0068] Using a lightweight basic network architecture, and with the Sobel gradient three-channel input design, a refined model is built, specifically for sub-pixel accuracy correction of tiny residuals.

[0069] The entire architecture balances robustness in large displacement detection with sub-pixel accuracy in small displacement measurement.

[0070] Example 2:

[0071] This embodiment provides a specific example of step 1 in embodiment 1.

[0072] The steps of spot image preprocessing and spot energy extraction include:

[0073] Step 101:

[0074] Adaptive gamma enhancement is performed on the input raw spot grayscale image. First, the image is divided into foreground and background using the maximum inter-class variance method, and the average pixel value of the foreground is calculated. Combined with the maximum pixel value of the image Calculate the adaptive gamma threshold The specific method for calculating the gamma threshold is as follows:

[0075]

[0076] in =0.2, when When the value is large, It can limit its size to prevent brighter backgrounds from being identified as light spots and thus participating in subsequent interpolation shifts.

[0077] Based on the results of local mean filtering with a kernel size of 5, the image pixel size is kept greater than 5. To improve the brightness, a gamma transformation of γ=15 is applied to other areas to enhance the contrast between the light spot and the background, highlighting the effective area of ​​the light spot.

[0078] Step 102:

[0079] The gamma-enhanced image is subjected to adaptive threshold segmentation using the maximum inter-class variance method to obtain a binarized image. The maximum connected region is extracted by contour retrieval, and noise and interference regions are filtered out to generate a binary mask containing only the target spot, thus completing the accurate localization of the spot region.

[0080] Step 103:

[0081] The mask is dilated with a kernel size of 15 to expand the spot area. Pixels outside the dilated area are taken as pure background pixels. Median filtering is used to calculate the background brightness value to avoid interference from abnormal pixels on the background estimation.

[0082] Step 104:

[0083] The expanded mask is subjected to Gaussian blurring with a kernel size of 45 to obtain a soft mask, which makes the energy transition at the edge of the spot smooth and improves the energy stability during the displacement process.

[0084] Step 105:

[0085] The background value is subtracted from the original image, and then multiplied pixel by pixel with the soft mask to obtain the pure spot energy distribution. This completes the accurate extraction of spot energy, eliminates background and edge interference, and provides high-quality spot data for subsequent displacement simulation and model training.

[0086] Example 3:

[0087] This embodiment provides a specific example of step 2 in embodiment 1.

[0088] The steps for generating online displacement sample pairs include:

[0089] Step 201, Dataset Partitioning:

[0090] The single-spot video with a resolution of 640×480 was extracted frame by frame to obtain 1642 original spot images. These were divided into training source images and validation source images at a ratio of 9:1. After subsequent online generation, the source images can be expanded into image pairs to form the final training set and validation set. The training set is used for updating model parameters, and the validation set is used to monitor the model convergence status. The validation set uses a fixed random seed to ensure that the experiment is reproducible.

[0091] Step 202, randomly cut and paste:

[0092] The original spot image is randomly cropped. A 65-pixel boundary is reserved based on the spot center position (centerx, centery), spot radius (15 pixels), maximum displacement (20 pixels), and safety distance (30 pixels). The cropping start point and size are randomly selected to ensure the spot remains intact. The specific cropping size rules are as follows:

[0093] The left boundary range is [0, centerx-65], the right boundary range is [centerx+65, 512], the upper boundary range is [0, centery-65], and the lower boundary range is [centery+65, 480].

[0094] Paste the cropped light spot area onto a 512×512 canvas to make the light spots randomly distributed within the image.

[0095] Step 203, randomly flip:

[0096] Random flipping enhancements are performed on the spot regions, including horizontal flipping, vertical flipping, and diagonal flipping, to prevent the spots from occupying too much of a corner of the cropping block and to enrich the dataset.

[0097] Step 204, random displacement:

[0098] A block displacement sampling strategy is adopted, dividing the displacement of the coarse-tuned model into three intervals: small displacement 0~1 pixel, medium displacement 1~10 pixel, and large displacement 10~20 pixel, with a ratio of 2:3:5 for each interval; the displacement of the fine-tuned model is divided into three intervals: small displacement 0~0.2 pixel, medium displacement 0.2~0.5 pixel, and large displacement 0.5~1.2 pixel, with a ratio of 2:3:5; the displacement in the x and y directions is sampled independently and randomly assigned positive and negative directions, so that the model covers the entire displacement scene.

[0099] Step 205, Spot interpolation shift:

[0100] Subpixel displacement simulation is performed based on the energy distribution of the light spot. Keeping the total energy of the light spot constant, a displacement operation is performed on the effective energy region. Bicubic interpolation is used to ensure displacement accuracy, generating a displaced image and forming sample pairs.

[0101] Example 4:

[0102] This embodiment provides a specific example of steps 3 and 4 in embodiment 1.

[0103] Figure 1 The overall flowchart of the coarse adjustment, pre-alignment, and fine-tuning two-stage cascaded spot displacement measurement according to the example embodiment is shown.

[0104] like Figure 1 As shown, the overall process of coarse adjustment, pre-alignment, and fine-tuning cascaded spot displacement measurement is as follows:

[0105] First, the reference spot image is... Image of the light spot to be detected Simultaneously input the coarse adjustment model to obtain the global coarse displacement ( , Since the training labels of the coarse-tuned model are normalized (divided by 20) to ensure normal model convergence, inverse normalization (multiplying by 20) is required before alignment; the coarse shift is then used to... The light spot is aligned using sub-pixel interpolation shifting to obtain a pre-aligned image. Then and Input the refined model to obtain the sub-pixel residual displacement ( , Finally, the inverse normalized coarse displacement is added to the residual displacement to obtain the final high-precision displacement.

[0106] This embodiment constructs a three-level serially cascaded spot displacement measurement model consisting of a coarse adjustment model, a sub-pixel pre-alignment model, and a fine-tuning model. The overall approach is a step-by-step solution: first, the coarse adjustment model is used to quickly estimate the global displacement across the entire range; then, based on the coarse positioning results, sub-pixel alignment of the image is performed, compressing the remaining displacement to the sub-pixel range; finally, the fine-tuning model is used to learn the small residuals and perform high-precision correction, ultimately fusing the two levels of displacement to obtain a complete measurement result.

[0107] Figure 2 A coarse-tuned model network structure diagram of a weight-sharing twin network structure according to an example embodiment is shown.

[0108] like Figure 2As shown, the coarse-tuning model adopts a weight-shared Siamese network structure, which is divided into four parts: Siamese encoder, spatial attention mask module, feature fusion and regression head. The coarse-tuning model takes the reference spot image and the spot image to be detected as two inputs, which are fed into the Siamese encoder with the same structure and shared weights. The encoder is based on standard convolution and uses multi-layer depthwise separable convolution to construct dual residual branches, which realizes network lightweighting while extracting the global position and contour features of the spot.

[0109] The encoder output features are divided into two paths: one path serves as the backbone features, and the other path enters the spatial attention masking module. The spatial attention masking module compresses the multi-channel features into a single channel through a 1×1 convolution, and generates a spatial weight mask in the 0-1 interval through a sigmoid activation function. After the two masks are fused pixel by pixel, the backbone features are weighted pixel by pixel to enhance the features of the spot area and suppress background interference.

[0110] The weighted two features are subjected to feature difference operation to highlight the feature differences caused by displacement; the difference features are fed into the regression head, and after multi-layer convolution, batch normalization and linear rectified activation processing, the global coarse displacement in the x and y directions is directly output through global average pooling. , This allows for rapid and stable initial positioning of a large-scale displacement.

[0111] Depthwise separable convolution consists of depthwise convolution and pointwise convolution. The number of input channels in depthwise convolution is the same as the number of output channels; the number of input channels in pointwise convolution is equal to the number of output channels in depthwise convolution, and the number of output channels is configured independently for each layer. The specific parameters (number of channels, stride, dilation factor) of each depthwise separable convolution in the encoder are not limited to the typical values ​​listed in the table and can be adjusted according to the actual task requirements.

[0112] The specific parameters of each layer of the coarse-tuning model in this embodiment are shown in Table 1:

[0113] Table 1. Coarse-tuned model network parameters

[0114]

[0115] In Table 1, S represents the inverse normalized coarse displacement after step size (value is 1 or 2), and D represents the expansion coefficient (value is 1 or 2). The fill is the same as the expansion coefficient.

[0116] Subpixel interpolation shift alignment:

[0117] Based on the global coarse displacement output by the coarse-tuning model ( , ), for the reference spot image The light spot undergoes subpixel shifting, reuses the light spot mask and dilation processing obtained during light spot extraction, and employs bicubic interpolation to achieve high-precision alignment of the light spot, resulting in a pre-aligned image. .

[0118] After alignment, the remaining displacement between the two images is limited to a small range, which greatly improves the detection accuracy of larger displacements and ensures the stability of subpixel measurements.

[0119] Figure 3 A refined model network structure diagram of a high-resolution lightweight convolutional network structure according to an example embodiment is shown.

[0120] The refined model input is a three-channel feature tensor, which is composed of a single-channel grayscale spot image, a Sobel horizontal gradient map, and a Sobel vertical gradient map stacked together, which enhances the edge and gradient details of the spot and improves the sub-pixel residual learning ability.

[0121] The refined model employs a high-resolution, lightweight convolutional network, maintaining a relatively small downsampling rate to preserve the edge and sub-pixel detail information of the light spot to the maximum extent. Its network structure... Figure 3 As shown.

[0122] like Figure 3 As shown, the refinement model in this embodiment adopts a weight-shared Siamese network structure, taking the three-channel feature tensors of the pre-aligned image and the image to be detected as input. The model is divided into four parts: a Siamese encoder, feature difference operation, difference aggregation, and a regression head. The pre-aligned image... Compared with the original image to be detected Using the Sobel three-channel features of two images as input, gradient information is introduced to enhance the edge and detail features of the light spot, thereby improving the learning ability of sub-pixel residuals. The two sets of three-channel feature tensors are fed into a twin encoder with identical structure and shared weights. Through multi-layer convolution, batch normalization, and a leaky linear rectified activation function, high-resolution light spot features are extracted, preserving sub-pixel detail information to the maximum extent.

[0123] The two sets of features output by the encoder are fed into the feature difference operation module, which highlights the minute displacement differences of the light spot through feature difference operation; the difference features are then fed into the difference aggregation module, and after processing by convolution, batch normalization and activation function, they are sent to the regression head; the regression head consists of multiple convolution and batch normalization layers, and outputs the sub-pixel residual displacements in the x and y directions through global average pooling. , It completes high-precision correction of minute displacements that were not corrected in the coarse-tuned model.

[0124] The specific parameters of each layer of the refined model are shown in Table 2:

[0125] Table 2 Refined Model Network Parameters

[0126] Displacement fusion output:

[0127] The global coarse displacement output by the coarse-tuned model is inversely normalized and then added component-by-componently to the sub-pixel residual displacement output by the fine-tuned model to obtain the final displacement:

[0128]

[0129] In the formula, 20 is the displacement normalization coefficient of the coarse-tuned model, corresponding to the displacement mapping range (0~20 pixels) during the training phase of the coarse-tuned model.

[0130] By employing a two-stage cascaded structure of coarse and fine adjustments, this model simultaneously achieves robustness against large displacements and sub-pixel measurement accuracy for small displacements, enabling stable and high-precision spot displacement detection across the entire measurement range.

[0131] Example 5:

[0132] This embodiment provides a specific example of model training based on the above embodiments.

[0133] In this embodiment, a step-by-step independent training strategy is adopted for the coarse-tuning model and the fine-tuning model. First, the coarse-tuning model is trained to achieve stable estimation of large-range displacement, and then the fine-tuning model is trained to achieve accurate correction of sub-pixel residuals. Both models are trained using online generated samples, and the training process is optimized independently without interfering with each other.

[0134] Step 1, Training Environment and Basic Settings:

[0135] Model training is performed in a general-purpose CPU / GPU deep learning environment, with a maximum training iteration count of 300 epochs. An early stopping strategy is employed to avoid overfitting: the early stopping patience value is set to 30 for coarse-tuning and 40 for fine-tuning. Samples are loaded in fixed batches during training, with a batch size of 64, and a data loading thread is enabled to improve training efficiency.

[0136] Step 2, coarse-tuning model training:

[0137] The coarse-tuned model employs a coarse-tuned network structure and is trained using a dedicated spot dataset.

[0138] Label preprocessing: The training labels are normalized by dividing the true displacement value by 20 and mapping it to the 0~1 interval, which improves the stability and convergence speed of model training.

[0139] Data augmentation: Enhances the model's robustness to noise, blur, and brightness variations through stronger random perturbations, adapting to the coarse model's training objective of "fast convergence + large-scale displacement perception".

[0140] Gaussian noise (100% probability of success): Adds random Gaussian noise to the image with a noise standard deviation sigma sampling range of [0.001, 0.005] (the image has been normalized to the range of [0, 1]), ensuring that the noise intensity is sufficient to simulate the interference of the real acquisition environment, but without destroying the core features of the light spot.

[0141] Gaussian blur (30% probability of activation): Apply Gaussian blur to the image with a 30% probability. The standard deviation of the blur kernel, sigma, has a sampling range of [0.1, 0.3], simulating slight defocusing or motion blur during the imaging process.

[0142] Brightness perturbation (40% probability of activation): Adjusts image brightness with a 40% probability. The brightness scaling factor alpha sampling range is [0.90, 1.1] (overall brightness), and the bias term beta sampling range is [-0.03, 0.03] (base brightness offset), simulating light source intensity fluctuations or exposure deviations.

[0143] Execution logic: All enhancement strategies are independently determined to determine their effectiveness probability, and multiple enhancements can be stacked; after enhancement, the image pixel values ​​are cropped to the range of [0, 1] to avoid numerical overflow.

[0144] Optimizer settings: AdamW optimizer is used, with an initial learning rate of 1×10⁻⁶. - ³, the weight decay coefficient is set to 1×10 - ³ enables rapid parameter updates and effective regularization.

[0145] Learning rate scheduling: An adaptive learning rate decay strategy is adopted to verify the minimum loss as the monitoring target. The decay factor is set to 0.5, the patience value is set to 5, and the loss reduction threshold is 1×10. -4 The minimum learning rate is 1×10. -7 .

[0146] Loss function: Total loss Loss due to displacement regression With mask constraint loss The loss consists of two weighted components: the basic regression loss uses the SmoothL1 loss with a β parameter set to 0.015; the mask constraint loss is used to prevent spatial attention mask degradation, and its calculation formula is as follows:

[0147]

[0148] Therefore, the total loss is calculated as follows:

[0149]

[0150] Training objective: Through end-to-end training, enable the coarse-tuned model to achieve fast and stable global displacement estimation across the full range of 0-20 pixels. After the normalized displacement result is output by the coarse-tuned model inference, it needs to be multiplied by 20 to restore the true pixel displacement before being used for subsequent sub-pixel pre-alignment operations.

[0151] Figure 4 The diagram illustrates the training loss versus validation loss curves during the coarse-tuned model training process according to an example embodiment. Figure 4 In this context, train_loss represents the training loss, val_loss represents the validation loss, the dependent variable (vertical axis) is the logarithmic loss value with base 10, and the independent variable (horizontal axis) is the number of training epochs.

[0152] like Figure 4 As shown, both the training loss and validation loss of the coarse-tuned model decrease rapidly with the increase of iteration rounds. After 100 training rounds, the loss tends to converge smoothly, while the validation loss does not show a significant upward trend, and no overfitting phenomenon is observed. The early stopping strategy (patience value 30) can stably terminate the training, indicating that the coarse-tuned model learns sufficiently and has good generalization performance in a wide range of displacement estimation tasks.

[0153] Step 3, Refine the model training:

[0154] The refined model employs a refined network structure and is trained independently using a dedicated refined sample set.

[0155] Optimizer settings: AdamW optimizer is used, with an initial learning rate of 1×10⁻⁶. -3 Without setting additional weight decay, fine-grained fitting capability is preserved.

[0156] Data augmentation: Improves model generalization by gently perturbating while preserving details, adapting to the training objective of "high-precision displacement regression" of the fine model.

[0157] Gaussian noise (100% probability of activation): The noise intensity is significantly reduced, and the standard deviation sigma sampling range is [0.0002, 0.001]. It only simulates extremely low intensity sensor noise to avoid obscuring fine features such as the edge of the light spot.

[0158] Gaussian blur (15% probability of activation): The activation probability is halved (15%), the standard deviation of the blur kernel sigma is sampled in the range of [0.1, 0.25], only a very slight blur is applied, and the gradient information of the spot outline is preserved.

[0159] Brightness perturbation (40% probability of taking effect): The probability is consistent with the coarse model, but the perturbation range is narrowed: the sampling range of the brightness scaling factor alpha is [0.95, 1.05], and the sampling range of the bias term beta is [-0.02, 0.02], so as to preserve the brightness distribution characteristics of the spot to the greatest extent.

[0160] Additional feature fusion: After the fine model enhancement, Sobel gradients (x / y directions) are extracted from the image, and the "original image + x gradient + y gradient" are stacked into a 3-channel input to enhance the positional information of the light spot edge and assist in high-precision displacement regression.

[0161] Execution logic: Same as the coarse model, the enhancement strategy takes effect independently and supports overlay, and the final pixel value is cropped to the range of [0, 1].

[0162] Learning rate scheduling: An adaptive learning rate decay strategy is adopted, with a decay factor set to 0.8, a patience value set to 20, and a loss reduction threshold of 1×10. -5 The minimum learning rate is 1×10. -7 .

[0163] Loss function: Only SmoothL1 loss is used for displacement residual regression, with the β parameter set to 0.015. No mask constraint loss is added, focusing on sub-pixel accuracy optimization.

[0164] Training objective: To enable the refined model to learn with high precision the tiny residuals within ±1 pixel after coarse alignment, and output subpixel-level corrected displacement.

[0165] Figure 5 The training loss and validation loss curves are shown during the training process of the refined model according to the example embodiment. Figure 5 In this context, train_loss represents the training loss, val_loss represents the validation loss, the dependent variable (vertical axis) is the logarithmic loss value with base 10, and the independent variable (horizontal axis) is the number of training epochs.

[0166] like Figure 5 As shown, the training loss and validation loss of the refined model decrease steadily and converge gradually, with no drastic fluctuations in the loss curve. The validation loss remains low throughout, and there is no overfitting or underfitting. Combined with the early stopping strategy (patience value 40), stable training can be completed, indicating that the refined model has excellent fitting effect and strong training stability in the sub-pixel residual displacement learning task.

[0167] Example 6:

[0168] This embodiment provides a specific example of conducting a displacement measurement performance experiment based on the above embodiment.

[0169] To verify the actual performance of the subpixel spot displacement measurement method based on twin networks described in the above embodiments, experimental tests were conducted from four dimensions: measurement accuracy, anti-interference robustness, real-time performance and lightweight design, and label-free scene trajectory tracking. All tests were performed on actual collected spot video data.

[0170] Two light spot videos, video1 and video2, were used in the testing phase. The imaging environment of the two videos was basically the same, but the light spot motion trajectory and background interference were different. Video2 was used for the accuracy comparison experiment and the robustness experiment. The light spot in the video made reciprocating vibration motion to verify the model's measurement accuracy and generalization ability. Video1 was used for the label-free test. The light spot in the video had a clear continuous displacement path in the x-direction to verify the model's continuous trajectory tracking ability.

[0171] It is important to note that to ensure consistency in the distribution of training and testing data, the model performs uniform interpolation and shifting on the input images during both the training and testing phases. During training, the input to the coarse-tuned model is taken from cropped blocks of the original image, requiring zero-shift interpolation using the mask region used in the online generation phase; the other input is generated through interpolation and shifting, requiring no further zero-shift interpolation. The input processing rules for the refined model remain consistent with those for the coarse-tuned model. During testing, in the accuracy comparison and robustness experiments, zero-shift interpolation is required on the input to the coarse-tuned model; in the unlabeled test, both images are cropped blocks of the original image, therefore zero-shift interpolation is required before inputting into both the coarse-tuned and refined models, with the zero-shift interpolation performed based on its own mask.

[0172] Experiment 1, Accuracy Comparison Experiment:

[0173] This experiment compares the cascaded model with traditional spot displacement measurement methods (template matching, phase correlation, SIFT matching) and a standalone coarse-tuning model. The evaluation metrics are x-direction displacement error, y-direction displacement error, and the proportion of high-precision samples. The proportion of high-precision samples includes those with errors less than 0.01 pixels, 0.05 pixels, and 0.1 pixels; higher values ​​indicate higher measurement accuracy. The experimental results are shown in Table 3.

[0174] Table 3. Accuracy Comparison Test Results

[0175]

[0176] Experimental results show that traditional methods (template matching, phase correlation, SIFT matching) are severely affected by background and noise interference, with dx and dy errors both greater than 8.8 pixels, and lack subpixel measurement capability. The coarse-tuning model alone can achieve large-scale displacement estimation, with dx and dy errors reduced to 0.219798 pixels and 0.184002 pixels, respectively, but the proportion of subpixel high-precision measurements is extremely low. The cascaded model of coarse and fine tuning has a dx error of only 0.012203 pixels and a dy error of only 0.012947 pixels. The proportion of samples with errors less than 0.01 pixels reaches 82.75%, and the proportion of samples with errors less than 0.05 pixels reaches 99.12%. The measurement accuracy is far superior to traditional methods and the coarse-tuning model alone, fully verifying the significant advantages of the cascaded architecture in subpixel displacement measurement.

[0177] Experiment 2, Robustness Experiment:

[0178] To test the model's anti-interference capability in complex imaging environments, this experiment simulates common Gaussian noise interference and brightness fluctuation interference in actual measurements. Seven sets of comparative experiments were set up (original image, weak / medium / strong Gaussian noise, low / medium / high brightness disturbance), with total displacement error and high precision ratio as evaluation indicators.

[0179] Gaussian noise:

[0180] Weak noise: The standard deviation sigma has a sampling range of [0, 0.0005].

[0181] Medium noise: standard deviation sigma sampling range is [0, 0.001].

[0182] Strong noise: the standard deviation sigma has a sampling range of [0, 0.005];

[0183] Brightness perturbation:

[0184] Low brightness: Brightness scaling factor alpha sampling range [0.95, 1.05], bias term beta sampling range [-0.02, 0.02].

[0185] Medium brightness: Brightness scaling factor alpha sampling range [0.90, 1.10], bias term beta sampling range [-0.05, 0.05].

[0186] High brightness: brightness scaling factor alpha sampling range [0.85, 1.15], bias term beta sampling range [-0.08, 0.08].

[0187] The experimental results are shown in Table 4:

[0188] Table 4. Robustness test results

[0189]

[0190] Experimental results show that under weak and medium noise and low and medium brightness perturbations, the total error of the model is less than 0.02 pixels, and the proportion of samples with errors less than 0.05 pixels remains above 96.5%. Even under strong noise and high brightness perturbations, the model still maintains stable measurement capabilities, with total errors of 0.025934 pixels and 0.025322 pixels, respectively, and the proportion of samples with errors less than 0.05 pixels is higher than 92%, proving that the method of this application has excellent robustness to noise and brightness changes.

[0191] Experiment 3, Time Consumption and Parameter Quantity Verification:

[0192] To verify the lightweight and real-time performance of the model, this experiment was conducted on a hardware platform with an Intel Core i5-12400F CPU, 32GB of memory, and an NVIDIA RTX 3060 GPU (12GB). Only the forward inference time of the cascaded model was counted, excluding image preprocessing time.

[0193] Test results show that the average time for a single inference of the cascaded model is 57.86ms, which can meet the requirements of real-time measurement. The coarse-tuned model has 12,624 parameters, the fine-tuned model has 6,408 parameters, and the total number of parameters of the two-level cascaded model is only 19,032. It belongs to the ultra-lightweight deep learning model with low computational cost and low memory usage, and is very easy to deploy on embedded terminals and portable precision measurement equipment.

[0194] Experiment 4, Unlabeled Test:

[0195] To verify the model's practicality in scenarios without manual annotation, this experiment used the light spot in the first frame of video1 as a reference. A 150×150 light spot region was cropped and pasted into the center of a 512×512 canvas as a reference image. Subsequent frames all used the same preprocessing method and were input into the cascaded model for continuous displacement detection. The results are as follows: Figure 6 As shown.

[0196] Figure 6 The graph shows the results of continuous spot displacement detection in a label-free scene using a cascaded model according to an example embodiment. Figure 6 In this context, dx is the offset on the x-axis, and dy is the offset on the y-axis. The positive direction is to the right in the x-direction and the positive direction is downward in the y-direction. The dependent variable (x-axis) is the offset in pixels, and the independent variable (y-axis) is the current frame of the image of the light spot to be detected.

[0197] Depend on Figure 6 It is known that the cascaded model disclosed in this application can stably track the continuous displacement of the light spot, and the displacement curve is smooth without sudden changes or obvious jumps. It can accurately restore the real motion trajectory of the light spot and still has excellent displacement detection stability and tracking ability in actual application scenarios without labels or manual annotation.

[0198] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A sub-pixel spot displacement measurement method based on Siamese networks, characterized in that, Includes the following steps: S10, Spot Energy Extraction: The method of combining gamma adaptive enhancement, maximum inter-class variance threshold segmentation and soft mask weighting is used to automatically locate the spot region and extract the effective spot energy. S20, online data generation and augmentation, uses an online sample generation strategy that includes random cropping, random placement, random flipping and random displacement to dynamically construct sample pairs during training; S30, Construct a lightweight basic network architecture with lightweight convolutional layers, regression branches and global average pooling layers as the core. The lightweight basic network adopts a weight-sharing Siamese network structure. S40, a spatial attention module is added to the lightweight basic network architecture to build a coarse-tuning model, and a fine-tuning model is built by combining the Sobel gradient three-channel input with the lightweight basic network architecture; the coarse-tuning model is used to achieve fast coarse localization of the light spot over a large area, and the image pre-alignment of the light spot region is completed based on the coarse localization result; and sub-pixel residual correction is performed through the fine-tuning model. The coarse-tuning model employs a weight-shared Siamese network structure. It includes two coarse-tuning encoders, two spatial attention masking modules, and two pixel-wise feature weighting modules. The model also includes a pixel-wise masking fusion module, a feature difference operation module, a regression head module, and a global average pooling module. The two coarse-tuning encoders are structurally identical and share weights. They are constructed using multi-layer depthwise separable convolutions based on standard convolutions. The inputs to the two coarse-tuning encoders are a reference spot image and a spot image to be detected, respectively. The outputs of each encoder are split into two paths: one serves as the backbone feature, and the other enters the spatial attention masking module. The spatial attention mask module includes ordinary convolution and sigmoid activation function. The spatial attention mask module compresses multi-channel features into a single channel through ordinary convolution and generates a spatial weight mask through sigmoid activation function. Both spatial weight masks are fed into the pixel-by-pixel mask fusion module for pixel-by-pixel fusion, and then fed into the two feature pixel-by-pixel weighting modules for pixel-by-pixel weighting with the backbone features. The two weighted features are fed into the feature difference operation module for feature difference operation to obtain the difference features. The difference features are fed into the regression head module, and after multi-layer convolution, batch normalization and linear rectified activation processing, they are fed into the global average pooling module, which outputs global coarse displacements in the x and y directions after global average pooling processing. The refinement model input is a three-channel feature tensor, which is composed of a single-channel grayscale spot image, a Sobel horizontal gradient map, and a Sobel vertical gradient map stacked together. The refinement model adopts a high-resolution lightweight convolutional network, including two refinement model encoders, a refinement model feature difference operation module, a difference aggregation module, a refinement model regression head module, and a refinement model global average pooling module. The two refinement model encoders are structurally identical and share weights. The refinement model encoder is constructed by sequentially connecting multiple ordinary convolution, batch normalization, and leaky linear rectified activation modules. The inputs of the two refinement model encoders are the three-channel feature tensor of the pre-aligned image and the three-channel feature tensor of the spot image to be detected, respectively. The outputs of the two refinement model encoders are both fed into the refinement model feature difference operation module. The output of the refinement model feature difference operation module is sequentially passed through the difference aggregation module, the refinement model regression head module, and the refinement model global average pooling module to output sub-pixel residual displacement.

2. The sub-pixel spot displacement measurement method based on Siamese networks according to claim 1, characterized in that, Step S10, the extraction of light spot energy, includes the following steps: S101, Adaptive gamma enhancement is performed on the input original spot grayscale image; S102, perform maximum inter-class variance adaptive threshold segmentation on the gamma-enhanced image to obtain a binarized image; extract the maximum connected region through contour retrieval, filter out noise and interference regions, and generate a binary mask containing only the target spot to complete the accurate localization of the spot region. S103, the mask is dilated by a preset kernel size to expand the spot area, the pixels outside the dilated area are taken as pure background pixels, and the background brightness value is calculated by median filtering to avoid the interference of abnormal pixels on the background estimation. S104, the expanded mask is subjected to Gaussian blurring with a preset kernel size to obtain a soft mask, which makes the energy transition at the edge of the spot smooth and improves the energy stability during the displacement process; S105: Subtract the background value from the original image, and then multiply it pixel by pixel with the soft mask to obtain the pure spot energy distribution, thus completing the accurate extraction of spot energy.

3. The sub-pixel spot displacement measurement method based on Siamese networks according to claim 2, characterized in that, Step S101, which involves adaptive gamma enhancement of the input original spot grayscale image, includes: The image is divided into foreground and background using the maximum inter-class variance method, and the average pixel value of the foreground is calculated. ; Calculate the adaptive gamma threshold ,in The maximum pixel value of the image. The preset adjustment coefficient; Based on the local mean filtering result with a preset kernel size, maintain the image pixel size greater than 100%. To reduce the brightness of the background, a gamma transform with a gamma value > 1 is applied to other areas to suppress the background brightness.

4. The sub-pixel spot displacement measurement method based on Siamese networks according to claim 1, characterized in that, Step S20, the online data generation and expansion, includes the following steps: S201, Dataset partitioning: The single spot video is extracted frame by frame to obtain multiple frames of original spot images. These are then divided into training source images and validation source images according to a preset ratio. After subsequent online generation, the source images can be expanded into image pairs to form the final training set and validation set. The training set is used for updating model parameters, and the validation set is used to monitor the model convergence status. The validation set uses a fixed random seed. S202, Randomly cut and paste: Based on the center position of the light spot, the radius of the light spot, and the maximum displacement reserved boundary range, randomly select the cutting starting point and size to ensure that the light spot is intact and not cut off; paste the cut light spot area onto the canvas so that the light spots are randomly distributed within the picture; S203, random flipping, performs random flipping enhancement on the spot region, including horizontal flipping, vertical flipping and diagonal flipping, to prevent the spot from occupying too much of a corner of the cropping block and to enrich the dataset; S204, random displacement, adopts a block displacement sampling strategy, divides the displacement of the coarse-tuned model and the fine-tuned model into multiple displacement intervals, and samples the displacement of each interval according to a preset ratio; the displacement in the x and y directions is sampled independently and randomly assigned positive and negative directions, so that the model covers the full range of displacement scenarios; S205, Spot Interpolation Displacement, simulates sub-pixel displacement based on spot energy distribution, keeps the total spot energy constant, performs displacement operation on the effective energy region, uses bicubic interpolation to ensure displacement accuracy, generates the displacement image, and forms sample pairs.

5. The sub-pixel spot displacement measurement method based on Siamese networks according to claim 1, characterized in that, Step S40, which uses the coarse adjustment model to achieve fast coarse localization of the light spot over a large area and performs pre-alignment of the light spot region image based on the coarse localization result, includes: S401, input the reference spot image and the spot image to be detected into the coarse adjustment model at the same time to obtain the global coarse displacement; S402, the global coarse displacement is inversely normalized to obtain the inversely normalized coarse displacement; S403, the pre-aligned image is obtained by sub-pixel interpolation displacement alignment of the light spot image by inverse normalization coarse displacement. Step S40, which involves performing sub-pixel residual correction using the refined model, includes: S404, input the pre-aligned image and the image of the spot to be detected into the refinement model to obtain the sub-pixel residual displacement; S405, the inverse normalized coarse displacement is added to the sub-pixel residual displacement to obtain the final high-precision displacement.

6. The sub-pixel spot displacement measurement method based on Siamese networks according to claim 5, characterized in that, The depthwise separable convolution includes depthwise convolution and pointwise convolution. The number of input channels and the number of output channels of the depthwise convolution are the same. The number of input channels of the pointwise convolution is equal to the number of output channels of the depthwise convolution. The number of output channels of the pointwise convolution is independently configured by each layer of depthwise separable convolution.

7. The sub-pixel spot displacement measurement method based on Siamese networks according to claim 5, characterized in that, Step S403, the subpixel interpolation displacement alignment, includes performing subpixel shifting on the spot of the reference spot image based on the global coarse displacement output by the coarse adjustment model, reusing the spot mask and dilation processing obtained during spot extraction, and using bicubic interpolation to perform high-precision alignment of the spot to obtain a pre-aligned image.

8. The sub-pixel spot displacement measurement method based on Siamese networks according to claim 5, characterized in that, In step S405, the inverse normalized coarse displacement is added to the sub-pixel residual displacement to obtain the final high-precision displacement, including: The global coarse displacement output by the coarse-tuned model is inversely normalized and then added component-by-componently to the sub-pixel residual displacement output by the fine-tuned model to obtain the final displacements in the x and y directions. and , Where n is the preset displacement normalization coefficient of the coarse-tuned model, corresponding to the displacement mapping interval during the training phase of the coarse-tuned model. and These are the global coarse displacements in the x and y directions output by the coarse-tuned model, respectively. and These are the subpixel residual displacements in the x and y directions output by the refined model, respectively.

Images