Method for improving image splicing precision of scanning pen

Abstract

The invention provides a method for improving image splicing precision of a scanning pen. The method comprises the following steps: S1, constructing a convolutional neural network; s2, training a neural network model, namely training the convolutional neural network constructed in the step S1; and S3, a reasoning stage: processing a network output result. According to the method, the convolution model with a simple structure is used, so that the precision is greatly improved on the premise of ensuring the speed. Compared with a traditional image splicing method, ORB precision is greatly improved, compared with a deep learning-based method, SuperPoint training is simple, the multi-scale feature problem is solved, a result is obtained from end to end, and complex post-processing is not needed.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for improving the accuracy of image splicing of scanning translator pens. Background Art

[0002] Image stitching technology refers to the process of stitching two images together to form a larger image by finding common key points. Among them, the ORB (Oriented FAST and RotatedBRIEF) algorithm commonly used in the prior art is a feature point extraction and matching algorithm based on FAST corner detection and BRIEF binary descriptor. The SuperPoint algorithm is a feature point extraction and matching algorithm based on deep learning. It uses a network composed of a convolutional neural network (CNN) to learn the location and descriptors of feature points, and uses non-maximum suppression (NMS) and Harris corner detection to obtain the final feature points. FPN (Feature Pyramid Network) is a network structure used to solve the problem of multi-scale feature representation in target detection and semantic segmentation.

[0003] However, traditional image stitching methods, such as the ORB algorithm, usually require manual setting of parameters and thresholds, and these methods are often inefficient and error-prone for stitching large-size and high-resolution images. Deep learning-based methods such as the SuperPoint network use bilinear interpolation for feature upsampling, which cannot capture features of different scales well and requires extremely complex post-processing, such as non-maximum suppression (NMS), which requires manual adjustment of the threshold in maximum suppression. Summary of the invention

[0004] In order to solve the above problems, the purpose of this application is to use a convolutional model with a simple structure, which has greatly improved the accuracy while ensuring the speed. Compared with the traditional image stitching method ORB, the accuracy has been greatly improved. Compared with the deep learning-based method SuperPoint, the training is simpler, the multi-scale feature problem is solved, and the results are obtained end-to-end without the need for complex post-processing.

[0005] Specifically, the present invention provides a method for improving the accuracy of image splicing of a scanning pen, the method comprising the following steps:

[0006] S1. Constructing a convolutional neural network

[0007] The convolutional neural network uses a Backbone, Neck, and Head structure. The Backbone uses the resnet18 network for feature extraction and reduces the image features by 32 times. The Neck uses the FPN network to upsample the feature map reduced by 32 times back to the original image size. The upsampling here uses the nearest neighbor difference algorithm, which directly copies adjacent pixels. The Head part uses a convolution operation with a convolution kernel of 1x1 and a dimension of 2.

[0008] S2. Train the neural network model, i.e., the convolutional neural network built in step S1;

[0009] First, the input image is a single-channel grayscale image with a width of 96 pixels and a height of 96 pixels. After passing through the neural network, the output feature map is the same size as the input image and the number of channels is changed to 2, that is, (2,96,96), which means that each pixel on the 96x96 image is classified into two categories. Here, a convolution with a dimension of 2 and a convolution kernel of 1 is used for binary classification.

[0010] The loss function for measuring the classification results uses the cross entropy loss function. In deep learning, cross entropy is usually used to measure the difference between the model prediction value and the true label, that is, the difference between the probability distribution of the model output and the probability distribution of the true label. The formula is: L = -1 / N*Σ(y_i*log(p_i)+(1-y_i)*log(1-p_i)).

[0011] Where N represents the number of samples, y_i represents the true label of the i-th sample, and p_i represents the predicted probability of the i-th sample;

[0012] S3. Inference phase: Processing network output results

[0013] According to step S2, the output size of the network is (2, 96, 96). The torch.argmax() function is used to obtain the index of the maximum value in the output result. The argmax function is used to return the index of the maximum value in the tensor. Because each pixel on the image is classified into two categories (the method is the same as the two categories in step S1), the index values ​​are only 0 and 1. For each pixel, if the index value is 1, it means that the current pixel is a key point.

[0014] In step S1, the resnet18 network is used for feature extraction, and the extraction process is assumed to be a Backbone module, further comprising:

[0015] Input image: size = 1*96*96;

[0016] After two layers of convolution to extract features, the size becomes [16*48*48], where 16 is the number of channels and 48 is the width and height of the output feature map;

[0017] After two layers of convolution operations to extract features, the size becomes [48*24*24], where 48 is the number of channels and 24 is the width and height of the output feature map;

[0018] After two layers of convolution operations to extract features, the size becomes [128*12*12], where 128 is the number of channels and 12 is the width and height of the output feature map;

[0019] After two layers of convolution operations to extract features, the size becomes [256*6*6], where 256 is the number of channels and 6 is the width and height of the output feature map.

[0020] In step S1, the FPN process further includes:

[0021] The size of the image after double upsampling is [256*12*12];

[0022] 1×1 convolutional layer, input channel 256, output channel 128, at this time the feature map size becomes [128*12*12];

[0023] 2x upsampling, the size is [128, 24, 24], the 1×1 convolution layer has 128 input channels and 48 output channels, and the size becomes [48, 24, 24];

[0024] 2x upsampling, size is [48, 48, 48], 1×1 convolution layer has 48 input channels and 16 output channels, and the size becomes [16, 48, 48];

[0025] After 2x upsampling, the size is [16,96,96].

[0026] The Head part uses a convolution operation with a convolution kernel of 1x1 and a dimension of 2; the process is:

[0027] 1) The convolution kernel size used is 1x1, which means that each convolution kernel is a 1x1 matrix that performs sliding calculation operations on the feature map;

[0028] 1×1 convolutional layer, with 16 input channels, 2 output channels, and size [2, 96, 96];

[0029] 2) Since the dimension is 2, the output of 2 dimensions will be calculated for each sliding position of the convolution kernel;

[0030] Finally, the output image is: size = 2*96*96.

[0031] In step S3, assuming that the output result is [2.24, 5.67], after comparison, 5.67 is greater than 2.24. Since the initial value of the index is 0, the index of the maximum value 5.67 returned is 1.

[0032] Therefore, the advantages of this application are:

[0033] 1. No additional NMS (non-maximum) post-processing is required;

[0034] 2. Using FPN (Feature Pyramid Network) instead of interpolation for image upsampling can better capture features of different scales, thereby improving the performance and stability of the model.

[0035] The existing technology directly uses upsampling without learnable parameters (i.e., directly uses the difference method) to expand the feature map back to the original image size after extracting features through the resnet18 network. However, this method uses FPN (Feature Pyramid Network) instead of interpolation for image upsampling, which has many learnable parameters and can better capture features of different scales, thereby improving the performance and stability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0037] Figure 1 It is a schematic diagram of the overall network structure for building a convolutional neural network.

[0038] Figure 2(1) and Figure 2(2) are schematic diagrams of the output results. The white dots in Figure 2(2) are the pixels with an index value of 1 after the image in Figure 2(1) is extracted through the neural network and finally classified into two categories, i.e., the key points finally obtained.

[0039] Figure 3 is a schematic diagram of the process of this method. DETAILED DESCRIPTION

[0040] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0041] This method belongs to the field of computer vision and proposes a method for improving the accuracy of scanning pen image stitching. Figure 3 As shown, the main implementation steps of the method are as follows:

[0042] Step S1. Construct a convolutional neural network, such as Figure 1 As shown,

[0043] The convolutional neural network uses the Backbone, Neck, and Head structures, where Backbone uses the resnet18 network for feature extraction. The extraction process is as follows: Figure 1 As shown in the Backbone module on the left,

[0044] Input image: size = 1*96*96;

[0045] After two layers of convolution to extract features, the size becomes [16*48*48], where 16 is the number of channels and 48 is the width and height of the output feature map;

[0046] After two layers of convolution operations to extract features, the size becomes [48*24*24], where 48 is the number of channels and 24 is the width and height of the output feature map;

[0047] After two layers of convolution operations to extract features, the size becomes [128*12*12], where 128 is the number of channels and 12 is the width and height of the output feature map;

[0048] After two layers of convolution operations to extract features, the size becomes [256*6*6], where 256 is the number of channels and 6 is the width and height of the output feature map;

[0049] The image features are reduced by 32 times, and Neck uses the FPN network to upsample the feature map reduced by 32 times back to the original image size. The upsampling here uses the nearest neighbor difference algorithm (that is, directly copying adjacent pixels). The FPN process is as follows Figure 1 The FPN module on the right is shown:

[0050] The size of the image after double upsampling is [256*12*12];

[0051] 1×1 convolutional layer, input channel 256, output channel 128, at this time the feature map size becomes [128*12*12];

[0052] 2x upsampling, the size is [128, 24, 24], the 1×1 convolution layer has 128 input channels and 48 output channels, and the size becomes [48, 24, 24];

[0053] 2x upsampling, size is [48, 48, 48], 1×1 convolution layer has 48 input channels and 16 output channels, and the size becomes [16, 48, 48];

[0054] After 2x upsampling, the size is [16,96,96];

[0055] The head part uses a convolution operation with a convolution kernel of 1x1 and a dimension of 2. The process is:

[0056] 1) The convolution kernel size used is 1x1, which means that each convolution kernel is a 1x1 matrix that performs sliding calculation operations on the feature map.

[0057] 2) Since the dimension is 2, each time the convolution kernel is sliding, the output in two dimensions will be calculated respectively; Figure 1 As shown in the Head module;

[0058] 1×1 convolutional layer, with 16 input channels, 2 output channels, and size [2, 96, 96];

[0059] Finally, the output image: size = 2*96*96;

[0060] Step S2. Train the neural network model, that is, train the convolutional neural network built in step S1: first, the image input is a single-channel grayscale image with a width of 96 pixels and a height of 96 pixels. After passing through the neural network, the output is a feature map with the same size as the input image and the number of channels changed to 2, that is, (2,96,96), which means that each pixel on the 96x96 image is classified into two categories (here, a convolution with a dimension of 2 and a convolution kernel of 1 is used for binary classification);

[0061] The loss function for measuring the classification results uses the cross entropy loss function; in deep learning, cross entropy is usually used to measure the difference between the model prediction value and the true label, that is, the difference between the probability distribution of the model output and the probability distribution of the true label. The formula is:

[0062] L=-1 / N*Σ(y_i*log(p_i)+(1-y_i)*log(1-p_i)),

[0063] Where N represents the number of samples, y_i represents the true label of the i-th sample, and p_i represents the predicted probability of the i-th sample.

[0064] S3. Inference phase: Processing network output results

[0065] According to step S2, the output size of the network is (2,96,96). The torch.argmax() function is used to obtain the index of the maximum value in the output result. The argmax function is used to return the index of the maximum value in the tensor. For example, the output result is [2.24,5.67]. After comparison, 5.67 is greater than 2.24. Since the initial value of the index is 0, the index of the maximum value 5.67 is 1. Because each pixel on the image is classified into two categories here (the method is the same as the two categories in step S1), the index values ​​are only 0 and 1. For each pixel, if the index value is 1, it means that the current pixel is a key point. The output result is visualized as shown in Figure 2(1) and Figure 2(2). The white dot in Figure 2(2) is the pixel with an index value of 1 after the image in Figure 2(1) is extracted by the neural network and finally classified into two categories, that is, the key point obtained at the end.

[0066] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for improving the accuracy of image splicing of scanning translator pen, characterized in that: The method comprises the following steps: S1. Constructing a convolutional neural network The convolutional neural network uses a Backbone, Neck, and Head structure. The Backbone uses the resnet18 network for feature extraction and reduces the image features by 32 times. The Neck uses the FPN network to upsample the feature map reduced by 32 times back to the original image size. The upsampling here uses the nearest neighbor difference algorithm, which directly copies adjacent pixels. The Head part uses a convolution operation with a convolution kernel of 1x1 and a dimension of 2. S2. Train the neural network model, i.e., the convolutional neural network built in step S1; First, the input of the image is a single-channel grayscale image with a width of 96 pixels and a height of 96 pixels. After passing through the neural network, the output is a feature map with the same size as the input image and the number of channels changed to 2, that is, (2,96,96), which means that each pixel on the 96x96 image is classified into two categories. Here, a convolution with a dimension of 2 and a convolution kernel of 1 is used for binary classification. For each pixel in the binary classification, if the index value is 1, it means that the current pixel is a key point. The loss function for measuring the classification results uses the cross entropy loss function; in deep learning, cross entropy is usually used to measure the difference between the model prediction value and the true label, that is, the difference between the probability distribution of the model output and the probability distribution of the true label. The formula is: L=-1 / N*Σ(y_i*log(p_i)+(1-y_i)*log(1-p_i)), Where N represents the number of samples, y_i represents the true label of the i-th sample, and p_i represents the predicted probability of the i-th sample; S3. Inference phase: Processing network output results According to step S2, the output size of the network is (2, 96, 96). The torch.argmax() function is used to obtain the index of the maximum value in the output result. The argmax function is used to return the index of the maximum value in the tensor. Because each pixel on the image is classified into two categories here, the index values ​​are only 0 and 1. For each pixel, if the index value is 1, it means that the current pixel is a key point.

2. The method for improving the accuracy of scanning pen image splicing according to claim 1, characterized in that: In step S1, the resnet18 network is used for feature extraction, and the extraction process is assumed to be a Backbone module, further comprising: Input image: size = 1*96*96; After two layers of convolution to extract features, the size becomes [16*48*48], where 16 is the number of channels and 48 is the width and height of the output feature map; After two layers of convolution operations to extract features, the size becomes [48*24*24], where 48 is the number of channels and 24 is the width and height of the output feature map; After two layers of convolution operations to extract features, the size becomes [128*12*12], where 128 is the number of channels and 12 is the width and height of the output feature map; After two layers of convolution operations to extract features, the size becomes [256*6*6], where 256 is the number of channels and 6 is the width and height of the output feature map.

3. The method for improving the accuracy of image stitching of a scanning pen according to claim 2, characterized in that: In step S1, the FPN process further includes: The size of the image after double upsampling is [256*12*12]; 1×1 convolutional layer, input channel 256, output channel 128, at this time the feature map size becomes [128*12*12]; 2x upsampling, the size is [128, 24, 24], the 1×1 convolution layer has 128 input channels and 48 output channels, and the size becomes [48, 24, 24]; 2x upsampling, size is [48, 48, 48], 1×1 convolution layer has 48 input channels and 16 output channels, and the size becomes [16, 48, 48]; After 2x upsampling, the size is [16,96,96].

4. The method for improving the accuracy of image stitching of a scanning pen according to claim 3, characterized in that: In step S1, the Head part adopts a convolution operation with a convolution kernel of 1x1 and a dimension of 2; the process is: 1) The convolution kernel size used is 1x1, which means that each convolution kernel is a 1x1 matrix that performs sliding calculation operations on the feature map; 1×1 convolutional layer, with 16 input channels, 2 output channels, and size [2, 96, 96]; 2) Since the dimension is 2, the output of 2 dimensions will be calculated for each sliding position of the convolution kernel; Finally, the output image is: size = 2*96*96.

5. The method for improving the accuracy of image stitching of a scanning pen according to claim 1, characterized in that: In step S3, assuming that the output result is [2.24, 5.67], after comparison, 5.67 is greater than 2.

24. Since the initial value of the index is 0, the index of the maximum value 5.67 returned is 1.

Images