Splicing method and device for large-size high-resolution image

Abstract

The invention discloses a splicing method and device of a large-size high-resolution image. The method comprises the following steps: cutting an image pair overlapping region; inputting the cut image pair into a convolutional network for feature extraction; performing network prediction to generate a mean value and a variance; sorting the variances, and selecting the first k matched point pair data with the lowest variance as an output result of the network; reading the coordinate information to update the point pair data information, and generating a real point pair corresponding relation of the large-size image; generating a homography matrix by using point pair information, and acting on the image to complete final splicing; and repeating the steps from left to right and from top to bottom, when the upper and lower parts and the left and right parts are overlapped at the same time, respectively cutting the input network twice to output two results, and splicing the two results to complete the splicing of the whole image. According to the method, a dense correspondence algorithm between high-resolution images is established, the feature information corresponding precision level during image registration is improved, and the problem of low image splicing precision of large-size images is solved.

Description

Technical Field

[0001] The present invention relates to the technical field related to dense correspondence in image stitching, and in particular to a stitching method and device for large-size and high-resolution images. Background Art

[0002] Image stitching technology refers to the technology of stitching two or more overlapping images into a panoramic image. The core problem of image stitching technology lies in image registration. Traditional methods choose to find feature descriptors such as SIFT, SURF, ORB for image registration, or use more accurate network structures such as SuperPoint and D2-net to generate feature descriptions for registration. However, the above methods have low accuracy when facing large-scale high-resolution image data. They cannot achieve satisfactory results.

[0003] Dense correspondence technology can represent the dense correspondence relationship between data with higher accuracy than the registration method based on feature points. In principle, it can achieve high-precision stitching of large-size and high-resolution images. However, general networks do not have the data throughput of high-resolution images, resulting in the current lack of application of dense correspondence technology in high-resolution image stitching. Summary of the invention

[0004] In view of the problem that most neural networks are currently unable to read 4K, 8K and other high-resolution large-size images for image registration, the present invention proposes to establish a dense correspondence algorithm between high-resolution images to improve the accuracy of feature information correspondence during image registration and solve the problem of low precision in large-size image stitching.

[0005] The technical solution of the present invention is:

[0006] The present invention discloses a method and device for stitching large-size and high-resolution images. The method comprises:

[0007] S1, cropping two images to be stitched with overlapping areas in a preset cropping direction, and recording coordinate information used in cropping;

[0008] S2, input the cropped image pair into the convolutional network to extract features, and perform correlation calculation on the obtained features to obtain a correlation matrix of the image pair;

[0009] S3, inputting the correlation matrix into the subsequent network prediction output mean and variance, wherein the mean represents the target value of the coordinate information of the cropped image pair, and the variance represents the confidence between the coordinate information of the cropped image pair and the target value;

[0010] S4, sort the variance results, and select the first k point pairs with the lowest variance as the output results of the network;

[0011] S5, after reading the coordinate information in S1, update the point pair data information to generate a real point pair correspondence relationship of the large-size image;

[0012] S6, determining whether there is overlap between the two images to be spliced ​​in a cropping direction perpendicular to the preset cropping direction, and if so, cropping the two images to be spliced ​​having the overlapping area in the direction, recording the coordinate information used in cropping, and repeatedly executing S2-S5 to obtain a corresponding relationship between the point pairs corresponding to the overlapping area;

[0013] S7, using the point pair information obtained in S5 and S6 to generate a homography matrix, and applying it to the image to complete the final stitching. Specifically, in S1, the two images to be stitched are cropped according to the image overlap ratio, and the cropped image data is ensured to be much smaller than the original image size and can contain all the image overlap areas without exceeding the network throughput.

[0014] Specifically, the convolutional network in S2 is composed of an encoder and two decoders, wherein one decoder is used to realize the mean prediction of the probability model; the other decoder realizes the prediction of the variance of the probability model; the results of the two decoders are fused to output the final probability model.

[0015] Specifically, in S2, when training the model, a series of training data are perspective-changed, and a mask area is manually added to add noise, and the corresponding two images are recorded, and the corresponding transformation mapping is used as the label information of the flow.

[0016] Specifically, the network in S3 sets discretely distributed variance data from large to small to ensure the convergence of the network. The result of inference prediction is the preset discrete variance, which is combined into the final result.

[0017] Specifically, the final output result of the network in S3 is a probability distribution, the loss is calculated as the difference between the sampling result and the annotation of the final predicted distribution, and the optimization direction is to minimize the loss.

[0018] Specifically, the point pair information read in S5 is the corresponding coordinate correspondence between the two cropped images, which is updated using the coordinate information during cropping to generate the real point pair coordinate correspondence information of the large-size high-resolution image.

[0019] Specifically, the updating method specifically includes: respectively obtaining the coordinates of the lower right corner of the stitched image (x, y) and the upper left corner of the stitched image (0, 0), using the designed hyperparameters to perform cropping with a width of w and a height of h, inputting the cropped image pair into a convolutional network, calculating the corresponding coordinate relationship of the image pair, the coordinates of the stitched image do not need to be translated, and an offset (xw, yh) is added to all the coordinates of the stitched image to update the coordinate correspondence.

[0020] Specifically, in S7, a homography matrix is ​​generated by using the real point-to-coordinate correspondence relationship and applied to the image to be stitched, and a suitable weight is selected for the stitching area to perform linear superposition to generate the final result.

[0021] In addition, an embodiment of the present invention further provides a large-size high-resolution image stitching device, comprising:

[0022] An image cropping module, used to crop two images to be spliced ​​with overlapping areas in a preset cropping direction, and record coordinate information used in cropping;

[0023] A feature extraction and correlation calculation module, used for inputting the cropped image pair into the convolutional network to extract features, and performing correlation calculation on the obtained features to obtain a correlation matrix of the image pair;

[0024] A mean and variance prediction module, used for inputting the correlation matrix into a subsequent network prediction output mean and variance, wherein the mean represents the target value of the coordinate information of the cropped image pair, and the variance represents the confidence between the coordinate information of the cropped image pair and the target value;

[0025] The point pair data selection module is used to sort the variance results and select the first k point pair data with the lowest variance as the output result of the network;

[0026] A point pair correspondence generation module is used to update the point pair data information after reading the coordinate information in the image cropping module to generate a point pair correspondence of a real large-size image;

[0027] An overlapping region judgment module is used to judge whether there is overlap between the two images to be spliced ​​in a cropping direction perpendicular to a preset cropping direction. If so, the two images to be spliced ​​with the overlapping region are cropped in the direction, the coordinate information used in cropping is recorded, and the feature extraction and correlation calculation module, the mean and variance calculation module, the point pair data selection module, and the point pair correspondence generation module are repeatedly executed to obtain the point pair correspondence corresponding to the overlapping region;

[0028] A coordinate information updating module is used to update point pair data information to generate a real point pair correspondence relationship;

[0029] The homography matrix generation module is used to generate the homography matrix according to the obtained point pair information, and act on the image to complete the final stitching.

[0030] The beneficial effects of the present invention are:

[0031] The present invention is based on a probability density correspondence model and applies the probability density correspondence model to the field of image registration, thereby solving the problem that network models such as convolution cannot handle large-size high-resolution images and achieving high-precision stitching of high-resolution images. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Flowchart of a large-size high-resolution image stitching method provided by an embodiment of the present invention

[0033] Figure 2 A schematic diagram of network input content provided by an embodiment of the present invention;

[0034] Figure 3 A schematic diagram of a network architecture provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0036] Example

[0037] Figure 1 A flow chart of a large-size high-resolution image stitching method provided by an embodiment of the present invention specifically includes the following steps:

[0038] S1. Crop two images to be stitched with overlapping areas in a preset cropping direction, and record coordinate information used in cropping.

[0039] The images to be stitched are large-size and high-resolution images.

[0040] For example, see Figure 2 Since the two images to be stitched may have overlapping areas in different directions, they may need to be stitched in the left-right direction and the up-down direction at the same time. Therefore, the present application performs the cropping and stitching of the overlapping areas according to the preset cropping direction to solve the problem that the images to be stitched have both up-down stitching and left-right stitching. The preset cropping direction can be the left-right direction or the up-down direction.

[0041] Specifically, in S1, the two images to be stitched are cropped according to the ratio of image overlap, ensuring that the cropped image data is much smaller than the original image size and can contain all image overlap areas without exceeding the network throughput.

[0042] S2. Input the cropped image pair into the convolutional network for feature extraction, and perform correlation calculation on the obtained features to obtain the correlation matrix of the image pair.

[0043] For more information, see Figure 3 The convolutional network in S2 consists of an encoder and two decoders, one of which is used to realize the mean prediction of the probability model; the other decoder realizes the prediction of the variance of the probability model; the results of the two decoders are fused to output the final probability model.

[0044] Optionally, the encoder in this embodiment can select a convolutional network such as ResNet, ConvNeXt or a Transformer network such as ViT to realize feature extraction of two image data. After completing the feature extraction and generating a feature correlation matrix, the two decoders of the network can also be designed as a convolutional network or a Transformer network.

[0045] Specifically, when training the model, S2 rotates and translates a series of training data, and manually adds noise to the mask area. The corresponding two images are recorded, and the corresponding transformation mapping is used as the label information of the flow. By designing a targeted data generation process, the collection of training set data is facilitated and the high robustness of the training model is guaranteed.

[0046] S3. Input the correlation matrix into the subsequent network prediction output mean and variance, where the mean represents the target value of the coordinate information of the cropped image pair, and the variance represents the confidence between the coordinate information of the cropped image pair and the target value.

[0047] Specifically, the network in S3 sets discretely distributed variance data from large to small to ensure the convergence of the network. The result of inference and prediction is the preset discrete variance, which is combined into the final result.

[0048] Specifically, the final output result of the network in S3 is a probability distribution, the loss is calculated as the difference between the sampling result and the annotation of the final predicted distribution, and the optimization direction is to minimize the loss.

[0049] S4. Sort the variance results and select the first k matched point pairs with the lowest variance as the output results of the network.

[0050] S5. After reading the coordinate information in step 1, the point pair data information is updated to generate a point pair correspondence relationship of a real large-size image.

[0051] Specifically, the point pair information read in S5 is the corresponding coordinate correspondence between the two cropped images, which is updated using the coordinate information during cropping to generate the real point pair coordinate correspondence information of the large-size high-resolution image.

[0052] Specifically, the updating method specifically includes: respectively obtaining the coordinates of the lower right corner of the stitched image (x, y) and the upper left corner of the stitched image (0, 0), using the designed hyperparameters to perform cropping with a width of w and a height of h, inputting the cropped image pair into a convolutional network, calculating the corresponding coordinate relationship of the image pair, the coordinates of the stitched image do not need to be translated, and an offset (xw, yh) is added to all the coordinates of the stitched image to update the coordinate correspondence.

[0053] S6. Determine whether there is overlap between the two images to be spliced ​​in a cropping direction perpendicular to the preset cropping direction. If so, crop the two images to be spliced ​​with the overlapping area in this direction, record the coordinate information used in cropping, and repeat S2-S5 to obtain the corresponding relationship between the point pairs corresponding to the overlapping area.

[0054] S7. Generate a homography matrix using the point pair information obtained in S5 and S6, and apply it to the image to complete the final stitching.

[0055] Specifically, in S7, a homography matrix is ​​generated by using the real point-to-coordinate correspondence relationship and applied to the image to be stitched, and a suitable weight is selected for the stitching area to perform linear superposition to generate the final result.

[0056] Specifically, when there are multiple overlapping areas between the spliced ​​image and the image to be spliced, see Figure 2 , the convolutional network is input twice, once for the red area and once for the blue area, the results of the two outputs of the network are spliced ​​and sorted, and a set of point pair correspondences are output to generate a homography matrix, which is applied to the image to complete image stitching.

[0057] In addition, an embodiment of the present invention further provides a large-size high-resolution image stitching device, comprising:

[0058] An image cropping module, used to crop two images to be spliced ​​with overlapping areas in a preset cropping direction, and record coordinate information used in cropping;

[0059] A feature extraction and correlation calculation module, used for inputting the cropped image pair into the convolutional network to extract features, and performing correlation calculation on the obtained features to obtain a correlation matrix of the image pair;

[0060] A mean and variance prediction module, used for inputting the correlation matrix into a subsequent network prediction output mean and variance, wherein the mean represents the target value of the coordinate information of the cropped image pair, and the variance represents the confidence between the coordinate information of the cropped image pair and the target value;

[0061] The point pair data selection module is used to sort the variance results and select the first k point pair data with the lowest variance as the output result of the network;

[0062] A point pair correspondence generation module is used to update the point pair data information after reading the coordinate information in the image cropping module to generate a point pair correspondence of a real large-size image;

[0063] An overlapping region judgment module is used to judge whether there is overlap between the two images to be spliced ​​in a cropping direction perpendicular to a preset cropping direction. If so, the two images to be spliced ​​with the overlapping region are cropped in the direction, the coordinate information used in cropping is recorded, and the feature extraction and correlation calculation module, the mean and variance calculation module, the point pair data selection module, and the point pair correspondence generation module are repeatedly executed to obtain the point pair correspondence corresponding to the overlapping region;

[0064] A coordinate information updating module is used to update point pair data information to generate a real point pair correspondence relationship;

[0065] The homography matrix generation module is used to generate the homography matrix according to the obtained point pair information, and act on the image to complete the final stitching.

[0066] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for stitching large-size high-resolution images, characterized in that: include: S1, cropping two images to be stitched with overlapping areas in a preset cropping direction, and recording coordinate information used in cropping; S2, input the cropped image pair into the convolutional network to extract features, and perform correlation calculation on the obtained features to obtain a correlation matrix of the image pair; S3, inputting the correlation matrix into the subsequent network prediction output mean and variance, wherein the mean represents the target value of the coordinate information of the cropped image pair, and the variance represents the confidence between the coordinate information of the cropped image pair and the target value; S4, sort the variance results, and select the first k point pairs with the lowest variance as the output results of the network; S5, after reading the coordinate information in S1, update the point pair data information to generate a real point pair correspondence relationship of the large-size image; S6, determining whether there is overlap between the two images to be spliced ​​in a cropping direction perpendicular to the preset cropping direction, and if so, cropping the two images to be spliced ​​having the overlapping area in the direction, recording the coordinate information used in cropping, and repeatedly executing S2-S5 to obtain a corresponding relationship between the point pairs corresponding to the overlapping area; S7. Generate a homography matrix using the point pair information obtained in S5 and S6, and apply it to the image to complete the final stitching.

2. The method according to claim 1, characterized in that The S1 specifically includes: The two images to be stitched are cropped according to the ratio of image overlap, and the cropped image data is ensured to be much smaller than the original image size and to contain all image overlap areas without exceeding the network throughput.

3. The method according to claim 1, characterized in that The convolutional network in S2 is composed of an encoder and two decoders, one of which is used to realize the mean prediction of the probability model; the other decoder realizes the prediction of the variance of the probability model; the results of the two decoders are fused to output the final probability model.

4. The method according to claim 3, characterized in that When training the model, a series of training data are transformed in perspective, and a mask area is manually added to add noise. The corresponding two images are recorded, and their corresponding transformation mapping is used as the label information of the flow.

5. The method according to claim 1, characterized in that: The convolutional neural network in S2 sets the variance data of discrete distribution from large to small, and the result of inference prediction is the preset discrete variance.

6. The method according to claim 1, characterized in that The S5 specifically includes: The point pair information read is the corresponding coordinate correspondence between the two cropped images. It is updated using the coordinate information during cropping to generate the real point pair coordinate correspondence information of the large-size and high-resolution image.

7. The method according to claim 6, characterized in that The updating method specifically includes: Get the coordinates of the lower right corner of the stitched image (x, y) and the upper left corner of the stitched image (0, 0) respectively, use the designed hyperparameters to crop the image with a width of w and a height of h, input the cropped image pair into the convolutional network, calculate the corresponding coordinate relationship of the image pair, the coordinates of the stitched image do not need to be translated, add an offset (xw, yh) to all the coordinates of the stitched image, and update the coordinate correspondence.

8. The method according to claim 1, characterized in that The S7 specifically includes: The homography matrix is ​​generated by using the real point-to-point coordinate correspondence and applied to the image to be stitched. The appropriate weight is selected for the stitching area to perform linear superposition to generate the final result.

9. A large-size high-resolution image stitching device, characterized in that: include: An image cropping module, used to crop two images to be spliced ​​with overlapping areas in a preset cropping direction, and record coordinate information used in cropping; A feature extraction and correlation calculation module, used for inputting the cropped image pair into the convolutional network to extract features, and performing correlation calculation on the obtained features to obtain a correlation matrix of the image pair; A mean and variance prediction module, used for inputting the correlation matrix into a subsequent network prediction output mean and variance, wherein the mean represents the target value of the coordinate information of the cropped image pair, and the variance represents the confidence between the coordinate information of the cropped image pair and the target value; The point pair data selection module is used to sort the variance results and select the first k point pair data with the lowest variance as the output result of the network; A point pair correspondence generation module is used to update the point pair data information after reading the coordinate information in the image cropping module to generate a point pair correspondence of a real large-size image; An overlapping region judgment module is used to judge whether there is overlap between the two images to be spliced ​​in a cropping direction perpendicular to a preset cropping direction. If so, the two images to be spliced ​​with the overlapping region are cropped in the direction, the coordinate information used in cropping is recorded, and the feature extraction and correlation calculation module, the mean and variance calculation module, the point pair data selection module, and the point pair correspondence generation module are repeatedly executed to obtain the point pair correspondence corresponding to the overlapping region; A coordinate information updating module is used to update point pair data information to generate a real point pair correspondence relationship; The homography matrix generation module is used to generate the homography matrix according to the obtained point pair information, and act on the image to complete the final stitching.

Images