Semantic guidance multi-homography estimation-based unsupervised image splicing method
By combining semantic segmentation and multi-homography estimation, the problems of misalignment artifacts and homography inconsistency in multi-planar scenes in image stitching are solved, achieving efficient image stitching results, especially in low-light and low-texture scenes.
Patent Information
- Application Number
- CN202411498839.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing image stitching methods suffer from misalignment artifacts and inconsistent homography estimation when dealing with multi-planar scenes, especially in low-light and low-texture scenes. Furthermore, traditional and deep learning methods have limited effectiveness in multi-planar scenes.
A semantically guided multi-homography estimation method is adopted. The image is segmented into multiple planar regions through a semantic segmentation network. Eight positive AC bases are used for parameterized homography transformation. Combined with semantic plane distribution consistency, a homography transformation network is constructed for training and stitching.
It significantly improves the planar distribution consistency and accuracy of image stitching, achieving precise, consistent and robust image stitching results.
Smart Images

Figure CN119006276A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of image stitching, and in particular to an unsupervised image stitching method based on semantically guided multi-homography estimation. Background Art
[0002] Image stitching is a practical computer vision application that aims to construct an image with a wider field of view of a scene from captured images with a narrow field of view. This technology plays an important role in many fields such as biology, medicine, autonomous driving, surveillance video and virtual reality.
[0003] Existing image stitching schemes are divided into traditional methods and deep learning-based methods. Traditional image stitching methods extract features from input images and identify feature points that can be used for matching. By calculating the relationship between these feature points, the geometric transformation parameters of the image are determined to achieve accurate alignment between images. However, these methods rely heavily on the detection quality of feature points and often perform poorly in scenes with insufficient lighting and low texture. Deep learning-based methods use models such as convolutional neural networks (CNNs) or fully convolutional networks (FCNs) to estimate the transformation parameters between images. Compared with traditional methods, these methods have stronger feature extraction and expression capabilities and can improve the accuracy and robustness of image stitching to a certain extent. However, existing deep image stitching methods mainly align images by estimating a single global homography. When there are multiple planes in the image, these methods have serious misalignment artifacts. In order to alleviate the misalignment artifacts, some methods reject outlier areas by learning the main plane area mask and only select reliable areas for homography estimation. However, these methods are still essentially a global homography transformation and do not solve the problem of multiple planes. The latest methods solve the multi-plane problem by dividing the image into multiple uniform grid cells and estimating a local homography transformation for each grid cell. However, these methods divide the image into uniform grids, which may result in a grid containing multiple planes and a plane being divided into multiple grids, destroying the distribution of the planes. The limitations of a single homography are not completely solved, and the estimated homography is inconsistent when the main planes of adjacent grids are different. Summary of the invention
[0004] In order to overcome at least one deficiency in the prior art, the present application provides an unsupervised image stitching method based on semantically guided multi-homography estimation.
[0005] In a first aspect, a method for constructing a homography transformation network based on semantically guided multi-homography estimation is provided, comprising: A natural image dataset is obtained, where samples in the natural image dataset are image pairs, and the image pairs include a reference image and a target image; each image in the image pair is an image containing a multi-plane region; Each image in the sample is segmented using a semantic segmentation model to obtain a mask set corresponding to each image. The mask set includes multiple masks, each mask corresponds to a plane area, and the mask has a semantic label. Preprocessing is performed on the two mask sets of each image pair to obtain a matching plane region mask pair set, wherein the matching plane region mask pair set includes multiple mask pairs, and the two masks in each mask pair belong to different mask sets and have the same semantic label; Each mask pair in the matched plane region mask pair set is multiplied element-by-element with the image pair to obtain a plane region image pair set; the plane region image pair set includes a plurality of plane region image pairs; Constructing a homography transformation network; the homography transformation network includes a feature extraction module, a correlation calculation module, a regression network, and a parameterized homography module; the homography transformation network is used to obtain a homography transformation flow of each plane region based on each plane region image pair in the plane region image pair set; The homography transformation network is trained based on a set of plane region image pairs to obtain a trained homography transformation network.
[0006] In one embodiment, two mask sets of each image pair are preprocessed to obtain a set of matching plane region mask pairs, including: For each mask set, select masks with an area greater than 5000 pixels to form the processed mask set; The masks in the two processed mask sets that have the same semantic label and whose mask size difference is less than the set value constitute a mask pair; All mask pairs constitute the set of matched planar region mask pairs.
[0007] In one embodiment, the feature extraction module is used to extract features from each image in the plane region image pair to obtain a feature map; The correlation calculation module is used to perform correlation calculation on two feature maps corresponding to the plane area image pair to obtain a correlation map between the feature maps; The regression network is used to obtain the weights of eight orthogonal basis based on the correlation graph between feature graphs; The parameterized homography module is used to perform parameterized homography according to the weights of eight orthogonal basis values and eight orthogonal basis values, and obtain a homography transformation flow of a planar region corresponding to a planar region image.
[0008] In one embodiment, the homography transformation network is trained based on a set of planar region image pairs, and the loss function used in the training process is:
[0009]
[0010]
[0011] in, is the loss function, is the feature intensity loss, is the transformation consistency loss, is the weight of feature strength loss, is the weight of the transformation consistency loss, is the label of the plane region, is the number of plane regions, represents the feature extraction operation, represents the homography transformation, is the target image, The target image corresponds to The mask of the plane region, is the reference image, The reference image corresponds to The mask of the plane region, is the first The homography transformation flow of a planar region, represents element-wise multiplication, is 1 norm, is a matrix of all 1s of the same size.
[0012] In a second aspect, an unsupervised image stitching method based on semantically guided multi-homography estimation is provided, comprising: Obtain an image pair to be stitched, perform preliminary segmentation on each image of the image pair to be stitched using a semantic segmentation model, and obtain a mask set corresponding to each image; Preprocessing two mask sets of the image pair to be spliced to obtain a matching plane region mask pair set, wherein the matching plane region mask pair set includes a plurality of mask pairs; Each mask pair in the set of matched plane region mask pairs is multiplied element-by-element with the image pair to be spliced to obtain a set of plane region image pairs; Inputting the set of plane region image pairs into the trained homography transformation network to obtain the homography transformation stream of each plane region; the trained homography transformation network is obtained by the above-mentioned homography transformation network construction method based on semantically guided multi-homography estimation; The homography transformation flow of the planar region is used to transform the corresponding planar region in the natural image of the image pair to be stitched, so as to obtain the aligned planar region; all the aligned planar regions are fused to obtain the stitched image.
[0013] In one embodiment, two mask sets of the image pair to be stitched are preprocessed to obtain a set of matching plane region mask pairs, including: For each mask set, select masks with an area greater than 5000 pixels to form the processed mask set; The masks in the two processed mask sets that have the same semantic label and whose mask size difference is less than the set value constitute a preliminary mask pair; For each mask in the preliminary mask pair, determine the neighboring mask of the mask, and merge the neighboring mask into the mask to form a final mask pair; the neighboring mask is the mask that is adjacent to the mask but not matched into a mask pair and the mask whose area is less than 1000 pixels; All final mask pairs constitute the set of matched planar region mask pairs.
[0014] In a third aspect, a homography transformation network construction device based on semantically guided multi-homography estimation is provided, comprising: The data set acquisition module is used to acquire a natural image data set, wherein the samples in the natural image data set are image pairs, and the image pairs include a reference image and a target image; each image in the image pair is an image containing a multi-plane region; A first preliminary segmentation module is used to perform preliminary segmentation on each image in the sample using a semantic segmentation model to obtain a mask set corresponding to each image, where the mask set includes multiple masks, each mask corresponds to a plane area, and the mask has a semantic label; A first preprocessing module is used to preprocess the two mask sets of each image pair to obtain a matching plane area mask pair set, where the matching plane area mask pair set includes multiple mask pairs, and the two masks in each mask pair belong to different mask sets and have the same semantic label; A first image pair set acquisition module, used for performing element-by-element multiplication of each mask pair in the matched plane region mask pair set with the image pair to obtain a plane region image pair set; the plane region image pair set includes a plurality of plane region image pairs; A network construction module is used to construct a homography transformation network; the homography transformation network includes a feature extraction module, a correlation calculation module, a regression network, and a parameterized homography module; the homography transformation network is used to obtain a homography transformation flow of each plane region based on each plane region image pair in the plane region image pair set; The network training module is used to train the homography transformation network based on the set of plane region image pairs to obtain the trained homography transformation network.
[0015] In a fourth aspect, an unsupervised image stitching device based on semantically guided multi-homography estimation is provided, comprising: The second preliminary segmentation module is used to obtain the image pair to be spliced, and perform preliminary segmentation on each image of the image pair to be spliced using a semantic segmentation model to obtain a mask set corresponding to each image; A second preprocessing module is used to preprocess the two mask sets of the image pair to be spliced to obtain a matching plane region mask pair set, wherein the matching plane region mask pair set includes a plurality of mask pairs; A second image pair set acquisition module is used to perform element-by-element multiplication of each mask pair in the matched plane region mask pair set with the image pair to be spliced to obtain a plane region image pair set; A homography transformation module is used to input a set of plane region image pairs into a trained homography transformation network to obtain a homography transformation stream for each plane region; the trained homography transformation network is obtained by the above-mentioned homography transformation network construction method based on semantically guided multi-homography estimation; The fusion module is used to transform the corresponding plane regions in the natural images in the image pair to be stitched by using the homography transformation flow of the plane region to obtain the aligned plane regions; all the aligned plane regions are fused to obtain the stitched image.
[0016] In a fifth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, it can implement the above-mentioned homography transformation network construction method based on semantically guided multi-homography estimation, or the above-mentioned unsupervised image stitching method based on semantically guided multi-homography estimation.
[0017] In a sixth aspect, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the above-mentioned homography transformation network construction method based on semantically guided multi-homography estimation, or the above-mentioned unsupervised image stitching method based on semantically guided multi-homography estimation.
[0018] Compared with the prior art, the present application has the following beneficial effects: the unsupervised image stitching method based on semantically guided multi-homography estimation of the present application takes advantage of the consistent distribution of semantic segmentation planes to realize the combination of semantic planes and multi-homography estimation, breaking the limitations of traditional methods and deep learning methods in image stitching. The present application uses a semantic segmentation network to segment the image, and post-processes the results of the preliminary segmentation to obtain multiple plane regions. Under the same computational cost, the present application significantly improves the distribution consistency of the plane; the use of eight orthogonal basis functions to parameterize homography improves the accuracy of irregular plane homography estimation, thereby achieving accurate, consistent and robust image stitching. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The present application may be better understood by referring to the following description given in conjunction with the accompanying drawings, which together with the following detailed description are included in this specification and form a part of this specification. In the drawings: Figure 1 A flowchart of a method for constructing a homography transformation network based on semantically guided multi-homography estimation according to an embodiment of the present application is shown; Figure 2 A schematic diagram of the structure of the regression network is shown; Figure 3 A schematic diagram of the structure of a convolutional block is shown; Figure 4 shows a schematic diagram of the structure of a linear block; Figure 5 A flowchart of an unsupervised image stitching method based on semantically guided multi-homography estimation according to an embodiment of the present application is shown; Figure 6 A structural block diagram of a homography transformation network construction device based on semantically guided multi-homography estimation according to an embodiment of the present application is shown; Figure 7 A structural block diagram of an unsupervised image stitching device based on semantically guided multi-homography estimation according to an embodiment of the present application is shown. DETAILED DESCRIPTION
[0020] The exemplary embodiments of the present application will be described below in conjunction with the accompanying drawings. For the sake of clarity and conciseness, not all features of the actual embodiments are described in the specification. However, it should be understood that many implementation-specific decisions can be made in the process of developing any such actual embodiments in order to achieve the specific goals of the developer, and these decisions may vary from embodiment to embodiment.
[0021] It is also necessary to explain here that, in order to avoid obscuring the present application due to unnecessary details, only the device structure closely related to the scheme according to the present application is shown in the drawings, while other details that are not closely related to the present application are omitted.
[0022] It should be understood that the present application is not limited to the described implementation forms due to the following description with reference to the accompanying drawings. In this article, where feasible, the embodiments can be combined with each other, features between different embodiments can be replaced or borrowed, and one or more features can be omitted in one embodiment.
[0023] This application aims to solve the problem that traditional image stitching methods are heavily dependent on the quality of feature detection and are not suitable for low-light and low-texture scenes. In addition, the deep learning-based methods in the prior art for estimating global homography and multi-homography estimation based on image grid division cannot completely solve the limitation of single homography. This application provides a homography transformation network construction method based on semantic-guided multi-homography estimation. Figure 1 A flowchart of a method for constructing a homography transformation network based on semantically guided multi-homography estimation according to an embodiment of the present application is shown, see Figure 1 , methods include: Step S11, obtaining a natural image data set, where samples in the natural image data set are image pairs, and the image pairs include a reference image and a target image; each image in the image pair is an image containing multiple plane regions.
[0024] Step S12, using a semantic segmentation model to perform preliminary segmentation on each image in the sample, and obtaining a mask set corresponding to each image, the mask set includes multiple masks, each mask corresponds to a plane area, and the mask has a semantic label.
[0025] Here, the semantic segmentation model can adopt existing models such as U-Net and DeepLab. After the image of the multi-plane area is segmented by the semantic segmentation model, each plane area corresponds to a mask, and the mask has a semantic label. Here, the semantic label refers to the specific category of the target in the plane area.
[0026] Step S13, preprocessing the two mask sets of each image pair to obtain a matching plane area mask pair set, wherein the matching plane area mask pair set includes multiple mask pairs, and the two masks in each mask pair belong to different mask sets and have the same semantic label.
[0027] Here, the mask set is preprocessed to screen out masks that meet the training conditions, and at the same time, the masks in the two mask sets of the image pair are matched to form a mask pair.
[0028] Step S14, multiply each mask pair in the matched plane area mask pair set by the image pair element by element to obtain a plane area image pair set; the plane area image pair set includes multiple plane area image pairs, and each plane area image pair includes 2 plane area images.
[0029] Step S15, constructing a homography transformation network; the homography transformation network includes a feature extraction module, a correlation calculation module, a regression network, and a parameterized homography module; the homography transformation network is used to obtain a homography transformation flow of each plane area based on each plane area image pair in the plane area image pair set.
[0030] Step S16, training the homography transformation network based on the set of plane region image pairs to obtain a trained homography transformation network.
[0031] In this embodiment, a semantic segmentation model is used to segment the image, and the preliminary segmentation results are post-processed to obtain multiple plane area image pairs. A homography transformation network is trained based on the multiple plane area image pairs. The trained homography transformation network is used to obtain the homography transformation flow of each plane area for the image pairs to be spliced in the inference stage, breaking the limitations of traditional methods and deep learning methods in image stitching.
[0032] In one embodiment, in step S13, preprocessing is performed on two mask sets of each image pair to obtain a matching plane region mask pair set, including: For each mask set, select the masks with an area greater than 5000 pixels to form the processed mask set; Masks in two processed mask sets that have the same semantic label and whose mask size difference is less than a set value constitute a mask pair; here, the set value may be 20%.
[0033] All mask pairs constitute the set of matched planar region mask pairs.
[0034] In this embodiment, the goal during training is to obtain mask pairs that are as accurate as possible, and results with larger areas in different categories are selected to reduce the impact of segmentation errors. Small or mismatched masks are not used for network training.
[0035] In one embodiment, in step S15, the feature extraction module is used to extract features from each image in the plane area image pair to obtain a feature map; here, the feature extraction module can be, for example, a feature extraction network using Resnet50 or Resnet18 as the backbone network to extract a feature map with a resolution of 1 / 16 of the original image.
[0036] The correlation calculation module is used to perform correlation calculation on two feature maps corresponding to the plane area image pair to obtain a correlation map between the feature maps; Here, the correlation graph between feature graphs can be obtained using the following formula: First, calculate the correlation between feature maps:
[0037] in, For relevance, is the feature map of the plane region image pair corresponding to the reference image, is the feature map of the plane area image pair corresponding to the target image, for The pixel position in for The pixel position in ; is the activation function; represents transpose; Then, the correlation Convert to a two-channel correlation graph:
[0038] in, is the correlation diagram, is the motion characteristic in the horizontal direction, is the motion characteristic in the vertical direction, To use correlation and the width of the feature map Do the modulo operation, To use correlation and the width of the feature map Perform a division operation. Finally, we get the feature correlation graph of the two channels, whose shape is (B, 2, 32, 32), where B represents the size of the batch_size.
[0039] The regression network is used to obtain the weights of eight orthogonal basis based on the correlation graph between feature graphs; Figure 2 The schematic diagram of the structure of the regression network is shown. The regression network includes three convolutional blocks, Rearrange (rearrangement), and linear blocks connected in sequence.
[0040] Figure 3 A schematic diagram of the structure of a convolution block is shown, which includes a convolution layer (Conv), an activation function layer (ReLU), a convolution layer (Conv), an activation function layer (ReLU), and a maximum pooling layer (MaxPool) connected in sequence.
[0041] Figure 4 A schematic diagram of the structure of a linear block is shown, which includes a linear layer (Linear), an activation function layer (ReLU), a linear layer (Linear), an activation function layer (ReLU), and a linear layer (Linear) connected in sequence.
[0042] Here, see Figure 2 and Figure 3, the correlation map is input into the first convolution block to obtain a feature map of shape (B, 64, 16, 16), the feature map is input into the second convolution block to obtain a feature map of shape (B, 128, 8, 8), and then input into the third convolution block to obtain a feature map of shape (B, 256, 4, 4). Then, the feature map is input into rearrange for rearrangement to obtain a tensor of shape (B, 4096); tensor input Figure 4 In the linear block shown, a tensor of size (B, 8) is obtained, i.e., the weights of 8 orthogonal basis.
[0043] The parameterized homography module is used to perform parameterized homography according to the weights of eight orthogonal basis values and eight orthogonal basis values, and obtain a homography transformation flow of a planar region corresponding to a planar region image.
[0044] Here, we first obtain 8 orthogonal exchange bases. The specific process is as follows: Initialize the homography matrix H, which is a 3*3 matrix, H=[[h1,h2,h3], [h4,h5,h6], [h7,h8,1]], where h1, h2, h3, h4, h5, h6, h7, h8 are all elements of H. The initialized values of h1 and h5 are 1, and the values of h2, h3, h4, h6, h7, h8 are 0; Each time the value of one element in h1, h2, h3, h4, h5, h6, h7, and h8 is modified, that is, from 1 to 0, or from 0 to 1, an updated homography matrix is obtained; Transform an original image based on the updated homography matrix to obtain a transformed image; Subtracting the original image from the transformed image will give a flow basis. After 8 modifications, 8 flow bases are obtained. Use the largest flow basis to normalize each flow basis, and obtain 8 normalized flow basis; The 8 normalized flow bases are orthogonally decomposed to obtain 8 orthogonal flow bases.
[0045] Then, the homography is parameterized with eight orthogonal bases according to their weights, including: The 8 orthogonal basis are multiplied by the weights of the corresponding orthogonal basis and accumulated to obtain the homography transformation flow, using the following formula:
[0046] in, is the homography transformation flow, For the The weights of the orthogonal AC basis, For the A positive exchange basis.
[0047] In one embodiment, in step S16, the homography transformation network is trained based on the set of plane region image pairs, and the loss function used in the training process is:
[0048]
[0049]
[0050] in, is the loss function, is the feature intensity loss, is the transformation consistency loss, is the weight of feature strength loss, is the weight of the transformation consistency loss, is the label of the plane region, is the number of plane regions, represents the feature extraction operation, represents the homography transformation, is the target image, The target image corresponds to The mask of the plane region, is the reference image, The reference image corresponds to The mask of the plane region, is the first The homography transformation flow of a planar region, represents element-wise multiplication, is 1 norm, is a matrix of all 1s of the same size.
[0051] In this embodiment, the parameter optimizer used in training is Adam, the initial learning rate is 0.0001, and the weight decay factor is 0.01.
[0052] An embodiment of the present application also provides an unsupervised image stitching method based on semantically guided multi-homography estimation, which is applied to the stitching of underground coal mine monitoring video screens. The relevant mining equipment and coal rock background in the underground coal mine monitoring video form multiple planes. The existing image stitching technology is difficult to achieve accurate stitching screens with consistent plane distribution. The unsupervised image stitching method based on semantically guided multi-homography estimation of this embodiment can effectively improve the accuracy of coal and rock monitoring screen stitching and the consistency of plane distribution. Figure 5 FIG. 1 shows a flowchart of an unsupervised image stitching method based on semantically guided multi-homography estimation according to an embodiment of the present application, see Figure 5 , methods include: Step S51, obtaining a pair of images to be stitched, performing preliminary segmentation on each image of the pair of images to be stitched using a semantic segmentation model, and obtaining a mask set corresponding to each image.
[0053] Here, obtaining the image pair to be stitched includes selecting two surveillance videos of the same area from the underground coal mine surveillance video, and extracting two surveillance pictures at the same time.
[0054] Step S52 , preprocessing is performed on the two mask sets of the image pair to be spliced to obtain a matching plane region mask pair set, where the matching plane region mask pair set includes a plurality of mask pairs.
[0055] Step S53, performing element-by-element multiplication of each mask pair in the set of matched plane region mask pairs and the image pair to be spliced to obtain a set of plane region image pairs; Step S54, inputting the set of plane region image pairs into the trained homography transformation network to obtain a homography transformation stream for each plane region; the trained homography transformation network is obtained according to the homography transformation network construction method based on semantically guided multi-homography estimation of the aforementioned embodiment; Step S55, use the homography transformation flow of the plane area to transform the corresponding plane area in the natural image of the image pair to be stitched to obtain the aligned plane area; all the aligned plane areas are fused to obtain the stitched image, that is, an accurate coal and rock video monitoring picture with consistent plane distribution is obtained.
[0056] The specific implementation process of each step in the above embodiment can refer to the content of the above embodiment, which will not be repeated here.
[0057] In one embodiment, step S52, preprocessing is performed on two mask sets of the image pair to be stitched to obtain a matching plane region mask pair set, including: For each mask set, select masks with an area greater than 5000 pixels to form the processed mask set; The masks in the two processed mask sets that have the same semantic label and whose mask size difference is less than the set value constitute a preliminary mask pair; For each mask in the preliminary mask pair, determine the neighboring mask of the mask, and merge the neighboring mask into the mask to form a final mask pair; the neighboring mask is the mask that is adjacent to the mask but not matched into a mask pair and the mask whose area is less than 1000 pixels; All final mask pairs constitute the set of matched planar region mask pairs.
[0058] The unsupervised image stitching method based on semantically guided multi-homography estimation in this embodiment takes advantage of the consistent distribution of semantic segmentation planes to achieve the combination of semantic planes and multi-homography estimation, breaking the limitations of traditional methods and deep learning methods in image stitching. This application uses a semantic segmentation network to segment the image, and post-processes the results of the preliminary segmentation to obtain multiple plane regions. Under the same computational cost, this application significantly improves the distribution consistency of the plane; uses eight orthogonal basis variables to parameterize homography, which improves the accuracy of irregular plane homography estimation, thereby achieving accurate, consistent and robust image stitching.
[0059] Using the same inventive concept as the method for constructing a homography transformation network based on semantically guided multi-homography estimation, this embodiment also provides a corresponding homography transformation network construction device based on semantically guided multi-homography estimation. Figure 6 The structural block diagram of a homography transformation network construction device based on semantically guided multi-homography estimation according to an embodiment of the present application is shown, including: The data set acquisition module 61 is used to acquire a natural image data set, where the samples in the natural image data set are image pairs, where the image pairs include a reference image and a target image; each image in the image pair is an image containing a multi-plane region; A first preliminary segmentation module 62 is used to perform preliminary segmentation on each image in the sample using a semantic segmentation model to obtain a mask set corresponding to each image, where the mask set includes multiple masks, each mask corresponds to a plane area, and the mask has a semantic label; A first preprocessing module 63 is used to preprocess the two mask sets of each image pair to obtain a matching plane region mask pair set, where the matching plane region mask pair set includes a plurality of mask pairs, and the two masks in each mask pair belong to different mask sets and have the same semantic label; The first image pair set acquisition module 64 is used to perform element-by-element multiplication of each mask pair in the matched plane region mask pair set with the image pair to obtain a plane region image pair set; the plane region image pair set includes a plurality of plane region image pairs; The network construction module 65 is used to construct a homography transformation network; the homography transformation network includes a feature extraction module, a correlation calculation module, a regression network, and a parameterized homography module; the homography transformation network is used to obtain a homography transformation flow of each plane region based on each plane region image pair in the plane region image pair set; The network training module 66 is used to train the homography transformation network based on the set of plane region image pairs to obtain a trained homography transformation network.
[0060] The apparatus for constructing a homography transformation network based on semantically guided multi-homography estimation in this embodiment has the same inventive concept as the method for constructing a homography transformation network based on semantically guided multi-homography estimation in the above text. Therefore, the specific implementation method of the apparatus can be seen in the embodiment part of the method for constructing a homography transformation network based on semantically guided multi-homography estimation in the above text, and its technical effect corresponds to the technical effect of the above method, which will not be repeated here.
[0061] Using the same inventive concept as the unsupervised image stitching method based on semantically guided multi-homography estimation, this embodiment also provides a corresponding unsupervised image stitching device based on semantically guided multi-homography estimation. Figure 7 The structure block diagram of the unsupervised image stitching device based on semantically guided multi-homography estimation according to an embodiment of the present application is shown, including: A second preliminary segmentation module 71 is used to obtain an image pair to be spliced, and to perform preliminary segmentation on each image of the image pair to be spliced using a semantic segmentation model to obtain a mask set corresponding to each image; A second preprocessing module 72 is used to preprocess the two mask sets of the image pair to be spliced to obtain a matching plane region mask pair set, where the matching plane region mask pair set includes a plurality of mask pairs; The second image pair set acquisition module 73 is used to perform element-by-element multiplication of each mask pair in the matched plane region mask pair set with the image pair to be spliced to obtain a plane region image pair set; A homography transformation module 74 is used to input the set of plane region image pairs into a trained homography transformation network to obtain a homography transformation stream for each plane region; the trained homography transformation network is obtained according to the homography transformation network construction method based on semantically guided multi-homography estimation of the aforementioned embodiment; The fusion module 75 is used to transform the corresponding plane regions in the natural images in the image pair to be stitched using the homography transformation flow of the plane regions to obtain aligned plane regions; all aligned plane regions are fused to obtain a stitched image.
[0062] The unsupervised image stitching device based on semantically guided multi-homography estimation of this embodiment has the same inventive concept as the unsupervised image stitching method based on semantically guided multi-homography estimation mentioned above. Therefore, the specific implementation method of the device can be seen in the implementation example part of the unsupervised image stitching method based on semantically guided multi-homography estimation in the above text, and its technical effect corresponds to the technical effect of the above method, which will not be repeated here.
[0063] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it is used to implement the above-mentioned homography transformation network construction method based on semantically guided multi-homography estimation, or the unsupervised image stitching method based on semantically guided multi-homography estimation.
[0064] An embodiment of the present application provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, it is used to implement the above-mentioned homography transformation network construction method based on semantically guided multi-homography estimation, or the unsupervised image stitching method based on semantically guided multi-homography estimation.
[0065] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A homography transformation network construction method based on semantically guided multi-homography estimation, characterized in that: include: Acquire a natural image dataset, wherein samples in the natural image dataset are image pairs, wherein the image pairs include a reference image and a target image; and each image in the image pairs is an image containing a multi-plane region; Performing preliminary segmentation on each image in the sample using a semantic segmentation model to obtain a mask set corresponding to each image, wherein the mask set includes a plurality of masks, each mask corresponds to a plane area, and the mask has a semantic label; Preprocessing the two mask sets of each image pair to obtain a matching plane region mask pair set, wherein the matching plane region mask pair set includes a plurality of mask pairs, and the two masks in each mask pair belong to different mask sets and have the same semantic label; Each mask pair in the matched plane region mask pair set is multiplied element-by-element with an image pair to obtain a plane region image pair set; the plane region image pair set includes a plurality of plane region image pairs; Constructing a homography transformation network; the homography transformation network includes a feature extraction module, a correlation calculation module, a regression network, and a parameterized homography module; the homography transformation network is used to obtain a homography transformation flow of each plane area based on each plane area image pair in the plane area image pair set; The homography transformation network is trained based on the set of plane region image pairs to obtain a trained homography transformation network.
2. The method according to claim 1, characterized in that in, Preprocess the two mask sets of each image pair to obtain a set of matching plane area mask pairs, including: For each mask set, select the masks with an area greater than 5000 pixels to form the processed mask set; The masks in the two processed mask sets that have the same semantic label and whose mask size difference is less than the set value constitute a mask pair; All mask pairs constitute the matched planar region mask pair set.
3. The method according to claim 1, characterized in that The feature extraction module is used to extract features from each image in the plane area image pair to obtain a feature map; The correlation calculation module is used to perform correlation calculation on the two feature maps corresponding to the plane area image pair to obtain a correlation map between the feature maps; The regression network is used to obtain weights of eight orthogonal basis sets based on the correlation graph between the feature graphs; The parameterized homography module is used to perform parameterized homography according to the weights of the eight orthogonal basis variables and the eight orthogonal basis variables to obtain a homography transformation flow of the planar region corresponding to the planar region image pair.
4. The method according to claim 1, characterized in that in, The homography transformation network is trained based on the set of plane region image pairs, and the loss function used in the training process is: in, is the loss function, is the feature intensity loss, is the transformation consistency loss, is the weight of feature strength loss, is the weight of the transformation consistency loss, is the label of the plane region, is the number of plane regions, represents the feature extraction operation, represents the homography transformation, is the target image, The target image corresponds to The mask of the plane region, is the reference image, The reference image corresponds to The mask of the plane region, is the first The homography transformation flow of a planar region, represents element-wise multiplication, is 1 norm, is a matrix of all 1s of the same size.
5. An unsupervised image stitching method based on semantically guided multi-homography estimation, characterized in that: include: Obtaining an image pair to be stitched, and performing preliminary segmentation on each image of the image pair to be stitched using a semantic segmentation model to obtain a mask set corresponding to each image; Preprocessing the two mask sets of the image pair to be spliced to obtain a matching plane region mask pair set, wherein the matching plane region mask pair set includes a plurality of mask pairs; Multiplying each mask pair in the matched plane region mask pair set by the image pair to be spliced element by element to obtain a plane region image pair set; Inputting the set of plane region image pairs into the trained homography transformation network to obtain a homography transformation stream for each plane region; the trained homography transformation network is obtained according to the homography transformation network construction method based on semantically guided multi-homography estimation according to any one of claims 1 to 4; The homography transformation flow of the plane area is used to transform the corresponding plane area in the natural image of the image pair to be stitched to obtain aligned plane areas; all aligned plane areas are fused to obtain a stitched image.
6. The method according to claim 5, characterized in that in, Preprocessing is performed on the two mask sets of the image pair to be stitched to obtain a matching plane region mask pair set, including: For each mask set, select the masks with an area greater than 5000 pixels to form the processed mask set; The masks in the two processed mask sets that have the same semantic label and whose mask size difference is less than the set value constitute a preliminary mask pair; For each mask in the preliminary mask pair, determine the neighboring mask of the mask, and merge the neighboring mask into the mask to form a final mask pair; the neighboring mask is a mask that is adjacent to the mask but not matched to form a mask pair and a mask with an area less than 1000 pixels; All final mask pairs constitute the set of matched planar region mask pairs.
7. A homography transformation network construction device based on semantically guided multi-homography estimation, characterized in that: include: A data set acquisition module is used to acquire a natural image data set, wherein the samples in the natural image data set are image pairs, wherein the image pairs include a reference image and a target image; and each image in the image pair is an image containing a multi-plane region; A first preliminary segmentation module, used to perform preliminary segmentation on each image in the sample using a semantic segmentation model to obtain a mask set corresponding to each image, wherein the mask set includes a plurality of masks, each mask corresponds to a plane area, and the mask has a semantic label; A first preprocessing module is used to preprocess the two mask sets of each image pair to obtain a matching plane area mask pair set, wherein the matching plane area mask pair set includes a plurality of mask pairs, and the two masks in each mask pair belong to different mask sets and have the same semantic label; a first image pair set acquisition module, configured to perform element-by-element multiplication of each mask pair in the matched plane region mask pair set with an image pair to obtain a plane region image pair set; the plane region image pair set includes a plurality of plane region image pairs; A network construction module, used for constructing a homography transformation network; the homography transformation network includes a feature extraction module, a correlation calculation module, a regression network, and a parameterized homography module; the homography transformation network is used for obtaining a homography transformation flow of each plane area based on each plane area image pair in the plane area image pair set; The network training module is used to train the homography transformation network based on the set of plane area image pairs to obtain a trained homography transformation network.
8. An unsupervised image stitching device based on semantically guided multi-homography estimation, characterized in that: include: A second preliminary segmentation module is used to obtain a pair of images to be spliced, and perform preliminary segmentation on each image of the pair of images to be spliced using a semantic segmentation model to obtain a mask set corresponding to each image; A second preprocessing module, configured to preprocess the two mask sets of the image pair to be spliced to obtain a matching plane region mask pair set, wherein the matching plane region mask pair set includes a plurality of mask pairs; a second image pair set acquisition module, configured to perform element-by-element multiplication of each mask pair in the matched plane region mask pair set with the image pair to be spliced, to obtain a plane region image pair set; A homography transformation module, used for inputting the set of plane region image pairs into the trained homography transformation network to obtain a homography transformation stream for each plane region; the trained homography transformation network is obtained according to the homography transformation network construction method based on semantically guided multi-homography estimation according to any one of claims 1 to 4; The fusion module is used to transform the corresponding plane areas in the natural images in the image pair to be stitched by using the homography transformation flow of the plane area to obtain aligned plane areas; all aligned plane areas are fused to obtain a stitched image.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the homography transformation network construction method based on semantically guided multi-homography estimation as described in any one of claims 1 to 4, or the unsupervised image stitching method based on semantically guided multi-homography estimation as described in any one of claims 5 to 6.
10. A computer program product, characterized in that The invention comprises a computer program / instruction, which, when executed by a processor, implements the homography transformation network construction method based on semantically guided multi-homography estimation as described in any one of claims 1 to 4, or the unsupervised image stitching method based on semantically guided multi-homography estimation as described in any one of claims 5 to 6.
Citation Information
Patent Citations
Shielding area determination method and device and segmentation model training method
CN115049820A
Image registration method, computer program product, storage medium and electronic device
CN115810033A
Image alignment method and device, computer readable medium and electronic equipment
CN118691481A
Systems and methods for processing multi-modal images
WO2024095557A1
