A method and device for predicting homography matrix of image

Abstract

The present invention discloses a method and device for predicting homography matrix of an image pair, belonging to the field of computer vision technology. The prediction method comprises: obtaining an image pair to be predicted, the image pair comprising a first image and a second image; predicting a first homography matrix of the first image transformed to the second image by a first method; transforming the first image based on the first homography matrix to obtain a third image; predicting a second homography matrix of the third image transformed to the second image by a second method; and calculating the final homography matrix based on the first homography matrix and the second homography matrix. The present invention improves the accuracy of homography matrix prediction.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a method and a device for predicting an image homography matrix. Background Art

[0002] The homography matrix can be used to describe the mapping relationship between two planes. It is one of the basic tasks for realizing UAV attitude control, autonomous driving, and robotic arm motion path planning based on the visual system. Theoretically, at least 4 pairs of key points with mapping relationships are required to calculate the 8 degrees of freedom represented by the homography matrix. The convenience of obtaining key points in the visual system has prompted the method of calculating the homography matrix based on image features to be widely studied. However, the complexity of the application scenarios often brings great challenges to these algorithms. Therefore, the visual homography matrix calculation model has become one of the research hotspots of many scholars.

[0003] Traditional homography matrix solution is based on various detectors to obtain explicit key feature points, including Harris, SIFT, ORB, SURF, LBP and other feature points. The advantage of explicit feature points is that the features are highly interpretable, and the process of pairwise matching of feature points can be intuitively displayed. The computing power requirement is relatively low, but it is highly sensitive to environmental changes. In subsequent improvements, these discrete feature points are combined with features such as straight lines, curves, T-shaped intersection segments, and colors to improve the calculation accuracy, but the applicability to smooth images is still insufficient. The improved homography matrix calculation model combined with the powerful feature extraction capability of deep learning technology may combine explicit and implicit features from key points to shapes, objects and other levels in the image pair. However, the data annotation accuracy required for model training is extremely high, and slight deviations in manual annotation will lead to increased errors. Therefore, existing research mainly constructs data sets based on the homography transformation of a single image itself, thereby converting model training into unsupervised learning. Related models mainly directly predict the position transformation deviations of 4 pairs of key points and indirectly calculate the homography matrix, or directly predict the 8 degrees of freedom values ​​in the homography matrix. Currently, the performance improvement of these models mainly depends on the improvement of network structure, such as the introduction of ST network, so that the image pixel features and their position information can be back-propagated in the neural network at the same time, or the introduction of multi-level hierarchical method, so as to subdivide the single homography matrix calculation with a large span between image pairs into multiple small span calculation processes.

[0004] The homography matrix is ​​predicted using the position deviation of four pairs of key points. This method directly uses a convolutional neural network to extract features from two images, and finally uses a fully connected network to predict four sets of coordinates, that is, eight coordinate position deviations. Since the training data comes from the same image and its homography transformed image, it belongs to unsupervised learning, and the construction of the training data set is simple, but it also makes the prediction of the real image pair unstable.

[0005] The homography matrix is ​​predicted based on its 8 degrees of freedom. Directly using a convolutional neural network to predict the 8 degrees of freedom of the homography matrix often results in large errors. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies of the prior art and to provide a method and device for predicting a homography matrix of an image.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] A first aspect of the present invention discloses a method for predicting a homography matrix of an image, comprising:

[0009] Acquire an image pair to be predicted, the image pair comprising a first image and a second image;

[0010] Predicting a first homography matrix for transforming the first image to the second image using a first method;

[0011] transforming the first image based on the first homography matrix to obtain a third image;

[0012] Predicting a second homography matrix for transforming the third image to the second image using a second method;

[0013] A final homography matrix is ​​calculated based on the first homography matrix and the second homography matrix.

[0014] Furthermore, before predicting the first homography matrix of the first image transformed into the second image by using the first method, the prediction method further includes:

[0015] removing moving objects in the first image and the second image according to a preset rule;

[0016] The first image and the second image are binarized respectively.

[0017] Further, predicting a first homography matrix of the first image transformed into the second image using the first method includes:

[0018] Performing feature extraction on the first image to obtain a first feature map; performing feature extraction on the second image to obtain a second feature map;

[0019] Perform several convolution operations on the first feature map and the second feature map respectively;

[0020] The first feature map and the second feature map having the same number of convolution operations are respectively spliced ​​in parallel to obtain a first spliced ​​image;

[0021] Perform convolution and full connection operations on each first spliced ​​image to obtain a first prediction result, wherein the first prediction result includes position deviations of four pairs of key points;

[0022] Taking a weighted average of all first prediction results to obtain a second prediction result;

[0023] Generate a first homography matrix for transforming the first image to the second image according to the second prediction result;

[0024] Before parallel splicing of the first feature map and the second feature map, a row of all-0 rows is added between every two adjacent rows in the first feature map and the second feature map, and then a first blank image of all-0 feature values ​​of the same size is added between the first feature map and the second feature map.

[0025] Furthermore, convolution and full connection operations are performed on each first spliced ​​image to obtain a first prediction result, which includes position deviations of four pairs of key points, including:

[0026] Perform convolution and full connection operations on the current first spliced ​​image to obtain a third prediction result, wherein the third prediction result includes position deviations of four pairs of key points;

[0027] If the current first stitched image has a previous level first stitched image, generating a first prediction result of the current first stitched image according to the first prediction result of the previous level first stitched image and the third prediction result of the current first stitched image;

[0028] If the current first stitched image does not have a previous level first stitched image, determining the third prediction result of the current first stitched image as the first prediction result of the current first stitched image;

[0029] The number of convolution operations for generating the feature map of the previous level first stitched image is one more than the number of convolution operations for generating the feature map of the current first stitched image.

[0030] Further, predicting a second homography matrix of the third image transformed to the second image using the second method includes:

[0031] Performing feature extraction on the second image and the third image respectively to obtain a second feature map and a third feature map;

[0032] Adding a feature map composed of horizontal axis coordinates and a feature map composed of vertical axis coordinates between the second feature map and the third feature map, and then stitching them together to obtain a second stitched image;

[0033] Perform convolution and full connection operations on the second spliced ​​image to obtain 8 degrees of freedom values;

[0034] A second homography matrix for transforming the third image to the second image is generated according to the eight degrees of freedom values.

[0035] Further, feature extraction is performed on the second image and the third image respectively to obtain a second feature map and a third feature map, including:

[0036] Performing feature extraction on the second image to obtain a second feature map;

[0037] Performing feature extraction on the third image to obtain a third feature map;

[0038] The second feature map and the third feature map are cropped respectively.

[0039] A second aspect of the present invention discloses a device for predicting an image homography matrix, comprising:

[0040] An image acquisition module, used to acquire an image pair to be predicted, wherein the image pair includes a first image and a second image;

[0041] A first neural network is used to predict a first homography matrix for transforming the first image to the second image using a first method;

[0042] An image transformation module, used for transforming the first image based on the first homography matrix to obtain a third image;

[0043] A second neural network is used to predict a second homography matrix of the third image transformed into the second image using a second method;

[0044] The matrix calculation module is used to calculate a final homography matrix based on the first homography matrix and the second homography matrix.

[0045] Furthermore, the prediction device further comprises:

[0046] The preprocessing module is used to remove moving objects in the first image and the second image according to preset rules, and to binarize the first image and the second image.

[0047] Furthermore, the first neural network comprises:

[0048] A first image processing channel, used for performing feature extraction on the first image to obtain a first feature map, and performing a plurality of convolution operations on the first feature map;

[0049] A second image processing channel is used to extract features from the second image to obtain a second feature map, and to perform a plurality of convolution operations on the second feature map;

[0050] A plurality of splicing modules are used to parallel splice the first feature map and the second feature map with the same number of convolution operations to obtain a first spliced ​​image; wherein, before parallel splicing the first feature map and the second feature map, a row of all zeros is added between every two adjacent rows in the first feature map and the second feature map, and then a first blank image with all zero feature values ​​of the same size is added between the first feature map and the second feature map;

[0051] A plurality of first prediction modules, used for performing convolution and full connection operations on the first spliced ​​image to obtain a first prediction result, wherein the first prediction result includes position deviations of four pairs of key points;

[0052] A second prediction module is used to perform weighted averaging on all first prediction results to obtain a second prediction result, and to generate a first homography matrix for transforming the first image to the second image according to the second prediction result;

[0053] The first image processing channel and the second image processing channel have the same structure and share parameters.

[0054] Furthermore, the second neural network comprises:

[0055] A feature extraction module, used to extract features from the second image and the third image respectively to obtain a second feature map and a third feature map;

[0056] A feature map stitching module, used to obtain a feature map composed of horizontal axis coordinates and a feature map composed of vertical axis coordinates, and to add a feature map composed of horizontal axis coordinates and a feature map composed of vertical axis coordinates between the second feature map and the third feature map, and then stitch them together to obtain a second stitched image;

[0057] The third prediction module is used to perform convolution and full connection operations on the spliced ​​image to obtain 8 degrees of freedom values, and generate a second homography matrix for transforming the third image to the second image according to the 8 degrees of freedom values.

[0058] The beneficial effects of the present invention are:

[0059] (1) The method of the present invention incorporates position information when improving the convolution calculation process in the first stage, thereby using 4-point position deviations to achieve the prediction of the macroscopic numerical value of the homography matrix between image pairs; in the second stage, position information is added to the input image features to improve the accuracy of the direct homography matrix 8-degree-of-freedom numerical prediction, thereby assisting in the fine-tuning of the overall homography matrix;

[0060] (2) The present invention improves the feature fusion method in the traditional two-way branch prediction process of four pairs of key point deviations, replaces channel serial splicing with channel parallel splicing, and improves the accuracy of the prediction results;

[0061] (3) After the present invention makes a macro prediction of the homography matrix based on four pairs of key points, the homography matrix is ​​fine-tuned in combination with the predicted eight degrees of freedom values, thereby improving the accuracy of the homography matrix prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A flow chart of a method for predicting a homography matrix of an image in the present invention;

[0063] Figure 2 is another flow chart of the method for predicting the homography matrix of an image in the present invention;

[0064] Figure 3 A schematic diagram of the structure of the first neural network in the present invention;

[0065] Figure 4 A schematic diagram of the image mirror in the present invention;

[0066] Figure 5 Schematic diagram of the structure of the second neural network in the present invention. DETAILED DESCRIPTION

[0067] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0068] See also Figures 1 to 5 The present invention provides a method and device for predicting an image homography matrix:

[0069] The first aspect of this embodiment discloses a method for predicting an image homography matrix, such as Figure 1 and Figure 2 As shown, the prediction method includes steps S100 to S500. Figure 2 A and B in the figure respectively represent the first image and the second image in an original image pair in an embodiment, A' and B' represent the first image and the second image after preprocessing, A'' and B'' respectively represent the first feature map and the second feature map after adding all-0 eigenvalues, H1 represents the first homography matrix, H2 represents the second homography matrix, H represents the final homography matrix, and T represents the image obtained by transforming A' using H1.

[0070] Step S100: Obtain an image pair to be predicted, wherein the image pair includes a first image and a second image.

[0071] Step S200: Predict a first homography matrix for transforming a first image into a second image using a first method.

[0072] In some embodiments, a first method is used to predict a first homography matrix for transforming a first image to a second image, including: based on the first image and the second image, using a first neural network to predict four pairs of key point position deviations, generating a homography matrix for transforming the first image to the second image based on the four pairs of key point position deviations, and recording the homography matrix as a first homography matrix.

[0073] In some embodiments, predicting a first homography matrix for transforming a first image into a second image using a first method includes steps S210 to S260.

[0074] Step S210: extract features from the first image to obtain a first feature map; extract features from the second image to obtain a second feature map.

[0075] Specifically, feature extraction is performed on the first image and the second image respectively to obtain a first feature map and a second feature map of the same size.

[0076] Step S220: Perform several convolution operations on the first feature map and the second feature map respectively.

[0077] Specifically, the same convolution operation is performed on the first feature map and the second feature map, respectively. For example, parameters such as the convolution kernel and the convolution step size when the first convolution operation is performed on the first feature map are the same as the convolution kernel and the convolution step size when the first convolution operation is performed on the second feature map, and parameters such as the convolution kernel and the convolution step size when the second convolution operation is performed on the first feature map are the same as the convolution kernel and the convolution step size when the second convolution operation is performed on the second feature map, and so on.

[0078] Step S230. Parallel splicing is performed on the first feature map and the second feature map with the same number of convolution operations to obtain a first spliced ​​image; before parallel splicing is performed on the first feature map and the second feature map, a row of all zeros is added between every two adjacent rows in the first feature map and the second feature map, and then a first blank image of all zero eigenvalues ​​of the same size is added between the first feature map and the second feature map.

[0079] Specifically, the first feature map and the second feature map that have not undergone convolution operation (that is, the number of convolution operations between the first feature map and the second feature map is 0) are spliced ​​in parallel in two rows to obtain a first spliced ​​image; the first feature map and the second feature map that have undergone one convolution operation are spliced ​​in parallel to obtain a first spliced ​​image; and so on, to obtain several first spliced ​​images.

[0080] Before parallel splicing of the first feature map and the second feature map, a row of all-0 rows is added between every two adjacent rows in the first feature map, and a row of all-0 rows is added between every two adjacent rows in the second feature map. Then, a first blank image of all-0 feature values ​​of the same size is added to the first feature map and the second feature map (that is, the size of the first blank image is the same as that of the first feature map and the second feature map), and then a convolution operation is performed to obtain the first spliced ​​image.

[0081] Step S240: Perform convolution and full connection operations on each first spliced ​​image to obtain a first prediction result, where the first prediction result includes position deviations of 4 pairs of key points.

[0082] Specifically, convolution and full connection operations are performed on each first spliced ​​image to obtain a corresponding first prediction result, wherein the first prediction result includes position deviations of four pairs of key points. If there are multiple first spliced ​​images, multiple first prediction results are obtained.

[0083] In some embodiments, convolution and full connection operations are performed on each first stitched image respectively to obtain a first prediction result, and the first prediction result includes the position deviations of 4 pairs of key points, including: convolution and full connection operations are performed on the current first stitched image to obtain a third prediction result, and the third prediction result includes the position deviations of 4 pairs of key points; if there is a previous level first stitched image for the current first stitched image, the first prediction result of the current first stitched image is generated according to the first prediction result of the previous level first stitched image and the third prediction result of the current first stitched image; if there is no previous level first stitched image for the current first stitched image, the third prediction result of the current first stitched image is determined as the first prediction result of the current first stitched image; wherein the number of convolution operations for generating the feature map of the previous level first stitched image is one more than the number of convolution operations for generating the feature map of the current first stitched image.

[0084] In this embodiment, the prediction result of the first spliced ​​image is transmitted, thereby improving the accuracy of the prediction result.

[0085] Generating a first prediction result of a current first stitched image according to a first prediction result of a previous level first stitched image and a third prediction result of a current first stitched image includes: adding the first prediction result of the previous level first stitched image and the third prediction result of the current first stitched image to obtain the first prediction result of the current first stitched image; or averaging the first prediction result of the previous level first stitched image and the third prediction result of the current first stitched image to obtain the first prediction result of the current first stitched image.

[0086] Step S250: Perform weighted averaging on all first prediction results to obtain a second prediction result.

[0087] The second prediction result includes the position deviations of 4 pairs of key points.

[0088] Before weighted averaging all the first prediction results, weights of the first prediction results corresponding to the first spliced ​​images of different scales are set.

[0089] In this embodiment, the first spliced ​​images of multiple scales are used to calculate the second prediction result, thereby improving the accuracy of the calculation result.

[0090] Step S260: Generate a first homography matrix for transforming the first image into the second image according to the second prediction result.

[0091] In some embodiments, the first image and the second image are input into a first neural network, and the first neural network outputs position deviations of four pairs of key points based on the input first image and the second image; then a first homography matrix is ​​generated based on the position deviations of the four pairs of key points.

[0092] The structure of the first neural network is as follows Figure 3 As shown, A' and B' represent the first image and the second image after preprocessing, BP represents the bridging process, A'' and B'' represent the first feature map and the second feature map after adding all-0 eigenvalues, respectively, loss3, loss2, loss1 represent the loss functions corresponding to the first bridging process, the second bridging process and the third bridging process, respectively, and M represents the first blank image. When the first neural network fuses two branches (the structures of the two branches in the first neural network are the same, and the parameters of the two branches are shared), the traditional serial splicing feature map pair (the feature map pair includes the first feature map and the second feature map) is improved to a parallel splicing feature map pair. Specifically, before the parallel splicing feature map pair, horizontal single-pixel all-0 eigenvalues ​​are added at intervals in the two original feature maps, and then the first blank image with all-0 eigenvalues ​​of the same size is added between the two feature maps. At this time, the convolution operation is improved as follows Figure 3 As shown in the figure, due to the regular addition of a large number of 0 pixels, a single convolution operation will equally extract the features of the pixels at the same position in the feature map, thereby improving the accuracy of the homography matrix prediction that is highly correlated with the position information. Since this process is like using a bridge connecting two branches, it is called a bridging process.

[0093] The backbone network of the first neural network is improved on the CSP-Darknet53 structure, with an input feature size of 640×640×1 and an output feature size of 64×64×256. Figure 3The three bridging processes in the figure are the first bridging process, the second bridging process and the third bridging process from left to right. The size of the input feature map of the first bridging process is 64×64×512, and 512 convolution kernels are used to perform the improved convolution operation; the size of the input feature map of the second bridging process is 32×32×512, and 256 convolution kernels are used to perform the improved convolution operation; the size of the input feature map of the third bridging process is 16×16×512, and 128 convolution kernels are used to perform the improved convolution operation. The first neural network is similar to the homography Net, and finally passes through a fully connected operation. It finally outputs 8×1 feature values ​​for regression operation through a fully connected network, directly predicting the position deviation of four pairs of key points. The regression network uses the Euclidean distance to calculate the loss function, as shown in formula (1):

[0094]

[0095] Among them, p(x) represents the predicted coordinate deviation and q(x) represents the actual coordinate deviation.

[0096] In addition, in order to utilize the multi-scale characteristics, the predicted output results of each bridge process will be passed to other bridge processes step by step, that is, the loss function is formed as shown in formula (2), where Obtained through experiments, it is used to balance the weights of different bridging processes.

[0097]

[0098] Among them, loss k Represents the loss function corresponding to the k-th BP (bridging process).

[0099] When the first neural network is trained, the two images input have the same content but different orientations, thus avoiding errors caused by manual labeling due to different input image contents.

[0100] Step S300: transform the first image based on the first homography matrix to obtain a third image.

[0101] Step S400: Predict a second homography matrix for transforming the third image into the second image using a second method.

[0102] In some embodiments, a second method is used to predict a second homography matrix for transforming the third image to the second image, including: based on the third image and the second image, using a second neural network to predict 8 degree of freedom values, generating a homography matrix for transforming the third image to the second image based on the 8 degree of freedom values, and recording the homography matrix as a second homography matrix.

[0103] In some embodiments, predicting a second homography matrix for transforming the third image to the second image using a second method includes steps S410 to S440.

[0104] Step S410: Perform feature extraction on the second image to obtain a second feature map; perform feature extraction on the third image to obtain a third feature map.

[0105] In some embodiments, the second feature map and the third feature map are cropped respectively to avoid a situation where a large amount of black images appear in one of the images when the orientation difference between the second image and the third image is relatively large. For example, the second feature map and the third feature map are cropped respectively with the center of the original image as the center and half of the length and width of the original image as the length and width.

[0106] Step S420. Add a feature map consisting of horizontal axis coordinates and a feature map consisting of vertical axis coordinates between the second feature map and the third feature map, and then stitch them together to obtain a second stitched image.

[0107] Direct prediction of the homography matrix requires the convolutional neural network to have complex coordinate calculation capabilities. To this end, the present invention adds coordinate information corresponding to each pixel in the image on the basis of the feature channels contained in the traditional image, that is, adds two feature channels to express the x and y coordinates of the pixel points respectively. At the same time, in order to prevent the overfitting problem caused by the added coordinate information, such as Figure 4 As shown, each set of image pairs is required to be mirrored left and right and up and down on the basis of the same coordinate feature layer. When mirroring the image left and right and up and down, first construct the x-coordinate information feature channel and the y-coordinate information feature channel, that is, the x-coordinate information feature channel corresponds to each row of the matrix that stores values ​​from 0 to w, where w is the width of the second image, and the y-coordinate information feature channel corresponds to each column of the matrix that stores values ​​from 0 to h, where h is the height of the second image; left and right mirroring refers to mirroring the numerical elements of each row in the matrix corresponding to the feature channel, such as 0 to w, to w to 0; up and down mirroring refers to mirroring the numerical elements of each column in the matrix corresponding to the feature channel, such as 0 to h, to h to 0.

[0108] Step S430: Perform convolution and full connection operations on the second stitched image to obtain 8 degrees of freedom values.

[0109] Step S440: Generate a second homography matrix for transforming the third image into the second image according to the eight degrees of freedom values.

[0110] In some embodiments, the third image and the second image are input into the second neural network to obtain 8 degrees of freedom values, and then a second homography matrix is ​​generated according to the 8 degrees of freedom values. The structure of the second neural network is as follows: Figure 5As shown, the backbone network of the second neural network is still improved on the CSP-Darknet53 structure, with an input feature size of 640×640×1 and an output feature size of 8×8×512. A full connection operation is then performed to directly predict the 8 degrees of freedom values ​​of the homography matrix.

[0111] like Figure 5 As shown, the training process of the second neural network is as follows: obtain two input images, perform preprocessing operations such as personnel removal and binarization on the first input image and the second input image, then use the first neural network to calculate the first homography matrix of the first input image to the second input image, and use the first homography matrix to transform the first input image into the third input image. Perform feature extraction on the second input image and the third input image to obtain the second feature map and the third feature map, then crop the second feature map and the third feature map, then add the feature map composed of the horizontal axis coordinate and the feature map composed of the vertical axis coordinate between the cropped second feature map and the third feature map, and then splice the second feature map, the third feature map, the feature map composed of the horizontal axis coordinate and the feature map composed of the vertical axis coordinate into a second spliced ​​image. Perform convolution and full connection operations on the second spliced ​​image to obtain 8 degrees of freedom values, and generate a second homography matrix based on the 8 degrees of freedom values. Based on the second homography matrix, perform operations such as coordinate position transformation and pixel sampling on the third feature map to obtain a new third feature map, and solve the loss function loss4 based on the second feature map and the new third feature map, thereby realizing unsupervised learning. Figure 5 The Grid Generator and Sampler in the figure belong to the SPT module, which are used for coordinate position transformation and pixel sampling respectively. Figure 5 Where A' and B' represent the first image and the second image after preprocessing, H1 represents the first homography matrix, H2 represents the second homography matrix, T represents the image obtained by transforming A' using H1, and T * represents the image cropped from T, B * A represents the sub-image cropped from B'. * represents the new third feature map.

[0112] The loss function of the second neural network refers to the twin network, as shown in formula (3), where q v It takes a value of 0 or 1, indicating whether the predicted category of the pixels at the same position in the two images is the same. v represents the probability of the classification prediction and m is the total number of pixels. Since the image contains only black and white pixels, this model essentially performs binary classification.

[0113]

[0114] Step S500: Calculate a final homography matrix based on the first homography matrix and the second homography matrix.

[0115] The final homography matrix is ​​calculated as:

[0116] H=H1*H2

[0117] Where H represents the final homography matrix, H1 represents the first homography matrix, and H2 represents the second homography matrix.

[0118] In some embodiments, before using the first method to predict the first homography matrix of the first image transformed into the second image, the prediction method also includes: removing moving targets in the first image and the second image according to preset rules; and binarizing the first image and the second image respectively.

[0119] The homography transformation problem in competitive sports (such as live videos of basketball, handball, ice hockey and other games) is different from other scenes. Since there are a large number of randomly moving players in competitive sports videos, directly solving the homography matrix for two frames of images will lead to large errors. For this reason, the present invention preprocesses all image pair data. The preprocessing process mainly includes player detection and segmentation, and black and white binarization of the image after removing the players.

[0120] A second aspect of this embodiment discloses a device for predicting an image-to-homography matrix, the device comprising an image acquisition module, a first neural network, an image transformation module, a second neural network and a matrix calculation module.

[0121] The image acquisition module is used to acquire an image pair to be predicted, wherein the image pair includes a first image and a second image.

[0122] The first neural network is used to predict a first homography matrix for transforming the first image to the second image using a first method.

[0123] In some embodiments, the first neural network includes a first image processing channel, a second image processing channel, a plurality of stitching modules, a plurality of first prediction modules and a second prediction module.

[0124] The first image processing channel is used to extract features from the first image to obtain a first feature map, and to perform several convolution operations on the first feature map.

[0125] The second image processing channel is used to extract features from the second image to obtain a second feature map, and perform several convolution operations on the second feature map.

[0126] A plurality of splicing modules are used to parallel splice the first feature map and the second feature map with the same number of convolution operations to obtain a first spliced ​​image; wherein, before parallel splicing the first feature map and the second feature map, a row of all-0 rows is added between every two adjacent rows in the first feature map and the second feature map, and then a first blank image with all-0 feature values ​​of the same size is added between the first feature map and the second feature map.

[0127] A plurality of first prediction modules are used to perform convolution and full connection operations on the first spliced ​​image to obtain a first prediction result, wherein the first prediction result includes position deviations of four pairs of key points.

[0128] The second prediction module is used to perform weighted averaging on all first prediction results to obtain a second prediction result, and to generate a first homography matrix for transforming the first image into the second image according to the second prediction result.

[0129] The image transformation module is used to transform the first image based on the first homography matrix to obtain a third image.

[0130] The second neural network is used to predict a second homography matrix for transforming the third image to the second image using a second method.

[0131] In some embodiments, the second neural network includes a feature extraction module, a feature map concatenation module and a third prediction module.

[0132] The feature extraction module is used to extract features from the second image and the third image respectively to obtain a second feature map and a third feature map.

[0133] The feature map stitching module is used to obtain the feature map composed of the horizontal axis coordinates and the feature map composed of the vertical axis coordinates, and to add the feature map composed of the horizontal axis coordinates and the feature map composed of the vertical axis coordinates between the second feature map and the third feature map, and then stitch them together to obtain a second stitched image.

[0134] The third prediction module is used to perform convolution and full connection operations on the spliced ​​image to obtain 8 degrees of freedom values, and generate a second homography matrix for transforming the third image to the second image according to the 8 degrees of freedom values.

[0135] The matrix calculation module is used to calculate a final homography matrix based on the first homography matrix and the second homography matrix.

[0136] In some embodiments, the two images input by the first neural network during each training are images with the same content; when training, the second neural network needs to use the third image obtained by the first neural network, and at this time, the two images input by the first neural network have different contents; and finally, the homography matrix between the two images with different contents is predicted jointly by the first neural network and the second neural network.

[0137] In some embodiments, the prediction device further includes a preprocessing module, which is used to remove moving objects in the first image and the second image according to preset rules, and to binarize the first image and the second image.

[0138] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system or device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0139] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A method for predicting homography matrix of an image pair, It is characterized in that include: Acquire an image pair to be predicted, the image pair comprising a first image and a second image; Predicting a first homography matrix for transforming the first image to the second image using a first method; Transforming the first image based on the first homography matrix to obtain a third image; Predicting a second homography matrix for transforming the third image to the second image using a second method; Calculating a final homography matrix based on the first homography matrix and the second homography matrix; Predicting a first homography matrix for transforming a first image to a second image using a first method includes: Performing feature extraction on the first image to obtain a first feature map; performing feature extraction on the second image to obtain a second feature map; Perform several convolution operations on the first feature map and the second feature map respectively; The first feature map and the second feature map having the same number of convolution operations are respectively concatenated in parallel to obtain a first concatenated image; Perform convolution and full connection operations on each first spliced ​​image to obtain a first prediction result, wherein the first prediction result includes position deviations of four pairs of key points; Taking a weighted average of all first prediction results to obtain a second prediction result; Generate a first homography matrix for transforming the first image to the second image according to the second prediction result; Before parallel splicing of the first feature map and the second feature map, a row of all zeros is added between every two adjacent rows in the first feature map and the second feature map, and then a first blank image of all zero feature values ​​of the same size is added between the first feature map and the second feature map; Convolution and full connection operations are performed on each first spliced ​​image to obtain a first prediction result, which includes position deviations of 4 pairs of key points, including: Perform convolution and full connection operations on the current first spliced ​​image to obtain a third prediction result, wherein the third prediction result includes position deviations of four pairs of key points; If the current first stitched image has a previous level first stitched image, generating a first prediction result of the current first stitched image according to the first prediction result of the previous level first stitched image and the third prediction result of the current first stitched image; If the current first stitched image does not have a previous level first stitched image, determining the third prediction result of the current first stitched image as the first prediction result of the current first stitched image; The number of convolution operations for generating the feature map of the previous level first stitched image is one more than the number of convolution operations for generating the feature map of the current first stitched image; Predicting a second homography matrix for transforming the third image to the second image using the second method includes: Performing feature extraction on the second image and the third image respectively to obtain a second feature map and a third feature map; Adding a feature map composed of horizontal axis coordinates and a feature map composed of vertical axis coordinates between the second feature map and the third feature map, and then stitching them together to obtain a second stitched image; Perform convolution and full connection operations on the second spliced ​​image to obtain 8 degrees of freedom values; A second homography matrix for transforming the third image to the second image is generated according to the eight degrees of freedom values.

2. The method for predicting an image pair homography matrix according to claim 1, It is characterized in that Before predicting the first homography matrix of the first image transformed into the second image by using the first method, the prediction method further includes: removing moving objects in the first image and the second image according to a preset rule; The first image and the second image are binarized respectively.

3. The method for predicting an image pair homography matrix according to claim 1, It is characterized in that Feature extraction is performed on the second image and the third image respectively to obtain a second feature map and a third feature map, including: Performing feature extraction on the second image to obtain a second feature map; Performing feature extraction on the third image to obtain a third feature map; The second feature map and the third feature map are cropped respectively.

4. A device for predicting a homography matrix of an image, It is characterized in that include: An image acquisition module, used to acquire an image pair to be predicted, wherein the image pair includes a first image and a second image; A first neural network is used to predict a first homography matrix for transforming the first image to the second image using a first method; An image transformation module, used for transforming the first image based on the first homography matrix to obtain a third image; A second neural network is used to predict a second homography matrix of the third image transformed into the second image using a second method; A matrix calculation module, used for calculating a final homography matrix based on the first homography matrix and the second homography matrix; The first neural network comprises: A first image processing channel, used for performing feature extraction on the first image to obtain a first feature map, and performing a plurality of convolution operations on the first feature map; A second image processing channel is used to extract features from the second image to obtain a second feature map, and to perform a plurality of convolution operations on the second feature map; A plurality of splicing modules are used to parallel splice the first feature map and the second feature map with the same number of convolution operations to obtain a first spliced ​​image; wherein, before parallel splicing the first feature map and the second feature map, a row of all zeros is added between every two adjacent rows in the first feature map and the second feature map, and then a first blank image with all zero feature values ​​of the same size is added between the first feature map and the second feature map; A plurality of first prediction modules, used for performing convolution and full connection operations on the first spliced ​​image to obtain a first prediction result, wherein the first prediction result includes position deviations of four pairs of key points; A second prediction module is used to perform weighted averaging on all first prediction results to obtain a second prediction result, and to generate a first homography matrix for transforming the first image to the second image according to the second prediction result; The first image processing channel and the second image processing channel have the same structure and share parameters; Convolution and full connection operations are performed on each first spliced ​​image to obtain a first prediction result, which includes position deviations of 4 pairs of key points, including: Perform convolution and full connection operations on the current first spliced ​​image to obtain a third prediction result, wherein the third prediction result includes position deviations of four pairs of key points; If the current first stitched image has a previous level first stitched image, generating a first prediction result of the current first stitched image according to the first prediction result of the previous level first stitched image and the third prediction result of the current first stitched image; If the current first stitched image does not have a previous level first stitched image, determining the third prediction result of the current first stitched image as the first prediction result of the current first stitched image; The number of convolution operations for generating the feature map of the previous level first stitched image is one more than the number of convolution operations for generating the feature map of the current first stitched image; The second neural network comprises: A feature extraction module, used to extract features from the second image and the third image respectively to obtain a second feature map and a third feature map; A feature map stitching module, used to obtain a feature map composed of horizontal axis coordinates and a feature map composed of vertical axis coordinates, and to add a feature map composed of horizontal axis coordinates and a feature map composed of vertical axis coordinates between the second feature map and the third feature map, and then stitch them together to obtain a second stitched image; The third prediction module is used to perform convolution and full connection operations on the spliced ​​image to obtain 8 degrees of freedom values, and generate a second homography matrix for transforming the third image to the second image according to the 8 degrees of freedom values.

5. The device for predicting an image pair homography matrix according to claim 4, It is characterized in that The prediction device also includes: The preprocessing module is used to remove moving objects in the first image and the second image according to preset rules, and to binarize the first image and the second image.

Images