Image splicing method and system based on key point and homography estimation

Abstract

The invention provides an image splicing method and system based on key points and homography estimation, and the method comprises the steps: obtaining a first visual angle image and a second visual angle image of a same object; determining an incremental search window by adopting an incremental search strategy; determining a correctly matched key point pair set in an incremental search window; carrying out outlier elimination processing on the correctly matched key point pair set, and determining a homography matrix of a first stage; performing homography estimation processing on the image area of the first view angle image and the image area of the second view angle image corresponding to the incremental search window by adopting a preset deep neural network, and determining a homography matrix of a second stage; determining a spliced image according to the homography matrix of the first stage and the homography matrix of the second stage; and performing color correction optimization on the spliced image to determine a target spliced image. According to the image splicing method and device, the homography relation of the overlapping areas of the two adjacent view angle images is determined from coarse to fine, and the precision and robustness of image splicing are improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and in particular to an image stitching method and system based on key point and homography estimation. Background Art

[0002] The traditional image stitching task is to use the features of the overlapping parts of multi-view images to match adjacent areas, and then transform the adjacent view images according to the transformation relationship calculated after matching, so as to make the camera extrinsic parameters of different view images consistent, and finally stitch them together to form a full-view image.

[0003] The key to traditional image stitching methods is to find a feature that is invariant to both brightness and contrast, so as to ensure that the matching algorithm can still accurately match the same feature when the lighting and shadow conditions of images from different perspectives are different. However, for microscopic images, since the structure of microorganisms is relatively simple, for example, Paramecium cells only include structures such as the nucleus and cell fluid, and the texture of these structures is single, it is difficult to extract robust features that are invariant to brightness and contrast. Therefore, the above traditional image stitching methods are difficult to achieve high-precision stitching effects. Summary of the invention

[0004] In view of the defects in the prior art, the purpose of the present invention is to provide an image stitching method and system based on key point and homography estimation.

[0005] To achieve the above object, according to one aspect of the present disclosure, a method for image stitching based on key points and homography estimation is provided, comprising:

[0006] Acquire a first-perspective image and a second-perspective image corresponding to the same object, wherein the first perspective and the second perspective are adjacent perspectives;

[0007] Using an incremental search strategy to translate the first perspective image and the second perspective image to perform first key point matching, and determining an incremental search window and a window translation amount corresponding to the incremental search window;

[0008] Performing second key point matching on an image area of ​​the first viewing angle image corresponding to the incremental search window and an image area of ​​the second viewing angle image corresponding to the incremental search window to determine a set of correctly matched key point pairs, wherein the set of correctly matched key point pairs includes image coordinates of each correctly matched key point;

[0009] Performing outlier elimination processing on the correctly matched key point pair set to determine the homography matrix of the first stage;

[0010] Using a preset deep neural network to perform homography estimation processing on an image area of ​​the first viewing angle image corresponding to the incremental search window and an image area of ​​the second viewing angle image corresponding to the incremental search window, to determine a homography matrix of the second stage;

[0011] According to the homography matrix of the first stage and the homography matrix of the second stage, transform and stitch the first perspective image and the second perspective image to determine a stitched image;

[0012] The stitched images are subjected to color correction optimization processing to determine a target stitched image.

[0013] Optionally, the step of using an incremental search strategy to translate the first perspective image and the second perspective image to perform first key point matching, and determining an incremental search window and a window translation amount corresponding to the incremental search window, comprises:

[0014] translating the first perspective image and the second perspective image according to a preset window step size and a preset window threshold;

[0015] Performing surf feature extraction processing on the image of each window area to determine the number of key points corresponding to each window area;

[0016] Determine a window area in which the number of key points in the window area is greater than the preset window threshold as the incremental search window;

[0017] The overlapping area of ​​the first viewing angle image and the second viewing angle image is traversed to determine an average offset corresponding to each of the incremental search windows.

[0018] Optionally, performing second key point matching on an image area of ​​the first viewing angle image corresponding to the incremental search window and an image area of ​​the second viewing angle image corresponding to the incremental search window to determine a set of correctly matched key point pairs includes:

[0019] According to a preset distance threshold, a K nearest neighbor algorithm is used to perform distance matching on each surf feature sub-region of the image region of the first perspective image corresponding to the incremental search window and each surf feature sub-region of the image region of the second perspective image corresponding to the incremental search window, and a correctly matched key point pair is determined, each of the correctly matched key point pairs including a key point of the first perspective image and a key point of the second perspective image;

[0020] The image coordinates of the key points of the first perspective image and the image coordinates of the key points of the second perspective image in the correctly matched key point pairs are obtained to determine the correctly matched key point pair set.

[0021] Optionally, the method of using a K nearest neighbor algorithm to perform distance matching on each surf feature sub-region of the image area of ​​the first perspective image corresponding to the incremental search window and each surf feature sub-region of the image area of ​​the second perspective image corresponding to the incremental search window according to a preset distance threshold to determine a correctly matched key point pair includes:

[0022] Determine the nearest neighbor distance between each surf feature sub-image area of ​​the first viewing angle image corresponding to the incremental search window and each surf feature sub-image area of ​​the second viewing angle image corresponding to the incremental search window by using the K nearest neighbor algorithm;

[0023] If the nearest neighbor distance is less than the preset distance threshold, the surf feature of the image area of ​​the first perspective image and the surf feature of the image area of ​​the second perspective image corresponding to the nearest neighbor distance are determined as the correctly matched key point pair.

[0024] Optionally, performing outlier elimination processing on the correctly matched key point pair set to determine a homography matrix of the first stage includes:

[0025] A random sampling consistency algorithm based on the least squares method is used to perform outlier elimination processing on the set of correctly matched key point pairs to determine the homography matrix of the first stage.

[0026] Optionally, the preset deep neural network includes a preset feature extractor, a preset mask extractor and a preset homography estimator.

[0027] Optionally, the using a preset deep neural network to perform homography estimation processing on an image area of ​​the first viewing angle image corresponding to the incremental search window and an image area of ​​the second viewing angle image corresponding to the incremental search window to determine a homography matrix of the second stage includes:

[0028] Inputting the image area of ​​the first viewing angle image corresponding to the incremental search window and the image area of ​​the second viewing angle image corresponding to the incremental search window into the preset mask extractor respectively, and determining the principal plane mask of the overlapping area of ​​the first viewing angle image and the principal plane mask of the overlapping area of ​​the second viewing angle image;

[0029] Inputting the image area of ​​the first viewing angle image corresponding to the incremental search window and the main plane mask of the overlapping area of ​​the first viewing angle image into the preset feature extractor to determine a first feature map of the overlapping area of ​​the first viewing angle image;

[0030] Inputting the image area of ​​the second perspective image corresponding to the incremental search window and the main plane mask of the overlapping area of ​​the second perspective image into the preset feature extractor to determine a second feature map of the overlapping area of ​​the second perspective image;

[0031] The first feature map of the overlapping area of ​​the first perspective image and the second feature map of the overlapping area of ​​the second perspective image are respectively input into the preset homography estimator to determine the homography matrix of the second stage.

[0032] Optionally, performing transformation and stitching processing on the first perspective image and the second perspective image according to the homography matrix of the first stage and the homography matrix of the second stage to determine the stitched image includes:

[0033] Using the homography matrix of the second stage to optimize the homography matrix of the first stage, and determining an optimized homography matrix;

[0034] Performing homography transformation on the second perspective image by using the optimized homography matrix to determine a homography transformed second perspective image;

[0035] Determine, according to the first perspective image and the homographically transformed second perspective image, an overlapping area of ​​the first image, an overlapping area of ​​the homographically transformed second perspective image, and a non-overlapping area of ​​the homographically transformed second perspective image;

[0036] The overlapping area of ​​the second perspective image after the homography transformation is replaced with the overlapping area of ​​the first perspective image, and the non-overlapping area of ​​the second perspective image after the homography transformation is circumscribed to the first perspective image to determine the stitched image.

[0037] Optionally, performing color correction optimization on the stitched image to determine a target stitched image includes:

[0038] Using a preset gradient operator to perform gradient calculation on the stitched image to determine gradient information of the stitched image, wherein the gradient information includes a gradient histogram;

[0039] Using a preset color correlation algorithm to perform color correlation calculation on each overlapping area of ​​the stitched image to determine color information of the stitched image, wherein the color information includes a color distance between each overlapping area and a distribution of the color distance;

[0040] According to the gradient information of the stitched image and the color information of the stitched image, the stitched image is color-corrected and optimized by using sliding window averaging to determine the target stitched image.

[0041] According to a second aspect of the present disclosure, there is provided an image stitching system based on key points and homography estimation, comprising:

[0042] An image acquisition module, used to acquire a first-perspective image and a second-perspective image corresponding to the same object, wherein the first perspective and the second perspective are adjacent perspectives;

[0043] A first key point matching module, configured to use an incremental search strategy to translate the first perspective image and the second perspective image to perform first key point matching, and determine an incremental search window and a window translation amount corresponding to the incremental search window;

[0044] A second key point matching module is used to perform second key point matching on the image area of ​​the first perspective image corresponding to the incremental search window and the image area of ​​the second perspective image corresponding to the incremental search window, and determine a set of correctly matched key point pairs, wherein the set of correctly matched key point pairs includes the image coordinates of each correctly matched key point;

[0045] A first-stage homography estimation module is used to perform outlier elimination processing on the correctly matched key point pair set to determine the homography matrix of the first stage;

[0046] A second-stage homography estimation module, configured to use a preset deep neural network to perform homography estimation processing on an image area of ​​the first viewing angle image corresponding to the incremental search window and an image area of ​​the second viewing angle image corresponding to the incremental search window, and determine a second-stage homography matrix;

[0047] An image transformation and stitching module, configured to perform transformation and stitching processing on the first perspective image and the second perspective image according to the homography matrix of the first stage and the homography matrix of the second stage, so as to determine a stitched image;

[0048] The color correction optimization module is used to perform color correction optimization processing on the stitched image to determine a target stitched image.

[0049] According to a third aspect of the present disclosure, a non-temporary computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method provided in the first aspect of the present disclosure are implemented.

[0050] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:

[0051] a memory having a computer program stored thereon;

[0052] A processor is used to execute the computer program in the memory to implement the steps of the method provided in the first aspect of the present disclosure.

[0053] Compared with the prior art, the embodiments of the present disclosure have at least one of the following beneficial effects:

[0054] Through the above technical scheme, adjacent first-view images and second-view images corresponding to the same object are obtained, and the incremental search strategy and feature sub-matching method are used to match the key points of the first-view image and the second-view image, and then the correct key point pair set is processed for outlier elimination, the homography matrix of the first stage is determined, a rough homography estimation is achieved, and the homography change relationship of the overlapping area of ​​the first-view image and the second-view image is determined; a preset deep neural network is used to perform homography estimation processing on the image area of ​​the first-view image corresponding to the incremental search window determined in the first stage and the image area of ​​the second-view image corresponding to the incremental search window, and the homography matrix of the second stage is obtained, so as to optimize the homography matrix of the first stage, improve the accuracy of the obtained homography matrix, and realize the homography transformation relationship of the overlapping area between the first-view image and the second-view image from coarse to fine; by performing color correction optimization processing on the stitched image, color consistency is achieved at the stitching edge of the image, and the stitching quality of the target stitched image is improved.

[0055] In an embodiment of the present disclosure, a preset deep neural network uses a preset mask extractor to extract the main plane mask of the overlapping area of ​​the second-view image, thereby achieving the effect of outlier elimination and improving the accuracy of homography estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Other features, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0057] Figure 1 The figure is a flowchart of an image stitching method based on key points and homography estimation according to an exemplary embodiment.

[0058] Figure 2 The present invention is a schematic diagram of a microscopic image stitching process of an image stitching method based on key points and homography estimation according to an exemplary embodiment.

[0059] Figure 3 It is a schematic diagram showing a homography estimation process based on a preset deep neural network according to an exemplary embodiment.

[0060] Figure 4 The present invention is a schematic diagram of a process for optimizing color correction of stitched microscopic images according to an exemplary embodiment.

[0061] Figure 5 The invention is a block diagram of an image stitching system based on key points and homography estimation according to an exemplary embodiment. DETAILED DESCRIPTION

[0062] The present disclosure is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present disclosure, but are not intended to limit the present disclosure in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present disclosure. These all fall within the scope of protection of the present disclosure.

[0063] Figure 1 The figure is a flowchart of an image stitching method based on key points and homography estimation according to an exemplary embodiment. Figure 2 The present invention is a schematic diagram of a microscopic image stitching process of an image stitching method based on key points and homography estimation according to an exemplary embodiment.

[0064] like Figure 1 , Figure 2 As shown, the present disclosure provides an image stitching method for key point and homography estimation, including S11 to S17. The image stitching method for key point and homography estimation provided by the present disclosure can be applicable to the image stitching task of multi-field microscope imaging.

[0065] S11, acquiring a first-perspective image and a second-perspective image corresponding to the same object.

[0066] The first perspective and the second perspective are adjacent perspectives, and microscopes with adjacent perspectives can be used to collect images of the same object from different perspectives as the first perspective image and the second perspective image.

[0067] S12, using an incremental search strategy to translate the first-view image and the second-view image to perform first key point matching, and determining an incremental search window and a window translation amount corresponding to the incremental search window.

[0068] Among them, the relative relationship between multi-view microscope images is a translation relationship, not a rotation relationship. Each lens rotates simultaneously. The overlapping area between adjacent microscope perspectives can be translated into the image of the perspective to be matched through an incremental search strategy, and key point matching is performed step by step. The determined incremental search window is the overlapping area of ​​the image of the perspective to be matched.

[0069] S13, performing second key point matching on the image area of ​​the first perspective image corresponding to the incremental search window and the image area of ​​the second perspective image corresponding to the incremental search window to determine a set of correctly matched key point pairs.

[0070] The image area of ​​the first perspective image corresponding to the incremental search window and the image area of ​​the second perspective image corresponding to the incremental search window are respectively the overlapping area of ​​the first perspective image and the overlapping area of ​​the second perspective image.

[0071] The correctly matched key point pair set includes the image coordinates of each correctly matched key point, the image coordinates of the key point located in the first-view image and the image coordinates of the key point located in the second-view image in the correctly matched key point pair.

[0072] In steps S12 and S13, the first key point matching and the second key point matching both use surf features to perform key point matching.

[0073] S14, performing outlier elimination processing on the set of correctly matched key point pairs to determine the homography matrix of the first stage.

[0074] S15, using a preset deep neural network to perform homography estimation processing on the image area of ​​the first perspective image corresponding to the incremental search window and the image area of ​​the second perspective image corresponding to the incremental search window, and determine the homography matrix of the second stage.

[0075] Among them, the preset deep neural network includes a preset feature extractor, a preset mask extractor and a preset homography estimator.

[0076] S16, performing transformation and stitching processing on the first-view image and the second-view image according to the homography matrix of the first stage and the homography matrix of the second stage to determine a stitched image.

[0077] S17, performing color correction optimization processing on the stitched image to determine a target stitched image.

[0078] In the present disclosure, S11 to S14 represent a rough homography estimation in the first stage, and S15 to S16 represent a fine homography estimation in the second stage.

[0079] Through the above technical scheme, adjacent first-view images and second-view images corresponding to the same object are obtained, and the incremental search strategy and feature sub-matching method are used to match the key points of the first-view image and the second-view image, and then the correct key point pair set is processed for outlier elimination, the homography matrix of the first stage is determined, a rough homography estimation is achieved, and the homography change relationship of the overlapping area of ​​the first-view image and the second-view image is determined; a preset deep neural network is used to perform homography estimation processing on the image area of ​​the first-view image corresponding to the incremental search window determined in the first stage and the image area of ​​the second-view image corresponding to the incremental search window, and the homography matrix of the second stage is obtained, so as to optimize the homography matrix of the first stage, improve the accuracy of the obtained homography matrix, and realize the homography transformation relationship of the overlapping area between the first-view image and the second-view image from coarse to fine; by performing color correction optimization processing on the stitched image, color consistency is achieved at the stitching edge of the image, and the stitching quality of the target stitched image is improved.

[0080] In a possible embodiment, after step S11, the first-view image and the second-view image may be enhanced to increase the number of key points, so as to improve the accuracy of subsequent key point matching.

[0081] In a possible embodiment, S12, using an incremental search strategy to translate the first-view image and the second-view image to perform first key point matching, and determining an incremental search window and a window translation amount corresponding to the incremental search window, may include S21 to S24.

[0082] S21, translating the first-view image and the second-view image according to a preset window step size and a preset window threshold.

[0083] S22, performing surf feature extraction processing on the image of each window area to determine the number of key points corresponding to each window area.

[0084] Among them, each surf feature represents a key point.

[0085] S23, determining a window area in which the number of key points in the window area is greater than a preset window threshold as an incremental search window.

[0086] Among them, in order to ensure that the number of key points is sufficient to meet the robustness requirements of calculating the homography transformation, the preset window threshold of the present invention is set to 20, that is, when the number of key points in the current translation window area is greater than 20, the current window area is determined to be an incremental search window, that is, an overlapping area.

[0087] S24, traversing the overlapping area of ​​the first-view image and the second-view image, and determining an average offset corresponding to each incremental search window.

[0088] Among them, during the translation process of the first-perspective image and the second-perspective image, the average offset corresponding to each incremental search window is recorded, and the first-perspective image and the second-perspective image are continued to be translated along the current direction until the number of key points in the translation window area is no more than 20, indicating that the overlapping area of ​​the first-perspective image and the second-perspective image has been traversed, and according to the average offset corresponding to each incremental search window, the complete overlapping area of ​​the first-perspective image and the second-perspective image can be determined.

[0089] In a possible embodiment, S13, a feature sub-pair is used to perform a second key point matching on the image area of ​​the first perspective image corresponding to the incremental search window and the image area of ​​the second perspective image corresponding to the incremental search window to determine a set of correctly matched key point pairs, including: S31 to S32.

[0090] Among them, the second key point matching disclosed in the present invention can also use surf features to perform key point matching, which can ensure matching accuracy and matching speed.

[0091] S31, according to the preset distance threshold, the K nearest neighbor algorithm is used to perform distance matching on each surf feature sub-region of the image area of ​​the first perspective image corresponding to the incremental search window and each surf feature sub-region of the image area of ​​the second perspective image corresponding to the incremental search window, and determine the correctly matched key point pairs.

[0092] Each correctly matched key point pair includes a key point of the first-view image and a key point of the second-view image.

[0093] The preset distance is set to 0.3 times the next nearest neighbor distance, and the K nearest neighbor algorithm is K-Nearest Neighbors, KNN algorithm.

[0094] In the above step S22, the surf features of the image area of ​​the first viewing angle image corresponding to the incremental search window and the surf features of the image area of ​​the second viewing angle image corresponding to the incremental search window may be determined.

[0095] S32, obtaining the image coordinates of the key points of the first-view image and the image coordinates of the key points of the second-view image in the correctly matched key point pairs, and determining a set of correctly matched key point pairs.

[0096] The image coordinates of the key points in the first perspective image are represented by the image coordinates of the key points in the first perspective image in the first perspective image, and the image coordinates of the key points in the second perspective image are represented by the image coordinates of the key points in the second perspective image in the second perspective image.

[0097] In a possible embodiment, S32, according to a preset distance threshold, uses the K nearest neighbor algorithm to perform distance matching on each surf feature sub-region of the image area of ​​the first perspective image corresponding to the incremental search window and each surf feature sub-region of the image area of ​​the second perspective image corresponding to the incremental search window to determine the correctly matched key point pairs, and may also include S321 to S322.

[0098] S321, using a K nearest neighbor algorithm to determine the nearest neighbor distance between each surf feature sub-region of the image region of the first perspective image corresponding to the incremental search window and each surf feature sub-region of the image region of the second perspective image corresponding to the incremental search window.

[0099] S322: If the nearest neighbor distance is less than a preset distance threshold, determine that the surf feature of the image area of ​​the first perspective image and the surf feature of the image area of ​​the second perspective image corresponding to the nearest neighbor distance are correctly matched key point pairs.

[0100] That is, the nearest neighbor distance is less than 0.3 times the nearest neighbor distance, and the two corresponding surf features are determined to be the correctly matched key point pairs.

[0101] Repeat steps S321 to S322 until each incremental search window is traversed, that is, the overlapping area of ​​the first-view image and the second-view image is traversed.

[0102] The key points in the set of correctly matched key point pairs screened out through the above steps S31 to S33 come from the first-view image and the second-view image. The image of the microscope where they are located is three-dimensional in the world coordinate system, and they may be in different planes. It is necessary to perform outlier elimination processing to screen the key point pairs to retain the key points on the same plane, and then calculate the homography matrix of the homography transformation relationship between the first-view image and the second-view image.

[0103] In a possible embodiment, S14, performing outlier elimination processing on the correctly matched key point pair set to determine the homography matrix of the first stage includes:

[0104] The random sampling consistency algorithm based on the least squares method is used to eliminate outliers from the set of correctly matched key point pairs and determine the homography matrix of the first stage.

[0105] Among them, the random sampling consensus algorithm of the least squares method is the RANSAC algorithm based on the least squares method.

[0106] The image coordinates of each key point in the set of correctly matched key points after outlier elimination are used to establish a set of eight-variable linear equations to parse out each position parameter of the third-order homography matrix as the homography matrix of the first stage, denoted as M1.

[0107] In the first stage homography estimation process, due to the simple structure of the microscopic image, even if the first-view image and the second-view image are enhanced, the number of matched key points is still limited, which will affect the accuracy of the homography matrix in the first stage and further affect the image stitching effect. Therefore, the present disclosure also provides a second stage homography estimation process.

[0108] Figure 3 It is a schematic diagram showing a homography estimation process based on a preset deep neural network according to an exemplary embodiment.

[0109] like Figure 3As shown, in a possible embodiment, S15, using a preset deep neural network to perform homography estimation processing on the image area of ​​the first perspective image corresponding to the incremental search window and the image area of ​​the second perspective image corresponding to the incremental search window to determine the homography matrix of the second stage, may include S51 to S54.

[0110] S51, input the image area of ​​the first perspective image corresponding to the incremental search window and the image area of ​​the second perspective image corresponding to the incremental search window into a preset mask extractor respectively, and determine the principal plane mask of the overlapping 4 areas of the first perspective image and the principal plane mask of the overlapping area of ​​the second perspective image.

[0111] Among them, a preset mask extractor is used as an outlier elimination module to achieve the effect of outlier elimination and improve the accuracy of homography estimation.

[0112] S52, inputting the image area of ​​the first perspective image corresponding to the incremental search window and the main plane mask of the overlapping area of ​​the first perspective image into a preset feature extractor to determine a first feature map of the overlapping area of ​​the first perspective image.

[0113] The image area of ​​the first-perspective image corresponding to the incremental search window is the overlapping area of ​​the first-perspective image. A preset feature extractor is used to extract the depth feature feature1 of the overlapping area of ​​the first-perspective image, and a first feature map is output.

[0114] S53, inputting the image area of ​​the second perspective image corresponding to the incremental search window and the main plane mask of the overlapping area of ​​the second perspective image into a preset feature extractor to determine a second feature map of the overlapping area of ​​the second perspective image.

[0115] Among them, the image area of ​​the second perspective image corresponding to the incremental search window is the overlapping area of ​​the second perspective image, and a preset feature extractor is used to extract the depth feature feature2 of the overlapping area of ​​the second perspective image, and output a second feature map.

[0116] In steps S52 to S53 of the present disclosure, the preset feature extractor shares parameters in the process of extracting the depth feature feature1 of the overlapping area of ​​the first-view image and extracting the depth feature feature2 of the overlapping area of ​​the second-view image.

[0117] S54, inputting the first feature map of the overlapping area of ​​the first perspective image and the second feature map of the overlapping area of ​​the second perspective image into a preset homography estimator respectively to determine the homography matrix of the second stage.

[0118] Wherein, the preset homography estimator shares parameters in the process of performing homography estimation on the first feature map and the second feature map respectively.

[0119] The first feature map of the overlapping area of ​​the first-perspective image is input into the preset homography estimator, and the homography matrix H_12 of the second stage is output; the second feature map of the overlapping area of ​​the second-perspective image is input into the preset homography estimator, and the homography matrix H_21 of the second stage is output. The matrix parameters of the second-stage homography matrix H_12 and the second-stage homography matrix H_21 are the same, and both are expressed as the homography matrix of the second stage.

[0120] The homography matrix H_12 of the second stage is expressed as:

[0121]

[0122] The homography matrix H_21 of the second stage is expressed as:

[0123]

[0124] In a possible embodiment, S16, performing transformation and stitching processing on the first-view image and the second-view image according to the homography matrix of the first stage and the homography matrix of the second stage to determine a stitched image, includes: S61 to S64.

[0125] S61, using the homography matrix of the second stage to optimize the homography matrix of the first stage, and determining an optimized homography matrix.

[0126] In the present disclosure, in the homography estimation process of the first stage, the insufficient number of key points will affect the accuracy of the outlier elimination processing in the first stage, and affect the accuracy of the homography matrix in the first stage, and then affect the accuracy of image stitching. The homography estimation of the second stage is used to fine-tune the homography matrix of the first stage to optimize the homography matrix of the first stage.

[0127] The optimized homography matrix represents the homography transformation relationship between the overlapping area of ​​the first-view image and the overlapping area of ​​the second-view image.

[0128] S62, performing homography transformation on the second-view image using the optimized homography matrix to determine the second-view image after homography transformation.

[0129] S63, determining, according to the first perspective image and the second perspective image after homography transformation, an overlapping area of ​​the first image, an overlapping area of ​​the second perspective image after homography transformation, and a non-overlapping area of ​​the second perspective image after homography transformation.

[0130] S64, replacing the overlapping area of ​​the second perspective image after homography transformation with the overlapping area of ​​the first perspective image, and circumscribing the non-overlapping area of ​​the second perspective image after homography transformation to the first perspective image, to determine a stitched image.

[0131] Since the microscope light has a halo at the edge of the stitched image, it will cause different imaging lighting conditions at each viewing angle, resulting in inconsistent colors at the stitching edge of the stitched image. The stitched image can be color corrected and optimized.

[0132] Figure 4 The present invention is a schematic diagram of a process for optimizing color correction of stitched microscopic images according to an exemplary embodiment.

[0133] like Figure 4 As shown, in a possible embodiment, S17, performing color correction optimization on the stitched image to determine a target stitched image, includes: S71 to S73.

[0134] S71, using a preset gradient operator to perform gradient calculation on the stitched image to determine gradient information of the stitched image.

[0135] The gradient information includes a gradient histogram.

[0136] The preset gradient operator may adopt the Sobel operator, the Prewitt operator and the Scharr operator, so as to calculate the gradient components of the image in the horizontal and vertical directions of the mosaic image, and then determine the gradient amplitude and direction of each pixel point, which may be realized by a convolution operation during the gradient calculation.

[0137] In the present disclosure, the preset gradient operator adopts the Sobel operator, and the gradient magnitude and direction of each pixel point are counted as a gradient histogram.

[0138] S72, using a preset color correlation algorithm to perform color correlation calculation on each overlapping area of ​​the stitched image to determine color information of the stitched image.

[0139] Among them, color correlation represents the similarity or correlation between different color spaces in visual perception. Common color spaces include red, green, and blue (RGB), hue, saturation, and brightness (HSV), brightness, a channel, and b channel (Lab). It can be measured using a color-based distance metric, statistical distribution characteristics, or a perception model of the human visual system. In the present disclosure, a color-based distance metric is used to measure color correlation.

[0140] The color information includes the color distance between each overlapping area and the distribution of the color distance.

[0141] S73, performing color correction optimization on the stitched image by using sliding window averaging according to the gradient information and the color information of the stitched image, and determining a target stitched image.

[0142] The sliding window averaging method is used to average the RGB values ​​in each sliding window of the stitched image to achieve color consistency.

[0143] Based on an image stitching method based on key points and homography estimation provided by the present invention, two stages of homography estimation are performed and color correction optimization is performed on the stitched image. The homography estimation algorithm is applied to the microscope image stitching scene in a multi-field microscope for the first time, and the homography transformation relationship of overlapping areas in adjacent viewing angles is determined from coarse to fine, and the overlapping areas are first subjected to homography transformation and then stitched.

[0144] Figure 5 The invention is a block diagram of an image stitching system based on key points and homography estimation according to an exemplary embodiment.

[0145] Based on the same concept, the present disclosure also provides an image stitching system 100 based on key points and homography estimation, such as Figure 5 As shown, it includes: an image acquisition module 110, a first key point matching module 120, a second key point matching module 130, a first stage homography estimation module 140, a second stage homography estimation module 150, an image transformation and stitching module 160, and a color correction and optimization module 170.

[0146] An image acquisition module 110 is used to acquire a first-perspective image and a second-perspective image corresponding to the same object, wherein the first perspective and the second perspective are adjacent perspectives;

[0147] A first key point matching module 120 is used to use an incremental search strategy to translate the first view image and the second view image to perform first key point matching, and determine an incremental search window and a window translation amount corresponding to the incremental search window;

[0148] A second key point matching module 130 is used to perform second key point matching on the image area of ​​the first perspective image corresponding to the incremental search window and the image area of ​​the second perspective image corresponding to the incremental search window, and determine a set of correctly matched key point pairs, where the set of correctly matched key point pairs includes the image coordinates of each correctly matched key point;

[0149] The first stage homography estimation module 140 is used to perform outlier elimination processing on the set of correctly matched key point pairs to determine the homography matrix of the first stage;

[0150] A second-stage homography estimation module 150 is used to use a preset deep neural network to perform homography estimation processing on the image area of ​​the first-view image corresponding to the incremental search window and the image area of ​​the second-view image corresponding to the incremental search window to determine the homography matrix of the second stage;

[0151] An image transformation and stitching module 160 is used to perform transformation and stitching processing on the first-view image and the second-view image according to the homography matrix of the first stage and the homography matrix of the second stage to determine a stitched image;

[0152] The color correction optimization module 170 is used to perform color correction optimization processing on the stitched image to determine a target stitched image.

[0153] Through the above technical scheme, adjacent first-view images and second-view images corresponding to the same object are obtained, and the incremental search strategy and feature sub-matching method are used to match the key points of the first-view image and the second-view image, and then the correct key point pair set is processed for outlier elimination, the homography matrix of the first stage is determined, a rough homography estimation is achieved, and the homography change relationship of the overlapping area of ​​the first-view image and the second-view image is determined; a preset deep neural network is used to perform homography estimation processing on the image area of ​​the first-view image corresponding to the incremental search window determined in the first stage and the image area of ​​the second-view image corresponding to the incremental search window, and the homography matrix of the second stage is obtained, so as to optimize the homography matrix of the first stage, improve the accuracy of the obtained homography matrix, and realize the homography transformation relationship of the overlapping area between the first-view image and the second-view image from coarse to fine; by performing color correction optimization processing on the stitched image, color consistency is achieved at the stitching edge of the image, and the stitching quality of the target stitched image is improved.

[0154] Regarding the embodiment of the above system, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0155] Based on the same concept as above, in another embodiment of the present disclosure, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor is used to execute an image stitching method based on key points and homography estimation when executing the program.

[0156] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0157] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0158] The processor is used to execute the computer program stored in the memory to implement the various steps in the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0159] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0160] In an embodiment of the present disclosure, a non-temporary computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of an image stitching method based on key points and homography estimation in any of the above embodiments are implemented.

[0161] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0163] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0165] Although the preferred embodiments of the present disclosure have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present disclosure.

[0166] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. An image stitching method based on key points and homography estimation, characterized in that: include: Acquire a first-perspective image and a second-perspective image corresponding to the same object, wherein the first perspective and the second perspective are adjacent perspectives; Using an incremental search strategy to translate the first perspective image and the second perspective image to perform first key point matching, and determining an incremental search window and a window translation amount corresponding to the incremental search window; Performing second key point matching on an image area of ​​the first viewing angle image corresponding to the incremental search window and an image area of ​​the second viewing angle image corresponding to the incremental search window to determine a set of correctly matched key point pairs, wherein the set of correctly matched key point pairs includes image coordinates of each correctly matched key point; Performing outlier elimination processing on the correctly matched key point pair set to determine the homography matrix of the first stage; Using a preset deep neural network to perform homography estimation processing on an image area of ​​the first viewing angle image corresponding to the incremental search window and an image area of ​​the second viewing angle image corresponding to the incremental search window, to determine a homography matrix of the second stage; According to the homography matrix of the first stage and the homography matrix of the second stage, transform and stitch the first perspective image and the second perspective image to determine a stitched image; The stitched image is subjected to color correction optimization processing to determine a target stitched image.

2. The method according to claim 1, characterized in that The step of using an incremental search strategy to translate the first perspective image and the second perspective image to perform first key point matching, and determining an incremental search window and a window translation amount corresponding to the incremental search window, comprises: translating the first perspective image and the second perspective image according to a preset window step size and a preset window threshold; Performing surf feature extraction processing on the image of each window area to determine the number of key points corresponding to each window area; Determine a window area in which the number of key points in the window area is greater than the preset window threshold as the incremental search window; The overlapping area of ​​the first viewing angle image and the second viewing angle image is traversed to determine an average offset corresponding to each of the incremental search windows.

3. The method according to claim 1, characterized in that The performing second key point matching on the image area of ​​the first viewing angle image corresponding to the incremental search window and the image area of ​​the second viewing angle image corresponding to the incremental search window to determine a set of correctly matched key point pairs includes: According to a preset distance threshold, a K nearest neighbor algorithm is used to perform distance matching on each surf feature sub-region of the image region of the first perspective image corresponding to the incremental search window and each surf feature sub-region of the image region of the second perspective image corresponding to the incremental search window, and a correctly matched key point pair is determined, each of the correctly matched key point pairs including a key point of the first perspective image and a key point of the second perspective image; The image coordinates of the key points of the first perspective image and the image coordinates of the key points of the second perspective image in the correctly matched key point pairs are obtained to determine the correctly matched key point pair set.

4. The method according to claim 3, characterized in that The method of using a K nearest neighbor algorithm to perform distance matching on each surf feature sub-region of the image region of the first viewing angle image corresponding to the incremental search window and each surf feature sub-region of the image region of the second viewing angle image corresponding to the incremental search window according to a preset distance threshold to determine a correctly matched key point pair includes: Determine the nearest neighbor distance between each surf feature sub-image area of ​​the first viewing angle image corresponding to the incremental search window and each surf feature sub-image area of ​​the second viewing angle image corresponding to the incremental search window by using the K nearest neighbor algorithm; If the nearest neighbor distance is less than the preset distance threshold, the surf feature of the image area of ​​the first perspective image and the surf feature of the image area of ​​the second perspective image corresponding to the nearest neighbor distance are determined as the correctly matched key point pair.

5. The method according to claim 1, characterized in that The step of performing outlier elimination processing on the correctly matched key point pair set to determine the homography matrix of the first stage includes: A random sampling consistency algorithm based on the least squares method is used to perform outlier elimination processing on the set of correctly matched key point pairs to determine the homography matrix of the first stage.

6. The method according to claim 1, characterized in that The preset deep neural network includes a preset feature extractor, a preset mask extractor and a preset homography estimator; The method of using a preset deep neural network to perform homography estimation processing on an image area of ​​the first viewing angle image corresponding to the incremental search window and an image area of ​​the second viewing angle image corresponding to the incremental search window to determine a homography matrix of the second stage includes: Inputting the image area of ​​the first viewing angle image corresponding to the incremental search window and the image area of ​​the second viewing angle image corresponding to the incremental search window into the preset mask extractor respectively, and determining the principal plane mask of the overlapping area of ​​the first viewing angle image and the principal plane mask of the overlapping area of ​​the second viewing angle image; Inputting the image area of ​​the first viewing angle image corresponding to the incremental search window and the main plane mask of the overlapping area of ​​the first viewing angle image into the preset feature extractor to determine a first feature map of the overlapping area of ​​the first viewing angle image; Inputting the image area of ​​the second perspective image corresponding to the incremental search window and the main plane mask of the overlapping area of ​​the second perspective image into the preset feature extractor to determine a second feature map of the overlapping area of ​​the second perspective image; The first feature map of the overlapping area of ​​the first perspective image and the second feature map of the overlapping area of ​​the second perspective image are respectively input into the preset homography estimator to determine the homography matrix of the second stage.

7. The method according to claim 1, characterized in that The step of performing transformation and splicing processing on the first perspective image and the second perspective image according to the homography matrix of the first stage and the homography matrix of the second stage to determine a spliced ​​image includes: Using the homography matrix of the second stage to optimize the homography matrix of the first stage, and determining an optimized homography matrix; Performing homography transformation on the second perspective image by using the optimized homography matrix to determine a homography transformed second perspective image; Determine, according to the first perspective image and the homographically transformed second perspective image, an overlapping area of ​​the first image, an overlapping area of ​​the homographically transformed second perspective image, and a non-overlapping area of ​​the homographically transformed second perspective image; The overlapping area of ​​the second perspective image after the homography transformation is replaced with the overlapping area of ​​the first perspective image, and the non-overlapping area of ​​the second perspective image after the homography transformation is circumscribed to the first perspective image to determine the stitched image.

8. The method according to claim 1, characterized in that The step of performing color correction optimization on the stitched image to determine a target stitched image includes: Using a preset gradient operator to perform gradient calculation on the stitched image to determine gradient information of the stitched image, wherein the gradient information includes a gradient histogram; Using a preset color correlation algorithm to perform color correlation calculation on each overlapping area of ​​the stitched image to determine color information of the stitched image, wherein the color information includes a color distance between each overlapping area and a distribution of the color distance; According to the gradient information of the stitched image and the color information of the stitched image, the color correction optimization is performed on the stitched image by using sliding window averaging to determine the target stitched image.

9. An image stitching system based on key points and homography estimation, characterized in that: include: An image acquisition module, used to acquire a first-perspective image and a second-perspective image corresponding to the same object, wherein the first perspective and the second perspective are adjacent perspectives; A first key point matching module, configured to use an incremental search strategy to translate the first perspective image and the second perspective image to perform first key point matching, and determine an incremental search window and a window translation amount corresponding to the incremental search window; A second key point matching module is used to perform second key point matching on the image area of ​​the first perspective image corresponding to the incremental search window and the image area of ​​the second perspective image corresponding to the incremental search window, and determine a set of correctly matched key point pairs, wherein the set of correctly matched key point pairs includes the image coordinates of each correctly matched key point; A first-stage homography estimation module is used to perform outlier elimination processing on the correctly matched key point pair set to determine the homography matrix of the first stage; A second-stage homography estimation module, configured to use a preset deep neural network to perform homography estimation processing on an image area of ​​the first viewing angle image corresponding to the incremental search window and an image area of ​​the second viewing angle image corresponding to the incremental search window, and determine a second-stage homography matrix; An image transformation and stitching module, configured to perform transformation and stitching processing on the first perspective image and the second perspective image according to the homography matrix of the first stage and the homography matrix of the second stage, so as to determine a stitched image; The color correction optimization module is used to perform color correction optimization processing on the stitched image to determine a target stitched image.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

Images