An image stitching method and device based on an electric microscope

Abstract

The application discloses an image splicing method and device based on an electric microscope, and the method comprises the following steps: capturing a local atlas of a sample by using the electric microscope; wherein the local atlas comprises a plurality of local images of the sample containing a capturing sequence; determining an overlapping area of adjacent local images according to the capturing sequence, moving a grating ruler according to the overlapping area, and splicing the local images; determining an optimal splicing seam of the overlapping area, and fusing and splicing the overlapping area according to the optimal splicing seam to obtain a fusion area; judging whether the fusion area has a splicing seam; if the fusion area has the splicing seam, performing Poisson fusion on the splicing seam; otherwise, obtaining a global image spliced by all the local images. The method solves the problems of low fusion image quality, noise and long time consumption existing in the prior art image fusion method, realizes continuous and natural splicing of the plurality of local images, and effectively improves the visual effect of image splicing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to an image splicing method and device based on an electric microscope. BACKGROUND

[0002] Image splicing is an important research direction in the field of computer vision, which aims to splice multiple images with overlapping regions into a seamless panoramic image, thereby obtaining a larger observation field of view. In recent years, image splicing technology has gradually been applied to electric microscope imaging. It allows multiple local images with overlapping regions captured by an electric microscope to be spliced into a seamless panoramic image covering the entire sample range, thereby obtaining a larger observation field of view and providing more detailed and comprehensive information for microscopic observation. This technology is widely used in the research of biology, material science, medicine and other fields.

[0003] The two most critical steps of image fusion are image registration and image fusion. Image registration refers to finding the mapping relationship of two images to be spliced in space during the splicing process, thereby establishing the geometric relationship between the images, so that the corresponding feature points in the images are one-to-one corresponding in space to achieve the effect of information fusion. Image fusion is to fuse multiple images into a new image, and the fused image can more accurately and comprehensively display sample or scene information. Existing image fusion methods include transformation-based methods, pixel-based methods and model-based methods. Among them, the transformation-based method refers to the wavelet transform-based image fusion method. This method can realize the fusion of images with different resolutions and energies. However, due to the errors and distortions of wavelet transform, it will result in low image fusion quality. The pixel-based method refers to a method of extracting some specific pixels from the images to be fused and combining them into a new image. This method does not consider the spatial structure of the image and the mutual relationship between the pixels, which can easily lead to artifacts and noise in the fused image. The model-based method refers to a method of analyzing and processing two input images by establishing a mathematical model or a probability model. However, this method requires a large amount of calculation and training, and has high implementation difficulty and time complexity. SUMMARY

[0004] The present application provides an image splicing method based on an electric microscope, which solves the problems of low fusion image quality, noise and long time consumption existing in the prior art image fusion method, and realizes an image splicing method based on an electric microscope that can solve the above problems.

[0005] In a first aspect, the embodiments of the present application provide an image stitching method based on an electric microscope, comprising: capturing a local atlas of a sample by using an electric microscope; wherein the local atlas comprises a plurality of local images of the sample containing a capturing sequence; determining an overlapping area of adjacent local images according to the capturing sequence, moving a grating ruler according to the overlapping area, and stitching the local images according to the displacement of the grating ruler; determining an optimal stitching seam of the overlapping area, dividing the overlapping area according to the optimal stitching seam, and performing weighted fusion stitching on the overlapping area according to the division result to obtain a fusion area; judging whether the fusion area has a stitching seam; if the fusion area has the stitching seam, performing local Poisson fusion on the stitching seam; otherwise, obtaining a global image after stitching of all the local images.

[0006] With reference to the first aspect, in a possible implementation manner, before the determination of the optimal stitching seam of the overlapping area, the method further includes: adjusting the exposure and brightness of the overlapping area to make the exposure and brightness of the overlapping area of adjacent local images consistent.

[0007] With reference to the first aspect, in a possible implementation manner, the adjusting of the exposure and brightness of the overlapping area includes: respectively determining the average values of RGB three channels of adjacent local images; respectively calculating the difference between each channel of the local image and the corresponding average value to obtain an exposure difference; and performing exposure compensation and brightness correction on the overlapping area of adjacent local images according to the exposure difference.

[0008] With reference to the first aspect, in a possible implementation manner, the determination of the optimal stitching seam of the overlapping area includes: determining a color difference, a geometric difference and a texture difference of the overlapping area; constructing an energy function according to the color difference, the geometric difference and the texture difference; traversing the pixel points of the overlapping area row by row, and determining the energy value of the traversed pixel points through the energy function; and the path formed by the pixel point with the minimum energy value is the optimal stitching seam.

[0009] With reference to the first aspect, in a possible implementation manner, the calculation formula of the geometric difference is as follows:

[0010] ; wherein, ,

[0011] . In the formula, C represents the color difference of the overlapping area, represents the geometric difference of the overlapping area, and respectively represent the Sobel operators of the overlapping area in the x direction and the y direction, a difference in gray scale values of the pixel points in the horizontal direction in the overlapping region, a difference in gray scale values of the pixel points in the vertical direction in the overlapping region, a difference in gray scale values of the pixel points in the horizontal direction in the overlapping region, a difference in gray scale values of the pixel points in the horizontal direction in the overlapping region,

[0012] The energy function is as follows:

[0013] wherein, the energy function, the color difference of the overlapping region, the geometric difference of the overlapping region, the texture difference.

[0014] In combination with the first aspect, in a possible implementation manner, the weighted fusion splicing of the overlapping region according to the division result to obtain a fused region comprises:

[0015] The weighted fusion method is adopted to fuse and splice the pixel points in the overlapping region, and the pixel points outside the best splicing seam adjust their weight coefficients according to their distances from the best splicing seam, and the specific formula is as follows:

[0016] ;

[0017] wherein, + =1, , + =1, ; wherein, coordinates of the overlapping region to be fused, and the weight coefficients of the two adjacent local images to be spliced, and respectively represent the weight coefficients of the pixel points of the overlapping region on the left and right sides of the best splicing seam, and represent pixel points on the two adjacent local images to be spliced, the best splicing seam, H represents the width of the overlapping region, and d represents the distance of the pixel points of the overlapping region from the best splicing seam.

[0018] With reference to the first aspect, in a possible implementation manner, after the global image after splicing of all the local images is obtained, the method further includes: detecting whether the global image has black edges; if the global image has black edges, traversing the global image row by row and / or column by column, and performing a black edge removing step until the global image is traversed; the black edge removing step includes: determining the number of black pixel points in a current row and / or column being traversed, and determining whether the number of black pixel points in the current row and / or column exceeds a preset threshold; if the number of black pixel points in the current row and / or column exceeds the preset threshold, changing all pixel points in the current row and / or column to black, and taking a next row and / or column of the current row and / or column as the current row and / or column to perform the black edge removing step; otherwise, taking the next row and / or column of the current row and / or column as the current row and / or column to perform the black edge removing step; determining non-black pixel points in the global image, and cutting off pixel points outside a minimum bounding rectangle of the non-black pixel points to remove black edges of the global image; if the local images do not have black edges, the global image is not processed.

[0019] With reference to the first aspect, in a possible implementation manner, the locally Poisson fusing the splicing seam includes: determining gradient information of the fusion region, and determining pixel values of a minimum rectangular region around the splicing seam according to the gradient information; performing Poisson fusing on the splicing seam in the minimum rectangular region; and performing smoothing processing on a splicing position of the minimum rectangular region and the fusion region.

[0020] With reference to the first aspect, in a possible implementation manner, the fusing and splicing the overlapping region according to the division result to obtain a fusion region includes: for a pixel point on the best splicing seam, determining a splicing seam weight according to a pixel value of the pixel point and a width of the overlapping region; for a pixel point other than the best splicing seam, determining a pixel point weight of the pixel point according to a distance of the pixel point from the best splicing seam; and fusing the best splicing seam according to the splicing seam weight, the pixel point weight, and a pixel value of the corresponding pixel point to obtain the fusion region.

[0021] In a second aspect, the embodiments of the present application provide an image stitching device based on an electric microscope, characterized in that the device comprises: a capturing module configured to capture a local image set of a sample by using an electric microscope; wherein the local image set comprises a plurality of local images of the sample and the local images comprise a capturing sequence; a stitching module configured to determine an overlapping area of adjacent local images according to the capturing sequence, move a grating ruler according to the overlapping area, and stitch the local images according to the displacement of the grating ruler; a fusion module configured to determine an optimal stitching seam of the overlapping area, divide the overlapping area according to the optimal stitching seam, and perform weighted fusion stitching on the overlapping area according to the division result to obtain a fusion area; and an optimization module configured to determine whether the fusion area has a stitching seam; if the fusion area has the stitching seam, perform local Poisson fusion on the stitching seam; otherwise, obtain a global image after stitching of all the local images.

[0022] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0023] By using the grating ruler to stitch the local images, the embodiments of the present application can achieve more accurate stitching and reduce stitching error and calculation amount; by fusing the local images according to the optimal stitching seam, the embodiments of the present application can make the stitching place smoothly transition, the stitching effect is more natural, and the calculation resources and time consumed are less. The embodiments of the present application effectively solve the problems of low fusion image quality, noise and long time consumption existing in the prior art image fusion method, and further implement an image stitching method based on an electric microscope, which can continuously and naturally stitch a plurality of local images together and effectively improve the visual effect of image stitching. Moreover, the embodiments of the present application avoid a large amount of data calculation or feature extraction and consume less time. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0025] Figure 1 A flowchart of an image stitching method based on an electric microscope provided by the embodiments of the present application;

[0026] Figure 2 A flowchart of determining an optimal stitching seam of an overlapping area provided by the embodiments of the present application;

[0027] Figure 3 A flowchart of detecting and removing black edges of a global image provided by the embodiments of the present application;

[0028] Figure 4 This is a schematic diagram of the structure of an image stitching device based on an electric microscope provided in an embodiment of this application;

[0029] Figure 5 This is a schematic diagram of two adjacent partial images to be stitched and the optimal stitching seam provided in an embodiment of this application;

[0030] Figure 6 An example diagram of the optimal splicing seam obtained by calculating the original energy function provided in the embodiments of this application;

[0031] Figure 7 An example diagram of the optimal splicing seam obtained by calculating the energy function of this application, provided for embodiments of this application;

[0032] Figure 8 Provided for the embodiments of this application Figure 6 The final result after fusion;

[0033] Figure 9 Provided for the embodiments of this application Figure 7 The final result after merging. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0035] The following description of some technologies involved in the embodiments of this application is provided to aid understanding and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, some descriptions of well-known functions and structures are omitted in the following description.

[0036] Figure 1 This is a flowchart of an image stitching method based on an electric microscope provided in an embodiment of this application, including steps 101 to 106. Figure 1 This is merely one execution order shown in the embodiments of this application and does not represent the only execution order for an image stitching method based on an electric microscope. The execution order can be adjusted to achieve the desired final result. Figure 1 The steps shown can be performed in parallel or in reverse order.

[0037] Step 101: Capture a local atlas of the sample by using the motorized microscope. In the embodiment of the present application, the local atlas includes multiple local images of the sample captured in sequence. Specifically, multiple local images of the sample are captured in sequence by using the motorized microscope, and the multiple local images are stored in the local atlas according to the capturing sequence. In addition, adjacent local images have an overlapping area, and in the local atlas, there are one, two, three or four overlapping areas of local images.

[0038] Those skilled in the art should understand that the multiple local images of the sample are captured in sequence by using the motorized microscope, and the capturing sequence here is not strictly limited to the capturing sequence of the motorized microscope, but the adjacent relationship of the local images, that is, adjacent local images have partially the same overlapping area, so as to splice to restore the overall image of the sample, so as to avoid the situation that the local images are spliced disorderly, resulting in the overall image of the sample cannot be correctly displayed.

[0039] Step 102: Determine the overlapping area of adjacent local images according to the capturing sequence, move the grating ruler according to the overlapping area, and splice the local images according to the displacement of the grating ruler. In the embodiment of the present application, the width of the overlapping area of the adjacent two local images is calculated, and then the grating ruler is moved according to the calculated width of the overlapping area. The grating ruler can provide high-precision and real-time position information, so as to obtain the relative position between the local images by measuring and recording the displacement of the grating ruler. Therefore, the grating ruler can be placed in the overlapping area of adjacent local images, and then the position relationship between the overlapping areas of the local images can be calculated by comparing the displacement difference of the grating ruler in the overlapping areas of different local images. Then, different local images are spliced together according to the position relationship, so as to realize the construction of the global image. This method avoids the problem that too much time and computing resources are consumed in the feature extraction and matching process in the traditional image splicing and fusion, and can also effectively process low feature images.

[0040] The grating ruler is a high-precision measuring tool, which can be controlled in both electric and manual modes. The electric mode realizes automatic movement by inputting the displacement amount in the x and y directions, while the manual mode controls the movement distance of the grating ruler by rotating the knob. When the resolution of the image is high, the displacement distance of the grating ruler corresponds to the change of the image pixels, for example, if the minimum scale of the grating ruler is 0.02 microns, then the displacement of 0.02 microns corresponds to the movement of one pixel in the image. However, when the resolution is low, the minimum scale of the grating ruler cannot correspond to one pixel in the image, and feature matching can be performed in the overlapping area to further splice. When using this method for feature matching, it is not necessary to find feature points in the entire image for matching, but only local search is needed, which greatly reduces the time of feature matching. Therefore, the position of the overlapping area of the two adjacent local images can be inferred according to the displacement amount of the grating ruler in the x and y directions, so as to determine the size of the overlapping area. The movement distance of the grating ruler can be specified in advance, and automatic splicing of the image can be realized. Of course, according to the needs of the user, the splicing mode can also be selected independently. In addition, the grating ruler in the present application has high precision, and can realize image splicing without or with very little overlap. Compared with the traditional splicing method, the present application has lower requirements for the overlapping area, greatly improving the splicing efficiency. However, the present application selects about 20% of the overlapping area when splicing, so that the best splicing seam can be found in the rectangular overlapping area, and the splicing seam can be eliminated through subsequent processing operations.

[0041] In addition, the displacement of the grating ruler can be observed and recorded in real time, so that the position information in the local image can be quickly obtained, the additional calculation and processing steps are reduced, and time and computing resources can be saved.

[0042] Exemplarily, if the overlapping area is 50%, the grating ruler also moves 50% to ensure that the two adjacent local images can be spliced together.

[0043] In the embodiment of the present application, before the local image is spliced by moving the grating ruler according to the overlapping area, the grating ruler also needs to be placed in the field of view of the electric microscope and fixed on the microscope stage. Then the grating ruler is accurately positioned and calibrated.

[0044] In addition, when splicing with the grating ruler, if there is a splicing error, feature matching can be performed on the overlapping area to further complete the splicing. If it can be accurately spliced, it is not necessary. This method of performing feature matching on a specific area is more efficient and shorter in matching time than the method of performing feature matching on the entire image in the prior art.

[0045] In the embodiments of the present application, the first local image and the second local image can be spliced according to the capture order, and after the splicing is completed, one local image is added according to the capture order for splicing. A person skilled in the art can also obtain and place multiple local images in sequence, and then splice the multiple local images according to the overlapping regions of the multiple local images at the same time to improve the splicing efficiency.

[0046] Step 103: determining the optimal seam of the overlapping region, dividing the overlapping region according to the optimal seam, and performing weighted fusion splicing on the overlapping region according to the division result to obtain a fused region. In the embodiments of the present application, before the optimal seam of the overlapping region is determined, the exposure and brightness of the overlapping region can also be adjusted to make the exposure and brightness of the overlapping regions of the two adjacent local images consistent. As shown in FIG. 3, I Figure 5 s and I t are two adjacent local images to be spliced, I between the two vertical dashed lines is the overlapping region of the two adjacent local images to be spliced, H is the width of the overlapping region, and the bending curve in the overlapping region is the optimal seam.

[0047] The method for adjusting the exposure and brightness of the overlapping region is as follows.

[0048] The average values of the RGB three channels of the two adjacent local images are calculated respectively. Specifically, the average values of the RGB three channels of the two adjacent local images to be spliced are calculated, that is, the average values of the color values of the red, green and blue three channels of the two local images are calculated.

[0049] The exposure difference is calculated by calculating the difference between each channel of the local image and the corresponding average value. Specifically, the difference between the color values of the red, green and blue three channels of the two adjacent local images to be spliced and the average value of the local image is calculated, that is, the exposure difference.

[0050] The exposure compensation and brightness correction of the overlapping region of the two adjacent local images are performed according to the exposure difference. Specifically, the exposure compensation and brightness of the two adjacent local images to be spliced are adjusted according to the calculated exposure difference, so that the exposure and brightness of the two adjacent local images to be spliced are more consistent.

[0051] The step of determining the optimal seam of the overlapping region is as shown in FIG. 4, which includes steps 201 to 204, and is specifically as follows. Figure 2

[0052] ​​Step 201: determining color difference, geometric difference and texture difference of the overlapping region. In the embodiment of the present application, the color difference is obtained by taking the absolute value of the difference of the RGB three channels of the corresponding pixel points in the overlapping region of the two adjacent local images to be spliced, and then taking the average value. The texture difference is obtained by calculating through the gray level co-occurrence matrix method, and by comparing the contrast and energy difference of the two adjacent local images to obtain the difference value.

[0053] wherein, for the pixel points on the edge of the overlapping region, the difference value is directly calculated with the pixel value at the corresponding position; otherwise, the horizontal or vertical difference value is calculated by the average value of the current pixel point and the adjacent upper and lower or left and right pixel points. Then, the gradient difference of the overlapping region is obtained according to the horizontal and vertical differences. Finally, the geometric difference of the overlapping region is obtained according to the gradient difference, and is applied to the part of finding the best splicing seam. This method can more accurately calculate the geometric difference of the two images in the overlapping region, and improve the splicing effect.

[0054] The calculation formula of the geometric difference is as follows:

[0055] wherein, ,

[0056] . In the formula, represents the geometric difference of the overlapping region, and respectively represent the Sobel operator of the overlapping region in the x direction and the y direction, represents the difference of the gray value of the pixel points in the horizontal direction in the overlapping region, represents the difference of the gray value of the pixel points in the vertical direction in the overlapping region, and represent the pixel points on the two adjacent local images to be spliced. Exemplarily, , .

[0057] Step 202: constructing an energy function according to the color difference, the geometric difference and the texture difference. In the embodiment of the present application, the energy function is as follows:

[0058] ; in the formula, represents the energy function, represents the color difference of the overlapping region, represents the geometric difference of the overlapping region, Color difference represents the color difference between the two images. The color difference is calculated by calculating the difference between the absolute values of the RGB three channels of the pixels in the overlapping area, and then calculating the average value. The texture difference is calculated by the gray level co-occurrence matrix method. Before calculating the texture difference, first calculate the co-occurrence matrix of the two adjacent local images, and then calculate the texture difference value by comparing the difference between the contrast and energy values of the two adjacent local images.

[0059] Step 203: Traverse the pixel points of the overlapping area row by row, and determine the energy value of the traversed pixel points through the energy function. Specifically, start traversing the energy value of the pixel points from the first row of the overlapping area, and then find the pixel point corresponding to the minimum energy value according to the cumulative energy value of the pixel points in the last row, until the last row of pixel points in the overlapping area is traversed.

[0060] Step 204: The path formed by the pixel point with the minimum energy value is the best stitching seam. In the embodiment of the present application, the path formed by the pixel point corresponding to the minimum energy value in each row during the traversal process is the best stitching seam.

[0061] The present application can dynamically confirm the best stitching seam through color difference, geometric difference and texture difference.

[0062] In the embodiment of the present application, for the pixel points on the best stitching seam, the stitching seam weight is determined according to the pixel value and the width of the overlapping area; for the pixel points other than the best stitching seam, the pixel point weight is determined according to the distance from the best stitching seam; and the fusion area is fused by the best stitching seam according to the stitching seam weight, the pixel point weight and the pixel value of the corresponding pixel point. The specific formula is as follows:

[0063] .

[0064] Wherein, + =1, , + =1, . In the formula, represents the coordinates of the overlapping area to be fused, and represent the weight coefficients of the two adjacent local images to be stitched, and respectively represent the weight coefficients of the pixel points of the overlapping area on the left and right sides of the best stitching seam, and represent the pixel points on the two adjacent local images to be stitched, represents the best stitching seam, H represents the width of the overlapping area, and d represents the distance of the pixel points of the overlapping area from the best stitching seam.

[0065] Figure 6 is the best seam obtained by considering the difference of the current pixel point only. This method may not capture the local difference well, resulting in an inaccurate best seam. Figure 7 The method of using the average of the differences of three adjacent pixel points to calculate the current texture difference in the embodiment can reduce the influence of noise, better capture the local difference, and increase the influence of the texture difference, so that the found best seam is more accurate. By comparing the positions of the best seams in Figure 6 and Figure 7 , it can be found that the best seam in Figure 7 is basically divided along the black circle part, and such a division effect is more in line with the perception ability of the human eye to the salient object. In addition, by calculating the difference between the pixel values on both sides of the best seam as an evaluation index, the index of Figure 6 is 0.1106, while the index value in Figure 7 is 0.0539. Comparing the data, the texture difference on both sides of the best seam found by the method of the present application is smaller, and the loss when the seam is disconnected is smaller.

[0066] By comparing Figure 8 and Figure 9 , the difference in effects brought by the two stitching methods can be observed. In Figure 8 , the best seam calculated using the original energy function is used, and an improved fusion method is used for stitching, resulting in the appearance of a clear white stitching line at the seam. In Figure 9 , the best seam of the present application and the same fusion method are used, and almost no stitching seam can be seen at the stitching. It can be seen that the quality of the best seam generated by the improved method is slightly better than that of the original method.

[0067] Step 104: Determine whether there is a seam in the fusion region. Specifically, according to the above steps, it is determined whether there is a seam in the fusion region, and if there is, step 105 is performed, otherwise, step 106 is performed, which is as follows.

[0068] Step 105: Perform local Poisson fusion on the seam. Specifically, when there is a seam in the fusion region, Poisson fusion is used to smoothly diffuse the seam to the fusion region, so that the stitching of the local image is more natural. The seam is a horizontal or vertical line at the junction of the overlapping region and the non-overlapping region, as shown in Figure 5 , the two vertical dashed lines in the middle are the seam, and the I region between the dashed lines is the overlapping region, and the I s and I t are non-overlapping regions.

[0069] In the embodiment of the present application, gradient information of the fusion region is determined, and pixel values of the minimum rectangular region around the splicing seam are determined according to the gradient information; the splicing seam in the minimum rectangular region is subjected to Poisson fusion; and the splicing part of the minimum rectangular region and the fusion region is subjected to smoothing processing.

[0070] Step 106: obtaining the global image after splicing of all local images. When there is no splicing seam in the fusion region, the image after splicing of all local images is the global image of the sample.

[0071] In an embodiment of the present application, the global image after splicing can be further optimized, and the specific steps are as shown in Figure 3

[0072] Step 301: detecting whether there is a black border in the global image. Specifically, whether there is a black border in the global image is detected, and if there is, steps 302 to 308 are executed, otherwise, the global image has no black border and needs no optimization.

[0073] Step 302: traversing the global image row by row and / or column by column. In the embodiment of the present application, the global image can have a black border only at the upper and lower boundaries, and only row-by-row traversal is needed, and the global image can be optimized by row only. The global image can also have a black border only at the left and right boundaries, and only column-by-column traversal is needed, and the global image can be optimized by column only. The global image can also have a black border at the upper, lower, left and right boundaries, and row-by-row and column-by-column traversal is needed, and the global image can be optimized by row and column. There is no priority between row-by-row and column-by-column, and the global image can be optimized by row and column as long as the black border is removed.

[0074] Step 303: determining the number of black pixel points in the current row and / or column being traversed. Specifically, the black pixel point is a pixel point with black color, i.e. a pixel point with RGB value of (0, 0, 0). The proportion of the black pixel points in the current row and / or column being traversed is counted, and the number of the black pixel points is obtained according to the size of the global image by proportion conversion.

[0075] Step 304: judging whether the number of the black pixel points in the current row and / or column is greater than a preset threshold. Specifically, whether the number of the black pixel points in the current row and / or column is greater than the preset threshold is compared. If the number of the black pixel points in the current row and / or column is greater than the preset threshold, steps 305 to 306 are executed, otherwise, step 306 is executed.

[0076] ​In the embodiments of the present application, the preset threshold is exemplarily set as 100. Those skilled in the art can also set the preset threshold in multiple stages according to actual conditions, for example, an initial value is first set for the preset threshold to process the global image, if the processing effect cannot meet the requirements, the initial value is modified to process the global image again until the global image processed according to the modified initial value meets the requirements.

[0077] In addition, the number of black pixel points in each row and / or column of the global image can also be counted first, and the preset threshold is set according to the overall level of the number of black pixel points in the rows and / or columns of the global image. For example, the number of black pixel points in the rows and / or columns is 132, 1114, 3050, 1231, 2201, 3018 and 31, and the initial value of the preset threshold is set as 30, if the processing effect cannot meet the requirements, the initial value can be modified to 100, 1000, … according to the number of black pixel points in the rows and / or columns until the black edge removal effect meets the processing requirements.

[0078] It should be noted that the present application does not limit the specific value and setting method of the preset threshold, as long as the processing requirements of removing the black edge and retaining the actual content of the global image in the present application can be met.

[0079] Step 305: all pixel points in the current row and / or column are changed to black. Specifically, if the number of black pixel points in the current row and / or column exceeds the preset threshold, all pixel points in the current row and / or column are modified to black pixel points.

[0080] Step 306: whether the global image is traversed completely is determined. Specifically, whether the global image is traversed completely by row and / or column is determined, that is, whether the current row and / or column is the last row and / or column of the global image is determined. If the global image is traversed completely, step 308 is executed, and if the global image is not traversed completely, steps 307 to 308 are executed.

[0081] Step 307: the next row and / or column of the current row and / or column is taken as the current row and / or column. Specifically, if the global image is not traversed completely, the next row and / or column of the current row and / or column is taken as the current row and / or column, and steps 303 to 304 are executed.

[0082] Step 308: non-black pixel points in the global image are determined, and pixel points outside the minimum circumscribed rectangle of the non-black pixel points are cropped to remove the black edge of the global image. Specifically, if the global image is traversed completely, the range of non-black pixel points in the global image is obtained, and the minimum circumscribed rectangle of the range where the non-black pixel points are located is constructed, and the pixel points outside the minimum circumscribed rectangle are cropped according to the minimum circumscribed rectangle, that is, the global image without black edge after optimization is obtained.

[0083] Although the present application provides method operation steps such as embodiments or flowcharts, more or less operation steps can be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one of the many execution orders, and does not represent the only execution order. In actual device or client product execution, the method order shown in the embodiments or the drawings can be executed in sequence or in parallel (for example, in a parallel processor or multi-thread processing environment).

[0084] As shown in Figure 4 The embodiments of the present application also provide an image stitching device 400 based on an electric microscope. The device comprises a capturing module 401, a stitching module 402, a fusion module 403 and an optimization module 404, and specifically as follows.

[0085] The capturing module 401 is used to capture a local image set of a sample by using an electric microscope. The local image set comprises multiple local images of the sample containing a capturing order. The capturing module 401 is specifically used to sequentially capture multiple local images of the sample by using the electric microscope, and store the multiple local images in the local image set according to the capturing order, and the adjacent local images have a certain overlapping area. In the local image set, there are one, two, three or four overlapping areas of the local images.

[0086] Those skilled in the art should understand that the multiple local images of the sample are sequentially captured by using the electric microscope. The capturing order here is not strictly limited to the capturing order of the electric microscope, but the adjacent relationship of the local images, that is, the adjacent local images have partially the same overlapping area, so that the whole image of the sample can be restored by stitching, so as to avoid the situation that the local images are stitched disorderly, resulting in that the whole image of the sample cannot be correctly displayed.

[0087] The stitching module 402 is used to determine the overlapping area of the adjacent local images according to the capturing order, and move the grating ruler for stitching the local images according to the overlapping area. The stitching module 402 is specifically used to calculate the width of the overlapping area of the adjacent two local images, and then move the grating ruler according to the calculated width of the overlapping area. Exemplarily, if the overlapping area is 50%, the grating ruler is also moved by 50% to ensure that the adjacent two local images can be stitched together.

[0088] In the embodiments of the present application, before moving the grating ruler for stitching the local images according to the overlapping area, the grating ruler needs to be placed in the field of view of the electric microscope and fixed on the microscope stage. Then the grating ruler is accurately positioned and calibrated.

[0089] In the embodiments of the present application, the first local image and the second local image can be spliced first, and then one local image is added in sequence each time after the splicing is completed. Alternatively, a plurality of local images can be obtained and placed in sequence, and then spliced according to the overlapping regions of the plurality of local images at the same time, so as to improve the splicing efficiency.

[0090] The fusion module 403 is configured to determine the optimal seam of the overlapping region and perform weighted fusion splicing on the overlapping region according to the optimal seam to obtain a fusion region. Before determining the optimal seam of the overlapping region, the fusion module 403 can further adjust the exposure and brightness of the overlapping region to make the exposure and brightness of the overlapping region of the two adjacent local images consistent. As shown in FIG. 4, I Figure 5 s and I t are two adjacent local images to be spliced, I between the two vertical dashed lines is the overlapping region of the two adjacent local images to be spliced, H is the width of the overlapping region, and the bending curve in the overlapping region is the optimal seam.

[0091] The method for adjusting the exposure and brightness of the overlapping region is specifically as follows.

[0092] The average values of the RGB three channels of the two adjacent local images are calculated respectively. Specifically, the average values of the RGB three channels of the two adjacent local images to be spliced are calculated, that is, the average values of the color values of the red, green and blue three channels of the two local images are calculated.

[0093] The exposure difference is calculated by calculating the difference between each channel of the local image and the corresponding average value. Specifically, the difference between the color values of the red, green and blue three channels of the two adjacent local images to be spliced and the average value of the local image is calculated, that is, the exposure difference.

[0094] The exposure compensation and brightness correction of the overlapping region of the two adjacent local images are performed according to the exposure difference. Specifically, the exposure compensation and brightness of the two adjacent local images to be spliced are adjusted according to the calculated exposure difference, so that the exposure and brightness of the two adjacent local images to be spliced are more consistent.

[0095] The color difference and the geometric difference of the overlapping region are determined. In the embodiments of the present application, the color difference is obtained by subtracting the values of the corresponding pixel points in the overlapping region of the two adjacent local images to be spliced.

[0096] The calculation formula of the geometric difference is as follows:

[0097] ; wherein, ,

[0098] ​wherein, denotes the geometric difference of the overlapping region, and denote the Sobel operators of the overlapping region in x direction and y direction respectively, denotes the difference of the gray value of the pixel points in the horizontal direction in the overlapping region, denotes the difference of the gray value of the pixel points in the vertical direction in the overlapping region, and denote the pixel points on the adjacent two local images to be spliced. Exemplarily, , .

[0099] An energy function is constructed according to the color difference and the geometric difference of the overlapping region. In the embodiment of the present application, the energy function is as follows:

[0100] wherein, denotes the energy function, denotes the color difference of the overlapping region, denotes the geometric difference of the overlapping region, denotes the texture difference. The color difference is obtained by calculating the difference of the absolute values of the RGB three channels of the pixel points in the overlapping region, and then calculating the average value. The texture difference is calculated by the gray level co-occurrence matrix method. Before calculating the texture difference, the co-occurrence matrix of the adjacent two local images is calculated first, and then the texture difference value is calculated by comparing the difference of the contrast and the energy value of the adjacent two local images.

[0101] The pixel points of the overlapping region are traversed row by row, and the energy value of the traversed pixel points is determined by the energy function. Specifically, the energy value of the pixel points is traversed from the first row of the overlapping region, and then the pixel point corresponding to the minimum energy value is found according to the cumulative energy value of the pixel points in the last row, until the last row of the pixel points of the overlapping region is traversed.

[0102] The path formed by the pixel point with the minimum energy value is the best splicing seam. In the embodiment of the present application, the path formed by the pixel point corresponding to the minimum energy value in each row in the traversal process is the best splicing seam.

[0103] In the embodiment of the present application, the pixel points in the overlapping region are fused and spliced by using the fade-in and fade-out weighted fusion method, and the pixel points outside the best splicing seam adjust their weight coefficients according to the distance from the best splicing seam, and the specific formula is as follows:

[0104] .

[0105] wherein, + =1, , + =1, . In the formula, represents the coordinates of the overlapping region to be fused, and represents the weight coefficients of the two adjacent partial images to be spliced, and respectively represent the weight coefficients of the pixel points of the overlapping region on the left and right sides of the optimal seam, and represents the pixel points on the two adjacent partial images to be spliced, represents the optimal seam, H represents the width of the overlapping region, and d represents the distance of the pixel points of the overlapping region from the optimal seam.

[0106] The optimization module 404 is configured to determine whether there is a seam in the fusion region. If there is a seam in the fusion region, the seam is subjected to Poisson fusion. Otherwise, the global image after splicing of all the partial images is obtained. The optimization module 404 is specifically configured to determine whether there is a seam in the fusion region, and if there is a seam, the seam is subjected to Poisson fusion. Specifically, when there is a seam in the fusion region, the Poisson fusion is used to smoothly diffuse the seam to the fusion region, so that the splicing of the partial images is more natural.

[0107] If there is no seam in the fusion region, the image spliced by all the partial images is the global image of the sample. In an embodiment of the present application, the global image after splicing can be further optimized, specifically as follows.

[0108] It is detected whether there is a black border in the global image. Specifically, it is detected whether there is a black border in the global image. If there is no black border, the global image does not need to be optimized. If there is a black border, the global image is traversed row by row and / or column by column. In an embodiment of the present application, the global image can have a black border only at the upper and lower boundaries, so only row-by-row traversal is needed, and the global image can be optimized only by row. The global image can also have a black border only at the left and right boundaries, so only column-by-column traversal is needed, and the global image can be optimized only by column. The global image can also have a black border at the upper, lower, left and right boundaries, so row-by-row and column-by-column traversal is needed, and the global image can be optimized by row and column. There is no priority between row-by-row and column-by-column traversal, as long as the global image is optimized by row and column to remove the black border.

[0109] The number of black pixel points in the current row and / or column traversed is determined. Specifically, a black pixel point is a pixel point with black color, i.e., a pixel point with RGB value of (0, 0, 0). The proportion of black pixel points in the current row and / or column traversed is counted, and the number of black pixel points is obtained by proportion conversion according to the size of the global image.​

[0110] It is judged whether the number of black pixel points in the current row and / or column exceeds a preset threshold. Specifically, it is compared whether the number of black pixel points in the current row and / or column counted is greater than the preset threshold.

[0111] In the embodiments of the present application, the preset threshold is exemplarily set to 100. Those skilled in the art can also set a plurality of preset thresholds according to actual conditions, for example, an initial value is first set for the preset threshold to process the global image, if the processing effect cannot meet the requirements, the initial value is modified and then the global image is processed until the global image processed according to the modified initial value meets the requirements.

[0112] In addition, the number of black pixel points in each row and / or column in the global image can also be counted first, and the preset threshold is set in turn according to the overall level of the number of black pixel points in the row and / or column in the global image. For example, the number of black pixel points in the row and / or column is 132, 1114, 3050, 1231, 2201, 3018 and 31 respectively, and the initial value of the preset threshold is set to 30, and the initial value can be modified to 100, 1000, … according to the number of black pixel points in the row and / or column until the black border effect meets the processing requirements.

[0113] It should be noted that the present application does not limit the specific value and setting method of the preset threshold, as long as it can meet the processing requirements of removing the black border and retaining the actual content of the global image in the present application.

[0114] If the number of black pixel points in the current row and / or column is greater than the preset threshold, all pixel points in the current row and / or column are changed to black. Specifically, if the number of black pixel points in the current row and / or column exceeds the preset threshold, all pixel points in the current row and / or column are modified to black pixel points.

[0115] If the number of black pixel points in the current row and / or column is not greater than the preset threshold, it is judged whether the global image is traversed completely. Specifically, it is judged whether the global image is traversed completely by row and / or column, that is, whether the current row and / or column is the last row and / or column of the global image. If the global image is not traversed completely, the next row and / or column of the current row and / or column is taken as the current row and / or column to continue to traverse until the global image is traversed completely.

[0116] If the global image is traversed completely, the non-black pixel points in the global image are determined, and the pixel points outside the minimum circumscribed rectangle of the non-black pixel points are cropped to remove the black border of the global image. Specifically, if the global image is traversed completely, the range of non-black pixel points in the global image is obtained, and the minimum circumscribed rectangle of the range where the non-black pixel points are located is constructed, and the pixel points outside the minimum circumscribed rectangle are cropped according to the minimum circumscribed rectangle, that is, the global image without black border after optimization is obtained.

[0117] Some of the modules in the apparatus described in the present application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform particular tasks or implement particular abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0118] The apparatus or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above apparatus is described as various modules with functions. In the implementation of the embodiments of the present application, the functions of the modules can be implemented in one or more software and / or hardware. Of course, the modules implementing certain functions can also be implemented by a combination of multiple sub-modules or sub-units.

[0119] The methods, apparatuses or modules described in the present application can be implemented in a computer-readable program code in any appropriate manner, for example, the controller can take the form of, for example, a microprocessor or a processor, and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro) processor, logic gates, switches, application specific integrated circuits (Application Specific Integrated Circuit; abbreviated as: ASIC), programmable logic controllers and embedded microcontrollers, examples of the controller include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a pure computer-readable program code manner, the same function can also be implemented by logically programming the method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers. Therefore, such a controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the devices for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0120] In addition, each functional module in various embodiments of the present application can be integrated in one processing module, or each module can exist independently, or two or more modules can be integrated in one module.

[0121] The storage medium includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a cache, a hard disk drive (HDD), or a memory card. The storage medium can be used to store computer program instructions.

[0122] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary hardware. Based on such an understanding, the technical solutions of the present application can be embodied in the form of a software product or in the form of data migration in the implementation process. The computer software product can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the method described in various embodiments or some parts of the embodiments.

[0123] The various embodiments in the specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. The whole or part of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, small computers, large computers, distributed computing environments including any of the above systems or devices, etc.

[0124] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the present application.

Claims

1. An image stitching method based on an electric microscope, characterized by, The method comprises: capturing a local atlas of a sample by using an electron microscope; wherein the local atlas comprises a plurality of local images of the sample containing a capturing sequence; determining an overlapping area of adjacent local images according to the capturing sequence, moving a grating ruler according to the overlapping area, and stitching the local images according to the displacement of the grating ruler; determining an optimal seam of the overlapping area, dividing the overlapping area according to the optimal seam, and performing weighted fusion stitching on the overlapping area to obtain a fusion area; wherein the determination of the optimal seam of the overlapping area comprises: determining color difference, geometric difference and texture difference of the overlapping area; constructing an energy function according to the color difference, the geometric difference and the texture difference; traversing pixel points of the overlapping area row by row, and determining an energy value of the traversed pixel points by the energy function; a path formed by the pixel points with the minimum energy value is the optimal seam; a calculation formula of the geometric difference is as follows: ; wherein, , ； wherein, denotes the geometric difference of the overlapping region, and denotes the Sobel operator of the overlapping region in the x direction and the y direction, respectively, denotes the difference of the gray value of the pixel in the horizontal direction in the overlapping region, denotes the difference of the gray value of the pixel in the vertical direction in the overlapping region, and denotes the pixel on the adjacent two local images to be spliced. the energy function is as follows: ; wherein, represents the energy function, represents the color difference of the overlapping area, represents the geometric difference of the overlapping area, represents the texture difference; judging whether the fusion area has a seam; if the fusion area has the seam, performing local Poisson fusion on the seam; otherwise, obtaining a global image after stitching of all the local images.

2. The method of claim 1, wherein, Before the determination of the optimal seam of the overlapping area, the method further comprises: adjusting exposure and brightness of the overlapping area to make exposure and brightness of the overlapping area of adjacent local images consistent.

3. The method of claim 2, wherein, The adjustment of the exposure and brightness of the overlapping area comprises: determining average values of RGB three channels of adjacent local images respectively; calculating a difference value of each channel of the local image and the corresponding average value to obtain an exposure difference; performing exposure compensation and brightness correction on the overlapping area of adjacent local images according to the exposure difference.

4. The method of claim 1, wherein, The weighted fusion stitching of the overlapping area according to the division result to obtain the fusion area comprises: performing fusion stitching on pixel points in the overlapping area by using a weighted fusion method, and adjusting a weight coefficient of a pixel point outside the optimal seam according to a distance of the pixel point from the optimal seam, and a specific formula is as follows: ； wherein, + = 1, , + = 1, ; in the formula, represents the coordinates of the overlapping region to be fused, and represents the weight coefficient of the two adjacent local images to be spliced, and respectively represent the weight coefficients of the pixel points of the overlapping region on the left and right sides of the optimal seam, and represents the pixel points on the two adjacent local images to be spliced, represents the optimal seam, H represents the width of the overlapping region, and d represents the distance of the pixel points of the overlapping region from the optimal seam.

5. The method of claim 1, wherein, After obtaining the global image after stitching of all the local images, the method further comprises: detecting whether the global image has black edges; if the global image has black edges, traversing the global image row by row and / or column by column, and performing a black edge removal step until the global image is traversed; the black edge removal step comprises: determining a number of black pixel points of a current row and / or column being traversed, and judging whether the number of black pixel points in the current row and / or column exceeds a preset threshold; if the number of black pixel points in the current row and / or column exceeds the preset threshold, changing all pixel points in the current row and / or column to black, and taking a next row and / or column of the current row and / or column as the current row and / or column to perform the black edge removal step; otherwise, taking a next row and / or column of the current row and / or column as the current row and / or column to perform the black edge removal step. Determine non-black pixels in the global image, and crop pixels outside the minimum circumscribed rectangle of the non-black pixels to remove black edges of the global image. If the local image does not have black edges, the global image is not processed.

6. The method of claim 1, wherein, The local Poisson fusion of the seam includes: Determine gradient information of the fusion region, and determine pixel values of a minimum rectangular region around the seam according to the gradient information; Poisson fuse the seam in the minimum rectangular region; Smooth the joint of the minimum rectangular region and the fusion region.

7. The method of claim 1, wherein, The fusion and splicing of the overlapping region according to the division result includes: For a pixel point on the best seam, determine a seam weight according to its pixel value and the width of the overlapping region; For a pixel point other than the best seam, determine a pixel point weight according to its distance from the best seam; Fuse the best seam according to the seam weight, the pixel point weight, and the pixel value of the corresponding pixel point to obtain the fusion region.

8. An image stitching device based on an electromicroscope, characterized by, It includes: A capture module configured to capture a local image set of a sample using an electric microscope; wherein the local image set includes multiple local images of the sample containing a capture sequence; A splicing module configured to determine an overlapping region of adjacent local images according to the capture sequence, move a grating ruler according to the overlapping region, and splice the local images according to the displacement of the grating ruler; A fusion module configured to determine a best seam of the overlapping region, divide the overlapping region according to the best seam, and perform weighted fusion and splicing of the overlapping region according to the division result to obtain a fusion region; wherein the determination of the best seam of the overlapping region includes: determining color difference, geometric difference, and texture difference of the overlapping region; constructing an energy function according to the color difference, the geometric difference, and the texture difference; traversing pixel points of the overlapping region row by row, and determining energy values of the traversed pixel points through the energy function; a path formed by the pixel point with the minimum energy value is the best seam. The formula for calculating the geometric difference is as follows: ; wherein, , ； wherein, represents the geometric difference of the overlapping region, and respectively represent the Sobel operator of the overlapping region in the x direction and the y direction, represents the difference of the gray value of the pixel point in the horizontal direction in the overlapping region, represents the difference of the gray value of the pixel point in the vertical direction in the overlapping region, and represent the pixel points on the adjacent two local images to be spliced. The energy function is as follows: ; wherein, represents the energy function, represents the color difference of the overlapping area, represents the geometric difference of the overlapping area, represents the texture difference; An optimization module configured to determine whether the fusion region has a seam; if the fusion region has the seam, perform local Poisson fusion of the seam; otherwise, obtain a global image after splicing of all the local images.

Images