Microscopic image real-time splicing method for quickly eliminating accumulated errors

Abstract

The application discloses a kind of fast elimination of cumulative error's real-time splicing method of microscopic image, comprising: obtaining current frame microscopic image, carries out brightness correction and position correction;Obtain the adjacent spliced microscopic image of current frame microscopic image in the spliced large field microscopic image adjacent to current frame microscopic image, and carry out template matching, obtain the registration degree and offset of current frame microscopic image in corresponding direction;Determine whether the registration degree and offset of current frame microscopic image each direction is matched correctly;Using the method based on fast elimination of cumulative error to obtain the absolute position coordinate of current frame microscopic image;Current frame microscopic image is spliced to the spliced large field microscopic image, and the real-time update of spliced large field microscopic image is completed and is shown.The application is applied to the field of image splicing, can greatly eliminate cumulative error in splicing process, optimize the problem caused by matching failure, effectively improve the splicing quality and splicing speed of large field microscopic image.

Description

Technical Field

[0001] This invention relates to the field of image stitching technology, specifically a real-time stitching method for microscopic images that rapidly eliminates accumulated errors. Background Technology

[0002] Microscopes are widely used in industrial inspection, medical analysis, and scientific research fields such as lasers and semiconductors. However, in practical applications, due to the limited field of view of microscopes, only a portion of the sample to be observed can be captured. In recent years, with the rapid development of image processing technology, image processing techniques can be used to synthesize microscopic images of the sample under the microscope from various fields of view, thereby obtaining large-field-of-view microscopic images with a wider field of view and higher resolution.

[0003] Image stitching technology has received widespread attention in virtual reality, microscopic image processing, remote sensing, and military fields, and is currently a research hotspot. In microscopic image stitching, the rapid and accurate stitching of large-field-of-view microscopic images is a key concern. Existing image stitching methods can be divided into two main categories: one is a matching-then-fusion stitching method, where the final large-field-of-view microscopic image is only obtained after all images have been stitched together, making real-time stitching and real-time preview of large-field-of-view microscopic images impossible; the other is a registration-while-fusion stitching method, where images are stitched sequentially after registration. This places extremely high demands on the quality of microscopic image acquisition and the accuracy of the image registration algorithm. If matching fails, the stitching error accumulates along the stitching path, causing subsequent images to misalign. Therefore, there is an urgent need for a stitching method that can significantly eliminate the accumulated error in the stitching process, optimize the problems caused by matching failures, and effectively improve the stitching quality and speed of large-field-of-view microscopic images. Summary of the Invention

[0004] To address the problem of accumulated errors in the microscopic image stitching process in the prior art, this invention provides a real-time microscopic image stitching method that can quickly eliminate accumulated errors. This method can greatly eliminate accumulated errors in the stitching process, optimize the problems caused by matching failures, effectively improve the stitching quality and speed of large-field-of-view microscopic images, and enable real-time stitching of local microscopic images while acquiring microscopic images.

[0005] To achieve the above objectives, the present invention provides a real-time stitching method for microscopic images to rapidly eliminate accumulated errors, comprising the following steps:

[0006] Step 1: Obtain the current frame microscopic image obtained from the current frame scan, and perform brightness correction and position correction processing on the current frame microscopic image;

[0007] Step 2: Obtain the stitched large field-of-view microscopic image, and obtain the stitched adjacent microscopic images that are adjacent to the current frame microscopic image in the stitched large field-of-view microscopic image;

[0008] Step 3: Perform template matching between the current frame microscopic image and the stitched adjacent microscopic images to obtain the registration degree and offset of the current frame microscopic image in the direction corresponding to the stitched adjacent microscopic images;

[0009] Step 4: Determine whether the registration and offset of the current frame microscopic image in each direction are correctly matched;

[0010] Step 5: Based on the registration and offset matching results of the current frame microscopic image in each direction, and the absolute position coordinates of the stitched adjacent microscopic images in the stitched large field of view microscopic image, the absolute position coordinates of the current frame microscopic image in the stitched large field of view microscopic image are obtained by using a method based on fast elimination of accumulated errors.

[0011] Step 6: Based on the absolute position coordinates of the current frame microscopic image in the stitched large field of view microscopic image, stitch the current frame microscopic image into the stitched large field of view microscopic image to complete the real-time update and display of the stitched large field of view microscopic image;

[0012] Step 7: Repeat steps 1 to 6 until the large field-of-view microscopic images are stitched together.

[0013] Compared with the prior art, the present invention has the following beneficial technical effects:

[0014] 1. This invention can perform local microscopic image stitching while acquiring microscopic images, and display the stitched microscopic images in real time;

[0015] 2. In the process of image stitching, the present invention uses a method based on rapidly eliminating accumulated errors for image positioning, thereby effectively eliminating the impact of accumulated errors on the stitching effect.

[0016] 3. This invention corrects the brightness of the acquired microscopic images by performing brightness matrix calibration in advance, which can effectively eliminate the problem of inconsistent brightness in different parts of large-field microscopic images;

[0017] 4. This invention effectively solves the problem of the microscope objective and the motion platform not being completely perpendicular in practical applications by calibrating the positional relationship between the microscope objective and the motion platform in advance and correcting the position of the acquired microscope images. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a real-time microscopic image stitching method for rapidly eliminating accumulated errors in an embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of the snake-like scanning method from top to bottom in an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of the serpentine scanning method from left to right in an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of neighborhood matching of microscopic images in an embodiment of the present invention;

[0023] Figure 5 This is a schematic diagram of matching direction j=0 in an embodiment of the present invention;

[0024] Figure 6 This is a schematic diagram of the matching direction j=1 in an embodiment of the present invention;

[0025] Figure 7 This is a schematic diagram of the matching direction j=2 in an embodiment of the present invention;

[0026] Figure 8 This is a schematic diagram of the matching direction j=3 in an embodiment of the present invention;

[0027] Figure 9 This is a schematic diagram of the position index of the microscopic image in an embodiment of the present invention.

[0028] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] It should be noted that all directional indicators (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicator will also change accordingly.

[0031] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0032] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0033] This embodiment discloses a real-time microscopic image stitching method for rapidly eliminating accumulated errors. This stitching method is established after image stitching has been performed for a period of time, i.e., when a large-field-of-view microscopic image with a partially stitched portion is currently available (hereinafter defined as "stitched large-field-of-view microscopic image"). Reference Figure 1 The real-time stitching method for microscopic images in this embodiment specifically includes the following steps 1 to 7.

[0034] Step 1: Obtain the current frame microscopic image obtained from the current frame scan, and perform brightness correction and position correction processing on the current frame microscopic image.

[0035] In this embodiment, to quickly obtain a sequence of overlapping original microscopic images, two scanning methods can be adopted: a top-down serpentine scanning method and a left-to-right serpentine scanning method. Assuming the sample includes 3×3 fields of view to be scanned, the top-down serpentine scanning method... Figure 2 The scanning direction indicated by the middle arrow is a serpentine scanning pattern from left to right. Figure 3 The scanning direction is indicated by the middle arrow. In practice, the original microscopic image sequence obtained from the scan can be stored in the image stack first. When performing real-time stitching of microscopic images, the microscopic images are extracted frame by frame from the image stack according to the time sequence and used as the current frame microscopic image.

[0036] During the scanning process, considering the influence of the lens and light source on microscopic imaging, the acquired microscopic images will exhibit uneven brightness. This will result in different brightness levels in different parts of the generated large-field-of-view microscopic image. Therefore, before stitching the microscopic images, it is necessary to perform brightness correction on the original current frame microscopic image. In this embodiment, the brightness correction method is to multiply the original current frame microscopic image by a pre-calibrated brightness correction matrix to complete the brightness correction, that is:

[0037]

[0038] in, This represents the current frame photomicrograph after brightness correction, where I represents the original current frame photomicrograph, and B represents the original photomicrograph. C This represents the brightness correction matrix.

[0039] The brightness correction matrix needs to be calibrated in advance, and each microscope device only needs to be calibrated once. Considering that the acquired microscope images have a certain amount of noise, this embodiment uses the method of averaging multiple images to denoise the image. The specific steps are as follows:

[0040] First, multiple blank microscopic images without impurities are acquired using a microscope, and the average of these blank microscopic images is calculated to obtain a standard microscopic image.

[0041] Then, the standard microscopic image is converted into a grayscale image, and the average brightness of the grayscale image is calculated. Simultaneously, iterate through the grayscale image to find the value V with the highest brightness. max Then, the brightness correction matrix is ​​calculated as follows:

[0042]

[0043] Among them, B C (x, y) represents the brightness correction matrix B. C The brightness correction coefficient at the (x, y) coordinate in the grayscale image, where G(x, y) represents the grayscale value at the (x, y) coordinate in the grayscale image.

[0044] After brightness correction of the current frame of the microscopic image, position correction is performed. To improve stitching efficiency, most existing microscopic image stitching methods assume a perpendicular relationship between the microscope objective and the motion platform, meaning the motion platform coordinate system and the imaging plane coordinate system are parallel. This means that only the horizontal offset needs to be considered during matching, resulting in poor stitching quality. In practical applications, the microscope objective and the motion platform cannot be perfectly perpendicular; there is an angle between the motion platform coordinate system and the imaging plane coordinate system. Therefore, this embodiment calibrates the positional relationship between the microscope objective and the motion platform beforehand to obtain a position transformation matrix. Multiplying this matrix by the brightness-corrected current frame of the microscopic image achieves position correction, resulting in both brightness-corrected and position-corrected current frame microscopic images.

[0045] The position transformation matrix also needs to be pre-calibrated, and each microscope device only needs to be calibrated once. The calibration method can be Zhang Zhengyou's calibration method. Of course, other methods can also be used to calibrate the position transformation matrix in specific applications.

[0046] It is worth noting that in practical applications, it is not limited to obtaining the current frame microscopic image from the image stack before performing brightness and position correction. Alternatively, the brightness and position of the scanned microscopic image can be corrected first before storing it in the image stack.

[0047] Step 2: Obtain the stitched large field-of-view microscopic image, and obtain the stitched adjacent microscopic images that are adjacent to the current frame microscopic image in the stitched large field-of-view microscopic image.

[0048] Because the scanning methods used when acquiring microscopic images are top-down and left-to-right serpentine scanning, if a top-down or bottom-up serpentine scanning method is used, for the current frame of the microscopic image, the adjacent stitched microscopic images are located to its left (or right) and above (or below) the stitched large-field-of-view microscopic image, or only to its left or right (i.e., when the current frame of the microscopic image is the first frame of a column), or only above or below (i.e., when the current frame of the microscopic image is in the first column). If a left-to-right or right-to-left serpentine scanning method is used, for the current frame of the microscopic image, the adjacent stitched microscopic images are located to its left (or right) and above (or below) the stitched large-field-of-view microscopic image, or only above or below (i.e., when the current frame of the microscopic image is the first frame of a row), or only to its left or right (i.e., when the current frame of the microscopic image is in the first row).

[0049] Step 3: Perform template matching based on the current frame microscopic image and the stitched adjacent microscopic images to obtain the registration degree and offset of the current frame microscopic image in the direction of the corresponding stitched adjacent microscopic images.

[0050] In this embodiment, a template matching algorithm based on gradient integral images is used to sequentially calculate the registration degree and offset between each pair of adjacent microscopic images. Assuming the sample comprises 3×3 scanned regions, the microscopic images at the four corner points of the scanned region have two adjacent microscopic images, the microscopic images at the first row, first column, last row, and last column (excluding the four corner points) of the scanned region have three adjacent microscopic images, and all other locations in the scanned region have four adjacent microscopic images, i.e., ... Figure 4 As shown.

[0051] In the specific implementation process, the process of calculating the registration accuracy and offset of the current frame microscopic image in the direction of the corresponding stitched adjacent microscopic images is as follows:

[0052] Step 301: Define the current frame microscopic image as the image to be matched, and define the stitched adjacent microscopic images in direction j as the reference image j, where j = 0, 1, 2, 3; direction j = 0 indicates that the reference image is located above the image to be matched, i.e. Figure 5 As shown; direction j=1 indicates that the reference image is located to the left of the image to be matched, i.e. Figure 6 As shown; direction j=2 indicates that the reference image is located below the image to be matched, i.e. Figure 7 As shown; direction j=3 indicates that the reference image is located to the right of the image to be matched, i.e. Figure 8 As shown;

[0053] Step 302: Based on the matching direction, extract the region of interest (ROI) as a template image from the overlapping region of the reference image j; wherein, when j=0 or j=2, the overlapping region is 5% to 10% of the height of the reference image j; when j=1 or j=3, the overlapping region is 5% to 10% of the width of the reference image j; that is... Figures 5 to 8 As shown; since the scanning method set in this embodiment is a top-to-bottom serpentine scanning method, the matching direction of the image to be matched is 0 and 1 or 2 and 1 in the actual matching process;

[0054] Step 303: Obtain the corresponding position of the center point of the region of interest in the image to be matched. Using this position as the reference point, move the reference point in the image to be matched along the horizontal or vertical direction (i.e., along the width or height of the microscopic image) according to the set step size.

[0055] Step 304: Each time the reference point is moved, a region of the same size as the template image is selected in the image to be matched, with the reference point as the center point. The registration degree between the region and the template image is calculated, and the offset of the reference point is calculated at this time. The calculated registration degree and offset are then saved in the registration degree dataset and the offset dataset, respectively. The registration degree can be calculated using the standard squared difference matching method, the standard correlation matching method, or the normalized cross-correlation matching method, etc.

[0056] Step 305: When the reference point moves to the set boundary, stop moving and take the largest registration degree in the registration degree dataset and its corresponding offset as the registration degree and offset of the image to be matched in direction j.

[0057] When the image to be matched does not have a reference image j in the stitched large field-of-view micrograph, the registration degree and offset of the image to be matched in direction j are defined as 0.

[0058] In this embodiment, the gradient integral map method is used to calculate the region of interest, specifically:

[0059] First, calculate the gradient of the overlapping region of the reference image j. Specifically, the gradient of the image can be calculated using operators such as Sobel, Roberts, and Laplacian.

[0060] Secondly, after calculating the gradient of the overlapping region of the reference image j, the integral image of the gradient is then calculated. The formula for calculating the integral image is as follows:

[0061] I(x,y)=I(x-1,y)+I(x,y-1)-I(x-1,y-1)+g(x,y)

[0062] Where I(x,y)) represents the integral value at (x,y) in the integral image, I(x-1,y) represents the integral value at (x-1,y) in the integral image, I(x,y-1) represents the integral value at (x,y-1) in the integral image, and g(x,y) represents the gradient value at (x,y) in the image gradient.

[0063] Finally, based on the integral image, the sum of gradient values ​​in a rectangular region of a certain size at any location in the image gradient is calculated. The location of the rectangular region with the largest sum of gradient values ​​is the region of interest. Using the integral image can greatly speed up the calculation of the sum of gradient values ​​in any rectangular region of the image gradient, because it only requires one calculation.

[0064] This embodiment uses the template matching method to calculate the registration and offset of two adjacent microscopic images. The main reason is that the template matching method has very high matching accuracy and calculation efficiency, which well satisfies the balance between efficiency and accuracy in the image stitching process.

[0065] Step 4: Determine whether the registration and offset of the current frame microscopic image in each direction are correctly matched. The specific implementation process is as follows:

[0066] For the matching result determination of the current frame microscopic image in direction j, if the registration degree of the current frame microscopic image in direction j is greater than the first threshold and the offset of the current frame microscopic image in direction j is greater than the second threshold, then the current frame microscopic image is determined to be correctly matched in direction j; otherwise, the current frame microscopic image is determined to be incorrectly matched in direction j. This embodiment compares the registration degree and offset with the first threshold and the second threshold respectively, taking into account the following two factors: one is that when the overlapping area of ​​the reference image or the image to be matched has no target or very few target objects, noise is easily identified as a target point; the other is that the content of the microscopic image is simple and contains a large number of similar targets, which will generate a large number of similar target regions.

[0067] Step 5: Based on the registration and offset matching results of the current frame microscopic image in each direction, and the absolute position coordinates of the adjacent stitched microscopic images in the stitched large field of view microscopic image, the absolute position coordinates of the current frame microscopic image in the stitched large field of view microscopic image are obtained by using a method based on fast elimination of accumulated errors.

[0068] In the specific implementation process, based on the matching results of the current frame microscopic image in various directions, different methods are used to calculate the absolute position coordinates of the current frame microscopic image in the large field of view microscopic image. The specific implementation process can be divided into the following two cases based on the matching results of the current frame microscopic image in various directions:

[0069] Scenario 1: When the current frame of the microscopic image is correctly matched in at least one direction, the calculation process of the absolute position coordinates of the current frame of the microscopic image in the stitched large field-of-view microscopic image specifically includes the following steps:

[0070] Step 501: Calculate the confidence level of the current frame microscopic image in each direction, as follows:

[0071] For adjacent stitched microscopic images whose orientations are not correctly matched or whose absolute position coordinates are not determined, the confidence level of the current frame microscopic image in the corresponding orientation is determined to be 0.

[0072] For a correctly matched orientation of an adjacent stitched micrograph with known absolute position coordinates, the confidence level of the current frame micrograph in the corresponding orientation is calculated based on the adjacent stitched micrograph and the preceding M-1 stitched micrographs.

[0073]

[0074] Where, m j This indicates the confidence level of the current frame microscopic image in direction j, s ij This represents the registration degree of the i-th frame micrograph in the j-th direction;

[0075] Step 502: Based on the offset of the current frame microscopic image in each direction and the absolute position coordinates of the adjacent stitched microscopic images, calculate the corresponding absolute position coordinates of the current frame microscopic image in each direction, as follows:

[0076]

[0077]

[0078]

[0079]

[0080]

[0081]

[0082]

[0083]

[0084] Where (x1, y1), (x2, y2), (x3, y3), and (x4, y4) represent the absolute position coordinates of the current frame microscopic image in the directions j=0, j=2, j=1, and j=3, respectively. d1, d2, d3, and d4 represent the absolute position coordinates of the current frame micrograph image in directions j=0, j=2, j=1, and j=3, respectively, corresponding to the stitched adjacent micrograph images. d1, d2, d3, and d4 represent the offsets of the current frame micrograph image in directions j=0, j=2, j=1, and j=3, respectively. b1 represents the overlapping area of ​​the current frame micrograph image in directions j=0 and j=2, and b2 represents the overlapping area of ​​the current frame micrograph image in directions j=1 and j=3.

[0085] Step 503: Based on the absolute position coordinates and confidence level of the current frame microscopic image in each direction, calculate the absolute position coordinates of the current frame microscopic image within the stitched large-field-of-view microscopic image, as follows:

[0086]

[0087]

[0088] Among them, (x p y p The coordinates () represent the absolute position coordinates of the current frame of the microscopic image within the stitched large field-of-view microscopic image. m0, m2, m1, and m3 represent the confidence levels of the current frame of the microscopic image in the directions j=0, j=2, j=1, and j=3, respectively. This represents the sum of confidence levels of the current frame's microscopic image in all directions;

[0089] Step 504: Mark the current frame microscopic image as having determined absolute position coordinates.

[0090] It is worth noting that during step 502, there may be... In cases where one, two, or three coordinates are unknown, the unknown coordinates can be set to (0, 0) or other arbitrary values. Therefore, if the absolute position coordinates of adjacent microscopic images stitched together in a certain direction are unknown, the confidence level of the current frame microscopic image in that direction is 0. In the absolute position coordinate calculation process of step 503, the corresponding term for the unknown coordinate is 0. Therefore, even if the unknown coordinates are set to (0, 0) or other arbitrary values, it will not affect the final calculation result.

[0091] This embodiment takes into account the cumulative effect of errors during the stitching process, and therefore adopts a method based on quickly eliminating accumulated errors to mitigate the impact of accumulated errors on stitching accuracy. By calculating the confidence level of the neighborhood of the unstitched image in each direction and weighting it into the location information, location information with high confidence level contributes significantly to the final location.

[0092] Scenario 2: When the current frame of the microscopic image is not correctly matched in any of the four directions, the specific process for calculating the absolute position coordinates of the current frame of the microscopic image in the stitched large field of view microscopic image is as follows:

[0093] The absolute position coordinates of the current frame of the microscopic image are obtained by mechanically stitching the previous stitched microscopic image within the stitched large-field microscopic image.

[0094] In the stitching method of this embodiment, the absolute position coordinates of each frame of the photomicrograph are determined frame by frame during the stitching process. Since the previous frame of the photomicrograph is necessarily a stitched adjacent photomicrograph in a certain direction of the current frame of the photomicrograph, when stitching the current frame of the photomicrograph, the absolute position coordinates of the current frame of the photomicrograph in the stitched large field of view photomicrograph can be obtained through one of the above two cases.

[0095] In this embodiment, the mechanical hard stitching method refers to the zero offset between adjacent microscopic images. The position information of the unstitched microscopic images is calculated based on the absolute position coordinates of the reference image, the size information of a single image, the position index of the unstitched microscopic images, and the scaling factor. This is the same as the method for calculating the absolute position coordinates of the initial image described below, and will not be elaborated upon in this embodiment.

[0096] Step 6: Based on the absolute position coordinates of the current frame microscopic image in the stitched large field of view microscopic image, that is, after obtaining the absolute position coordinates of the current microscopic image, the overlapping area of ​​two adjacent microscopic images is fused using a linear fusion method. This allows the current frame microscopic image to be stitched into the stitched large field of view microscopic image, completing the real-time update and display of the stitched large field of view microscopic image.

[0097] Step 7: Repeat steps 1 to 6 until the large field-of-view microscopic images are stitched together.

[0098] In a preferred embodiment, parallel processing technology is employed to improve the overall speed of the stitching process in order to achieve real-time microscopic image stitching. Parallel processing is a processing mechanism that can execute multiple tasks simultaneously. It distributes the various parts of the work to different processing processes (threads) for concurrent execution, which can effectively improve the system's operating speed. The entire stitching process of the microscopic image stitching method designed in this embodiment can be divided into three stages: image acquisition and preprocessing (i.e., step 1), image matching (i.e., steps 2, 3, and 4), and image stitching (i.e., steps 5 and 6). Multi-threading technology is used to enable parallel processing to further improve the operating speed.

[0099] Furthermore, although the stitching method in this embodiment is established after image stitching has been performed for a period of time, i.e., when a large-field-of-view microscopic image with a partially stitched portion is available, most image stitching algorithms require stitching the microscopic image sequence sequentially based on the offsets obtained during the matching process after all images have been matched, in order to obtain a large-field-of-view microscopic image. This involves calculating the absolute position of each microscopic image within the large-field-of-view microscopic image. The common practice in existing technologies is to start from the initial scanning position and sequentially fuse and stitch the matched microscopic images. However, considering that in some cases, the initial scanning position may acquire a blank or low-texture microscopic image, which cannot obtain effective registration and offset during the matching process and is therefore unsuitable as the initial image for stitching, this embodiment searches for the initial image based on a certain number of already matched microscopic images, i.e., after the image stack contains a certain number of microscopic images. The search process is as follows:

[0100] The sum of registration scores for each microscopic image in the image stack in each direction is calculated. The maximum value of the sum of registration scores is compared with a set third threshold. If it is greater than the set third threshold, the microscopic image with the largest sum of registration scores is used as the initial image for stitching. If it is less than or equal to the set third threshold, the number of matched microscopic images is dynamically increased (i.e., image scanning continues to increase the number of microscopic images in the image stack). The above process is repeated until the initial image for stitching is determined.

[0101] Once the initial image for stitching is determined, the microscopic images in the microscopic image set can be stitched together, specifically as follows:

[0102] First, the absolute position coordinates of the initial image within the large-field-of-view microscope image are calculated. Then, the initial image is stitched to the large-field-of-view microscope image based on these absolute position coordinates. Specifically, the calculation of the absolute position coordinates of the initial image within the large-field-of-view microscope image involves:

[0103] x0=(wS x )×i+δ d

[0104] y0=(hS y )×j+δ d

[0105] Where (x, y) represents the absolute position coordinates of the initial image within the large field-of-view micrograph, w and h represent the width and height of a single micrograph, and S... x and S y δ represents the scaling factor in the x and y directions. d This represents the safety value reserved for the backhaul difference, where i and j represent the position indices of the initial image in the m×n areas to be scanned. Figure 9 As shown;

[0106] The scaling factor is one of two calibration parameters that need to be calibrated before stitching. The calculation method is as follows: select several rich microscopic images, calculate the offset between these microscopic images and their four adjacent microscopic images, and then calculate the average value of the offset in the x direction (i.e. the width direction of the microscopic image) and the average value of the offset in the y direction (i.e. the height direction of the microscopic image) and use them as the scaling factor in the x and y directions.

[0107] After calculating the absolute position coordinates of the initial image in the large-field-of-view micrograph, all micrographs with timestamps preceding the initial image in the image stack are arranged sequentially in reverse timestamp order to form the first micrograph image sequence. Simultaneously, all micrographs with timestamps following the initial image are arranged sequentially in timestamp order to form the second micrograph image sequence. Then, referring to the absolute position coordinates of the initial image in the large-field-of-view micrograph, the micrographs in the first micrograph image sequence are stitched together sequentially from the initial image to the large-field-of-view micrograph. Similarly, the micrographs in the second micrograph image sequence are stitched together sequentially from the initial image to the large-field-of-view micrograph. The stitching process is the same as in case one or case two described above, and therefore will not be repeated. It is worth noting that when stitching together the micrographs in the second micrograph image sequence, the second micrograph image sequence is updated in real time as the number of images in the image stack increases. Stitching together existing or updated micrographs in the second micrograph image sequence is the real-time stitching defined in this embodiment.

[0108] Of course, in practice, the microscopic images in the first microscopic image sequence may not be arranged in reverse chronological order; they may be arranged in chronological order or randomly. When the microscopic images in the first microscopic image sequence are arranged in chronological order or randomly, the stitching process of each microscopic image in the first microscopic image sequence is as follows:

[0109] The first step is to define the microscopic images that have been stitched into the large field of view microscopic images as stitched microscopic images, and to define the microscopic images in the current first microscopic image sequence that have not been stitched into the large field of view microscopic images as microscopic images to be stitched.

[0110] The second step is to traverse the first microscopic image sequence from front to back, and sequentially stitch the microscopic images to be stitched into the large field of view microscopic image according to the above-mentioned case one or case two, and mark them as stitched microscopic images.

[0111] The third step is to repeat the traversal and stitching process from the second step, and so on, until all the microscopic images in the first microscopic image sequence have been stitched together.

[0112] After all the microscopic images in the first microscopic image sequence have been stitched together, the microscopic images in the second microscopic image sequence are stitched together into a large field of view microscopic image, starting from the initial image and following the sequence order.

[0113] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A method for real-time stitching of microscopic images to rapidly eliminate accumulated errors, characterized in that, The steps include the following: Step 1: Obtain the current frame microscopic image obtained from the current frame scan, and perform brightness correction and position correction processing on the current frame microscopic image; Step 2: Obtain the stitched large field-of-view microscopic image, and obtain the stitched adjacent microscopic images that are adjacent to the current frame microscopic image in the stitched large field-of-view microscopic image; Step 3: Perform template matching between the current frame microscopic image and the stitched adjacent microscopic images to obtain the registration degree and offset of the current frame microscopic image in the direction corresponding to the stitched adjacent microscopic images; Step 4: Determine whether the registration and offset of the current frame microscopic image in each direction are correctly matched; Step 5: Based on the registration and offset matching results of the current frame microscopic image in each direction, and the absolute position coordinates of the stitched adjacent microscopic images in the stitched large field of view microscopic image, the absolute position coordinates of the current frame microscopic image in the stitched large field of view microscopic image are obtained by using a method based on fast elimination of accumulated errors. When the current frame microscopic image is correctly matched in at least one direction, step 5 is specifically as follows: Step 501: Calculate the confidence level of the current frame microscopic image in each direction, as follows: For the orientation of the stitched adjacent microscopic images that are not correctly matched or for which the absolute position coordinates are not determined, the confidence level of the current frame microscopic image in the corresponding orientation is determined to be 0. For the orientation of the correctly matched adjacent microscopic images where the absolute position coordinates are determined, then based on the orientation of the adjacent microscopic images and the previously stitched images preceding them... The confidence level of the current frame micrograph in the corresponding direction is calculated from the -1 frame micrograph image, and is as follows: in, This indicates that the current frame microscopic image is in the direction Credibility, Indicates the first Frame micrographs at the first Registration in each direction; Step 502: Based on the offset of the current frame microscopic image in each direction and the absolute position coordinates of the stitched adjacent microscopic images, calculate the corresponding absolute position coordinates of the current frame microscopic image in each direction, as follows: in, , , , These respectively represent the orientation of the current frame microscopic image. =0、 =2、 =1、 =3 corresponds to the absolute position coordinates. , , , These respectively represent the orientation of the current frame microscopic image. =0、 =2、 =1、 =3 corresponds to the absolute position coordinates of the stitched adjacent microscopic images. , , , These respectively represent the orientation of the current frame microscopic image. =0、 =2、 =1、 Offset at =3, This indicates that the current frame microscopic image is in the direction =0、 The overlapping region is equal to 2. This indicates that the current frame microscopic image is in the direction =1、 =3 overlapping area; Step 503: Based on the absolute position coordinates and confidence level of the current frame microscopic image in each direction, calculate the absolute position coordinates of the current frame microscopic image within the stitched large field-of-view microscopic image, as follows: in, This indicates the absolute position coordinates of the current frame micrograph within the stitched large field-of-view micrograph. , , , These respectively represent the orientation of the current frame microscopic image. =0、 =2、 =1、 Credibility at =3, This represents the sum of the confidence levels of the current frame micrograph in all directions; Step 504: Mark the current frame microscopic image as having determined absolute position coordinates. Step 6: Based on the absolute position coordinates of the current frame microscopic image in the stitched large field of view microscopic image, stitch the current frame microscopic image into the stitched large field of view microscopic image to complete the real-time update and display of the stitched large field of view microscopic image; Step 7: Repeat steps 1 to 6 until the large field-of-view microscopic images are stitched together.

2. The real-time stitching method for rapidly eliminating accumulated errors in microscopic images according to claim 1, characterized in that, In step 1, the brightness correction and position correction processes are as follows: For the current frame microscopic image Brightness correction is performed as follows: in, This represents the current frame micrograph after brightness correction. Represents the brightness correction matrix; Then examine the microscopic images Position correction is performed as follows: microscopic images Multiplying by the position transformation matrix between the microscope objective and the motion platform yields the current frame microscope image after brightness and position correction.

3. The real-time stitching method for rapidly eliminating accumulated errors in microscopic images according to claim 2, characterized in that, The brightness correction matrix The acquisition process is as follows: Multiple blank microscopic images without impurities are acquired using a microscope, and the average of these blank microscopic images is calculated to obtain a standard microscopic image. The standard microscopic image is converted into a grayscale image, and the average brightness of the grayscale image is calculated. The system then iterates through the grayscale image to find the value with the highest brightness. Then, the brightness correction matrix is ​​calculated as follows: in, Represents the brightness correction matrix middle Brightness correction coefficient at coordinates In a grayscale image The grayscale value at the coordinates.

4. The real-time stitching method for rapidly eliminating accumulated errors in microscopic images according to claim 1, 2, or 3, characterized in that, Step 3 is as follows: Step 301: Define the current frame microscopic image as the image to be matched, and define the direction... The stitched adjacent microscopic images on the image serve as reference images. ,in, =0,1,2,3; Direction =0 indicates that the reference image is located above the image to be matched, and the direction is... =1 indicates that the reference image is located to the left of the image to be matched, and the direction is... =2 indicates that the reference image is located below the image to be matched, and the direction is... =3 indicates that the reference image is located to the right of the image to be matched; Step 302, in the reference image The overlapping region is used to extract the region of interest as a template image; where, when =0 or When =2, the overlapping area is the reference image. 5% to 10% of the height; when =1 or When =3, the overlapping area is the reference image. 5% to 10% of the width; Step 303: Obtain the corresponding position of the center point of the region of interest in the image to be matched, and move the reference point in the image to be matched along the horizontal or vertical direction by a set step size, using the position as the reference point. Step 304: Each time the reference point is moved, a region of the same size as the template image is selected in the image to be matched, with the reference point as the center point. The registration degree between the region and the template image is calculated, and the offset of the reference point at this time is calculated. The calculated registration degree and offset are then saved in the registration degree dataset and the offset dataset, respectively. Step 305: When the reference point moves to the set boundary, it stops moving, and the maximum registration degree in the registration degree dataset and its corresponding offset are taken as the value of the image to be matched in the direction. Registration accuracy and offset; Wherein, the image to be matched does not contain a reference image in the stitched large-field-of-view micrograph. When, the image to be matched is defined in the direction The registration accuracy and offset are 0.

5. The real-time stitching method for rapidly eliminating accumulated errors in microscopic images according to claim 4, characterized in that, In step 302, the region of interest is calculated using a gradient integral map, specifically as follows: Calculate the reference image The gradient of the overlapping region is calculated, and the integral image of the gradient is obtained. Based on the integral image, the sum of gradient values ​​of a rectangular region of a certain size at any position in the image gradient is calculated, and the position of the rectangular region with the largest sum of gradient values ​​is the region of interest.

6. The real-time stitching method for rapidly eliminating accumulated errors in microscopic images according to claim 4, characterized in that, Step 4 is as follows: For the current frame of the microscopic image in the direction Matching on the current frame micrograph, if the current frame micrograph is in the direction The registration accuracy on the current frame micrograph is greater than the first threshold, and the current frame micrograph is in the orientation... If the offset in the current frame of the microscopic image is greater than the second threshold, then it is determined that the current frame of the microscopic image is in the direction of... If the match is correct, otherwise the current frame microscopic image is determined to be in the correct orientation. The above was not matched correctly.

7. The real-time stitching method for rapidly eliminating accumulated errors in microscopic images according to claim 6, characterized in that, When the current frame microscopic image does not match correctly in any of the four directions, step 5 is as follows: The absolute position coordinates of the current frame micrograph are obtained by mechanically stitching the previous stitched micrograph image within the stitched large-field-of-view micrograph image, referencing the absolute position coordinates of the previously stitched micrograph image.

Images