Method for real-time generation of panoramic images of biological sections in an electron microscope
By increasing image size, determining the ROI region, reducing image size, calculating the homography matrix, and removing black edges, the microscope image stitching process was optimized, solving the problem of time-consuming microscope image stitching and achieving real-time stitching.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FUJIAN CHUANZHENG COMM COLLEGE
- Filing Date
- 2022-09-02
- Publication Date
- 2026-05-29
AI Technical Summary
Existing microscope image stitching methods are computationally intensive and time-consuming, making real-time image stitching impossible.
By enlarging the original image size to ensure that the coordinates of the stitched image are positive, the ROI region is determined, the image is scaled down proportionally and the homography matrix is calculated, a rectangle is constructed by combining the diagonal points and black edges are cut along the boundary, thus optimizing the image stitching process.
It significantly improves image stitching speed, enabling real-time stitching even without a graphics card, reducing computational load and time, and improving stitching efficiency.
Smart Images
Figure CN115578253B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of electron microscope image stitching technology, specifically to a method for real-time generation of panoramic images of biological sections from an electron microscope. Background Technology
[0002] In the fields of biology and medicine, researchers use electron microscopes to observe sections of samples. High-resolution electron microscopes can be used to observe local areas of a section, but cannot provide a global image. While low-magnification lenses can be used to view the entire section, the details are unclear. Obtaining high-resolution microscopic images of the entire section is crucial and valuable. However, creating globally detailed microscopic images is extremely expensive. Therefore, we aim to capture high-resolution, high-precision images by panning the microscope horizontally and vertically, then stitch them together to create a larger field-of-view image with rich and clear details from multiple focal points, facilitating a comprehensive view for researchers and scholars.
[0003] Existing methods for stitching microscope images are complex, computationally intensive, and time-consuming. Therefore, this invention provides a method that improves image stitching speed without utilizing the parallel computing capabilities of a graphics card, enabling real-time image stitching and achieving a smooth result. Summary of the Invention
[0004] To address the above problems, this invention provides a method for real-time generation of panoramic images of biological sections from an electron microscope.
[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:
[0006] A method for real-time generation of panoramic images of biological sections from an electron microscope includes the following steps:
[0007] S1: Enlarge the size of the original image to ensure that the coordinates of the stitched image are all positive.
[0008] S2: Identify the ROI region in the original image;
[0009] S3: Proportionally reduce the ROI region of the original image and the image to be stitched, calculate the stitching transformation homography matrix H1 between the ROI region of the original image and the reduced image to be stitched, and then multiply it by the magnification factor to obtain the homography matrix H2 of the image at the original size. Stitch the original image and the image to be stitched according to the homography matrix H2, and retain the coordinates of the top left and bottom right corners of the original image, the coordinates of the top left corner of the image to be stitched in the stitching area of the original image, and the size of the image to be stitched after processing by the homography matrix H2.
[0010] S4: Remove the black borders from the stitched image;
[0011] S5: Use the image with the blackened edges as the new original image, wait for the newly acquired images to be stitched together, and repeat S1-S5 until the biological slice image acquisition is complete.
[0012] Furthermore, in S1, the size of the original image is enlarged, specifically as follows:
[0013] Construct a rectangular blank area with a width and height larger than the original image, and embed the original image in the center of the rectangular area.
[0014] Furthermore, the distance from the top left corner of the original image to the left boundary of the rectangular blank area is △w, and the distance from the top left corner of the image to the top boundary of the rectangular blank area is △h; the maximum value of △w is less than the width of the image to be stitched, and the maximum value of △h is less than the height of the image to be stitched.
[0015] Furthermore, step S2 includes:
[0016] S21: When the number of stitches is 1, the area where the original image is located is taken as the ROI area;
[0017] S22: When the number of stitches is greater than 1, the area where the previous stitched image is located in the original image is taken as the ROI area.
[0018] Furthermore, the specific steps of S22 are as follows:
[0019] Based on the coordinates (last_x, last_y) of the top left corner of the previous image to be stitched into the larger image, and the width and height of the image after processing by the homography matrix H2, a rectangular region is constructed with (last_x, last_y) as the top left corner and the width and height of the previous image to be stitched as the width and height. This rectangular region is then used as the ROI region.
[0020] Furthermore, in S3, the reduction ratio and the magnification factor are reciprocals of each other.
[0021] Furthermore, S4 performs black border cropping on the stitched images, the specific steps of which include:
[0022] 1) Construct a rectangular area using the coordinates of the top left corner of the image to be stitched in S3 and the size of the image to be stitched after processing by the homography matrix H2. Locate the coordinates of the bottom right corner of this rectangular area. These coordinates are the coordinates of the bottom right corner of the image to be stitched in the original image stitching area.
[0023] 2) Compare the x and y coordinates of the top-left corner of the image to be stitched (preserved in S3) with the top-left corner coordinates of the original image, and select the smaller x value from the x coordinates. min and the smaller value y in the vertical axis minBased on the selected x and y coordinates (x) min y min Locate the first coordinate point;
[0024] 3) Compare the x and y coordinates of the bottom right corner located in step 1) with the bottom right corner coordinates of the original image retained in S3, and select the larger value X in the x coordinate. max and the larger value Y in the vertical axis max Based on the selected x and y coordinates (X) max Y max Locate the second coordinate point;
[0025] 4) Construct a large rectangle using the first and second coordinate points as diagonal points, and cut the black edges along the boundary of the large rectangle area.
[0026] The beneficial effects of this invention are as follows:
[0027] 1) This invention improves the process and method of image stitching, simplifies calculation, solves the problem of long image stitching time, and greatly improves the speed of image stitching. It can also realize real-time stitching of large-pixel full HD images without the aid of a graphics card.
[0028] 2) Image scaling technology speeds up the calculation time of the splicing transformation matrix.
[0029] 3) Compared with the traditional method of scanning and cutting black edges, the present invention significantly shortens the calculation time by determining the diagonal points to construct a rectangle and cutting black edges along the boundary of the rectangle. Attached Figure Description
[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0031] Figure 1 This is a schematic diagram of the process of the present invention;
[0032] Figure 2 This is an illustration of enlarging the original image size.
[0033] Figure 3 This is a schematic diagram of a traditional splicing algorithm;
[0034] Figure 4 This is a diagram illustrating the proportional scaling down of the original image and the image to be stitched together.
[0035] Figure 5A schematic diagram for calculating the homography matrix H1 between the ROI region of the original image and the image to be stitched;
[0036] Figure 6 A schematic diagram for obtaining the homography matrix H2 of an image at its original size;
[0037] Figure 7 This is a schematic diagram illustrating the stitching of the original image and the image to be stitched together based on the homography matrix H2.
[0038] Figure 8 A schematic diagram for locating the first and second coordinate points and constructing a large rectangle;
[0039] Figure 9 This is a schematic diagram illustrating the process of removing black borders from images according to the present invention;
[0040] Figure 10 This is a schematic diagram of the stitching of the initial original image (i.e., the first image) and the image to be stitched together. Detailed Implementation
[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0042] See attached document Figure 1 As shown, a method for real-time generation of panoramic images of biological sections from an electron microscope includes the following steps:
[0043] S1: Enlarge the original image so that the coordinates of the stitched image are all positive; specifically:
[0044] See attached document Figure 2 As shown, a rectangular blank area with a width and height larger than the original image is constructed, and the original image is embedded in the center of this rectangular area. The distance from the top left corner of the original image to the left boundary of the rectangular blank area is Δw, and the distance from the top left corner of the original image to the top boundary of the rectangular blank area is Δh. Δw and Δh are set based on the electron microscope's movement distance. The maximum value of Δw is less than the width of the image to be stitched, and the maximum value of Δh is less than the height of the image to be stitched. The minimum values of Δw and Δh are zero. The smaller the values of Δw and Δh, the smaller the area of the feature points in the first scan. (The image to be stitched is the image acquired by the microscope each time, and its size is fixed.)
[0045] Traditional stitching algorithms, when calculating the transformed coordinates, may produce negative values in the homography matrix due to different overlapping positions. However, computer screen coordinates are always positive, which complicates subsequent calculations by causing data conversion issues (see attached). Figure 3 (As shown). This invention enlarges the size of the original image so that the coordinates of the stitched image are all positive, which facilitates subsequent calculations.
[0046] S2: Determine the Region of Interest (ROI) in the original image; specifically including:
[0047] S21: When the stitching count is 1, the area where the original image is located is taken as the ROI (as shown in the attached image). Figure 10 (as shown);
[0048] S22: When the number of stitching operations is greater than 1, the area containing the previous stitched image in the original image is taken as the ROI region; specifically:
[0049] Based on the top-left corner coordinates (last_x, last_y) of the previous image to be stitched into the larger image, and its width and height dimensions (width, height) after processing with the homography matrix H2, a rectangular region is constructed with (last_x, last_y) as the top-left corner and the width and height of the previous image to be stitched as its width and height. This rectangular region is designated as the Region of Interest (ROI) (see attached). Figure 9 (As shown).
[0050] As the size of the stitched images increases, the computational load also increases. To control the computational load, a coordinate following method is adopted. The system automatically remembers the coordinates (last_x, last_y) of the top-left corner of the previous image when it is stitched into the larger image, as well as the length and width (width, height) of that image after processing with the homography matrix H2. When stitching the next image, a rectangle is formed with this point as the top-left corner and the image size as the width and height, and then stitched with the next acquired image. This method can effectively reduce the computational load and improve the image stitching speed.
[0051] S3: Proportionally reduce the ROI region in the original image to the size of the image to be stitched (as shown in the attached image). Figure 4 As shown in the figure), the stitching transformation homography matrix H1 between the reduced ROI region of the original image and the reduced image to be stitched is calculated (as shown in the attached figure). Figure 5 and 9 (As shown), then multiply by the magnification factor to obtain the homography matrix H2 of the image at the original size (as shown in the attached image). Figure 6(as shown); where the reduction ratio and the magnification factor are reciprocals of each other. For example, when the original image and the image to be stitched are proportionally reduced to 1 / 3 of their original size, the magnification factor is 3, which is the homography matrix H1 multiplied by 3; the original image and the image to be stitched are then stitched together according to the homography matrix H2 (as shown in the attached figure). Figure 7 As shown), and retain the coordinates of the top left and bottom right corners of the original image, the coordinates of the top left corner of the image to be stitched in the stitching area of the original image, and the size of the image to be stitched after processing by the homography matrix H2;
[0052] Image magnification inevitably leads to a large amount of computation during the feature point matching process. By proportionally reducing the image size, the detection time of feature points is reduced, the homography matrix is calculated, and then multiplied by the magnification factor to obtain a new homography matrix, which greatly shortens the image matching time.
[0053] S4: Remove the black borders from the stitched image; refer to the attached document. Figure 8-9 As shown, the specific steps include:
[0054] 1) Construct a rectangular area using the coordinates of the top left corner of the image to be stitched in S3 and the size of the image to be stitched after processing by the homography matrix H2. Locate the coordinates of the bottom right corner of this rectangular area. These coordinates are the coordinates of the bottom right corner of the image to be stitched in the original image stitching area.
[0055] 2) Compare the x and y coordinates of the top-left corner of the image to be stitched (preserved in S3) with the top-left corner coordinates of the original image, and select the smaller x value from the x coordinates. min and the smaller value y in the vertical axis min Based on the selected x and y coordinates (x) min y min Locate the first coordinate point;
[0056] 3) Compare the x and y coordinates of the bottom right corner located in step 1) with the bottom right corner coordinates of the original image retained in S3, and select the larger value X in the x coordinate. max and the larger value Y in the vertical axis max Based on the selected x and y coordinates (X) max Y max Locate the second coordinate point;
[0057] 4) Construct a large rectangle using the first and second coordinate points as diagonal points, and cut the black edges along the boundary of the large rectangle area.
[0058] To facilitate understanding, here is an example. (See attached image.) Figure 8The coordinates of the top left corner of the original image are (△w, △h) and the coordinates of the bottom right corner are (btm_x, btm_y). The coordinates of the top left corner of the newly stitched image are (last_x1, last_y1) and the coordinates of the bottom right corner are (btm_x1, btm_y1).
[0059] First, compare the coordinates (△w, △h) of the top left corner of the original image with the coordinates (last_x1, last_y1) of the top left corner of the stitched image, and select the smaller value x in the horizontal coordinate. min and the smaller value y in the vertical axis min From the diagram, we can see that △w < last_x1 and △h < last_y1, therefore x min =△w,y min = △h; therefore, the coordinates of the first coordinate point are located as (△w, △h).
[0060] Next, the horizontal and vertical left sides of the bottom right corner coordinates (btm_x, btm_y) of the original image and the bottom right corner coordinates (btm_x1, btm_y1) of the stitched image are compared, and the larger value X in the horizontal coordinate is selected. min and the larger value Y in the vertical axis min From the graph, we can see that btm_x < btm_x1 and btm_y < btm_y1, therefore X max =btm_x1,Y max =btm_y1; Therefore, the coordinates of the second coordinate point are located as (btm_x1, btm_y1).
[0061] Construct a large rectangle using the first coordinate point (△w, △h) and the second coordinate point (btm_x1, btm_y1) as diagonal points, and cut the black edges along the boundary of the large rectangle area.
[0062] Compared to the traditional method of scanning and cutting black edges, the method of constructing a rectangle by determining the diagonal points and cutting black edges along the rectangle boundary can significantly shorten the computation time.
[0063] S5: Use the image with the blackened edges as the new original image, wait for the newly acquired images to be stitched together, and repeat S1-S5 until the biological slice image acquisition is complete.
[0064] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for real-time generation of panoramic images of biological sections from an electron microscope, characterized in that, Includes the following steps: S1: Enlarge the original image size, specifically: Construct a rectangular blank area with a width and height greater than the original image, and embed the original image in the center of the rectangular area; S2: Identify the ROI region in the original image; S3: Proportionally reduce the ROI region of the original image and the image to be stitched, calculate the stitching transformation homography matrix H1 of the reduced ROI region of the original image and the reduced image to be stitched, and then multiply it by the magnification factor to obtain the homography matrix H2 of the image at the original size. Based on the homography matrix H2, stitch the original image and the image to be stitched, and retain the coordinates of the top left and bottom right corners of the original image, the coordinates of the top left corner of the image to be stitched in the stitching area of the original image, and the size of the image to be stitched after processing by the homography matrix H2; S4: Remove the black borders from the stitched image; S5: Use the image with black edges removed as the new original image, wait for the newly acquired images to be stitched together, and repeat S1-S5 until image acquisition is complete.
2. The method for real-time generation of panoramic images of biological sections from an electron microscope according to claim 1, characterized in that, The distance from the top left corner of the original image to the left boundary of the rectangular blank area is △w, and the distance from the top left corner of the image to the top boundary of the rectangular blank area is △. h The maximum value of △w is less than the width of the images to be stitched together. h The maximum value is less than the height of the images to be stitched together.
3. The method for real-time generation of panoramic images of biological sections from an electron microscope according to claim 1, characterized in that, Step S2 includes: S21: When the number of stitches is 1, the area where the original image is located is taken as the ROI area; S22: When the number of stitches is greater than 1, the area where the previous stitched image is located in the original image is taken as the ROI area.
4. The method for real-time generation of panoramic images of biological sections from an electron microscope according to claim 3, characterized in that, The specific steps of S22 are as follows: Based on the previous image to be stitched, stitch it to the top left corner of the larger image (coordinates). last_x, last_y ) and its width and height dimensions after processing by the homography matrix H2; in ( last_x, last_y A rectangular area is constructed with the top left corner as the top corner and (width, height) as the width and height. This rectangular area is then used as the ROI region.
5. The method for real-time generation of panoramic images of biological sections from an electron microscope according to claim 1, characterized in that, In S3, the reduction ratio and the magnification factor are reciprocals of each other.
6. The method for real-time generation of panoramic images of biological sections from an electron microscope according to claim 1, characterized in that, In S4, the black borders of the stitched images are cropped. The specific steps include: 1) Construct a rectangular region using the coordinates of the top-left corner of the image to be stitched (preserved in S3) and the size of the image to be stitched after processing by the homography matrix H2, and locate the coordinates of the bottom-right corner of the rectangular region. 2) Compare the x and y coordinates of the top-left corner of the image to be stitched (preserved in S3) with the top-left corner coordinates of the original image, and select the smaller x value from the x coordinates. min and the smaller value y in the vertical axis min Based on the selected x and y coordinates (x) min y min Locate the first coordinate point; 3) Compare the x and y coordinates of the bottom right corner located in step 1) with the bottom right corner coordinates of the original image retained in S3, and select the larger value X in the x coordinate. max and the larger value Y in the vertical axis max Based on the selected x and y coordinates (X) max Y max Locate the second coordinate point; 4) Construct a large rectangle using the first and second coordinate points as diagonal points, and cut the black edges along the boundary of the large rectangle area.