Automatic image splicing method and system for microscopic imaging

Abstract

The invention discloses an automatic image splicing method and system for microscopic imaging, and relates to the technical field of microscopic imaging. The method is executed by an upper computer in an automatic image splicing system, and comprises the following steps: after receiving a plurality of microscopic images from microscopic image acquisition equipment, firstly extracting description information of a corresponding feature point set for each microscopic image by adopting an SIFT feature extraction algorithm, and then extracting description information of a corresponding feature point set for each pair of microscopic images by adopting an SIFT feature extraction algorithm; according to the description information of the two corresponding feature point sets, a FLANNN algorithm is adopted to carry out feature point matching to obtain a corresponding matched feature point set, a HOMOGRAPHY algorithm is used to calculate a corresponding geometric transformation relation, finally image perspective transformation and array fusion are carried out based on a calculation result to obtain a spliced microscopic image, and finally the microscopic image is obtained. Therefore, the purposes of automation and high efficiency of microscopic image splicing can be achieved, inaccurate or discontinuous splicing results caused by human factors are avoided, more accurate and complete detection results are provided for researchers, the acquisition cost is reduced, and the acquisition efficiency is improved.

Description

Technical Field

[0001] This invention belongs to the field of microscopic imaging technology, specifically relating to an automated image stitching method and system for microscopic imaging. Background Technology

[0002] A microscopic imaging system is a comprehensive device that combines optical microscopy with digital imaging technology to achieve multi-scale observation and image recording of samples. With the development of microscopic imaging technology, traditional microscopes have limited fields of view when conducting large-scale observations, especially when high-resolution images are required, as the limited field of view of the microscope cannot meet the observation needs.

[0003] To address these issues, researchers typically employ a method of multiple shots followed by manual stitching for large-scale image stitching. However, this traditional approach is not only inefficient but also susceptible to the influence of operator experience and errors, leading to inaccurate or discontinuous stitching results. Therefore, there is an urgent need for an automated and efficient microscopic image stitching solution to meet the demands for stitching large-scale, high-resolution images during microscopic imaging. Summary of the Invention

[0004] The purpose of this invention is to provide an automated image stitching method, system, computer-readable storage medium, and computer program product for microscopic imaging, in order to solve the problems of low efficiency and inaccurate or discontinuous stitching results caused by human factors in existing image stitching schemes.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] In a first aspect, an automated image stitching method for microscopic imaging is provided, executed by a host computer in an automated image stitching system for microscopic imaging. The automated image stitching system further includes a microscopic image acquisition device communicatively connected to the host computer. The microscopic image acquisition device is used to scan a target sample located in a specified area to obtain multiple microscopic images that partially overlap in pairs. The partial overlap in pairs refers to any microscopic image in the multiple microscopic images partially overlapping with at least one other microscopic image.

[0007] The automated image stitching method includes:

[0008] Receive the multiple microscopic images from the microscopic image acquisition device;

[0009] For each of the multiple microscopic images, the SIFT feature extraction algorithm is used to extract the descriptive information of the corresponding feature point set;

[0010] For each pair of microscopic images in the plurality of microscopic images, based on the description information of the corresponding two feature point sets, the FLANNN algorithm is used to perform feature point matching to obtain the corresponding matching feature point set;

[0011] For each pair of microscopic images in the plurality of microscopic images where the matching feature point set satisfies the image stitching conditions, the HOMOGRAPHY algorithm is first used to calculate the corresponding geometric transformation relationship based on the corresponding matching feature point set. Then, based on the calculation result, the cv2.warpPerspective() function is used to perform image perspective transformation to transform any one of the corresponding microscopic images to the perspective of the other corresponding microscopic image, thereby obtaining a new array of pixel values ​​for the corresponding microscopic image. Finally, the new array of pixel values ​​for the corresponding microscopic image and the array of pixel values ​​for the other microscopic image are merged into one array by array overwriting to obtain the corresponding stitched microscopic image.

[0012] Based on the above-mentioned invention, a novel scheme for microscopic image stitching based on SIFT, FLANN, and HOMOGRAPHY algorithms is provided. This scheme is executed by a host computer in an automated image stitching system for microscopic imaging. After receiving multiple microscopic images from a microscopic image acquisition device, the SIFT feature extraction algorithm is first used to extract descriptive information of the corresponding feature point sets for each image. Then, for each pair of microscopic images, the FLANNN algorithm is used to match feature points based on the descriptive information of the corresponding two feature point sets, obtaining the corresponding matched feature point sets. The HOMOGRAPHY algorithm is then used to calculate the corresponding geometric transformation relationships. Finally, based on the calculation results, image perspective transformation and array fusion are performed to obtain the stitched microscopic image. This achieves automation and high efficiency in microscopic image stitching, avoiding inaccurate or discontinuous stitching results caused by human factors. It provides researchers with more accurate and complete detection results, reduces acquisition costs, improves acquisition efficiency, and facilitates practical application and promotion.

[0013] In one possible design, for each of the multiple microscopic images, the SIFT feature extraction algorithm is used to extract descriptive information of the corresponding feature point set, including:

[0014] For a specific microscopic image among the multiple microscopic images, the SIFT feature extraction algorithm is used to find key points in the corresponding different scale spaces to obtain the corresponding key point set;

[0015] For each key point in the set of key points, the corresponding size information, orientation information, scale information and / or histogram information are calculated;

[0016] The set of key points is used as the feature point set of a certain microscopic image, and the calculation results of each key point are used as the descriptive information of the feature point set.

[0017] In one possible design, for a specific microscopic image among the multiple microscopic images, the SIFT feature extraction algorithm is used to find key points in the corresponding different scale spaces, obtaining the corresponding key point set, including:

[0018] For a specific microscopic image among the multiple microscopic images, the cv2.SIFT_create() function is used to generate the corresponding difference-of-Gaussian pyramid;

[0019] Each pair of adjacent images within the same group in the Gaussian difference pyramid is compared and analyzed to obtain a Gaussian difference image that corresponds one-to-one with each pair of adjacent images within the same group.

[0020] Local extremum detection processing is performed on each of the Gaussian difference images to obtain the extremum set of each Gaussian difference image;

[0021] The extreme point sets of each Gaussian difference image are filtered to obtain a new extreme point set with unstable extreme points and erroneously detected extreme points removed.

[0022] For each extreme point in the new extreme point set, the corresponding pixel in a certain microscopic image is taken as the key point of the certain microscopic image.

[0023] In one possible design, local extremum point detection is performed on each of the Gaussian difference images to obtain the extremum point set of each Gaussian difference image, including:

[0024] For each pixel in each Gaussian difference image, the corresponding pixel is compared and analyzed with all its neighboring pixels. If the difference between the corresponding pixel and all its image domain neighbors and / or all its scale domain neighbors is positive or negative, the corresponding pixel is added to the extreme point set of the image.

[0025] In one possible design, for each key point in the key point set, the corresponding histogram information is calculated, including:

[0026] For a certain key point in the set of key points, the corresponding surrounding pixel region is divided into multiple blocks.

[0027] Calculate the intra-block gradient histogram of the multiple blocks, and use the calculation result as the histogram information of a certain key point.

[0028] In one possible design, for each pair of microscopic images in the plurality of microscopic images where the matching feature point set satisfies the image stitching conditions, the corresponding geometric transformation relationship is first calculated using the HOMOGRAPHY algorithm based on the corresponding matching feature point set, including:

[0029] For each pair of microscopic images in the plurality of microscopic images where the matching feature point set satisfies the image stitching conditions, the cv2.findHomography() function is used to calculate the homography matrix that reflects the geometric transformation relationship based on the corresponding matching feature point set.

[0030] Secondly, the present invention provides an automated image stitching system for microscopic imaging, comprising a host computer and a microscopic image acquisition device that is communicatively connected to the host computer;

[0031] The microscopic image acquisition device is used to scan a target sample located in a specified area to obtain multiple microscopic images that overlap in pairs, wherein the overlap in pairs means that any one of the multiple microscopic images partially overlaps with at least one other microscopic image.

[0032] The host computer is used to execute the automated image stitching method as described in the first aspect or any possible design in the first aspect.

[0033] In one possible design, the microscopic image acquisition device includes a Cartesian robotic arm module and a microscope module that are respectively communicatively connected to the host computer;

[0034] The Cartesian robotic arm module is used to move the microscope module to ensure that the imaging field of view of the microscope module covers the area where the target sample is located, and to parse the instructions from the host computer through the GRBL control board to drive the robotic arm to complete the task of capturing microscopic images.

[0035] The microscope module is used to scan the target sample to obtain the multiple microscopic images.

[0036] In one possible design, the microscope module includes an imaging sensor, an LED light source, and a USB interface;

[0037] The imaging sensor is used to record the optical information of the target sample and convert the optical information into the microscopic image;

[0038] The LED light source is used to provide radiant light to illuminate the target sample;

[0039] The USB interface is used to support the communication connection between the microscope module and the host computer, so as to support real-time data transmission and image saving.

[0040] In one possible design, the host computer includes a user interface unit, a Cartesian robotic arm control unit, an image data acquisition unit, a data management and storage unit, and an image stitching algorithm unit.

[0041] The user interface unit is used to provide users with a graphical user interface developed based on the PyQt5 toolkit, so as to support users to view real-time microscope preview images and perform all operation settings, including scanning area selection, time interval setting, microscope control and / or image display and saving.

[0042] The Cartesian robotic arm control unit is responsible for communication and control with the Cartesian robotic arm module, so as to ensure that the robotic arm can translate the microscope module according to a preset path;

[0043] The image data acquisition unit is responsible for communicating with the microscope module in order to acquire images in real time and display them to the user.

[0044] The data management and storage unit is responsible for the management and storage of image data;

[0045] The image stitching algorithm unit is used to execute the automated image stitching method as described in the first aspect or any possible design in the first aspect.

[0046] Thirdly, the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, perform the automated image stitching method as described in the first aspect or any possible design within the first aspect.

[0047] Fourthly, the present invention provides a computer program product, including a computer program or instructions, which, when executed by a computer, implement the automated image stitching method as described in the first aspect or any possible design in the first aspect.

[0048] The beneficial effects of the above scheme are:

[0049] (1) This invention creatively provides a new scheme for microscopic image stitching based on SIFT, FLANN and HOMOGRAPHY algorithms. The scheme is executed by the host computer in the automated image stitching system for microscopic imaging. After receiving multiple microscopic images from the microscopic image acquisition device, the SIFT feature extraction algorithm is used to extract the description information of the corresponding feature point set for each microscopic image. Then, for each pair of microscopic images, the FLANNN algorithm is used to match the feature points according to the description information of the two corresponding feature point sets to obtain the corresponding matching feature point set. The HOMOGRAPHY algorithm is used to calculate the corresponding geometric transformation relationship. Finally, the image perspective transformation and array fusion are performed based on the calculation results to obtain the stitched microscopic image. This can achieve the goal of automation and high efficiency in microscopic image stitching, avoid inaccurate or discontinuous stitching results due to human factors, provide researchers with more accurate and complete detection results, reduce acquisition costs, improve acquisition efficiency, and facilitate practical application and promotion.

[0050] (2) Automated microbial scanning and image stitching: Considering that in the existing technology, most microscopic image acquisition still requires manual intervention, which is not only inefficient but also prone to errors, and traditional scanning methods usually rely on manual operation of the microscope for observation and photography, and the image stitching process often requires manual processing; therefore, this invention uses a Cartesian robotic arm to automate scanning, which can significantly improve the efficiency and accuracy of scanning, avoid human error, and also automate efficient and accurate image acquisition and stitching of microbial samples, greatly improving the automation and accuracy of experimental operations;

[0051] (3) It can generate high-resolution ultra-large field-of-view images: Considering that traditional microscope images are usually limited by field of view and resolution, especially when it is necessary to view a large range of microbial samples, it is impossible to meet the requirements of detail and field of view at the same time; the present invention can stitch multiple low-resolution microscope images into a complete ultra-resolution image through an image stitching algorithm (i.e., SIFT+FLANN+HOMOGRAPHY). This not only expands the field of view but also preserves more detail information and provides high-quality microbial sample images; this stitching technology enables the observer to observe the entire culture sample comprehensively and at high resolution, improving the accuracy of data analysis;

[0052] (4) Efficient image stitching and processing: Considering that in the existing technology, image stitching usually faces the problems of poor stitching accuracy and long computing time, especially when processing large-scale samples, the stitching speed and image quality are difficult to guarantee. Therefore, this invention optimizes the stitching accuracy and speed by adopting the SIFT+FLANN+HOMOGRAPHY algorithm, and significantly accelerates the stitching process through multi-threading technology, so that it can still maintain high efficiency and accuracy in the case of a large number of image processing, thereby enabling researchers to quickly obtain high-resolution images after stitching, which greatly improves the efficiency of experiments;

[0053] (5) Capable of long-term automated operation: Considering that many existing microbial image acquisition devices and methods are prone to human fatigue or unstable operation during long-term use, making it impossible to perform continuous and accurate work, the automated scanning system of the present invention can support long-term continuous scanning and image stitching, and does not rely on manual operation. It can operate around the clock and is suitable for high-throughput screening and long-term monitoring experiments. In addition, users only need to set relevant parameters, and the system can automatically complete the entire scanning, acquisition and stitching process, ensuring efficient and stable data acquisition.

[0054] (6) The automatic microscope can combine software and hardware, directly control the software interface, set the movement path and observation sample in advance, and obtain super-resolution panoramic microscopic images through image stitching. This system is also widely used in many scenarios that require high-precision detection, such as biology, cell biology, medicine and electronics industry, and has important practical application value. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart illustrating an automated image stitching method for microscopic imaging provided in an embodiment of this application.

[0057] Figure 2 This is a schematic diagram of the structure of an automated image stitching system for microscopic imaging provided in an embodiment of this application.

[0058] Figure 3 An example diagram of the SIFT pyramid structure provided in the embodiments of this application. Detailed Implementation

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these embodiments without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0060] It should be understood that although the terms "first" and "second", etc., may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another. For example, the first object may be referred to as the second object, and similarly, the second object may be referred to as the first object, without departing from the scope of the exemplary embodiments of the invention.

[0061] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, or A and B exist simultaneously. Another example is A, B and / or C, which can mean that any one of A, B, and C or any combination thereof exists. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone or A and B exist simultaneously. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0062] Example

[0063] like Figures 1-2 As shown, the automated image stitching method for microscopic imaging provided in the first aspect of this embodiment can be executed, but is not limited to, by a host computer with certain computing resources in an automated image stitching system for microscopic imaging. The automated image stitching system also includes, but is not limited to, a microscopic image acquisition device that is communicatively connected to the host computer. The microscopic image acquisition device is used to scan a target sample located in a specified area to obtain multiple microscopic images that overlap in pairs. The overlap in pairs means that any one of the multiple microscopic images partially overlaps with at least one other microscopic image.

[0064] like Figure 2As shown, in the specific structure of the automated image stitching system, the microscopic image acquisition device includes, but is not limited to, a Cartesian robotic arm module and a microscope module that are respectively communicatively connected to the host computer; the Cartesian robotic arm module is used to move the microscope module to ensure that the imaging field of view of the microscope module covers the area where the target sample is located, and to parse the instructions from the host computer through the GRBL control board to drive the robotic arm to complete the microscopic image acquisition task; the microscope module is used to scan the target sample to obtain the multiple microscopic images; it can, but is not limited to, use a 50x to 500x magnification USB (Universal Serial Bus) protocol electron microscope (which has the following advantages: small size, easy to combine with Cartesian robotic arm; USB interface is suitable for all computer devices and has good compatibility; 50x to 500x magnification meets the observation needs of most cells, tissues and microorganisms). The GRBL (an open-source embedded firmware developed based on the Arduino / AVR328 chip, mainly responsible for G-code parsing and motion control instruction generation, featuring high performance and low cost) control board is an existing CNC (Computer Numerical Control) motion control solution based on parallel ports. It features high performance, low cost, and cross-platform compatibility, and therefore can be used as the drive subsystem of the Cartesian manipulator in this embodiment to smoothly and accurately move the microscope module on two axes (i.e., the X-axis and the Y-axis) to complete the image acquisition work. The microscope module, as the core module of the microscopic image acquisition device, communicates with the host computer to achieve image acquisition and storage using OpenCV (Open Source Computer Vision Library, an open-source cross-platform computer vision and machine learning software library). Specifically, the microscope module includes, but is not limited to, an imaging sensor, an LED (Light-emitting Diode) light source, and a USB (Universal Serial Bus) interface. The imaging sensor records the optical information of the target sample and converts it into the microscopic image. The LED light source provides emitted light to illuminate the target sample. The USB interface supports the communication connection between the microscope module and the host computer, enabling real-time data transmission and image saving. With the aforementioned LED light source configuration, good microscopic images can be obtained without the need for bottom-transmitted light in traditional optical microscopes, combined with a relatively long exposure time.

[0065] like Figure 1 As shown, the automated image stitching method includes, but is not limited to, the following steps S1 to S4.

[0066] S1. Receive the multiple microscopic images from the microscopic image acquisition device.

[0067] In step S1, the multiple microscopic images can be routinely transmitted from the microscope module to the local host computer via the USB interface after acquisition.

[0068] S2. For each of the multiple microscopic images, the SIFT feature extraction algorithm is used to extract the descriptive information of the corresponding feature point set.

[0069] In step S2, SIFT (Scale-invariant feature transform) aims to extract the position, scale, and rotation invariants of corresponding features by searching for features at different spatial scales. Its input is unaffected by size scaling and ambient lighting, and it has a certain tolerance for noise. Therefore, the SIFT feature extraction algorithm can be applied to this embodiment to extract descriptive information of the feature point sets of each microscopic image. Specifically, for each microscopic image among the multiple microscopic images, the SIFT feature extraction algorithm is used to extract the descriptive information of the corresponding feature point sets, including but not limited to the following steps S21 to S23.

[0070] S21. For a specific microscopic image among the multiple microscopic images, the SIFT feature extraction algorithm is used to find key points in the corresponding different scale spaces to obtain the corresponding key point set.

[0071] In step S21, the key points found are the pixels used as feature points; if no key points are found, the key point set is an empty set. Specifically, for a specific microscopic image among the multiple microscopic images, the SIFT feature extraction algorithm is used to find key points in the corresponding different scale spaces to obtain the corresponding key point set, including but not limited to the following steps S211-215.

[0072] S211. For a specific microscopic image among the multiple microscopic images, use the cv2.SIFT_create() function to generate the corresponding Gaussian difference pyramid.

[0073] In step S211, the cv2.SIFT_create() function is an existing function provided by the OpenCV library for creating SIFT feature detectors, thus enabling scale-space creation. By performing a scale transformation on the microscopic image, a scale-space representation sequence of the microscopic image at different scales can be obtained, such as... Figure 3The pyramid-structured scale-space representation sequence shown can be used to extract the main contour of the sequence (i.e., the Gaussian difference image in subsequent steps) so that the main contour can be used as a feature vector for key point extraction.

[0074] S212. Compare and analyze each pair of adjacent layers in the same group of the Gaussian difference pyramid to obtain each Gaussian difference image that corresponds one-to-one with each pair of adjacent layers in the same group.

[0075] In step S212, the specific process of the comparative analysis is based on existing technical means, for example... Figure 3 As shown.

[0076] S213. Perform local extremum detection processing on each of the Gaussian difference images to obtain the extremum set of each Gaussian difference image.

[0077] In step S213, the aforementioned local extrema can be either local maxima or local minima. More specifically, local extrema detection processing is performed on each of the Gaussian difference images to obtain the extrema set of each Gaussian difference image. This includes, but is not limited to, comparing and analyzing each pixel in each Gaussian difference image with all its neighboring pixels. If the difference between the corresponding pixel and all its image domain neighbors and / or all its scale domain neighbors is positive or negative, then the corresponding pixel is added to the extrema set of its respective image. For example, if the difference between a pixel in each Gaussian difference image and all its image domain neighbors and / or all its scale domain neighbors is positive, it indicates that the pixel is a local maximum; if the difference is negative, it indicates that the pixel is a local minimum. Therefore, both can be added to the extrema set of the image to which the pixel belongs, thus ensuring the accuracy of the local extrema detection results. Furthermore, if no local extrema are detected, the set of extrema can also be an empty set.

[0078] S214. The extreme point sets of each Gaussian difference image are filtered to obtain a new extreme point set with unstable extreme points and erroneously detected extreme points removed.

[0079] In step S214, considering that local extrema are sensitive to noise and edges, it is necessary to filter out unstable extrema and erroneously detected extrema. Furthermore, the specific process of this filtering is a prior art technique and will not be elaborated upon here.

[0080] S215. For each extreme point in the new extreme point set, the corresponding pixel in a certain microscopic image is taken as the key point of the certain microscopic image.

[0081] In step S215, since each layer of the Gaussian difference pyramid is obtained by downsampling a certain microscopic image, the extreme point will have a corresponding pixel in the microscopic image, and thus the key point can be obtained.

[0082] S22. For each key point in the set of key points, calculate the corresponding size information, orientation information, scale information and / or histogram information.

[0083] In step S22, the size information, orientation information, scale information, and / or histogram information are used to form the descriptive information of the corresponding key points / feature points. Describing the key points is a crucial step in subsequent matching; the description is the mathematical definition of the key points. The description scope includes not only the key points themselves but also points in the adjacent regions that contribute to them. Therefore, preferably, for each key point in the key point set, the corresponding histogram information is calculated, including but not limited to the following steps S221 to S222.

[0084] S221. For a certain key point in the set of key points, divide the corresponding surrounding pixel region into multiple blocks.

[0085] S222. Calculate the intra-block gradient histogram of the multiple blocks, and use the calculation result as the histogram information of a certain key point.

[0086] In step S222, the intra-block gradient histograms of the multiple blocks can form a unique vector for a specific key point, which can then be used as an abstract representation of that key point. This can be implemented using the existing function `sift.detectAndCompute()` provided by the OpenCV library. Furthermore, the specific calculation processes for the size, orientation, and scale information are existing methods and will not be elaborated here.

[0087] S23. The set of key points is used as the feature point set of a certain microscopic image, and the calculation results of each key point are used as the descriptive information of the feature point set.

[0088] S3. For each pair of microscopic images in the plurality of microscopic images, based on the description information of the corresponding two feature point sets, the FLANNN algorithm is used to perform feature point matching to obtain the corresponding matching feature point set.

[0089] In step S3, the FLANN (Fast_Library_for_Approximate_Nearest_Neighbors) algorithm is a fast nearest neighbor search algorithm capable of performing fast nearest neighbor search for high-dimensional features. In principle, FLANN optimizes the feature search path by constructing a random kd-tree data structure for the features, thereby reducing search time complexity and improving matching efficiency. In the specific process of feature point matching using the FLANNN algorithm, the FLANN interface provided by the OpenCV library can be called to create a FLANN matcher, such as the cv2.FlannBasedMatcher() function. Feature matching will obtain a list of relationships corresponding to two feature sets: a training set and a query set. To improve matching speed, FLANN trains the matcher to optimize its performance. The training set builds an index tree for the feature set, and the feature points in the query set are matched against the trained matcher; each feature point in the query set will find an optimal match. Furthermore, if two feature point sets of a pair of microscopic images have no matching feature points, then the corresponding matching feature point set can be an empty set.

[0090] S4. For each pair of microscopic images in the plurality of microscopic images where the matching feature point set satisfies the image stitching conditions, based on the corresponding matching feature point set, the HOMOGRAPHY algorithm is first used to calculate the corresponding geometric transformation relationship. Then, based on the calculation result, the cv2.warpPerspective() function is used to perform image perspective transformation to transform any one of the corresponding microscopic images to the perspective of the other corresponding microscopic image, thereby obtaining a new array of pixel values ​​for the corresponding microscopic image. Finally, the new array of pixel values ​​for the corresponding microscopic image and the array of pixel values ​​for the other microscopic image are merged into one array by array overwriting to obtain the corresponding stitched microscopic image.

[0091] In step S4, the image stitching conditions can be predetermined based on the calculation requirements of geometric transformation relationships. For example, it cannot be an empty set, or the number of feature points in the matching feature point set cannot be less than a preset number, etc. Homogrammy, also known as homography, is a concept in projective geometry, also called projective transformation. It maps a point (a three-dimensional homogeneous vector) on one projective plane to another. Homogrammy is a linear transformation about a three-dimensional homogeneous vector, which can be represented by a 3×3 non-singular matrix H. This matrix H is called the homogramming matrix. Using this matrix, a point on one projective plane can be projected onto another plane. Specifically, for each pair of microscopic images in the plurality of microscopic images whose matching feature point sets satisfy the image stitching conditions, the HOMOGRAPHY algorithm is first used to calculate the corresponding geometric transformation relationship based on the corresponding matching feature point set. This includes, but is not limited to: for each pair of microscopic images in the plurality of microscopic images whose matching feature point sets satisfy the image stitching conditions, the cv2.findHomography() function is used to calculate the homography matrix reflecting the geometric transformation relationship based on the corresponding matching feature point set. The cv2.findHomography() function is also provided by the OpenCV library; the homography matrix describes the projection relationship between the two images and is a key step in the subsequent image stitching. The cv2.warpPerspective() function is also provided by the OpenCV library and is used to perform spatial transformation on the image based on the homography matrix, so that the two images to be stitched are precisely aligned, thereby eliminating the deviation and overlapping areas between the images and achieving seamless stitching; the stitched image retains detailed information from different shooting areas, and finally generates a large field-of-view and high-resolution stitched image. Furthermore, after obtaining the stitched microscopic images of each pair of microscopic images, the image stitching of multiple stitched microscopic images can continue based on the aforementioned steps S2 to S4 until the final global stitched microscopic image of the multiple microscopic images is obtained. When a certain microscopic image is located in multiple pairs of microscopic images that meet the image stitching conditions in the matching feature point set, multiple other microscopic images in the multiple pairs of microscopic images can be transformed to the perspective of the certain microscopic image based on the aforementioned step S4 to obtain a new array of pixel values ​​of the multiple other microscopic images. Finally, the new array of pixel values ​​of the multiple other microscopic images and the array of pixel values ​​of the certain microscopic image are merged into one array by array overlay to obtain the stitched microscopic image based on the multiple pairs of microscopic images.

[0092] Therefore, based on the automated image stitching method described in steps S1 to S4 above, a new scheme for microscopic image stitching based on SIFT, FLANN, and HOMOGRAPHY algorithms is provided. This scheme is executed by the host computer in the automated image stitching system for microscopic imaging. After receiving multiple microscopic images from the microscopic image acquisition device, the SIFT feature extraction algorithm is first used to extract the descriptive information of the corresponding feature point sets for each microscopic image. Then, for each pair of microscopic images, the FLANNN algorithm is used to match feature points based on the descriptive information of the corresponding two feature point sets, obtaining the corresponding matched feature point sets. The HOMOGRAPHY algorithm is then used to calculate the corresponding geometric transformation relationship. Finally, based on the calculation results, image perspective transformation and array fusion are performed to obtain the stitched microscopic image. This achieves automation and high efficiency in microscopic image stitching, avoiding inaccurate or discontinuous stitching results caused by human factors. It provides researchers with more accurate and complete detection results, reduces acquisition costs, improves acquisition efficiency, and facilitates practical application and promotion.

[0093] like Figure 2 As shown, the second aspect of this embodiment provides a physical system for implementing the automated image stitching method described in the first aspect, including a host computer and a microscopic image acquisition device that is communicatively connected to the host computer;

[0094] The microscopic image acquisition device is used to scan a target sample located in a specified area to obtain multiple microscopic images that overlap in pairs, wherein the overlap in pairs means that any one of the multiple microscopic images partially overlaps with at least one other microscopic image.

[0095] The host computer is used to execute the automated image stitching method as described in the first aspect.

[0096] In one possible design, the microscopic image acquisition device includes a Cartesian robotic arm module and a microscope module that are respectively communicatively connected to the host computer;

[0097] The Cartesian robotic arm module is used to move the microscope module to ensure that the imaging field of view of the microscope module covers the area where the target sample is located, and to parse the instructions from the host computer through the GRBL control board to drive the robotic arm to complete the task of capturing microscopic images.

[0098] The microscope module is used to scan the target sample to obtain the multiple microscopic images.

[0099] In one possible design, the microscope module includes an imaging sensor, an LED light source, and a USB interface;

[0100] The imaging sensor is used to record the optical information of the target sample and convert the optical information into the microscopic image;

[0101] The LED light source is used to provide radiant light to illuminate the target sample;

[0102] The USB interface is used to support the communication connection between the microscope module and the host computer, so as to support real-time data transmission and image saving.

[0103] In one possible design, the host computer includes, but is not limited to, a user interface unit, a Cartesian robotic arm control unit, an image data acquisition unit, a data management and storage unit, and an image stitching algorithm unit;

[0104] The user interface unit is used to provide users with a graphical user interface developed based on the PyQt5 toolkit, so that users can view real-time microscope preview images and set all operations. These operations include, but are not limited to, scanning area selection, time interval setting, microscope control, and / or image display and saving (i.e., users can view real-time microscope preview images and adjust the microscope position or set other parameters as needed; users can set the scanning path through the interface so that the robotic arm can be automatically instructed to move to a preset position for image acquisition).

[0105] The Cartesian robotic arm control unit is responsible for communication and control with the Cartesian robotic arm module, so as to ensure that the robotic arm can translate the microscope module according to a preset path (i.e., for example, by cooperating with the GRBL control board of the robotic arm through G-code commands, the movement of the microscope in the scanning area can be precisely controlled to achieve automated scanning).

[0106] The image data acquisition unit is responsible for communicating with the microscope module in order to acquire images in real time and display them to the user.

[0107] The data management and storage unit is responsible for the management and storage of image data (thus ensuring that the acquired and stitched image files can be efficiently stored, classified, and retrieved).

[0108] The image stitching algorithm unit is used to execute the automated image stitching method as described in the first aspect.

[0109] The working process, working details and technical effects of the aforementioned system provided in the second aspect of this embodiment can be found in the automated image stitching method described in the first aspect, and will not be repeated here.

[0110] The third aspect of this embodiment provides a computer-readable storage medium storing instructions comprising the automated image stitching method as described in the first aspect. Specifically, the computer-readable storage medium stores instructions that, when executed on a computer, perform the automated image stitching method as described in the first aspect. The computer-readable storage medium refers to a data storage medium, and may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0111] The working process, working details and technical effects of the aforementioned computer-readable storage medium provided in the third aspect of this embodiment can be found in the automated image stitching method described in the first aspect, and will not be repeated here.

[0112] This fourth aspect of the embodiment provides a computer program product, including a computer program or instructions, which, when executed by a computer, implement the automated image stitching method as described in the first aspect. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0113] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An automated image stitching method for microscopic imaging, characterized in that, Be executed by a host computer in an automated image stitching system for microscopic imaging, wherein the automated image stitching system further comprises a microscopic image acquisition device communicatively connected to the host computer, the microscopic image acquisition device being configured to scan a target sample located in a designated area to obtain a plurality of microscopic images each partially overlapping with at least one other microscopic image; The automated image stitching method comprises: receiving the plurality of microscopic images from the microscopic image acquisition device; for each microscopic image in the plurality of microscopic images, extracting a corresponding set of feature points using a SIFT feature extraction algorithm to obtain description information of the set of feature points; for each pair of microscopic images in the plurality of microscopic images, performing feature point matching using a FLANNN algorithm based on the description information of the corresponding two sets of feature points to obtain a corresponding set of matched feature points; for each pair of microscopic images in the plurality of microscopic images and satisfying an image stitching condition, using a HOMOGRAPHY algorithm to calculate a corresponding geometric transformation relationship based on the corresponding set of matched feature points, then performing image perspective transformation using a cv2.warpPerspective() function based on the calculation result to transform any one of the corresponding microscopic images to the perspective of the other corresponding microscopic image to obtain a new array of pixel values of the any one microscopic image, and finally fusing the new array of pixel values of the any one microscopic image and the array of pixel values of the other microscopic image into one array through array overlaying to obtain a corresponding stitched microscopic image.

2. The automated image stitching method of claim 1, wherein, For each microscopic image in the plurality of microscopic images, a SIFT feature extraction algorithm is used to extract the description information of the corresponding set of feature points, comprising: For a certain microscopic image in the plurality of microscopic images, a SIFT feature extraction algorithm is used to find key points in the corresponding different scale spaces to obtain a corresponding set of key points; For each key point in the set of key points, the size information, direction information, scale information and / or histogram information of the key point are calculated; The set of key points is taken as the set of feature points of the certain microscopic image, and the calculated results of the key points are taken as the description information of the set of feature points.

3. The automated image stitching method of claim 2, wherein, For a certain microscopic image in the plurality of microscopic images, a SIFT feature extraction algorithm is used to find key points in the corresponding different scale spaces to obtain a corresponding set of key points, comprising: For a certain microscopic image in the plurality of microscopic images, a cv2.SIFT_create() function is used to generate a corresponding Gaussian difference pyramid; Each pair of adjacent two layers of images in the same group in the Gaussian difference pyramid are compared and analyzed respectively to obtain each Gaussian difference image corresponding to each pair of adjacent two layers of images in the same group; Each Gaussian difference image is subjected to local extreme point detection processing to obtain a set of extreme points of the Gaussian difference image; Screening the extreme value point set of each Gaussian difference image to obtain a new extreme value point set from which non-stable extreme value points and false detected extreme value points are removed; For each extreme value point in the new extreme value point set, the corresponding pixel point in the certain micro image is taken as a key point of the certain micro image.

4. The automated image stitching method of claim 3, wherein, Local extreme value point detection processing is respectively performed on each Gaussian difference image to obtain an extreme value point set of each Gaussian difference image, including: For each pixel point in each Gaussian difference image, the corresponding pixel point and all adjacent pixel points are compared and analyzed, and if it is found that the difference between the corresponding pixel point and all image domain adjacent points and / or all scale domain adjacent points is positive or negative, the corresponding pixel point is added to the extreme value point set of the image.

5. The automated image stitching method of claim 2, wherein, For each key point in the key point set, the corresponding histogram information is calculated, including: For a certain key point in the key point set, the corresponding surrounding pixel region is divided into a plurality of sub-blocks; The intra-block gradient histogram of the plurality of sub-blocks is calculated, and the calculation result is taken as the histogram information of the certain key point.

6. The automated image stitching method of claim 1, wherein, For each pair of micro images in the plurality of micro images and the matching feature point set of which satisfies the image stitching condition, the corresponding geometric transformation relationship is calculated using the HOMOGRAPHY algorithm according to the corresponding matching feature point set, including: For each pair of micro images in the plurality of micro images and the matching feature point set of which satisfies the image stitching condition, the corresponding homography matrix reflecting the geometric transformation relationship is calculated using the cv2.findHomography() function according to the corresponding matching feature point set.

7. An automated image stitching system for microscopic imaging, characterized by, The host computer and a micro image acquisition device connected to the host computer are included; The micro image acquisition device is configured to scan a target sample located in a specified region to obtain a plurality of micro images that partially overlap with each other, wherein the partial overlap refers to that any micro image in the plurality of micro images partially overlaps with at least one other micro image. The host computer is configured to execute the automatic image stitching method according to any one of claims 1-6.

8. The automated image stitching system of claim 7, wherein, The micro image acquisition device includes a Cartesian mechanical arm module and a microscope module connected to the host computer respectively; The Cartesian mechanical arm module is configured to move the microscope module to ensure that the imaging field of view range of the microscope module covers the region where the target sample is located, and to parse instructions from the host computer through a GRBL control board to drive the mechanical arm to complete a micro image shooting task; The microscope module is configured to scan the target sample to obtain the plurality of micro images.

9. The automated image stitching system of claim 8, wherein, The microscope module includes an imaging sensor, an LED light source, and a USB interface; The imaging sensor is configured to record optical information of the target sample and convert the optical information into the micro image; The LED light source is configured to provide radiation light to illuminate the target sample; The USB interface is configured to support real-time data transmission and image saving by loading the communication connection between the microscope module and the host computer.

10. The automated image stitching system of claim 8, wherein, The host computer comprises a user operation interface unit, a Cartesian mechanical arm control unit, an image data acquisition unit, a data management and storage unit, and an image stitching algorithm unit. The user operation interface unit is used to provide a graphical user interface developed based on a PyQt5 toolkit for users, so as to support the users to view real-time microscope preview images and make settings of all operations, wherein the all operations include scanning area selection, time interval setting, microscope control, and / or image display and saving. The Cartesian mechanical arm control unit is used to be responsible for communication and control of the Cartesian mechanical arm module, so as to ensure that the mechanical arm can translate the microscope module according to a preset path. The image data acquisition unit is used to be responsible for communication of the microscope module, so as to acquire images in real time and show them to the user. The data management and storage unit is used to be responsible for management and storage of image data. The image stitching algorithm unit is used to execute the automatic image stitching method according to any one of claims 1 to 6.

Images