Microscope image information processing method, microscope image information processing system, and computer program

Abstract

Provided is a microscope image information processing method that avoids re-imaging of a sample and can be implemented by an existing computer system.　In a microscope image information processing method, a captured image (50) of a portion of a sample observed using a microscope is acquired and stored in a storage region (32) (ST1), when it is detected that the captured image (50) is stored in the storage region (32), feature point information which is information relating to a feature point in the captured image (50) is calculated, feature point information of a previous captured image and the feature point information of the captured image (50) are used (ST2) to execute matching processing of the feature point of the previous captured image and the feature point of the captured image (50), joining processing is executed on the basis of the result of the matching processing and a plurality of captured images (50) stored in the storage region (32) up to the present time to generate a joined image (52) (ST4), the joined image (52) is output for display, and the processing described above is repeated until imaging of the portion of the sample is terminated.

Description

Microscope image information processing method, microscope image information processing system, and computer program

[0001] The present invention relates to a method for processing microscopic image information.

[0002] A technology called virtual slides has been developed to digitally create high-resolution, large-area images of glass slides observed under a microscope. Because virtual slides are image data, they are easier to handle than storing the actual glass slides. Virtual slides can be used, for example, for remote pathological diagnosis and digital storage of pathological samples.

[0003] There are several methods for generating virtual slides, including offline stitching and real-time stitching (see, for example, Non-Patent Document 1). Offline stitching involves taking multiple images of a wide area of ​​a sample, such as a glass slide, and then stitching the captured images offline to generate a single image. Real-time stitching involves stitching multiple captured images of the sample while observing the slide to generate a single image. Another method for creating virtual slides is to use a device called a slide scanner, which automatically scans glass slides.

[0004] Alessandro Gherardi1 and Alessandro Bevilacqua, “Real-time whole slide mosaicing for non-automated microscopes in histopathology analysis”, [online], March 30, 2013, National Library of Medicine, [Retrieved April 18, 2022], Internet<URL: https: / / www.ncbi.nlm.nih.gov / pmc / articles / PMC3678752 / >

[0005] However, offline stitching does not allow users to preview the stitched images while photographing the slides. Therefore, when attempting to stitch the slides, problems such as stitching failure or missing images often necessitate re-photographing. Real-time stitching also requires dedicated software and a camera compatible with that software; ordinary digital cameras cannot be used. This makes it difficult to flexibly change the photographing conditions during the process. Furthermore, automated, dedicated devices for generating virtual slides, known as slide scanners, are generally very expensive, limiting the number of environments in which they can be installed.

[0006] The present invention has been made in view of the above problems.

[0007] In order to solve the above-mentioned problems, one aspect of the present invention is a microscope image information processing method executed by a computer system, the microscope image information processing method including: a first step of acquiring a photographed image of a portion of a sample observed using a microscope and saving the photographed image in a memory area; a second step of calculating feature point information, which is information relating to feature points of the photographed image, when it is detected that the photographed image has been saved in the memory area; a third step of performing a matching process between the feature points of the previous photographed image and the feature point information of the photographed image using the feature point information of the previous photographed image and the feature point information of the photographed image; a fourth step of performing a joining process based on the result of the matching process and a plurality of the photographed images that have been saved in the memory area up to now, to generate a joined image; and a fifth step of displaying and outputting the joined image, wherein the first step through the fifth step are repeated until the photographing of the portion of the sample is completed.

[0008] Another aspect of the present invention is a microscope image information processing system that performs a first step of acquiring a photographed image of a portion of a sample observed using a microscope and storing the image in a memory area; upon detecting that the photographed image has been stored in the memory area, performs a second step of calculating feature point information that is information regarding feature points of the photographed image; performs a third step of performing a matching process between the feature points of the previous photographed image and the feature point information of the current photographed image using the feature point information of the previous photographed image; performs a joining process based on the result of the matching process and a plurality of the photographed images that have been stored in the memory area up to now to generate a joined image; performs a fifth step of displaying and outputting the joined image; and repeats the first to fifth steps until the photographing of the portion of the sample is completed.

[0009] Another aspect of the present invention is a computer program that causes a computer system to execute the above-described microscope image information processing method.

[0010] 1 is a diagram illustrating an overview of the successive joining process in a microscopic image information processing method according to an embodiment of the present invention. FIG. 1 is a diagram illustrating an overview of the overall construction process in a microscopic image information processing method according to an embodiment of the present invention. FIG. 1 is a diagram illustrating an example of the results of matching a plurality of feature points detected in an Nth image with a plurality of feature points detected in the immediately preceding joined image. FIG. 2 is a diagram conceptually illustrating the recalculation process of a global transformation matrix using a connected undirected graph. FIG. 2 is a diagram conceptually illustrating the recalculation process of a global transformation matrix using a connected undirected graph. FIG. 3 is a diagram illustrating an example of a flowchart of the successive joining process in a microscopic image information processing method according to an embodiment of the present invention. FIG. 4 is a diagram illustrating an example of a joined image (part) generated by the successive joining process (without distortion correction). FIG. 5 is a diagram illustrating an example of a joined image (part) generated by the successive joining process (with distortion correction). FIG. 6 is a diagram illustrating an example of a flowchart of the process in step S120 (processing for image files from the second time onwards). FIG. 7 is a diagram illustrating an example of a flowchart of the overall construction process in a microscopic image information processing method according to an embodiment of the present invention. FIG. 8 is a diagram illustrating an example of a joined image (part) after bundle adjustment (distortion correction) of the overall construction process. FIG. 9 is a diagram illustrating an example of an image generated by joining all images. FIG. 10 is a diagram illustrating an example of a rectangular area divided into multiple tiles. FIG. 1 is a diagram showing the effect of additional matching processing of area-overlapping images in the sequential joining processing. FIG. 2 is a diagram showing the effect of additional matching processing of area-overlapping images in the sequential joining processing. FIG. 3 is a diagram showing the effect of reducing misalignment between images by calculating a Max Spanning Tree and performing simple bundle adjustment in the sequential joining processing. FIG. 4 is a diagram showing the effect of reducing misalignment between images by calculating a Max Spanning Tree and performing simple bundle adjustment in the sequential joining processing. FIG. 5 is a diagram showing the effect of distortion correction in the overall construction processing. FIG. 6 is a diagram showing the effect of distortion correction in the overall construction processing. FIG. 7 is a diagram showing an example of the hardware configuration of a computer device 30.

[0011] An embodiment of the present invention will be described in detail below with reference to the drawings. (Outline of Microscopic Image Processing Method) FIGS. 1 and 2 are diagrams illustrating an outline of a microscopic image processing method according to this embodiment. The microscopic image processing method according to this embodiment can be executed by a microscopic image processing system 1 including a conventional microscope 10, a camera 20, and a computer device 30. The microscopic image processing method according to this embodiment is broadly divided into two parts: a sequential joining process and an overall construction process. The sequential joining process generates a new joined image each time an image of a portion of a glass slide is captured, ultimately generating a joined image of the entire glass slide. The overall construction process is executed after the image capture of the glass slide is completed, and is executed on the joined image that has undergone the sequential joining process. The overall construction process mainly performs various processes to improve the quality of the joined image and saves the processed image. The sequential joining process mainly performs processes with little waiting time, while the overall construction process mainly performs processes that require time or computational cost.

[0012] FIG. 1 is a diagram illustrating an overview of the sequential joining process. While observing a glass slide specimen with a microscope 10, a user sequentially photographs each portion of the glass slide specimen with a camera 20 using camera software 40. Each time an image 50 of each portion of the glass slide specimen is photographed, the camera software 40 writes the photographed image 50 to a hard disk drive (HDD) 32 included in a computer device 30 (ST1). The joining software 42 also monitors the HDD 32 (ST2). Each time an image file 50 is detected as having been written to the HDD 32, the written image file 50 is read from the HDD 32 (ST3), and the joining process is performed (ST4). Each time an image 50 of each portion of the glass slide specimen is photographed and saved to the HDD 32, the photographed image is read from the HDD 32, and the joining process is performed. This process is repeated, and the joined image 52 is updated each time an image is photographed. Each time the joined image 52 is updated, the updated joined image 52 is previewed by the user on a display device such as a monitor. In the microscopic image information processing method according to this embodiment, the stitched image 52 being generated is previewed at any time by the user, allowing the user to confirm at any time that an appropriate stitched image 52 is being generated. This reduces the possibility of having to retake images after the image capture is complete, as occurs in conventional virtual slide generation using offline stitching.

[0013] More specifically, the sequential joining process can be performed as follows. That is, feature points of an image captured previously and stored on the HDD 32 are matched with feature points of a newly captured image 50 captured and stored on the HDD 32, and a transformation matrix between the two images is calculated. Based on the result of the match, feature points of other images stored on the HDD 32 that overlap with the destination region are matched with feature points of the previously captured image 50, and a transformation matrix is ​​calculated. By limiting the calculation target to only images with overlapping destination regions, rather than all stored captured images, it is possible to reduce user waiting time. Furthermore, by performing Max Spanning Tree calculation or simple bundle adjustment as necessary, it is possible to reduce the accumulation of misalignment between images.

[0014] 2 is a diagram illustrating an overview of the overall construction process. For example, when a user inputs that the sequential joining process is complete, the joining software 42 executes a process to improve the image quality (ST5), and stores the joined image 52 after the process execution in the HDD 32 (ST6).

[0015] More specifically, the overall recomposition process may be performed, for example, as follows: (1) bundle adjustment (calculating a transformation matrix that minimizes the misalignment between all feature point pairs and correcting lens distortion of the camera 20); (2) seam calculation (searching for the best breaks between images); (3) exposure correction (correcting exposure time and vignetting between images); (4) blending (combining images so that the breaks between images are less noticeable); and (5) writing the overall recomposition processed image to the HDD 32 of the computer device 30. Some of the above processes are performed by dividing the overall image into small regions (tiles). In this case, (2) to (5) or (3) to (5) may be processed for each tile. This makes it possible to reduce the amount of memory and calculation time used for a single process.

[0016] In the description of this embodiment, a virtual slide generated by imaging a glass slide specimen is used as an example, but this is merely an example. The microscope image information processing method according to this embodiment can also be applied to, for example, imaging cells cultured in a petri dish.

[0017] The sequential joining process and the overall construction process will be described in more detail below. (Sequential Joining Process) Referring to FIG. 3 , in order to perform the (N+1)th sequential joining (N: an integer equal to or greater than 1), a known method is used to detect multiple feature points, calculate feature quantities, and match the detected feature points for the Nth captured image (hereinafter, for convenience of explanation, referred to as the "Last(N) image") 50′ and the (N+1)th captured image (hereinafter, for convenience of explanation, referred to as the "New(N+1) image") 50 newly captured and stored on the HDD 32. FIG. 3 is a diagram showing an example of the results of matching multiple feature points detected in the Last(N) image 50′ with multiple feature points detected in the New(N+1) image 50. Note that, for convenience of explanation, only a portion of the matching result is shown by dashed lines in FIG. 3 . Next, based on the matching result, a transformation matrix R is calculated between the Last(N) image 50′ and the New(N+1) image 50. Ｎ，Ｎ＋１ is calculated. Ｎ，Ｎ＋１ is a matrix (affine transformation matrix) that indicates the movement distance and rotation amount between the Last(N) image 50′ and the New(N+1) image 50. That is, the following relational expression holds:

[0018]

[0019] Here, the matrix

[0020]

[0021] is a matrix representing the feature points of the New(N+1) image 50.

[0022]

[0023] is a matrix representing the feature points of the Last(N) image 50′. Ｎ，Ｎ＋１are sequentially stored in a storage area such as the HDD 32 of the computer device 30 in the form of an array or the like. Ｎ，Ｎ＋１ The degrees of freedom for the components a, b, c, d, e, and f may be reduced depending on the microscope's performance. For example, if the microscope's optical system and moving parts are sufficiently accurate, the relationship between the captured images can be considered a translation, so a = e = 1, b = d = 0, and only c and f may be variables (degrees of freedom: 2). Furthermore, if accuracy is somewhat poor or the slide specimen is loosely fixed, and rotation as well as translation must be considered, a = cos θ, b = sin θ, d = -sin θ, and e = cos θ may be used, and θ, c, and f may be used as variables (degrees of freedom: 3). In cases where accuracy is even poorer or the lens magnification changes during imaging, a = t cos θ, b = sin θ, d = -sin θ, and e = t cos θ may be used, and t, θ, c, and f may be used as variables (degrees of freedom: 4). Setting the appropriate degrees of freedom for the microscope and specimen can reduce the calculation time and memory consumption required for joining and improve joining accuracy.

[0024] Next, the global transformation matrix of the New(N+1) image 50 is calculated. The "global transformation matrix" is information that indicates how much the New(N+1) image 50 has moved and rotated from a specific image, with the New(N+1) image 50 being based on the specific image. Here, the first captured image is used as the reference, and the global transformation matrix of the New(N+1) image 50 is calculated as R Ｎ＋１ Then, the following relationship holds:

[0025]

[0026] That is, the global transformation matrix R of the New(N+1) image 50 Ｎ＋１ is a transformation matrix R between the immediately preceding captured image and the new captured image, calculated in each joining process prior to the Nth successive joining. １，２ , R ２，３ , ...R Ｎ，Ｎ＋１ The calculated global transformation matrix R of the New(N+1) image 50 can be calculated as the product of Ｎ＋１ is used to perform a joining process between the image data of a specific reference image (here, the first captured image) and the image data of the New(N+1) image 50, and a joined image 52 is generated.

[0027] By the way, the global transformation matrix R Ｎ＋１ It is possible to calculate R Ｎ＋１ In order to calculate multiple matrices R １，２ , R ２，３ , ...R Ｎ，Ｎ＋１ In this embodiment, the following processing is further performed in consideration of this point.

[0028] That is, the global transformation matrix R Ｎ＋１ By calculating this, the approximate destination (hereinafter referred to as the "destination area") of the New(N+1) image 50 from the reference image (first image) can be determined. Then, by comparing the destination area of ​​the New(N+1) image with the destination areas of each image calculated in the past, a search is performed to determine whether there are any previous joined images other than the Last(N) image 50' whose destination areas overlap with the New(N+1) image 50. Here, "overlapping destination areas" may mean, for example, that an overlap occurs when the overlapping area between the destination areas exceeds "0," or that an overlap occurs when the overlapping area between the destination areas exceeds a predetermined threshold. If it is determined that the destination region of the Kth (0<K<N) spliced ​​image 50′ overlaps with that of the New(N+1) image 50, feature points are also matched between the New(N+1) image 50 and the Kth spliced ​​image 50′, and a transformation matrix R between the Kth spliced ​​image 50′ and the New(N+1) image 50 is calculated. Ｋ，Ｎ＋１ The above series of processes is called "additional matching process of overlapping images." Based on this information, a connected undirected graph with each image as a vertex is calculated.

[0029] 4A and 4B are diagrams conceptually illustrating the recalculation process of the global transformation matrix using the graph. Before the recalculation of the global transformation matrix, as shown in FIG. 4A, a path was determined that traced each image in the order of the first image, the second image, the third image, the Kth image, the Nth image, and the N+1th image (New(N+1) image 50). By performing additional matching process of area overlapping images, for example, as shown in FIG. 4B, the Kth image and the N+1th image are linked. Then, based on the result of the matching process between the feature points of the Kth image and the feature points of the N+1th image, a transformation matrix R between the Kth image and the N+1th image is calculated. Ｋ，Ｎ＋１ is calculated. Furthermore, the optimal path for tracing the entire image can be determined, for example, by calculating a Max Spanning Tree. A Max Spanning Tree is a spanning tree in a weighted, connected, undirected graph that maximizes the total weight, and is known to be calculated using an algorithm such as the Kruskal algorithm. In the present invention, the number of matching feature points between images can be used as the weight of the graph, but other indicators may also be used as the weight. After calculating the Max Spanning Tree, the center of the tree is set as a new reference. This process results in a shorter overall route (fewer matrix multiplications and therefore less error) when the third image in FIG. 4B is used as the reference image, rather than a path that sequentially traces the first to (N+1)th images as shown in FIG. 4A. Therefore, the third image can be determined as the new reference image. In this case, the following relationship holds:

[0030]

[0031] The global transformation matrix R calculated as above Ｎ＋１By using the above algorithm to sequentially stitch the (N+1)th captured image, errors can be further reduced. (Overall Reconstruction Process) The overall reconstruction process is performed after all of the photographs of the glass slide have been taken, and improves the quality of the stitched image. More specifically, the overall reconstruction process in this embodiment performs the following steps: (1) bundle adjustment (calculating a transformation matrix that minimizes the misalignment of all feature point pairs and correcting lens distortion for the camera 20); (2) seam calculation (searching for the best break between images); (3) exposure correction (correcting exposure time and vignetting between images); (4) blending (combining images so that the break between images is inconspicuous); and (5) writing the overall reconstructed image to the HDD 32 of the computer device 30. Some of the above processes are performed by dividing the overall image into small regions (tiles). In this case, steps (2) to (5) or (3) to (5) may be performed for each tile. This reduces the amount of memory and calculation time required for a single process. (Bundle Adjustment) As described above, feature points are matched in the iterative joining process. The overall error can be calculated by calculating for all feature points the amount of error (reprojection error) between matched feature points when the feature points are moved and rotated as indicated by the transformation matrix. By minimizing this overall error using the least squares method, the iteratively joined image can be made into an image of overall higher quality. As an example, the Levenberg-Marquardt method can be used. The Levenberg-Marquardt method is one of the methods for solving nonlinear least squares problems.

[0032]

[0033] Here, in this embodiment, x ｉ : Component J of the transformation matrix of the image at processing time i ｉ : Jacobian at processing time i (the T on the right shoulder indicates transposition) r ｉ : error between feature points at processing time point i, λ: real number equal to or greater than zero that is adjusted according to the magnitude of the error, and I: identity matrix.

[0034] Using the above formula (1), parameters J and r are calculated for parameter x at a certain processing point. Furthermore, using the calculated parameters J and r, parameters J and r are further calculated for the next parameter x. By repeating this process, a global transformation matrix that minimizes the reprojection error between all matched feature points is calculated, and bundle adjustment can be performed using this. (Process Flow: Sequential Joining Process) FIG. 5 is a diagram showing an example of a flowchart of the sequential joining process.

[0035] First, the microscopic image information processing system 1 receives an input from a user specifying a folder on the HDD 32 to be monitored via an application such as the camera software 40 or the joining software 42 running on the computer device 30 (step S102). The joining software 42 checks for updates to the specified folder at regular intervals (step S104). Unless the joining software 40 receives an input indicating the end of the sequential joining process (step S106: No), the sequential joining process continues and the process proceeds to step S108.

[0036] In step S108, if the joining software 42 determines that the image file in the folder confirmed in step S104 has not been updated (step S108: No), the process returns to step S104. If the joining software 42 determines that the image file has been updated, i.e., if a new image of the slide glass specimen has been captured and saved in the folder (step S108: Yes), the software determines whether the image file update is the first saving of the image file, i.e., whether the first saved image of the slide glass specimen has been saved (step S110). If the image file update is not the first saving (step S110: No), the software executes processing for the second or subsequent saved image file (step S120). This processing will be described in detail later.

[0037] If the image file update is the first time the image file has been saved (step S110: Yes), the saved image file is read from the HDD 32, feature points and feature quantities for the image file are calculated, and the calculation results are stored in an array for saving information on feature points and feature quantities for each image (step S112). At this time, if distortion parameters and vignetting parameters are known in advance, corrections may be made when the image file is read. Distortion may occur in captured images due to the optical lens of the camera 20. It is generally known that the conversion from "undistorted true coordinates x, y, z" to "distorted post-capture coordinates u, v" can be calculated, for example, using the following formula: (See, for example, Zhengyou Zhang. A flexible new technique for camera calibration. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 22(11):1330-1334, 2000. https: / / www.microsoft.com / en-us / research / wp-content / uploads / 2016 / 02 / tr98-71.pdf)

[0038]

[0039] Here, k ｎ , p ｎ , c ｘ , c ｙ , f ｘ , f ｙ are distortion parameters, and these values ​​are estimated before distortion correction is performed. More specifically, before distortion correction is performed, a grid pattern in which squares of the same size are arranged is photographed with the same camera 20 as the camera 20 that photographs the glass slide sample. Then, the parameter k is calculated from the distortion of the photographed image. ｎ , p ｎ , c ｘ , c ｙ , f ｘ , f ｙThe value of can be estimated. Using the estimated parameter values, the distortion of the captured image can be corrected by performing an inverse transformation (u, v) → (x, y) (since this is a two-dimensional system, the Z coordinate is ignored). Note that since there is no analytical inverse function, the inverse transformation can be performed using an algorithm that calculates an approximate solution, such as Newton's method. Also, only some of the parameters may be considered (for example, the parameter k １ and the value of the parameter p １ (For example, only the value of parameter 1 is taken into consideration, and the values ​​of other parameters are regarded as "0"). Distortion correction may also be performed using other distortion models.

[0040] As described above, by correcting the distortion of the captured images and generating a joined image using each of the captured images after the distortion correction, it is possible to reduce misalignment in the joined image. Figures 6A and 6B are diagrams showing an example of a joined image (part) generated by the inventors of the present application using the microscope image information processing method according to this embodiment. Figure 6A shows a joined image obtained by sequentially joining captured images without performing distortion correction, while Figure 6B shows a joined image obtained by sequentially joining captured images after performing distortion correction. The error in the joined image of Figure 6A was approximately 3.32e+05, while the error in the joined image of Figure 6B was approximately 1.57e+04. Performing distortion correction on each captured image before sequential joining significantly reduced the error. The error (reprojection error) was calculated using the following procedure: (1) Using a global transformation matrix, calculate the coordinates of the feature points of each image in the reference coordinate system (the coordinate system of the reference image). (2) Calculate the deviation in the reference coordinate system between the matched feature points. Ideally, when the matched feature points are transferred to the reference coordinate system, they should overlap perfectly, but this does not happen due to distortion, etc. This is an error.

[0041] 5, the image file is then reduced by a predetermined ratio and stored in an array for saving image data (step S114). By storing the reduced image in RAM 33, HDD 32, or the like, it is possible to reduce memory consumption and storage capacity compared to storing an image at the original size. However, the image file may not be reduced in size and full-size image data may be stored in the array, and from this point on, the spliced ​​image may be generated and handled as a full-size image rather than a reduced image.

[0042] The global transformation matrix (unit matrix in this case) of the first captured image reduced in step S114 is stored in an array for saving global transformation matrices (step S116). A preview image is synthesized using the reduced image data saved in step S114 and the global transformation matrix (unit matrix) saved in step S116, and a preview is displayed. Note that a reduced image of the first captured image may also be displayed as a preview (step S118).

[0043] The above process continues until the joining software 42 receives an input from the user via an input device such as a keyboard or mouse indicating the end of the sequential joining process (step S106: Yes) for reasons such as the completion of the sequential joining process (i.e., the completion of virtual slide generation). When the joining software 42 receives an input from the user indicating the end of the sequential joining process, the software ends the sequential joining process and moves on to the overall construction process. (Processing Flow: Processing of Step S120) Figure 7 is a diagram showing an example of a flowchart of the process in step S120 (processing for the second and subsequent image files).

[0044] First, image n, which is an image of a portion of the glass slide specimen stored on the HDD 32 on the nth iteration (n: an integer equal to or greater than 2), is read, and feature points and feature quantities are calculated (step S1202). At this time, as in step S112 of FIG. 5 , if distortion parameters, vignetting parameters, etc. are known in advance, corrections may be made when the image file is read. Next, feature points are matched between image n and image n-1, which is the image captured on the (n-1)th iteration, and a transformation matrix between the two images (hereinafter referred to as an "image pair matrix") is calculated (step S1204). If the processing of step S1204 is successful (step S1206: Yes), the destination region of image n is calculated, and based on this calculation result, one or more images (hereinafter also referred to as "paired images"; image K in the example of FIG. 4B) whose destination regions overlap are selected from all stored images of the glass slide specimen (step S1208). Next, feature points are matched between image n and all paired images selected in step S1208, and the respective image pair matrices (transformation matrix R in the example of FIG. 4B) are calculated. Ｋ，Ｎ＋１ ) is calculated (step S1210). The feature point matching performed in step S1210 is a process with high computational costs, and performing feature point matching on all saved images would result in a long waiting time for the user. Therefore, by performing feature point matching only on images that overlap with the destination regions selected in step S1208, it is possible to reduce the waiting time for the user.

[0045] Next, the feature points and feature quantities of image n are stored in an array (step S1212). Image n is reduced by a predetermined ratio, and the image data of the reduced image is stored in an array (step S1214). The image pair matrix obtained in step S1210 is stored in an array (step S1216). A global transformation matrix is ​​calculated from the image pair matrix stored in step S1216 and stored in an array (step S1218). Note that the global matrix of image n can be calculated as the product of the image pair matrix between image n and its paired image (the image selected in step S1208; image K in the example of FIG. 4B) and the global transformation matrix of its paired image. Furthermore, in step S1218, the global transformation matrix of all saved images may be recalculated as needed. "As needed" may refer to, for example, "every predetermined number of times," "when input is received from the user," "when the reprojection error between all matched feature points reaches a certain value," etc. Furthermore, when the global transformation matrix of all images is recalculated, a Max Spanning Tree may be calculated with the number of feature matches as a weight. The image at the center of the tree (the third image in the example of FIG. 4B ) is set as the new reference image (the global transformation matrix of the third image is the identity matrix), and the global matrix for each image can be calculated by calculating the product of the image pair matrices in the order of the tree's edges. In addition to these, a small number of times (for example, two iterations) of bundle adjustment can be performed to calculate the global transformation matrix. Bundle adjustment can be performed by, for example, minimizing the reprojection error between all matched feature points using the Levenberg-Marquardt algorithm. Furthermore, the calculated image pair matrices can be updated based on the results of recalculating the global transformation matrices for all images (for example, if the global transformation matrices of image A and image B are Ma and Mb, respectively, the image pair matrices between image A and image B are the matrices Ma and Mb). －１ (This can be calculated by multiplying the product by .) This allows the next recalculation to start from a state with fewer errors.

[0046] A preview image is generated using the reduced image data saved in step S1214 and displayed on a display device such as a monitor (step S1220). Here, since the preview images up to image n-1 have already been combined, a preview image may be generated by joining the reduced-size image n to the preview image of image n-1. Furthermore, if the global transformation matrix is ​​recalculated for all saved images in step S1218, the preview images generated up to that point may be discarded and a new preview image may be generated. After this, the process proceeds to step S104 in FIG. 5.

[0047] Furthermore, if the processing of step S1204 is not successful in step S1206 (step S1206: No), feature points are matched between image n and all saved images, and an image pair matrix is ​​calculated (step S1222). As a result, if at least one image pair (one or more image pairs whose destination regions overlap) exists (step S1224: Yes), the processing proceeds to step S1212. If no image pair exists (step S1224: No), the processing proceeds to step S104 in FIG. 5.

[0048] Note that if the processing in step S1204 is not successful (step S1206: No), i.e., if matching of feature points between image n and image n-1 fails, this may occur, for example, when photographing a glass slide is restarted from a different field of view that is significantly different from the field of view photographed up to that point (because photographing the entire area of ​​the glass slide is not necessarily performed in a single stroke). Furthermore, the processing in step S1222 may be terminated once at least one image pair matrix has been calculated. In this case, the processing proceeds to step S1208.

[0049] 5 to 7, the data of the arrays storing the image pair matrices, the arrays storing the reduced images, the arrays storing the global transformation matrices, and the arrays storing the feature points and feature quantities of each image are carried over to the overall construction process described below. Furthermore, by storing all or part of this data as a file in a storage area such as the HDD 32 of the computer device 30, it is possible to postpone the overall construction process and perform it at a later date, or to execute the overall construction process on another computer device. (Process Flow: Overall Construction Process) FIG. 8 is a diagram showing an example of a flowchart of the overall construction process.

[0050] Bundle adjustment is performed on all images on which the sequential joining process has been performed. As described above, bundle adjustment can be performed by the Levenberg-Marquardt method (the above formula (1)) or the like (step S302). Here, if the distortion parameters of the optical lens of the camera 20 are not known in advance, it is possible to correct the optical distortion of the camera 20 by including distortion parameters common to all images in addition to the global transformation matrix as adjustment target parameters for bundle adjustment, and minimizing the reprojection error calculated after correcting the distortion of the feature point coordinates. That is, at this time, the parameter x in the above formula (1) includes the components of the transformation matrix of all images and the common distortion parameters. That is, the parameter x in formula (1) ｉ When distortion is not taken into account, the components (a, b, c, d) of the global transformation matrix of each image are arranged as (x ｉ = (R １ , R ２ , ..., R ｎ ). where (R １ = a １ , b １ , c １ , d １ )) If distortion is taken into account, x ｉ = (R １ , R ２ , ..., R ｎ , k １ , k ２ , p １ , p ２, ...), the distortion parameter of equation (2) is added. This becomes the common distortion parameter. This parameter x ｉ If we calculate the error r using （ｉ＋１） By repeating this process, a more improved global transformation matrix and distortion parameters can be obtained. １ = p １ By temporarily setting k = ... = 0, and repeatedly calculating equation (1), １ , p １ , ... are determined. Although it is also possible to perform calculations assuming distortion parameters specific to each image, using common distortion parameters makes it possible to reduce calculation time and is sufficiently practical when the position at which the slide glass specimen is observed is simply changed without changing the lens of the camera 20. Note that, since the distortion parameters are calculated using only the feature point coordinates in step S302, distortion correction is actually performed on all pixels of each image in the subsequent processing.

[0051] Next, distortion correction is performed on the reduced images (the reduced images stored in the arrays in steps S114 and S1214) (step S304). Note that if the distortion parameters are known in advance of the sequential joining process and distortion correction has already been performed on each captured image in step S112 of FIG. 5 and step S1202 of FIG. 7, the correction in this step is omitted. FIG. 9 shows an example of a joined image obtained by joining the same captured images as those shown in FIGS. 6A and 6B after performing distortion correction by including a common distortion parameter in the parameter x of Equation (1). The error in the joined image of FIG. 6A was approximately 3.32e+05, and the error in the joined image of FIG. 6B was approximately 1.57e+04. The error in the joined image of FIG. 9 was approximately 1.55e+03. It can be seen that the error in the joined image of FIG. 9 is even smaller than the joined image shown in FIG. 6B, in which the captured images were previously subjected to distortion correction and sequential joining.

[0052] Next, the seams between the captured images are calculated using the reduced images (step S306). By calculating the seams, the images can be joined at appropriate gaps. Existing methods for calculating seams can be used, such as a Voronoi diagram, dynamic programming, or graph cut. Because seam calculation takes time, in this embodiment, seam calculation is performed in a global construction process rather than a sequential joining process, so that even long user wait times are acceptable. While using full-size images would require a long time for seam calculation, this embodiment uses reduced images as described above, which contributes to shortening the calculation time.

[0053] Next, the size (width and height) of the image generated by joining all the captured images is calculated. FIG. 10 is a diagram showing an example of an image generated by joining all the captured images. Note that FIG. 10 is simplified for ease of explanation. Suppose there are six images (images 1 to 6), and the image generated by joining images 1 to 6 is image 60 configured to include all of images 1 to 6. In this step, the size (width w, height h) of image 60 is calculated as follows: (1) Using the global transformation matrix and distortion correction parameters obtained in step S302, the destination region (in the reference coordinate system) of each of images 1 to 6 is calculated (this can be calculated from the size (width and height) of each of images 1 to 6), and (2) a rectangular region including all of the destination regions obtained in (1) is calculated. The calculated region is then divided into multiple small regions (hereinafter referred to as "tiles," "tile images," etc.) (step S308). FIG. 11 is a diagram showing an example in which rectangular region 60 is divided into multiple tiles 65 and 66 (two here for ease of explanation). The images intersecting with tiles 65 and 66 are defined as tile image sets. In this example, the tile image set for tile 65 is images 1, 2, 3, 4, and 5, and the tile image set for tile 66 is images 4, 5, and 6. For example, combining tile 65 generates an image larger than the size of tile 65, but the protruding portion is cropped off. The size and shape of the tile can be determined as follows: (1) a rectangle of a predetermined size (e.g., 1024 pixels x 1024 pixels), (2) a user can specify the size in a configuration file, or (3) the size can be automatically determined in consideration of the memory of the computer device 30. Regarding (3), for example, the memory required to process one tile can be estimated by multiplying the number of images included in the tile image set by the size of one full-size image. Therefore, the tile size can be determined so that the required memory does not exceed 10% of the memory available in the software. The shape of the tiles may be a predetermined specific shape (such as a square) or may be a shape similar to each of the images 1 to 6 (such as a rectangle with an aspect ratio of 3:4). In the following steps S310 to S318, the full-size images are used to join the images.Furthermore, since processing the entire full-size image at once would require a large amount of memory and calculation time, the entire image is divided into tiles and steps S310 to S318 are executed for each tile.

[0054] First, in this step, images whose destination regions intersect with the tile being processed are identified. In the example of FIG. 11 , the tile image set for tile 65 is images 1, 2, 3, 4, and 5, and the tile image set for tile 66 is images 4, 5, and 6 (step S310). (The identified set of images is referred to as the "tile image set.") Next, each image file of the tile image set identified in step S310 is read from HDD 32 (step S312). For the tile image set read in step S312, distortion correction is performed based on distortion parameters (which may be known in advance or distortion parameters estimated by including a common distortion parameter in parameter x in equation (1) above). Furthermore, if the vignetting parameters are known in advance, image correction is performed based on the vignetting parameters. If the vignetting parameters are not known in advance, exposure correction is performed in the next step (step S314). The exposure correction method may be, for example, the method disclosed in M. Uyttendaele, A. Eden and R. Skeliski (2001) "Eliminating ghosting and exposure artifacts in image mosaics," Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, p. II (https: / / www.cs.jhu.edu / ~misha / ReadingSeminar / Papers / Uyttendaele01.pdf).

[0055] All images in the tile image set are moved and positioned according to the global transformation matrix calculated in step S302 (step S316). At this time, all images are joined using the seam information calculated in step S306. Blending may also be performed during the joining process. Various known blending methods, such as linear blending and multiband blending, may be used for blending. For example, the method disclosed in Richard Szeliski (2007), "Image Alignment and Stitching: A Tutorial," Foundations and Trends (R) in Computer Graphics and Vision: Vol. 2, No. 1, pp. 1-104 (https: / / www.nowpublishers.com / article / Details / CGV-009), may be used.

[0056] The image (tile) synthesized in step S316 is saved as a temporary file in a storage area such as the HDD 32 of the computer device 30 (step S318). The above steps S310 to S318 are executed for all tiles.

[0057] The temporary files saved for each tile in step S318 are read from a storage area such as the HDD 32 of the computer device 30 and written sequentially to the final file (step S320). As a result, the final virtual slide image that has undergone the overall configuration process is saved in a storage area such as the HDD 32 of the computer device 30.

[0058] 12A and 12B illustrate the effect of additional matching processing of overlapping area images in the sequential joining process. In the graph of FIG. 12A, the horizontal axis represents the image registration order, and the vertical axis represents the number of images for which matching was attempted. In the graph of FIG. 12B, the horizontal axis represents the image registration order, and the vertical axis represents the time required for matching. Three methods were used in the sequential joining process. The first method is a method in which, when a new image (New(N+1) image) is detected, matching is attempted with all images joined up to that point. The second method is a method in which additional matching processing of overlapping area images is performed. The third method is a method in which, when a new image (New(N+1) image) is detected, matching is attempted with only the most recently joined image (Last(N) image). In the first method, the number of images to be registered increases linearly as the image registration order increases. This increases the time required for matching, gradually increasing the user's waiting time. In the second method, the number of images for which matching was attempted fluctuates but remains approximately constant regardless of the image registration order. Therefore, the time required for matching is also nearly constant. In the third method, the number of images attempted to be matched is always 1, and therefore the time required for matching is also constant. However, when using the third method, the positional relationship between the new image (New(N+1) image) and images other than the immediately preceding spliced ​​image (Last(N) image) is unknown, so misalignment caused by repeated splicing or optical system distortion cannot be corrected by Max Spanning Tree calculation or bundle adjustment. In this way, by using the second method (additional matching processing of area-overlapping images), it is possible to correct misalignment between images while maintaining a constant time required for matching (user waiting time).

[0059] 13A and 13B show the effect of reducing misalignment between images during the sequential joining process using Max Spanning Tree calculation and simple bundle adjustment. FIG. 13A shows a graph for HE-stained slides of pancreatic cancer (320 images), and FIG. 13B shows a graph for HE-stained slides of liver (59 images). These graphs show the magnitude of misalignment (RMS: root mean square) between images at the end of the sequential joining process when Max Spanning Tree calculation and simple bundle adjustment are performed every 10 joins, and when these are not performed. Both graphs show that the misalignment between images is significantly reduced when Max Spanning Tree calculation and simple bundle adjustment are performed. If the misalignment is too large, it can be difficult for users to confirm whether an appropriate joined image has been generated. Therefore, it is effective to reduce the misalignment by Max Spanning Tree calculation or simple bundle adjustment as needed.

[0060] Figures 14A and 14B show the effect of distortion correction in the overall construction process. These figures particularly relate to the processing of step S304 described above. Figure 14A shows a graph for HE-stained slides of pancreatic cancer (320 images), and Figure 14B shows a graph for HE-stained slides of liver (59 images). These figures also show the magnitude of misalignment (RMS) when distortion correction is performed by including a common distortion parameter in parameter x of equation (1) described above, and when distortion correction is not performed. Note that RMS here is a numerical value that represents the misalignment per feature point pair in pixels. It can be seen that when distortion correction is performed, the misalignment is reduced to 0.5 pixels or less. This is a level of misalignment that is invisible to the naked eye, demonstrating the extremely significant effect of distortion correction.

[0061] (Configuration of Microscope Image Information Processing System) As shown in FIGS. 1 and 2 , a microscope image information processing system 1 that executes the microscope image information processing method according to this embodiment can be configured to include a microscope 10, a camera 20, and a computer device 30. Existing general computer devices can be used for the microscope 10, the camera 20, and the computer device 30. FIG. 15 is a diagram showing an example of the hardware configuration of the computer device 30. The computer device 30 can have the same hardware configuration as a general computer device, including a processor 31, a HDD 32, a random access memory (RAM) 33, a read-only memory (ROM) 34, a removable memory 35 such as a CD, DVD, USB memory, a memory stick, or an SD card, an input / output user interface (keyboard, mouse, touch panel, speaker, microphone, LED (light-emitting diode), etc.) 36, a wired / wireless communication interface 37 capable of communicating with other computer devices, a display 38, etc. The computer device 30 can realize each process of the microscope image information processing method of this embodiment described above by reading and executing computer programs (camera software 40, joining software 42, and various other computer programs) stored in, for example, the HDD 32 into a memory such as RAM 33 and various data to be processed.

[0062] Although the computer device 30 is illustrated as a single device in FIGS. 1 and 2 , it may be configured with two or more computers. For example, a first computer device may store images of each portion of the glass slide specimen on the HDD 32, and a second computer device may be configured separately from the first computer device to perform the sequential joining process and subsequent processes using the multiple captured images stored in the first computer device. In this case, for example, the captured images of the glass slide specimen temporarily stored on the HDD 32 of the first computer device may be automatically transmitted to the second computer device via wired or wireless communication, and the second computer device may use this as a trigger to perform the subsequent sequential joining process and overall construction process. Alternatively, for example, the first computer device may perform the sequential joining process and transmit the sequentially joined images obtained by the sequential joining process and data required for the overall construction process to the second computer device via wired or wireless communication. Then, the second computer device may perform the overall construction process using the transmitted joined images and data required for the overall construction process.

[0063] Furthermore, although the present embodiment has been described as storing the photographed images of each portion of the slide glass specimen and the joined images in the HDD 32, they may be stored in other recording media.

[0064] According to the microscope image information processing method of this embodiment, a sequential stitching process is performed while capturing images of each portion of the glass slide specimen, and previews of the stitched images are displayed to the user. The user can view the preview display and check each time whether the stitching process is being performed properly while continuing to capture the glass slide specimen. This avoids the need to re-photograph the glass slide specimen due to a final stitching failure, as occurs with conventional offline stitching.

[0065] Furthermore, the microscope image information processing method according to this embodiment does not require an automatic motorized stage or dedicated camera, as is the case with conventional real-time stitching, and can be realized using an existing imaging system (microscope, camera, and computer). This reduces the introduction cost of the system, and enables virtual slides to be generated using familiar captured images using a microscope that users are familiar with on a daily basis. Furthermore, a conventional method for creating virtual slides involves the use of a device called a slide scanner, which automatically scans glass slide specimens. However, such devices are very expensive. However, the microscope image information processing method according to this embodiment does not require such dedicated devices.

[0066] Furthermore, capturing images of fluorescent samples requires an exposure time of approximately 500 ms to 1 second. Furthermore, generating virtual slides of multiply stained samples requires manual filter switching. For example, with conventional real-time stitching, it was difficult to flexibly change the capture conditions in this way. In contrast, the microscope image information processing method according to this embodiment captures sample images one by one, making it possible to process fluorescent samples as well.

[0067] Furthermore, according to the microscopic image information processing method of this embodiment, a preview image can be generated using a reduced image, and a joined image of the original size can be generated in the overall construction process, which contributes to reducing memory usage and improving processing speed.

[0068] Although one embodiment of the present invention has been described above, it goes without saying that the present invention is not limited to the above-described embodiment and may be embodied in various different forms within the scope of the technical concept thereof.

[0069] Furthermore, the scope of the present invention is not limited to the exemplary embodiments shown and described, but includes all embodiments that achieve equivalent effects to those intended by the present invention. Furthermore, the scope of the present invention is not limited to the combination of inventive features defined by each claim, but can be defined by any desired combination of specific features from among all the respective disclosed features.

[0070] The following configurations also fall within the technical scope of the present invention: (1) A microscope image information processing method executed by a computer system, comprising: a first step of acquiring a photographed image of a portion of a sample observed using a microscope and storing it in a storage area; a second step of calculating feature point information that is information regarding feature points of the photographed image when it is detected that the photographed image has been stored in the storage area; a third step of performing a matching process between the feature points of the previous photographed image and the feature point information of the current photographed image using the feature point information of the previous photographed image; a fourth step of performing a joining process based on the result of the matching process and the multiple photographed images that have been stored in the storage area up to now to generate a joined image; and a fifth step of displaying and outputting the joined image, wherein the first to fifth steps are repeated until the photographing of the portion of the sample is completed. (2) The microscope image information processing method of (1) above, wherein the joined image generated in the fourth step is an image generated using the photographed images reduced at a predetermined ratio. (3) The microscope image information processing method of (1) or (2) above, wherein in the third step, a global transformation matrix of the captured images stored in the first step is calculated using a specific image from among the captured images stored in the storage area as a reference. (4) The microscope image information processing method of (3) above, wherein the global transformation matrix is ​​calculated using a Max Spanning Tree. (5) The microscope image information processing method of (4) above, wherein the global transformation matrix is ​​adjusted by minimizing, using a least squares method, the reprojection error between all feature points subjected to the matching process in the third step. (6) The microscope image information processing method of (1) above, further comprising, after the fifth step, a step of arranging, for a region including the joined image generated by repeating the first to fifth steps until imaging of a portion of the sample is completed, the region and all of the captured images in accordance with the global transformation matrix.(7) The microscope image information processing method according to (1) to (4), further comprising: a sixth step of dividing into a plurality of small regions an area including the joined image generated by repeating the first step to the fifth step until imaging of the portion of the sample is completed; and a seventh step of identifying, for each of the plurality of small regions, one or more of the captured images that intersect with the small region to be processed, and arranging the small region to be processed and the identified one or more of the captured images according to a global transformation matrix. (8) The microscope image information processing method according to (7), wherein in the sixth step, bundle adjustment is performed on the joined image generated by repeating the first step to the fifth step until imaging of the portion of the sample is completed, and then the joined image is divided into a plurality of small regions. (9) The microscope image information processing method according to (8), wherein the bundle adjustment is performed by evaluating the following equation defined by the Levenberg-Marquardt algorithm:

[0071]

[0072] However, x ｉ : Component J of the transformation matrix of the image at processing time i ｉ : Jacobian at processing time i (the T on the right shoulder indicates transposition) r ｉ : Error between feature points at processing time point i λ: Real number equal to or greater than zero that is adjusted according to the magnitude of the error I: Identity matrix (10) The method for processing microscopic image information described in (8) above, wherein the bundle adjustment is performed by evaluating the following equation defined by the Levenberg-Marquardt method:

[0073]

[0074] However, x ｉ : components of the transformation matrix of the image at processing time i and distortion correction parameters J ｉ : Jacobian at processing time i (the T on the right shoulder indicates transposition) r ｉλ: error between feature points at processing time point i λ: real number equal to or greater than zero adjusted according to the magnitude of the error I: unit matrix (11) The microscope image information processing method of (1) to (7) above, wherein the sample is a fluorescent sample. (12) The microscope image information processing method of (1) to (8) above, wherein the computer system includes a first computer device and a second computer device separate from the first computer device, the first step is executed on the first computer device, and the second to fifth steps are executed on the second computer device. (13) A microscope image information processing system that performs a first step of acquiring a photographed image of a portion of a sample observed using a microscope and storing it in a storage area, a second step of calculating feature point information that is information regarding feature points of the photographed image when it is detected that the photographed image has been stored in the storage area, a third step of performing a matching process between the feature points of the previous photographed image and the feature point information of the photographed image using the feature point information of the previous photographed image and the feature point information of the photographed image, a fourth step of performing a joining process based on a result of the matching process and the plurality of photographed images that have been stored in the storage area up to now to generate a joined image, and a fifth step of displaying and outputting the joined image, and repeating the first to fifth steps until the photographing of the portion of the sample is completed. (14) A computer program that causes a computer system to perform the microscope image information processing method of (1) to (12) above.

[0075] REFERENCE SIGNS LIST 1... Microscope image information processing system 10... Microscope 20... Camera 30... Computer device 31... Processor 32... Hard disk drive (HDD) 33... RAM 34... ROM 35... Removable memory 36... Input / output user interface 37... Communication interface 38... Display 40... Camera software 42... Bonding software 50, 50'... Photographed image of slide glass specimen

Claims

1. A method for processing microscope image information executed by a computer system, comprising: A first step of acquiring an image of a portion of a sample observed using a microscope and storing the image in a storage area; a second step of calculating feature point information, which is information about feature points of the captured image, triggered by detecting that the captured image has been stored in the storage area; a third step of executing a matching process between the feature points of the one or more captured images previously stored in the storage area and the feature points of the captured images stored in the storage area in the first step, using the feature point information of the one or more captured images previously stored in the storage area and the feature point information of the captured images calculated in the second step; a fourth step of performing a joining process using a result of the matching process, one or more of the captured images previously stored in the storage area, and the captured image stored in the storage area in the first step to generate a joined image; A fifth step of displaying and outputting the spliced ​​image; Including, A microscope image information processing method, comprising repeating the first to fifth steps each time an image of the sample is taken.

2. The fourth step, under certain conditions, performs a joining process using the result of the matching process, one or more of the captured images stored in the memory area up to the previous time, and the captured images stored in the memory area in the first step to generate a joined image, and, if the certain conditions are not met, performs a joining process using the result of the matching process, the joined image joined previously, and the captured images stored in the memory area in the first step to generate a joined image, The microscope image information processing method of claim 1, wherein the specific condition is that the number of times the sample is photographed is a multiple of a predetermined number, that input from a user has been received, or that the reprojection error between all feature points matched in the third step has reached a certain value.

3. The matching process in the third step comprises:

2. The microscope image information processing method of claim 1, further comprising matching the feature points of a captured image stored in the memory area in the first step with the feature points of the captured image stored in the memory area in the immediately preceding step among one or more captured images stored in the memory area up to the previous time, and if the matching fails, matching the feature points of a captured image stored in the memory area earlier with the feature points of the captured image stored in the memory area in the first step.

4. Detecting that the captured image has been stored in the memory area includes: receiving an input from a user specifying a folder to be monitored that exists in the storage area; 2. The microscope image information processing method according to claim 1, further comprising the step of detecting that the captured image has been saved in the memory area by checking at regular intervals whether the monitored folder has been updated.

5. 2. The microscope image information processing method according to claim 1, wherein the joined image generated in the fourth step is an image generated using the captured images reduced at a predetermined ratio.

6. 3. The microscope image information processing method of claim 2, wherein in the third step, under the specific conditions, a global transformation matrix of all of the captured images stored in the memory area is calculated, and when the specific conditions are not met, a global transformation matrix of the captured images stored in the first step is calculated based on a specific image among the captured images stored in the memory area.

7. A microscope image information processing method as described in Claim 6, wherein under the specific conditions, the global transformation matrix is ​​calculated by a Max Spanning Tree.

8. A microscope image information processing method as described in Claim 7, wherein, under the specific conditions, the global transformation matrix is ​​adjusted by minimizing the reprojection error between all feature points subjected to the matching process in the third step using the least squares method.

9. The microscope image information processing method of claim 1, further comprising, after the fifth step, a step of positioning an area including the joined image generated by repeating the first step to the fifth step until the photographing of the sample is completed, the area and all of the photographed images according to a global transformation matrix.

10. A sixth step of dividing an area including the joined image generated by repeating the first step to the fifth step into a plurality of small areas; For each of the plurality of small regions, Identifying one or more of the captured images that intersect with a small area to be processed; The small region to be processed and the one or more captured images identified are arranged according to a global transformation matrix. a seventh step of carrying out The method for processing microscope image information according to claim 1 , further comprising:

11. The microscope image information processing method of claim 10, wherein in the sixth step, a bundle adjustment is performed on the spliced ​​image generated by repeating the first to fifth steps, and then the spliced ​​image is divided into a plurality of small regions.

12. For each of the plurality of small regions, after the seventh step, an eighth step of joining the small area to be processed arranged in the seventh step and the identified one or more captured images; a ninth step of storing the image joined in the eighth step as a temporary file in a storage area; a tenth step of sequentially writing the plurality of temporary files saved by the ninth step for each of the plurality of small regions to a storage region; The method for processing microscope image information according to claim 10, further comprising:

13. The method for processing microscope image information according to claim 11 , wherein the bundle adjustment is performed by evaluating the following equation defined by the Levenberg-Marquardt method: [0010] however, x ｉ : Component of the transformation matrix of the image at processing time i J ｉ : Jacobian at processing time point i (the T on the right indicates transposition) r ｉ : Error between feature points at processing time i λ: A real number equal to or greater than zero that is adjusted according to the magnitude of the error I: identity matrix

14. The method for processing microscope image information according to claim 11 , wherein the bundle adjustment is performed by evaluating the following equation defined by the Levenberg-Marquardt method: [0025] however, x ｉ : Components of the transformation matrix of the image at processing time i and distortion correction parameters J ｉ : Jacobian at processing time point i (the T on the right indicates transposition) r ｉ : Error between feature points at processing time i λ: A real number equal to or greater than zero that is adjusted according to the magnitude of the error I: identity matrix The distortion correction parameters are distortion parameters common to all images at all processing points.

15. The method for processing microscope image information according to claim 1 , wherein the sample is a fluorescent sample.

16. the computer system includes a first computer device and a second computer device that is separate from the first computer device; the first step is executed on the first computer device; 2. The microscope image information processing method according to claim 1, wherein the second step to the fifth step are executed in the second computer device.

17. A first step of acquiring an image of a portion of a sample observed using a microscope and storing the image in a storage area is carried out; A second step of calculating feature point information, which is information related to feature points of the photographed image, is executed in response to detection that the photographed image has been stored in the storage area; a third step of executing a matching process between the feature points of the one or more captured images previously stored in the storage area and the feature points of the captured images stored in the storage area in the first step, using the feature point information of the one or more captured images previously stored in the storage area and the feature point information of the captured images calculated in the second step; a fourth step of performing a joining process using a result of the matching process, one or more of the captured images previously stored in the storage area, and the captured image stored in the storage area in the first step to generate a joined image; A fifth step of displaying and outputting the spliced ​​image is carried out; A microscope image information processing system which repeats the first to fifth steps every time an image of a portion of the sample is taken.

18. A computer program for causing a computer system to execute the microscope image information processing method according to any one of claims 1 to 16.

Images