Determining image registration quality in automated tissue anatomy process

Abstract

In one aspect, the invention relates to a method of quality control in an image registration process, the result of which is used in an automated anatomical process of anatomy of tissue from an anatomical slide on which tissue slices cut from a tissue sample are placed. The method (10) comprises the steps of: obtaining (11) a first image of a reference slide comprising a reference sample cut from the same tissue sample; obtaining (12) a second image of the anatomical slide; registering (20) the first image onto the second image wherein the transformation matrix is calculated as part of the image registration process to produce a best fit between the first image and the second image; calculating (30) a first quality metric for the image registration process based on a degree of overlap between the first image and the second image; calculating (40) a second quality metric for the image registration process based on parameters indicative of skewness and asymmetric scaling of the transformation matrix; comparing (50) each value of the calculated first and second quality metrics with a corresponding threshold to determine if each quality metric is satisfied; if the first or second quality metric is not satisfied, it is determined that the image registration quality is unsatisfactory, and the determination is reported (60) to the user.

Description

Technical Field

[0001] This invention relates to digital pathology, and more particularly to methods and systems for quality control in automated dissection processes to determine whether reliable automated dissection is feasible. Background Technology

[0002] For molecular testing purposes in oncology, multiple sections of tissue are dissected from glass slides. Generally, tissue is dissected from several slides of a single instance to obtain sufficient material for molecular testing. The need for automated dissection processes is increasing, including the application of digital pathology techniques. Regions of interest (ROIs) (from which tissue is to be collected) can be indicated by the pathologist on a reference image, capturing a reference image for a specific tissue section. The pathologist draws lines to identify the boundaries of the ROI; these lines are typically called annotations. Anatomical images of each slide, each containing another tissue section from the same sample to be dissected, are also captured. During image registration, the reference image is mapped onto each anatomical image, where the annotations of the ROI boundaries are also transferred to the anatomical image. After image registration, dissection tools are controlled based on the transferred annotation coordinates to mechanically remove biological material within the labeled boundaries from each anatomical slide.

[0003] An embodiment of a method for identifying tissue anatomical regions in a series of pathological slides is disclosed in WO2021 / 001564. First and second annotations are obtained for corresponding first and second reference images in the series, the first and second reference images being separated by several intermediate images. Annotations are then generated for the intermediate images between the reference images based on bidirectional image registration, and both types of annotations are subsequently transferred to each intermediate image. These annotations are then combined to obtain combined annotations for each intermediate image. The method includes determining a first set of registration parameters for registering the first reference image to the intermediate images, and determining a second set of registration parameters for registering the second image to the intermediate images, wherein the first and second sets of registration parameters are used, respectively, to propagate the first and second annotations to the intermediate images. The method may further include determining the fit quality of the annotations in the intermediate images, and if the fit quality does not meet a fit quality criterion, prompting the user to provide third annotations in a third image from the series of intermediate images between the first and second images.

[0004] The quality of the fit can be determined based on the shape difference or surface area overlap between the propagated first and second labels, or based on the deformation magnitude of the propagated first label and / or the deformation magnitude of the propagated second label. However, it is possible that the quality of the label fit appears satisfactory, while the quality of the image registration is unsatisfactory. Therefore, the determined result will produce false positives, which are, of course, undesirable.

[0005] As will be understood, the quality of the achieved anatomical results hinges on the removal of the tissue of interest from the anatomical slide. The overall anatomical outcome is influenced by several parameters associated with the equipment used in the automated dissection process. For example, misalignment between the imaging module and the dissection tools, as well as the accuracy of the dissection tool's movement, can affect the achieved quality. However, even without misalignment and with precise movement, if the annotation transfer to the anatomical slide is inaccurate due to poor image registration, the achieved anatomical result will be unsatisfactory.

[0006] There is still room for improvement. Summary of the Invention

[0007] In a first aspect, the present invention relates to a method for quality control in an image registration process, the results of which are used in an automated dissection process in which tissue is dissected from a dissecting slide and tissue sections cut from a tissue sample are placed on the dissecting slide. The method includes the following steps: A first image of a reference slide is obtained, the reference slide containing a reference sample cut from the same tissue sample; Obtain a second image of the anatomical slide; The first image is registered onto the second image, wherein the transformation matrix is ​​calculated as part of the image registration process to produce a best fit between the first and second images; The first quality metric of the image registration process is calculated based on the degree of overlap between the first and second images. A second quality metric for the image registration process is calculated based on parameters indicating the skewness of the transformation matrix and asymmetric scaling. Each calculated value of the first and second quality metrics is compared with its corresponding threshold to determine whether each quality metric is satisfied; and If either the first or second quality metric is not met, the image registration quality is determined to be unsatisfactory, and this determination is reported to the user.

[0008] In some implementations, a qualitative indicator such as "high" may be reported when each quality metric is met, and "low" may be reported when either the first or second quality metric is not met. The reporting step may further include reporting the calculated values ​​of the first and second quality metrics. If the image registration quality of a specific anatomical slide is determined to be unsatisfactory or low, the user may decide whether to reject the slide under investigation or to send an associated second image (which will be referred to as the anatomical image) to a pathologist for manual annotation. In other cases, after careful review, for example, of an overlay image (which shows a first image mapped onto the anatomical image (which will be referred to as the reference image)), if the quality determination appears to have produced a false negative, the user may decide to proceed with the dissection of the slide.

[0009] When satisfactory or high-quality conditions are determined, anatomical slides can be dissected automatically by a dissecting device equipped with dissecting tools to remove material from the slide. Suitable, the method further includes the steps of: identifying regions of interest (ROIs) to be dissected based on a reference slide, marking the boundaries of the identified ROIs on a reference image, and transferring the markings to the anatomical image. The transferred coordinates are transmitted to a controller of the dissecting device, which controls the movement of the tools to remove material from the identified ROIs.

[0010] Registering reference images onto a series of anatomical slides forms the basis for the automated transfer of ROI annotations for dissection. In the method of this invention, the quality of image registration is estimated based on at least two different metrics, which significantly reduces the possibility that the transferred annotations are actually unreliable and increases the likelihood that the quality of the dissection results will be satisfactory.

[0011] In some implementations, the step of estimating the image registration quality of each of a series of anatomical slides is performed before the ROI boundary annotation on the reference slide. This eliminates the need to transfer annotations to anatomical images that do not meet quality metrics. In other implementations, the quality of the image registration process is estimated and reported during or after the annotation of the reference image.

[0012] In the method of this invention, the first quality metric of the image registration process is the degree of overlap between the reference image and the anatomical image. Calculating the intersection-over-union (IoU) ratio is a preferred technique, where IoU represents the ratio of the overlapping area of ​​tissue presented in the image to the combined area of ​​tissue in the two images. The tissue area is determined by the number of pixels identified as tissue pixels, wherein the identification is based on intensity analysis. In embodiments using images captured using a transmission-based optical arrangement, tissue pixels are relatively bright and can be identified as pixels exceeding a threshold selected based on the background intensity level. Images can also be captured using a bright-field arrangement based on reflected light, in which case the tissue pixels are darker than the background.

[0013] Following the image registration step, the method appropriately further includes the following steps: The tissues presented in the reference and anatomical images are segmented to identify tissue pixels; and Determine the number of overlapping tissue pixels and the number of tissue pixels presented in the two combined images.

[0014] Therefore, IoU is calculated as the number of overlapping organization pixels. Divide by the total number of tissue pixels in the two combined images .

[0015] The IoU calculation is inversely proportional to the ratio of the surface area to the perimeter of objects in an image. In other words, any misalignment between images of "small" tissue objects results in a lower IoU value compared to an image calculated for an image of a "large" tissue object with the same degree of misalignment. This creates a bias that can be compensated for by scaling the applied IoU threshold based on the tissue area in the image, using a polynomial function of two empirically chosen points.

[0016] Therefore, the method may include the following additional step: setting an IoU threshold based on an area-related polynomial function, the polynomial function being related to the area of ​​the tissue, wherein the tissue area is obtained based on the number of tissue-identified pixels in the reference image during the step of segmenting the reference image.

[0017] In a preferred embodiment, the upper and lower limits of the IoU threshold are defined, for example, 0.85 and 0.5. Therefore, the method may include setting the IoU threshold to a value between the upper and lower limits based on a polynomial function related to area.

[0018] A second quality metric in the image registration process is based on the deformation of the anatomical image, which is required to achieve best-fit registration with a reference image. The image registration process includes the application of a registration algorithm. In one embodiment, a feature-based registration algorithm is applied, which identifies unique features in the image, such as corners or edges. Identified features are matched for image alignment, where transformation parameters are calculated based on the location of the matched features. In other embodiments, image registration can be performed using an active contour model, shape context, or a level set approach.

[0019] In practice, the registration process generates a transformation matrix comprising numerous elements, including xy-shifts, rotations, affine deformations, non-affine deformations, shearing, etc. Based on the process of cutting tissue and placing it on a slide, there is an expectation of actual deformation and mismatch between the slides. The transformation matrix properties should reflect this expectation range. Otherwise, it is highly likely that the registration process will not produce a good match, but rather be "masked" by features and produce local minima.

[0020] The registration algorithm returns a transformation matrix in the following form: in, and These are the coefficients that limit the translation; and, It is a coefficient that limits rotation, scaling, and skewness.

[0021] In the method of this invention, a subset of the transformation matrix contains coefficients that define rotation, scaling, and skewness, and the subset of the transformation matrix is ​​decomposed into a first matrix representing rotation and a second matrix representing skewness and scaling: in, Indicator with rotation The rotation matrix; and, and It is scaling in the x and y directions; and, and Limited to the second matrix The skewness in the middle.

[0022] The inventors have discovered that by applying an empirically estimated threshold, the condition number of the second matrix can be used to identify skewness and to identify asymmetric scaling between x and y.

[0023] Therefore, the step of calculating the second quality metric includes calculating the second matrix. The condition number represents the "score" of skewness versus asymmetric scaling. In a preferred embodiment, a maximum threshold for the condition number is set to 1.12. If the calculated condition number exceeds 1.12, the image registration quality is determined to be unsatisfactory.

[0024] In another improvement, the method of the present invention includes the following additional step: calculating a third quality metric for the image registration process, wherein the third metric is a symmetric scaling measure associated with the second matrix.

[0025] When the condition number of the calculated second matrix is ​​within a defined threshold, it can be assumed that the skewness is minimized and that the scaling between x and y is approximately identical. Therefore: Based on these assumptions, the 2-norm of the second matrix is ​​equal to When this value is too large or too small, the symmetric scaling metric is not satisfied.

[0026] Therefore, the step of calculating the third quality metric includes calculating the 2-norm of the second matrix, wherein upper and lower bounds for the calculated value are set. In a preferred embodiment, the lower bound is set to 0.9 and the upper bound is set to 1.1.

[0027] In another aspect, the present invention relates to a method for calculating a quantitative uncertainty parameter, which provides a measure of the spatial accuracy of image alignment. This represents the potential local inaccuracy based on the transferred annotations of image registration. The quantized uncertainty parameter can be calculated as a fourth quality measure in the image registration process.

[0028] The calculated values ​​of the indeterminate parameters may be reported to the user as part of the quality report of the image registration process, and / or as part of the quality report of the achieved dissection results, which is generated after automated dissection has been performed.

[0029] Preferably, the calculated value of the indeterminate parameter is relative to the indicator. It has been reported that the indicator is derived from the typical precision of the dissecting tools used in the dissection apparatus, for example... =0.1 mm. Therefore, the accuracy of the tool is used as a threshold, and the calculated value is compared with the threshold. If the calculated value is greater than... If this happens, an alert will be issued in the generated quality report.

[0030] In one implementation, the indeterminate parameter is calculated based on a first quality metric. The method includes the additional step of estimating the mean diameter of tissue objects detected during segmentation of the anatomical image. The value of the indeterminate parameter is obtained by multiplying the mean diameter by the IoU.

[0031] In another implementation, when the registration algorithm is feature-based, the indeterminate parameters are based on the transformation matrix. With individual transformation matrix The transformation matrix is ​​calculated by comparison. The individual transformation matrix is ​​the final result of the image registration process. This is calculated when mapping features detected in the reference image to corresponding features detected in the anatomical image. The distance between points (which is transformed using two matrices) provides a locally constrained measure of image registration quality. Therefore, the method may further include the following steps: Key points for identifying each feature in the reference image Each feature has been detected and matched to a corresponding feature in the anatomical image; use And use the individual transformation matrix associated with this feature. For each identified key point Perform transformations; and based on Determine the local spatial distance between two transformations. .

[0032] In one embodiment, the key points of the feature The center point of the feature is identified. Any other suitable point of the feature may be selected.

[0033] In one embodiment of a further improved method, the quantified indeterminate parameter is calculated as an average of determined spatial differences. This average can be calculated for all points in the image as a whole. In another embodiment, the method includes identifying tissue objects in an anatomy and determining the determined spatial differences of the tissue objects. Find the average value.

[0034] In yet another embodiment, the determined spatial difference obtained for each feature location It is interpolated and plotted as a heatmap.

[0035] In another embodiment of the method, wherein the identified ROI boundaries are marked on a reference image, the method further includes: Each determined spatial difference Mapped onto anatomical images; and Each difference is associated with a point labeled in the ROI, thereby calculating local values ​​of indeterminate parameters for all labeled points.

[0036] If any of the calculated local values ​​exceeds the threshold If so, the report includes an alert.

[0037] In a second aspect, the present invention relates to a system for assessing and reporting the image registration quality between a reference slide image and a series of anatomical slide images. The system includes an imaging module having an image sensor (such as a camera) and a processor programmed with an image registration algorithm and image processing software. The system further includes a display for displaying a quality control report. The processor is further programmed with instructions that cause the system to perform the method steps of the invention. Preferably, the system is further equipped with a user input interface that allows a user (pathologist) to annotate the image data.

[0038] In a preferred embodiment, the system is part of an anatomical apparatus. The apparatus is further equipped with movable anatomical tools and a controller for controlling tool movement based on coordinates received from the imaging module.

[0039] The dissection tools may be equipped with scrapers (such as those disclosed in WO2022 / 063695) or milling tips (such as those disclosed in US10876933). In other embodiments, the device may perform laser-based microdissection.

[0040] In a preferred embodiment of the dissection apparatus including the system according to the invention, the apparatus is further configured to evaluate the achieved dissection results. Additionally, the processor is programmed with instructions that cause the imaging module to capture a post-dissection image of the dissecting slide and compare it with a pre-dissection image, wherein ROI labels have been registered to the pre-dissection image. Furthermore, an image processing algorithm is applied, which performs the following steps: The achieved anatomical boundaries are detected by pixel-level comparison between pre-anatomical and post-anatomical images to determine intensity changes. The detected anatomical boundaries are compared with registered labeled boundaries to determine at least one indicator representing the quality of the achieved anatomical result; and Generate a quality report, which includes at least one defined quality indicator.

[0041] An embodiment of indicators used for the quality of the achieved anatomical results is the average deviation between the detected boundary and the labeled boundary, and the percentage of tissue dissected within the labeled boundary. A quality score can be assigned to each calculated quality indicator.

[0042] Furthermore, the quality of the image registration process affects the anatomical results achieved.

[0043] In another improvement to the method of the present invention, the quality score is assigned to a value already calculated for each quality metric in the image registration process.

[0044] Therefore, the overall quality score of the image registration process can be calculated based on the following: in, to These represent the quality scores associated with the first through fourth quality metrics, respectively.

[0045] Appropriately, a score of 0 is assigned to any calculated value that does not meet its corresponding threshold. In some embodiments, a value of 1 is assigned to a calculated value that meets the threshold, such that the overall rating... A score of 0 indicates dissatisfaction, while a score of 1 indicates satisfaction. In other embodiments, scores are assigned to one or more calculated values ​​to quantify the quality of a "satisfactory" image registration. The score assigned to a specific calculated value may be determined based on a normalization bias derived from a target value within an acceptable range of an applicable threshold. A score of 0 is assigned when the calculated value is outside the acceptable range.

[0046] As will be understood, in the implementation scheme that calculates only the first quality metric and the second quality metric, the overall rating is based on and .

[0047] In one embodiment of the dissection device, the device is configured to base its overall score on a determined image registration process. The quality rating of the achieved anatomical outcome is calculated. Appropriately, the quality rating is further determined using scores assigned to each quality indicator, calculated based on a comparison of pre- and post-anatomical images for the achieved anatomical outcome.

[0048] Therefore, the method and system of the present invention achieve several improvements that greatly reduce the risk of unreliable molecular assays from materials collected from anatomical slides. Other advantages will be explained in the following detailed description with reference to the accompanying drawings. Attached Figure Description

[0049] The invention will now be further illustrated with reference to the embodiments described below. In the accompanying drawings: Figure 1 A series of pathological slides are illustrated, one of which is a reference slide with tissue stained with H&E, and the remaining slides are anatomical slides without H&E staining. Figure 2 An embodiment of the method according to the invention for determining whether the quality of an image registration process is unsatisfactory for achieving reliable automated dissection results is shown, in which an image of a reference slide is mapped onto an image of an anatomical slide during the image registration process; Figure 3a An embodiment of a first image of a tissue slide is shown, the tissue slide comprising reference tissue sections cut from a sample; Figure 3b An embodiment of a second image of a tissue slide is shown, the tissue slide comprising tissue sections cut from the same sample; Figure 3c The overlap between the first and second images during the image registration process is shown; Figure 4 The sub-steps associated with the image registration step are shown, as well as the sub-steps associated with the calculation of a fourth metric quality in another embodiment of the method of the present invention; Figure 5 A system configured to carry out the method of the present invention is illustrated schematically; Figure 6 An embodiment of a dissection apparatus is illustrated schematically, the dissection apparatus being equipped with a system for carrying out the method of the present invention.

[0050] It should be noted that items with the same reference numerals in different figures have the same structural features and functions, or the same markings. Where the function and / or structure of a said item have already been explained, there is no need to repeat the explanation in the detailed description. Detailed Implementation

[0051] Pathological diagnostic studies of biomaterials (such as tissues and cells) form the basis for many treatment decisions, particularly in oncology. For example, performing genomic-based tests informs individual patients diagnosed with cancer about treatment options. Biomaterials / tissues can be obtained from biopsies or surgical procedures. Typical tissue preparation steps include, for example, formalin fixation and paraffin embedding. This formalin-fixed, paraffin-embedded (FFPE) biomaterial is then cut into thin sections, which are fixed onto glass slides. Other methods for obtaining and preparing tissue samples are known. For molecular testing, samples are selected from these sections to meet the testing requirements. Pathologists identify regions of interest (ROIs), and then the identified ROIs of each tissue slide are dissected to obtain sufficient material for molecular testing.

[0052] In automated dissection, the ROIs to be dissected from a series of tissue slides prepared from the same sample are indicated by a pathologist on a reference image of a reference slide via so-called annotations, which are drawn lines to indicate the boundaries of the identified ROIs. Typically, the reference slide is a H&E-stained slide from a reference sample of the same instance. The remaining slides in the series may be paraffin-embedded, unstained slides, or they may be dewaxed and optionally stained with nuclear solid red (NFR) or hematoxylin, staining which does not interfere with amplification and detection protocols used to identify the tissue material. These slides will be referred to as dissecting slides.

[0053] Figure 1 A series of tissue slides are shown, wherein tissue 5 in reference slide 1 is stained with H&E, and the remaining slides 2a to 2e are anatomical slides that are not stained with H&E.

[0054] Appropriately, the H&E reference slide 1 is scanned using a high-resolution scanner to allow for the identification of tumor cells and to provide the pathologist with H&E images (reference images) for ROI selection and the labeling of one or more ROI boundaries. Each anatomical slide 2a to 2e is then scanned to obtain anatomical images.

[0055] Then, registration is performed from the reference image to each anatomical image, and reference slide annotations are transferred to each anatomical image. Automated dissection can then be performed using a device equipped with dissecting tools whose movement is controlled to remove material from the slide based on the transferred annotation coordinates.

[0056] Registering a reference image onto an image on an anatomical slide forms the basis for the automated transfer of regions of interest (ROI) annotations used in dissection. Therefore, the quality of the image registration process is crucial for achieving reliable anatomical results for molecular analysis.

[0057] In a first aspect, the present invention relates to a method for quality control of an image registration process used in automated dissection, to determine whether the quality of the image registration is sufficient to achieve reliable and accurate dissection results. Figure 2 The steps of a preferred embodiment of the method 10 are shown.

[0058] In the first step 11, a first image of a reference slide containing tissue sections from the sample is obtained. This first image (i.e., the reference image) can be captured by a system configured to implement the method of the invention, or the reference image can be received or downloaded from an external source. In the next step 12, a second image of an anatomical slide having tissue sections from the same sample is captured or received to obtain an anatomical image. Appropriately, an anatomical image is obtained for each anatomical slide containing tissue sections from the same sample.

[0059] In subsequent step 20, during image registration, a reference image is registered to each anatomical image, wherein a transformation matrix is ​​calculated to produce a best fit between the reference image and each anatomical image. In one embodiment, a feature-based image registration algorithm (such as that described in US89229294) is applied.

[0060] In the method of the present invention, the quality of the image registration process is evaluated based on at least two different quality metrics.

[0061] In step 30, the first quality metric of the image registration process is calculated based on the overlap between the reference image and each anatomical image. A preferred embodiment of the overlap measurement is the intersection-to-union ratio (IoU). The IoU can be calculated via several sub-steps.

[0062] In the first sub-step, the reference image and each anatomical image are segmented to identify tissue pixels. An imaging module, including an optical arrangement and a light source, is used to capture images of the slide. Tissue pixels exhibit high intensity and can be identified as pixels exceeding a predetermined minimum intensity. Appropriately, this minimum intensity is defined based on the optical arrangement used, the type of tissue sample to be dissected, and the background intensity level.

[0063] In the second sub-step, the number of overlapping organization pixels is determined. And the total number of tissue pixels in the two combined images. .

[0064] Then, the IoU score is calculated as follows: Divide by .

[0065] In the next step 40, a second quality metric for the image registration process is calculated. The second metric is a parameter indicating both the skewness and scaling of the transformation matrix produced by image registration.

[0066] Image registration returns a transformation matrix in the following form: in, and These are the coefficients that limit the translation; and, It is a subset of the transformation matrix, containing coefficients that define rotation, scaling, and skewness.

[0067] The subset of transformation matrices mentioned above can be decomposed into: in, Indicator with rotation The rotation matrix; and, Indicator skewness and scaling matrix, where and It is a scaling in the x and y directions, and and Limit skewness.

[0068] Note that there is no unique solution here because there are 5 unknowns and 4 constraints. Given that the condition number of the rotation matrix is ​​always 1 (unity), and the condition number of the product of two matrices equals the product of the individual condition numbers, we can discard the rotation matrix. This leaves... .

[0069] There are two choices: (i) Both skewness values ​​are (almost) zero. In this case, the condition number equals... .

[0070] (ii) At least one skewness value is non-negligible. This has a significant impact on the condition number.

[0071] Therefore, by using empirically estimated thresholds, the condition number can be used to identify skewness as well as to identify asymmetric scaling between x and y.

[0072] Therefore, calculation step 40 appropriately includes decomposing the transformation matrix into a rotation matrix and skewness and scaling matrices, and calculating the condition number of the skewness and scaling matrices.

[0073] In the subsequent comparison step 50, each value calculated in steps 30 and 40 is compared with a corresponding threshold to determine whether the quality of the image registration process is unsatisfactory for achieving reliable automated anatomical results.

[0074] In sub-step 51, the IoU score is compared to an IoU threshold. The IoU threshold between the reference image and the anatomical image can be defined by an area-related polynomial. Based on the way IoU is calculated, it is inversely proportional to the ratio of the surface area to the perimeter of the object in the image. To compensate for this deviation, the IoU threshold is scaled by the tissue area in the reference image according to a polynomial function of two empirically selected points. In the depicted implementation, a lower and upper limit for the IoU threshold are defined, wherein the threshold is set to a value between the upper and lower limits based on the number of identified tissue pixels in the reference image during the sub-step of segmenting the reference image.

[0075] In one embodiment, the lower limit of the IoU threshold is defined as 0.5, and the upper limit is defined as 0.85. Appropriately, an area-related polynomial (used to set the threshold) can be derived empirically from previous calculations. If the calculated IoU value falls below the set threshold, the image registration quality is determined to be unsatisfactory.

[0076] When the calculated IoU value exceeds the IoU threshold, the overlap is satisfactory, but this is insufficient to indicate that the image registration process is of adequate quality. In sub-step 52, the calculated skewness and the condition number of the scaling matrix are compared with a predetermined threshold.

[0077] Appropriately, the threshold for the condition number is a hard-coded value derived empirically; in a preferred embodiment, the hard-coded value is set to 1.12. If the calculated condition number exceeds the threshold, the quality of the image registration process is determined to be unsatisfactory.

[0078] If either the first or second quality metric is not met, the method includes reporting the determined step 60 to the user. The report may also include the calculated value, possibly along with an alert, to carefully examine the associated anatomical images before allowing automated dissection of the slide to be studied, and / or to take corrective actions, such as sending the anatomical images to a pathologist for individual annotation.

[0079] Even if the calculated values ​​of the first and second quality metrics meet the corresponding criteria, there is still a risk that the image registration process may have insufficient quality. The calculated skewness and condition number of the scaling matrix may indicate the presence of negligible asymmetric scaling, but the overall scaling between the reference image and the anatomical image may still be unacceptably large or small.

[0080] Therefore, in a preferred embodiment, the method includes another step 70, in which a third quality metric is calculated and compared with a threshold, wherein the third metric is a parameter indicating symmetric scaling.

[0081] When comparison step 52 returns a result where the condition number of the calculated decomposed skewness and the asymmetric scaling matrix is ​​below a predetermined threshold, it can be assumed that the skewness is minimal and that the scaling between x and y is almost constant. Therefore: in, This indicates scaling when the scaling in x and y is the same.

[0082] Based on these assumptions, the skewness is equal to the 2-norm of the scaling matrix. Therefore, step 70 includes calculating the 2-norm of this matrix as a third quality metric. When the calculated value exceeds the upper limit or falls below the lower limit, the metric is not satisfied, and the user is prompted, as outlined above. In one embodiment, the lower limit is set to 0.9, and the upper limit is set to 1.1. Thus, the risk of unsatisfactory dissection results from the dissection of the slide under investigation is further reduced.

[0083] In one implementation of the method for calculating a third quality metric and comparing it with a threshold limit, the calculation and comparison can also be performed together with the calculation and comparison of the first and second metrics.

[0084] Example Figures 3a to 3c An embodiment of the registration of a first slide image with a second slide image is shown, wherein... Figure 3a A first image 305 (as a reference image) depicts a first glass slide containing tissue sections, and... Figure 3b A second image 310 depicts a second slide containing different tissue sections from the same tissue sample. During registration, the first image is mapped onto the second image, wherein... Figure 3c The results of the overlap are shown.

[0085] After registration, an IoU score is calculated. In the depicted embodiment, the IoU score is 0.967. The tissue area in the reference image is calculated based on identified tissue pixels, wherein in the given embodiment, the maximum value of the IoU threshold is set to 0.85 based on an area-related polynomial. The IoU score exceeds the threshold, and a first quality metric is satisfied.

[0086] For the image being depicted, the transformation matrix returned by the registration algorithm applied during the registration process is: .

[0087] One possible way to decompose the transformation matrix into rotation, scaling, and skewness is: .

[0088] Two scaling factors and It is close to 1, and the skewness is close to zero.

[0089] The condition number of the matrix is ​​calculated, resulting in a value of 1.002. Since this is less than the threshold of 1.12, the second quality metric is also satisfied.

[0090] Another check on image registration quality was performed by calculating the 2-norm of the decomposed skewness and scaling matrix to determine the overall scaling (symmetric scaling). The calculated value was 0.998, which is within the threshold of 0.9 to 1.1, and this quality metric was also satisfied.

[0091] Therefore, all scores fall within the limits of successful image registration, and it can be inferred that the annotations on the reference image can be reliably transferred to the second image. Appropriately, the successful determination should be reported to the user on the user interface. The report may also include the results of each calculated quality metric.

[0092] In another improvement, the method may include a step of calculating a fourth quality metric for the image registration process. The fourth metric is a quantified, indeterminate parameter that indicates the spatial accuracy of the image alignment during the registration process. This parameter provides a quantified indicator of the impact of registration quality on the accuracy of the transferred annotations, which is related to the accuracy of the achieved anatomical result.

[0093] Figure 4 A flowchart of certain steps according to one embodiment of another improved method is shown, in which sub-steps of image registration process 20 are depicted, and wherein, for simplicity, subsequent method steps 30 to 70, which calculate the first, second, and third quality metrics and compare them with their respective thresholds, have been omitted. Step 80, which calculates the fourth quality metric, may be performed after step 70, which calculates and compares the third quality metric. In other embodiments, a quantized indeterminate parameter is calculated after the steps of calculating and comparing the first and second quality metrics. In yet other embodiments not according to the invention, the quantized indeterminate parameter is calculated as an indicator of the quality of the image registration process, regardless of whether the first to third quality metrics as defined above are calculated.

[0094] In a preferred embodiment, the image registration process applied in step 20 is feature-based automated image registration using, for example, scale-invariant feature transform (SIFT). In sub-step 21, features in the reference image are detected and matched with corresponding features in the anatomical image. In sub-step 22, a transformation matrix is ​​computed for each corresponding feature pair.

[0095] The transformation matrix for each calculated matching feature pair is limited to: As previously outlined, the step of registering the reference image to the anatomical image includes sub-step 23, which calculates the transformation matrix for image registration as a whole. This transformation matrix is ​​defined as follows: By transforming the feature pairs' matrix Transformation matrix used for image registration By comparing the points of the matching feature pairs, the local spatial difference (i.e., distance) between the two transformations can be calculated. This provides a locally constrained measure of the image registration quality.

[0096] In substep 81, local spatial differences are calculated for each keypoint of the feature. ,in The distance between its own defined transformation and the overall transformation can be obtained through... To obtain.

[0097] In one embodiment, the key points for matching feature pairs are defined as the geometric center points of features detected in a reference image, which may be determined in sub-step 21 of detecting and matching features.

[0098] For each detected feature, identify the differences based on the identified key points. Furthermore, in substep 82, the indeterminate parameter (fourth quality metric) is calculated based on the average of determined values. In one embodiment, the determined difference is the average of all keypoints across the entire image.

[0099] In an alternative embodiment, the method includes another step of identifying tissue objects in reference images and / or anatomical images, and the step of calculating the average includes analyzing the differences. Calculate the average value, and the difference. Key points of features located within the tissue object are identified. The advantage of averaging values ​​calculated for features within the tissue object (rather than for all features in the image as a whole) is that the position of tissue objects in a series of anatomical slide images can change relative to each other. Therefore, averaging the entire image may not represent individual tissue objects. Another advantage is that matching features not including the tissue are very likely to be located outside the ROI to be dissected. Thus, a greater degree of deviation (uncertainty) in local spatial differences is permissible compared to the allowable deviation within the ROI. Therefore, the possibility of false negatives is reduced.

[0100] The permissible deviation depends on the typical accuracy of the dissecting tools and associated equipment used in subsequent automated dissections. The typical accuracy of the tools is... In the case of (e.g., 0.1 mm), the value An indefinite threshold is appropriately provided to be associated with the transfer of ROI annotations.

[0101] If the calculated value of the indeterminate parameter is greater than If the image registration quality is insufficient, this is determined and reported to the user. Preferably, the calculated value is also included in the report, as is the case when the value is acceptable.

[0102] In a second aspect, the present invention relates to a quality control system for an image registration process used in automated dissection procedures. Figure 5 An embodiment of a system configured to implement the method of the present invention is illustrated schematically. System 90 includes an imaging module 150 for capturing images of a series of tissue slides. Module 150 has an image sensor 155 and a processor 153 programmed with an image registration algorithm, image processing software, and instructions that cause the system to perform the steps of the method of the present invention. The system further includes a display 145 for reporting the results of quality control of the image registration process. Additionally, the system may include a user input interface 95 that allows a user (pathologist) to annotate the image data.

[0103] In another aspect, systems for quality control are integrated into the dissection equipment. Figure 6 An embodiment of such an autopsy apparatus is schematically illustrated. The apparatus 100 includes an autopsy tool 110 attached to a robotic arm 120, which is mounted to the apparatus housing 105. The robotic arm 120 includes a series of actuators for performing necessary movements during autopsy, these movements being controlled by a control unit 140 of the apparatus. Suitably, the robotic arm 120 includes: - A rotary actuator for rotating the tool about a vertical axis R, which is perpendicular to the platform 137 of the device, which supports a tissue slide 130 to be dissected; - An XY stage for translational movement, and - A Z-stage for vertical movement. The Z-stage may include hinge bearings for position control to ensure a constant and precise downward force is applied during dissection.

[0104] In the depicted embodiment, the dissecting tool 110 includes a scraper 115 for mechanically separating sample material from a tissue slide 130, which is preferably held in a slide holder 135 to prevent any movement of the slide during dissection. Typically, the slide holder holds several tissue slides from the same instance and is automatically loaded onto a support platform 137 of the device. The device is also equipped with a vacuum generator (not shown) that generates an upward airflow at the scraper to draw the scraped material into the interior of the dissecting tool 110.

[0105] The device also includes an imaging module / camera module 150 equipped with an image processor 153 configured to capture and process images of each tissue slide 130. The imaging module has an image sensor 155 and a lens 157 for image capture, and the device is further equipped with a light source 160 and filter elements 151, 152. In the depicted embodiment, images are captured by transmission, and the light source 160 is positioned below the glass tissue slide 130. A first filter element 151 is positioned between the light source 160 and the slide 130, and a second filter element 152 is positioned between the image sensor 155 and the lens 157. Suitably, the first and second filter elements are crossed polarizers.

[0106] Each tissue slide (including a H&E-stained reference slide) is scanned by imaging module 150, and a digital image of each captured slide is stored. The acquired reference image is mapped onto each anatomical image via an image registration algorithm, on which processor 153 is programmed. The processor is further configured to determine the quality of the image registration process by calculating at least a first quality metric and a second quality metric, as described above. If the first and second quality metrics are satisfied, a third quality metric is calculated. Preferably, processor 153 is further configured to perform a step of calculating a fourth quality metric. A quality report of the image registration process is generated for each anatomical slide and is displayed on display 145 of the device. The quality report contains an indication of whether the image registration quality is satisfactory or unsatisfactory, and may also include the value of each calculated quality metric. If the image registration quality of a particular anatomical slide is unsatisfactory, the report appropriately includes an alert and may recommend rejecting the slide for automated dissection or that corrective action should be taken.

[0107] A quality control report of the image registration process can be displayed before, after, or during the annotation of ROI boundaries on the reference image. After annotation, the processor is further configured to transfer the annotations to each anatomical slide based on the transformation matrix obtained from the image registration process. The transferred annotation coordinates are transmitted to the control unit 140 and used to guide the movement of the robotic arm 120 and the corresponding movement of the dissecting tool 110, so that material is removed from the ROI on each tissue slide.

[0108] As will be understood, it is important to dissect with high accuracy and precision the region of interest selected by the pathologist. In fact, several factors affect the accuracy of the achieved anatomical results. These factors include: the accuracy of registering the reference image to the anatomical image; the alignment of the imaging module and the dissecting tools; and the precision and accuracy of scraping and cutting movements on the tissue.

[0109] For automated processes, it is important to determine and evaluate the quality of the achieved dissection results in an automated and reliable manner, thereby determining whether the collected sample material can lead to successful molecular testing. In a preferred embodiment of the dissection apparatus including the system according to the invention, the apparatus is further configured to evaluate the achieved dissection results. Additionally, the processor is programmed with instructions to cause the imaging module to capture a post-dissection image of the dissected slide and compare it with a pre-dissection image, to which ROI labels have been registered. Furthermore, an image processing algorithm is applied, which performs the following steps: The achieved anatomical boundaries are detected by pixel-level comparison between pre-anatomical and post-anatomical images to determine intensity changes. The detected anatomical boundaries are compared with registered labeled boundaries to determine at least one indicator representing the quality of the achieved anatomical result; and Generate a quality report, which includes at least one defined quality indicator.

[0110] An embodiment of the quality indicator used for the achieved anatomical results is the average deviation between the detected boundary and the labeled boundary, and the percentage of tissue dissected from within the labeled boundary.

[0111] In addition to indicators of whether specific quality criteria have been met (e.g., the average deviation between the detected boundary and the labeled boundary is less than a predetermined maximum value), quality scores can be assigned to each determined quality indicator so that an overall quality rating of the anatomical results achieved for a specific slide can be calculated.

[0112] Appropriately, the results of image registration quality determination are also used as an indicator of the quality of the achieved anatomical results.

[0113] In another improvement to the method of the present invention, a quality rating for the image registration process is determined based on each calculated quality metric. A quality score is then calculated based on a corresponding threshold. , Values ​​can be assigned to each calculated quality metric. In one embodiment, a quality score of 1 or 0 is assigned to each quality metric, wherein a score of 1 is assigned when a corresponding threshold is met, and a score of 0 is assigned when the threshold is not met.

[0114] Therefore, an overall quality rating of the image registration process can be obtained, where Therefore, for a positive overall score of 1, each of the calculated quality measures must be satisfied, and a negative overall score of 0 is obtained when any of the calculated values ​​fails to meet its threshold.

[0115] In another embodiment, a score of 0 is assigned to any of the calculated metrics that did not meet the corresponding threshold, and based on its proximity to the corresponding threshold, the quantified score is assigned a "satisfactory" value. Therefore, the overall rating quantifies the quality of the registration process. For example, a first quality score... The results can be assigned to the IoU calculation, with 1 assigned for perfect overlap and a lower score assigned when the overlap is small. Similarly, the second score... The condition number of the skewness and scaling matrix can be assigned to a degree close to 1, and the third score... This can be assigned based on how close the symmetry scaling parameter is to 1. Furthermore, it can be based on the precision of the anatomical tools. Fourth quality score It can be assigned to indefinite parameters. For example, when the calculated value is significantly smaller than... When the calculated value is equal to 1, a score of 1 can be given; when the calculated value is equal to 1, a score of 1 can be given. When this occurs, a score of 0.8 can be given.

[0116] The two methods above can also be combined; for example, a score of 0 or 1 can be assigned based on whether the corresponding threshold is met. to And when the value of the calculated indeterminate parameter is less than the threshold At that time, for example, a quantization score between 0.8 and 1.0 was assigned to... .

[0117] Examples, embodiments, or optional features (whether or not indicated as non-limiting) should not be construed as limiting the claimed invention. It should be noted that the embodiments mentioned above are illustrative rather than limiting of the invention, and those skilled in the art will be able to devise many alternative embodiments without departing from the scope of the appended claims.

[0118] Figure Labels 1. Reference slide 2a-2b Anatomical Slide Series 5. Reference samples stained with H&E 10. Methods for quality control in image registration process 11 Obtain the reference image 12. Obtain anatomical images 20. Steps for registering the reference image onto the anatomical image 21. Detect and match feature pairs in an image. 22. Calculate the transformation matrix for each matching feature pair. 23. Calculate the transformation matrix for image registration as a whole. 30. Steps for calculating the first quality metric in the image registration process. 40. Steps for calculating the second quality metric in the image registration process. 50. Comparison Steps 51. Compare the first quality metric with the corresponding threshold. 52. Compare the second quality metric with the corresponding threshold. 60. Steps for reporting the results of image registration quality determination 70. Steps to calculate the third quality metric and compare it with the threshold. 80. Steps for calculating the fourth quality metric in the image registration process. 81 Confirm Use and Points to be transformed Local spatial differences between 82 Based on determined differences The average value is used to calculate the fourth quality metric. 83. Compare the fourth quality metric with the corresponding threshold. 90 Quality Control System 95 User input interface for image annotation 100 Dissection Equipment 105 Equipment casing 110 Dissection tools 115 Scrapers used in dissection 120 robotic arms including actuators to move dissecting tools 130 tissue slides 135 Slide Holder 137 Equipment support platform 140 Control Unit 145 monitor 150 Imaging Module 151,152 Contrast-enhanced filter elements 153 Image Processor 155 Image Sensor 157 Lens 160 light source 305 Reference image 310 Anatomical images of anatomical slides

Claims

1. A method (10) for quality control during image registration, the method comprising the following steps: Obtain a first image (305) of a reference slide (11) containing a reference sample (5) cut from a tissue sample; Obtain a second image (310) of (12) an anatomical slide (130) containing tissue sections cut from the same sample; The first image is registered (20) onto the second image, wherein the transformation matrix is ​​calculated in sub-step (23). This allows for the generation of an optimal fit between the first and second images. Based on the overlap between the first image and the second image, calculate the first quality metric of the image registration process (30); Based on the parameters indicating the skewness and scaling of the transformation matrix, calculate (40) the second quality metric of the image registration process; Each calculated value of the first and second quality metrics is compared with the corresponding threshold (50) to determine whether each quality metric is satisfied. If either the first quality metric or the second quality metric is not met, the image registration quality is determined to be unsatisfactory, and the determination (60) is reported to the user.

2. The method according to claim 1, wherein the first quality metric is the intersection-over-union ratio (IoU) of the first image (305) and the second image (310), and wherein the method further comprises the following sub-steps: The first and second images are segmented to identify the tissue pixels in each image; Calculate the number of overlapping tissue pixels in two images. And calculate the total number of tissue pixels in the two combined images. ; The step (30) of calculating the first quality metric includes calculating the IoU as Divide by .

3. The method of claim 2, wherein the calculated IoU value is compared with an IoU threshold, and wherein the method comprises the following additional step: The IoU threshold is set based on a polynomial function that depends on the tissue area, which is obtained from the number of identified tissue pixels in the first image.

4. The method according to any of the preceding claims, wherein the step (40) of calculating the second quality metric comprises decomposing a subset of the transformation matrix into a first matrix representing rotation and a second matrix representing skewness and scaling, the subset containing coefficients defining rotation, scaling and skewness, and wherein the parameter of the calculated second quality metric is the condition number of the second matrix.

5. The method according to claim 4, the method comprising another step (70) of calculating a third quality metric for the image registration process, wherein: The third quality metric is a parameter that indicates the symmetric scaling of the second matrix; Assuming the skewness is negligible, step (70) includes calculating the 2-norm of the second matrix; and If the calculated 2-norm value is not within the predetermined lower and upper thresholds, then the quality of the image registration process is deemed unsatisfactory.

6. The method according to any of the preceding claims, the method comprising an additional step (80) of calculating a fourth quality metric of the image registration process, wherein the fourth metric is a quantized indeterminate parameter that provides a measure of the spatial accuracy of the image alignment in the registration process (20).

7. The method according to claim 6, further comprising the following steps: The boundaries of the region of interest (ROI) to be dissected on the first image are marked; as well as Using the transformation matrix calculated in the sub-step (23) of image registration ( The ROI annotations are transferred to the second image. The coordinates of the transferred ROI are used to control the dissection tool (120) during automated dissection, and where: The calculated value of the variable parameter is compared with a threshold derived from the typical accuracy of the dissecting tools used in the automated dissection process, and if the variable parameter exceeds the threshold, the quality of the image registration is determined to be unsatisfactory.

8. The method according to claim 6 or 7, which is dependent on claim 2, further comprising determining the average diameter of tissue objects detected during a sub-step of identifying tissue pixels in a second image, wherein the step (80) of calculating quantized indeterminate parameters comprises multiplying the calculated IoU value by the determined average diameter.

9. The method according to claim 6 or 7, wherein the step of image registration (20) includes detecting (21) feature pairs in the first image and the second image, and calculating (22) the individual transformation matrix for each corresponding feature pair. ), and the calculation of the indeterminate parameter (80) includes: Key points identifying each feature in the first image (305) Each feature in the first image (305) has been detected and matched to the corresponding feature in the second image (310); Using the calculated transformation matrix of the image registration process ( And use the individual transformation matrix associated with this feature ( For each identified key point ( Transformation, and based on Determine the local spatial differences between the two transformations. .

10. The method of claim 9, wherein the indeterminate parameter is calculated based on the average of determined local spatial differences.

11. The method according to claim 9, which is dependent on claim 7, further comprising the steps of: Each spatial difference calculated Mapped onto the second image, and Each difference Points associated with ROI annotations on the first image (305) are used to calculate local values ​​of indeterminate parameters for all annotated points.

12. The method according to any one of claims 6 to 11, the method further comprising the following steps: The quality score is assigned to the value of each calculated quality metric; as well as The overall quality rating of the image registration process is calculated based on each assigned quality score.

13. A quality control system (90) for an image registration process, the system comprising an imaging module (150) and a display (145), the imaging module (150) having an image sensor (155) and a processor (153) programmed with an image registration algorithm and image processing software, wherein the processor (153) is further programmed with instructions that cause the system to perform the steps of the method according to any one of the preceding claims.

14. An apparatus for automated dissection, the apparatus comprising the quality control system (90) of claim 13, the apparatus (100) further comprising a movable dissection tool (110) and a controller (140) for controlling the movement of the tool based on coordinates received from the imaging module (150), wherein the received coordinates indicate the labeled boundaries of a region of interest (ROI) transferred from a first image (305) of a reference slide (1) to a second image of a dissection slide (130).

15. The device of claim 14, wherein the processor (153) is programmed to perform the method of claim 12 and is further programmed to evaluate the quality of the achieved anatomical result using an overall quality rating of the calculated image registration process.

Images