Computer-implemented method for quality control of digital images of samples

Abstract

A computer-implemented method is proposed for quality control of at least one digital image of a sample (112) placed on a slide (114). The method includes the following steps: a) providing at least one digital image of the sample (112) placed on the slide (114) using at least one imaging device (116) of a slide imaging device (110); b) determining the quality of the digital image by using at least one edge detection image filter to determine the sharpness value of a sub-region of at least one region of interest in the digital image and comparing the sharpness values ​​within the region of interest, wherein the quality of the region of interest is classified according to the comparison; c) generating at least one indication based on the classification of the quality, wherein steps b) and c) are performed automatically.

Description

Technical Field

[0001] This disclosure relates to a computer-implemented method for quality control of at least one digital image of a sample, a computer-implemented method for determining focus quality to determine at least one digital image of a sample placed on a slide, a computer-implemented method for determining a three-dimensional digital image of a sample placed on a slide using at least one imaging device of a slide imaging apparatus, a computer-implemented method for training machine and deep learning models to analyze at least one digital image of a sample placed on a slide, and a slide imaging apparatus. The apparatus and methods described herein are preferably used in digital pathology; however, many other applications are possible. Background Technology

[0002] Various methods and apparatuses are known for analyzing digital images used in digital pathology (i.e., digitization of tissue slides). Several problems may arise regarding the analysis of digital images used in digital pathology.

[0003] Specifically, to accurately analyze artifacts in digital images, integrated automated analysis of clinical digital workflows used in computational pathology is required. To avoid errors, out-of-focus areas are labeled in known techniques because these areas can hinder classification using algorithms. Labeling out-of-focus areas requires manual examination of digital images by laboratory professionals. This is time-consuming and particularly difficult to detect in-focus areas because slides need to be examined at high magnification (e.g., 40x). Janowczyk A, Zuo R, Gilmore H, Feldman M, Madabhushi A. HistoQC: “An Open-Source Quality Control Tool for Digital Pathology Slides.”, JCO Clin CancerInform. 2019; 3:1-7 describes a tool for performing quality control to identify and delineate artifacts and discover cluster-level outliers. This tool uses a combination of image metrics (e.g., color histogram, brightness, contrast), features (e.g., edge detectors), and supervised classifiers (e.g., light pen detection) to identify artifact-free regions on digitized slides. Typically, as mentioned above, automated tools for analyzing digital images require sharp images, necessitating quality control to ensure the algorithm's suitability for digital images. Current measurements for defining sharpness consider two methods: Michelson contrast and RMS contrast. The applied image settings, features, and supervised classifiers provide quality control for digital images. However, these strategies are based on mixed-signal results and do not directly define the applicability of the algorithm.

[0004] Furthermore, it is well known that multiple focusing depths, also known as z-stacking, have different effects on defocusing (see Kohlberger T, Liu Y, Moran M, et al., “Whole-Slide Image Focus Quality: Automatic Assessment and Impact on AI Cancer Detection,” J Pathol Inform, 2019). However, this approach increases scan time and file size to impractical levels. Moreover, it can be difficult to classify general defocusing. Typically, it can be checked using graphs to see if the focus coil is set to its nominal position. However, this approach may still leave room for interpretation.

[0005] In addition to tangential sections, 3D perspective views may be needed for digital pathology, such as to increase the orientation possibilities on the slides. 3D methods offer improved insights into architectural features and specific arrangements. Various attempts have been made to reconstruct 3D images from 2D images. Most attempts are based on manual segmentation of slides, which is a rather time-consuming and labor-intensive task, see, for example, Jansen I, Lucas M, Savci-Heijink CD et al., “Histopathology: ditch the slides, because digital and 3D are on show”, World J Urol. 2018; 36(4):549-555. doi:10.1007 / s00345-018-2202-12. In addition to the labor-intensive process, problems in the current literature are also limiting factors for the use of this technique. High-resolution 3D datasets are difficult to visualize. Therefore, minimizing the portion of the dataset shown is crucial. In addition, low out-of-plane resolution must be avoided.

[0006] Out-of-focus areas can indeed negatively impact the accuracy of algorithm performance. If the out-of-focus area is significant before a pathologist's examination, the slide needs to be rescanned; see, for example, Liu Y, Kohlberger T, Norouzi M, Dahl GE, Smith JL, Mohtashamian A et al., "Artificial Intelligence-Based Breast Cancer Nodal Metastasis Detection. Arch Pathol Lab Med. 2018". To train the algorithm, various tissue morphologies may need to be captured. To train the algorithm to recognize out-of-focus areas, this artifact is simulated using real focused images via synthetic Gaussian blur or bokeh blur. Kohlberger et al. (2019) showed that by adding contamination noise or JPEG artifacts, out-of-focus areas can become even more realistic, providing a good tool for training the algorithm. However, even with this approach, robust and reliable results may not be achievable with automated algorithms in cases where out-of-focus areas exist but the algorithm aborts.

[0007] Therefore, despite the implementation of the above technologies, there is still a need to improve the quality of digital images and the quality of automated analysis of digital images used in computational pathology.

[0008] Problems to be solved

[0009] Therefore, the object of the present invention is to provide a computer-implemented method for quality control of at least one digital image of a sample, a computer-implemented method for determining focus quality to determine at least one digital image of a sample placed on a slide, a computer-implemented method for determining a three-dimensional digital image of a sample placed on a slide using at least one imaging device of a slide imaging apparatus, a computer-implemented method for training machine and deep learning models to analyze at least one digital image of a sample placed on a slide, and a slide imaging apparatus, which at least partially avoid the disadvantages of such known devices and methods and at least partially solve the aforementioned challenges. Specifically, the quality of digital images and the quality of automated analysis of digital images for computational pathology should be improved. Summary of the Invention

[0010] This problem is solved by the following: a computer-implemented method for quality control of at least one digital image of a sample; a computer-implemented method for determining focus quality to determine at least one digital image of a sample placed on a slide; a computer-implemented method for determining a three-dimensional digital image of a sample placed on a slide using at least one imaging device of a slide imaging apparatus; a computer-implemented method for training machine and deep learning models to analyze at least one digital image of a sample placed on a slide; and a slide imaging apparatus having the features described in the independent claims. Advantageous embodiments that can be implemented individually or in any combination are set forth in the dependent claims and throughout the specification.

[0011] As used below, the terms “have,” “include,” or “contain,” or any of their arbitrary grammatical variations, are used in a non-exclusive manner. Thus, these terms can refer either to a situation where no other features exist in the entity described in this context besides those introduced by these terms, or to a situation where one or more other features exist. For example, the statements “A has B,” “A includes B,” and “A contains B” can refer either to a situation where no other elements exist in A besides B (i.e., where A is solely and uniquely composed of B), or to a situation where one or more other elements (such as element C, element D, or even other elements) exist in entity A besides B.

[0012] Furthermore, it should be noted that the terms "at least one," "one or more," or similar expressions indicating that a feature or element may exist once or more are generally used only once when the corresponding feature or element is introduced. In the following text, in most cases, when referring to the corresponding feature or element, the expressions "at least one" or "one or more" will not be used repeatedly, even though the corresponding feature or element may exist only once or more.

[0013] Furthermore, as used below, the terms “preferredly,” “more preferably,” “particularly,” “more particularly,” “specifically,” “more specifically,” or similar terms are used in combination with optional features without limiting the possibility of alternatives. Therefore, features introduced by these terms are optional features and are not intended to limit the scope of the claims in any way. As those skilled in the art will recognize, the invention can be carried out by using alternative features. Similarly, features introduced by “in one embodiment of the invention” or similar expressions are intended to be optional features without limiting alternative embodiments of the invention, without limiting the scope of the invention, and without limiting the possibility of combining features introduced in this way with other optional or non-optional features of the invention.

[0014] In a first aspect of the invention, a computer-implemented method is provided for quality control of at least one digital image of a sample placed on a glass slide.

[0015] As used herein, the term "computer-implemented" is a broad term and is given a common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, a process wholly or partially implemented using a data processing apparatus (such as a data processing apparatus including at least one processing unit). Therefore, the term "computer" can generally refer to an apparatus having at least one data processing apparatus (such as at least one processing unit), or a combination or network of apparatuses. Additionally, a computer may include one or more other components (such as at least one of a data storage device, an electronic interface, or a human-machine interface).

[0016] As used herein, the term "processing unit" is a broad term and is given a common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, any logic circuit configured to perform basic operations of a computer or system; and / or, generally, a device configured to perform computational or logical operations. In particular, a processing unit may be configured to process basic instructions that drive a computer or system. As an example, a processing unit may include at least one arithmetic logic unit (ALU), at least one floating-point unit (FPU) (such as a math coprocessor or numerical coprocessor), multiple registers (specifically registers configured to provide operands to the ALU and store the results of operations), and memory (such as L1 and L2 caches). In particular, a processing unit may be a multi-core processor. Specifically, a processing unit may be or may include a central processing unit (CPU). Additionally or alternatively, a processing unit may be or may include a microprocessor; therefore, specifically, the elements of the processing unit may be contained within a single integrated circuit (IC) chip. Alternatively or concurrently, the processing unit may be or may include one or more application-specific integrated circuits (ASICs) and / or one or more field-programmable gate arrays (FPGAs), etc.

[0017] As used herein, the term "sample" is a broad term and is given a common and conventional meaning to those skilled in the art, and is not limited to a particular or customary meaning. The term may specifically refer to, but is not limited to, biological samples, such as tissue samples. A sample may be or may include biological materials, such as tissue or smears. However, other types of samples may also be applicable.

[0018] As used herein, the term "slide" is a broad term and is given a common and customary meaning to those skilled in the art, and is not limited to a specific or customary meaning. The term may specifically refer to, but is not limited to, a substrate designated for placing a sample on the surface of a slide. In particular, for the purpose of bearing the sample without alteration during the handling of the slide, the substrate is mechanically stable and can therefore include any material that provides sufficient mechanical stability. Especially for the purpose of bearing biological samples, the substrate may preferably have a surface configured to be compatible with biological materials. For example, the slide is a glass slide, as glass is known to provide sufficient mechanical stability on the one hand and high compatibility with biological materials on the other. However, other types of materials for the slide are also feasible. For the purpose of generating a desired image of the sample, the slide may preferably be a plate having a 2D extension and a certain thickness, wherein the 2D extension of the plate may preferably be rectangular or circular in form, and wherein the thickness of the plate may be small compared to the dimension of the extension, preferably 20%, more preferably 10%, particularly 5%, or less than a measure of the linear range of the 2D extension of the plate.

[0019] Furthermore, the slide may particularly be in a form that enables imaging of a sample placed on the slide. As used herein, the terms “imaging” or “generating an image” are broad terms and are given a common and customary meaning to those skilled in the art, and are not limited to a specific or customary meaning. These terms may specifically refer to, but are not limited to, a 2D two-dimensional representation of at least one property of a sample, a two-dimensional representation also referred to by the term “image,” which is generally processed and displayed on a screen for a viewer to observe with their eyes, preferably without any further aids other than the viewer’s glasses. For this purpose, imaging apparatuses disclosed in more detail below are generally used. As used herein, the term “digital image” is a broad term and is given a common and customary meaning to those skilled in the art, and is not limited to a specific or customary meaning. This term may specifically refer to, but is not limited to, a discrete and discontinuous representation of an image. Thus, the term “digital image” may refer to a two-dimensional function f(x,y) where intensity and / or color values ​​are given for any x,y position in the digital image, where the position may be discretized, corresponding to a recorded pixel of the digital image.

[0020] As used herein, the term "quality" is a broad term and will be given a common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, the state of a digital image in relation to the suitability of the image being automatically analyzed. Specifically, the term "quality" refers to an indication of the sharpness, blurriness, or amount of defocus in a digital image.

[0021] As used herein, the term "sharpness" is a broad term and will be given a common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, measures of resolution and sharpness. Specifically, sharpness can refer to an attribute of a digital image that indicates the separability and / or distinguishability of different regions of the digital image by using one or more of color values, grayscale, intensity, and luminance. Sharpness may be related to the contrast between regions of a digital image (i.e., differences or variations in the luminance and / or color values ​​of the regions).

[0022] As used herein, the term "quality control" is a broad term and will be given a common and customary meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, monitoring and / or determining the sharpness of a digital image and / or the sharpness of at least one region of interest in a digital image.

[0023] The computer-implemented method includes the following steps, which can be performed in a given order. However, a different order is also possible. Furthermore, one or more, or even all, of the steps can be performed once or repeatedly. Additionally, the method steps can be performed in a time-overlapping manner or even in parallel. The method may further include other method steps not listed.

[0024] The method includes the following steps:

[0025] a) Using at least one imaging device of a slide imaging apparatus to provide at least one digital image of a sample placed on a slide;

[0026] b) Determine the quality of the digital image by using at least one edge detection image filter to determine the sharpness value of a sub-region of at least one region of interest and by comparing the sharpness values ​​within the region of interest, wherein the quality of the region of interest is classified according to the comparison;

[0027] c) Generate at least one indication based on the quality classification.

[0028] Steps b) and c) are executed automatically.

[0029] As used herein, the term "providing a digital image" is a broad term and will be given a common and customary meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, imaging and / or generating at least one digital image.

[0030] As used herein, the term "slide imaging apparatus" is a broad term and is to be given a common and customary meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, any device configured for imaging a sample placed on a slide. Furthermore, as used herein, the terms "apparatus" and "slide imaging apparatus" are broad terms and are to be given a common and customary meaning to those skilled in the art, and are not limited to a specific or customary meaning. These terms may specifically refer to, but are not limited to, an apparatus having the plurality of components disclosed in more detail below.

[0031] A slide imaging apparatus includes at least one imaging device. As used herein, the term "imaging device" is a broad term and is given a common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. The term may specifically refer to, but is not limited to, a device designated for generating a 2D representation of at least one visual attribute of a sample. In particular, the imaging device may be selected from a 2D camera or a line scan detector. However, other types of imaging devices may also be feasible.

[0032] Slide imaging apparatus may include a storage device that can hold multiple slides and is configured to store slides. As used herein, the term "storage device" is a broad term and is given a common and customary meaning to those skilled in the art, and is not limited to a specific or customary meaning. The term may specifically refer to, but is not limited to, a slide storage library designated for receiving individual slide holders or, alternatively, more than one slide holder simultaneously, wherein each slide holder is configured to hold more than one slide. Storage devices may be selected from slide trays or slide racks; however, other types of storage devices may also be feasible. The storage device may preferably be manually loaded with at least one slide; however, automatic loading of the storage device is also conceivable.

[0033] A slide imaging apparatus may include a supply device configured to supply at least one slide from a storage device to an imaging device. As used herein, the term "supply device" is a broad term and is given a common and customary meaning to those skilled in the art, and is not limited to a specific or customary meaning. The term may specifically refer to, but is not limited to, a device configured to transfer a slide from a storage device to an imaging device. For this purpose, the supply device may include a robotic arm.

[0034] As used herein, the term "region of interest" is a broad term and will be given a common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, a region or area of ​​any shape of a digital image to be analyzed. The region of interest may be the entire digital image or a portion of a digital image. The region of interest may be a region of an image that includes or is suspected of including at least one feature to be analyzed. The digital image may be a pixelated image comprising a plurality of pixels, arranged, for example, in a pixel array, such as a rectangular array, having m rows and n columns, where m and n are independently positive integers. The region of interest may be or may include a group of pixels comprising any number of pixels. As used herein, the term "subregion" is a broad term and will be given a common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, any portion or element of a region of interest comprising at least one pixel or group of pixels, particularly an image element. A subregion may be a square area of ​​the region of interest. The region of interest may include multiple subregions, such as multiple pixels.

[0035] As used herein, the term "edge" is a broad term and will be given a common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, areas in a digital image where the intensity and / or color values ​​have local variations and / or discontinuities.

[0036] As used herein, the term "edge detection image filter" is a broad term and will be given a common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, at least one image processing tool configured to identify edges in regions of interest. Preferably, the edge detection image filter can be a Laplacian filter. The Laplacian filter can be based on the second derivative of intensity and / or color values, and thus can determine local extrema. The edge detection image filter can be configured for edge detection based on local extrema. In particular, the edge detection filter can emphasize strong edges and / or gradients. Specifically, the method may include defining the sharpness of a digital image at the sub-region level, preferably at the pixel level, based on a parameter defined by the Laplacian filter. The edge detection filter can be configured to assign grayscale values ​​to each sub-region within the sub-region according to the corresponding local extrema. Using a Laplacian filter may be advantageous because it eliminates the need to rotate the kernel to obtain the x or y direction, as is done with first-derivative filters such as the Sobel or Canny filters.

[0037] This invention proposes using a Laplacian filter as an edge and / or gradient filter to define sharpness. The use of a Laplacian filter to define sharpness has not been previously proposed because it is a derivative filter commonly used to find rapidly changing regions in an image. For other applications such as image processing, such as edge detection and motion estimation, the discrete Laplacian operator is widely used, but not for defining sharpness and comparing sharpness values ​​within a region of interest.

[0038] As used herein, the term "sharpness value" is a broad term and will be given a common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, local extrema of intensity and / or color values. Sharpness value may also refer to color gradients or intensity gradients.

[0039] As used herein, the term "comparing sharpness values" is a broad term and will be given a common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, the process of comparing local extrema of subregions of a region of interest. This method may include determining the maximum local extremum of the region of interest. The maximum local extremum may be the highest local extremum of a subregion of the region of interest. This method may include the minimum local extremum of the region of interest. The minimum local extremum may be the lowest local extremum of a subregion of the region of interest. This method may include sorting the local extrema of the subregions in descending order from the maximum local extremum to the minimum local extremum. However, ascending order sorting is also possible. The comparison may include at least one mathematical operation for determining the relationship between the local extrema.

[0040] The quality of regions of interest is classified based on comparison. Maximum local extrema can be classified as highest resolution. Minimum local extrema can be classified as lowest resolution. Local extrema of sub-regions can be classified in descending order from maximum to minimum local extrema.

[0041] The method may also include setting a defocus threshold. As used herein, the term "defocus threshold" is a broad term and will be given a common and conventional meaning to those skilled in the art, and is not limited to a specific or custom-defined meaning. Specifically, the term may refer to, but is not limited to, a threshold that defines the minimum value of a maximum local extremum. For example, the defocus threshold may be set to a high-quality absolute sharpness that is achieved as the minimum value of a maximum local extremum in 99% of a set of controlled images. If the determined maximum local extremum in the region of interest is below the defocus threshold, the digital image may be rejected and / or classified as low quality. Using a defocus threshold allows for the prevention of identifying completely out-of-focus digital images.

[0042] As described above, the method includes the step of generating at least one indication based on a quality classification. As used herein, the term "indicator" is a broad term and will be given a common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. The term may specifically refer to, but is not limited to, outputs such as a visualization of the quality of a region of interest. For example, an indication may be a graphical indication including a heatmap (also denoted as a focus map). A heatmap can be generated by mapping sharpness values ​​of sub-regions according to their image coordinates. Generating a heatmap may include assigning color values ​​from a color space to each sharpness value in the sharpness values. For example, the color space may be the RGB color space. The RGB color space may be a color space with three color channels: one channel for red (R), one channel for green (G), and one channel for blue (B). The color space can range from 255, representing high-quality white, to 0, representing out-of-focus black. Therefore, the maximum value of white, 255, may represent the sharpest part of a digital image. A heatmap may include only one or two colors (such as only red and yellow color values), ranging from 255, representing high-quality white, to 0, representing out-of-focus black.

[0043] For example, a sub-region can correspond to pixels in a digital image. Generating a heatmap may include selecting a group of pixels within a region of interest, determining an average sharpness value for each pixel group within that group, and mapping the average sharpness value to its image coordinates. Specifically, generating a heatmap may include determining the average grayscale value within a pixel group. The average grayscale value can be determined by calculating the sum of all grayscale values ​​and dividing by the number of sub-regions of the image region, specifically the number of pixels. To calculate the average grayscale value, black pixels may be excluded or discarded. Black pixels can be pixels that have no organizational contrast and cannot be sharpened or blurred. Therefore, the term pixel can refer to a descriptive measurement.

[0044] The quality determined based on sharpness values ​​may be directly related to the applicability of classification algorithms used in computational pathology. For example, if the quality of a sub-region of a digital image is above a predefined threshold, at least one classification algorithm can be used to further evaluate the digital image. For example, the predefined threshold could be 15% of the maximum sharpness value. Similarly, if the percentage of a sub-region of a digital image is below a predefined threshold, at least one classification algorithm can be used to further evaluate the digital image. For example, the predefined threshold could be as high as 10% of the entire slide image. The generated indicators can allow the identification of artifacts caused by staining, tissue folding, and blurring.

[0045] This method may include displaying instructions, particularly heatmaps, using at least one user interface. As used herein, the term "user interface" is a broad term and will be given a common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, human-machine interfaces configured to display instructions, such as displays, screens, etc.

[0046] If necessary, the method may include automatic sorting of new scans. Specifically, in cases where the heatmap shows artifacts or blurring in unwanted image locations, the method may include automatically repeating steps a) through c).

[0047] This method can be executed automatically, preferably fully automatically, without any manual steps or user interaction. However, embodiments in which step a) includes manual steps as described above (such as loading a sample) may be feasible. Therefore, at least step b) determining the quality of the digital image and step c) generating the indication are performed fully automatically. Fully automated determination of quality and fully automated generation of indication may be superior to human eye control of digital images in identifying out-of-focus areas and artifacts.

[0048] Steps b) and c) can be performed using at least one control and evaluation device. As used further herein, the term "control and evaluation device" generally refers to any device configured to perform a specified operation, preferably by using at least one data processing device, and more preferably by using at least one processor and / or at least one application-specific integrated circuit (ASIC). Thus, by way of example, at least one control and evaluation device may include at least one data processing device having software code stored thereon, the software code comprising a plurality of computer commands. The control and evaluation device may provide one or more hardware elements for performing one or more specified operations, and / or may provide software running thereon to one or more processors for performing one or more specified operations. The control and evaluation device may include one or more programmable devices, such as one or more computers, application-specific integrated circuits (ASICs), digital signal processors (DSPs), or field-programmable gate arrays (FPGAs), configured to perform steps b) and c). However, additionally or alternatively, the control and evaluation device may also be implemented entirely or partially in hardware.

[0049] For example, the quality control described in this invention can be performed as follows. A digital image with a width of less than 12,000 pixels can be selected from the image pyramid. Only red channel values ​​can be used from the RGB color space. A binomial smoothing filter, approximating a Gaussian filter, can be applied to eliminate small high-frequency interference by transforming the intensity and / or color values ​​of pixels, for example, with a filter size of 7*7 pixels. In the next step, a Laplacian filter based on the second derivative can be applied for edge detection considering 8 connected pixels (3x3 pixels) in the neighborhood. For the heatmap, red channel values ​​in the RGB color space can be used, where white 255 represents the sharpest part of the image and black 0 represents out-of-focus or no information at all. The out-of-focus sub-region can be defined based on a predefined threshold of 12% of the maximum sharpness value. The sub-region exists within a directly related n pixels, which can be related to its neighborhood of 8 connected pixels. To avoid excessively small unsharpened areas, a threshold of 1.5% of the total number of pixels can be set, which can be calculated by converting the number of pixels n into a percentage of the total number of pixels in the image.

[0050] The method for quality control according to the invention allows for the determination of the sharpness of digital images. Therefore, when a digital image is out of focus, it can be marked to avoid one or more of the following: loss of blurry areas, delays due to poor slide images, and manual pre-scanning of the digital image. The method for quality control according to the invention allows for automated, integrated image analysis.

[0051] In another aspect of the invention, a computer-implemented method is disclosed for determining focus quality to determine at least one digital image of a sample placed on a glass slide. The method includes method steps that may be performed in a specific order. However, different orders are also possible. It is also possible to perform two or more method steps simultaneously, either fully or partially. Furthermore, one or more, or even all, of the method steps may be performed once or repeatedly, such as repeated once or multiple times. Additionally, the method may include additional method steps not listed.

[0052] The method includes the following steps:

[0053] i) Using at least one imaging device of a slide imaging apparatus to determine a z-stack of digital images of a sample placed on a slide, wherein the imaging device includes at least one transmission device having a focal length, and wherein the z-stack includes a plurality of digital images determined at at least three different distances between the transmission device and the slide.

[0054] ii) Determine information about the sharpness of multiple image regions of each digital image in a z-stacked digital image by using a computer-implemented method for quality control according to the present invention for quality control as described above or in more detail below;

[0055] iii) Generate a graphical indication of sharpness information based on the distance to each image region in the image region.

[0056] As described above, step ii) includes using a computer-implemented method for quality control according to the present invention for quality control, as described above or in more detail below. Therefore, for possible definitions, options, or embodiments, reference can be made to the description given above.

[0057] As used herein, the term "transmission device" is a broad term and will be given a common and customary meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, one or more optical elements having a focal length in response to an illuminating light beam. A transmission device may specifically include one or more of the following: at least one lens, for example, at least one lens selected from the group consisting of at least one refractive lens, at least one adjustable focusing lens, at least one aspherical lens, at least one spherical lens, and at least one Fresnel lens; at least one diffractive optical element; at least one multi-lens system. As used herein, the term "focal length" is a broad term and will be given a common and customary meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, the distance at which an incident collimated ray illuminating the transmission device can be focused, which can also be expressed as a focal point. Therefore, the focal length constitutes a measure of the transmission device's ability to converge an illuminating light beam.

[0058] The transmission device can form a coordinate system, where "z" is the coordinate along the optical axis. The coordinate system can be a polar coordinate system, where the optical axis of the transmission device forms the z-axis, and the distance from the z-axis and the polar angle can be used as additional coordinates. The coordinate along the z-axis can be considered as the longitudinal coordinate z.

[0059] As used herein, the term "z-stack" is a broad term and will be given a common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, a set of at least three digital images imaged at at least three different distances (i.e., at different z-levels) between the transport device and the slide. Specifically, two digital images may be defined on a layer above and below the focal plane. The z-stacked digital images may include high-magnification views, such as those with a magnification of 20x or 40x.

[0060] Step i) may include imaging the first digital image using an imaging device. For example, the first digital image may be imaged in a so-called seed focus plane, which is a recommended optimal focus plane automatically determined by a slide imaging device. For example, the slide imaging device may include at least one database storing information about the seed focus plane.

[0061] To determine the z-stack, two layers are added at a time. For example, two additional digital images can be imaged at a distance ±ε from the layer containing the focal point, where ε is a positive number. Specifically, one digital image can be imaged above the image plane of the first digital image, and another digital image can be imaged below the image plane of the first digital image. The distance between the z-stacked digital images can be defined based on the relative intensity and / or color variation determined in step b) of the method for quality control. Specifically, the distance can be defined such that the sharpness values ​​of the two layers' digital images can differ in the region of interest. The distance can be defined such that the sharpness values ​​of the two layers' digital images can differ beyond a predetermined tolerance, such as more than 5%. The distance ±ε from the layer containing the focal point can be selected based on accuracy. For example, the z-stack can include multiple digital images determined at a distance ε = 1 μm, which is defined by the depth of field and the size of a known object. However, other values ​​for the distance ε and / or non-equidistant distances are also possible. In the example of the z-stack, the distance ε can be 0.1 μm, 0.2 μm, 0.25 μm, or 0.5 μm. In addition, you can select 1, 3, 5, 7, 9, 11, 13 and 15 layers.

[0062] As used herein, the term "focus quality" is a broad term and will be given a common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, a measure of the amount of out-of-focus area at different z-levels. For example, if the sharpness value is above a predefined threshold, the focus quality can be determined to be good or adequate; otherwise, it can be determined to be poor or inadequate. For example, the predefined threshold could be 15% of the maximum sharpness value.

[0063] Graphical indicators may include sharpness values ​​plotted at the z-level, particularly for a predefined number of samples. The determination of sharpness information and the generation of graphical indicators can be performed using at least one control and evaluation device.

[0064] This method may include adjusting the focus settings of the transmission device for different distances between the transmission device and the slide according to graphical indications. The process of adjusting the height of the focusing mechanism assembly involves adjusting the depth of field. The height of the focusing mechanism, including the focusing coil, can be adjusted such that the highest average sharpness is obtained when it begins focusing at z=0. The focusing mechanism assembly, including the transmission device, can be set to its extreme values ​​without exceeding the specification level. To check the height setting of the nominal focus position of the focusing coil, the focusing coil is deactivated and a line track from the focusing camera is displayed. The line track represents the received signal based on the crenellated optics. The accuracy of the displayed curve indicates whether the height of the focusing mechanism assembly, including the focusing coil and the transmission device, is correctly adjusted. If the three colors of the upper horizontal bar are separated, the line track is acceptable. In known devices, the focusing mechanism assembly can be fixed with screws and must be adjusted manually. This invention proposes an automated method for adjusting the height of the focusing mechanism assembly based on quality control as described herein. This automation method is particularly useful because testing must be performed precisely and is time-consuming. In contrast to the manual method described by Kohlberger T, Liu Y, Moran M et al. in “Whole-Slide Image FocusQuality: Automatic Assessment and Impact on AI Cancer Detection”, J Pathol Inform, 2019, where numerous tests must be performed before an average quality metric is reached to define z=0 based on a strong “V”-shaped trend for the defocus class at the z-level, the method proposed in this invention is labor-efficient due to its automation.

[0065] The focus settings can be adjusted automatically to adjust the depth of field. Specifically, the determination of z-stack, the determination of information about sharpness, and the generation of graphic indicators can be performed automatically, such as by using at least one control and evaluation device.

[0066] The method may also include, for example, displaying graphical indicators by using at least one user interface.

[0067] The method for determining focus quality according to the present invention allows for a reduction in the statistical probability of obtaining a defocused area at z=0. When optimally set, the predicted probability of blurring at z=0 should be the lowest possible value, but its statistical bias is not 0%.

[0068] In another aspect of the invention, a computer-implemented method is disclosed for determining a three-dimensional digital image of a sample placed on a slide using at least one imaging device of a slide imaging apparatus. The imaging device includes at least one transmission device having a focal length. The method includes method steps that may be specifically performed in a given order. However, different orders are also possible. It is also possible to perform two or more method steps simultaneously, either wholly or partially. Furthermore, one or more, or even all, of the method steps may be performed once or repeatedly, such as repeated once or multiple times. In addition, the method may include additional method steps not listed.

[0069] The method includes the following steps:

[0070] - Determine the z-stack of two-dimensional digital images, wherein the z-stack is determined by imaging the sample at at least three different distances between the slide and the transport device, starting in a seed focusing plane, wherein the seed focusing plane is a recommended optimal focusing plane determined by a slide imaging device, wherein the first two-dimensional image imaged in the seed focusing plane has a first sharpness value, wherein the distance is defined based on the sharpness value, the sharpness value being determined by using a computer-implemented method for quality control according to the present invention for quality control as described above or in more detail below, such that the sharpness values ​​of the two-dimensional digital images determined at different distances are different from the first sharpness value and are different from each other;

[0071] - Select the focus area of ​​each digital 2D image in the z-stack by using an indication based on quality classification;

[0072] - A three-dimensional digital image is determined by combining information about the focal plane of the seed and the focal area of ​​each digital two-dimensional image of the z-stack.

[0073] For possible definitions, choices, or implementations, please refer to the description of the methods given above.

[0074] The method according to the present invention proposes a 3D image construction based on z-stacking, where z = 0 has the optimal focus value.

[0075] Digital images can be digital images magnified 20x or 40x.

[0076] The distance between the z-stacked digital images can be defined based on the relative intensity and / or color variation determined in step b) of the method for quality control. Specifically, the distance can be defined such that the sharpness values ​​of the digital images of the two layers can differ in the region of interest. The distance can be defined such that the sharpness values ​​of the digital images of the two layers can differ beyond a predetermined tolerance, such as exceeding at least 5%. The distance ±ε from the layer including the focal point can be selected based on accuracy. For example, z-stack can include multiple digital images determined at a distance ε = 1 μm, which is defined by the depth of field and the size of the known object. However, other values ​​for distance ε and / or non-equidistant distances are also possible. In the example of z-stack, distance ε can be 0.1 μm, 0.2 μm, 0.25 μm, 0.5 μm. Furthermore, 1, 3, 5, 7, 9, 11, 13, and 15 layers can be selected.

[0077] Slide imaging devices can define a focal plane for each slide by picking up a seed point (also referred to as a seeding point) near the highest vertical extension on the lower edge of the sample based on the z-stack. Therefore, an optimal focal plane, i.e., the seed plane, can be defined. The slide imaging device can be configured to continuously update the focal plane based on data derived from a dynamic forward-looking focus tracking mechanism during scanning of the digital image. Specifically, the separation mirror and crenellated optics allow the focusing camera next to the imaging camera to scan simultaneously to determine if the signal is in focus. This information can be directly used to adjust the output of the dynamic focusing coil in the focusing mechanism assembly, thereby adjusting the height of the transmission device. A forward-looking tracking mechanism can be applied for each layer of the z-stack.

[0078] The first step of the method may include imaging a first two-dimensional digital image in a seed plane and imaging at least two additional two-dimensional digital images at two different distances between the slide and the transmission device. In each two-dimensional digital image, due to the shallow depth of field, only the regions of the corresponding digital image located on the same z-plane are focused.

[0079] In the next step, all two-dimensional digital images can be analyzed, particularly by using control and evaluation devices to select “sharp” image regions from each two-dimensional image. As used herein, the term “focus area” is a broad term and will be given a common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, regions of a digital image having a sharpness value higher than a predefined threshold. For example, the predefined threshold could be 15% of the maximum sharpness value.

[0080] 3D information can be generated through focus depth. Specifically, a 3D image can be determined by combining the defined focus area with information about the corresponding focus plane. Since the focus plane is updated during the scanning of the digital image, low out-of-plane resolution can be largely avoided. By integrating height information, 3D digital image reconstruction is possible.

[0081] The method may also include displaying a three-dimensional digital image. The three-dimensional digital image can be displayed using at least one user interface.

[0082] This method can include coloring a 3D digital image by applying color information (i.e., a texture image) to individual voxels, 3D pixels. By texturing using x, y, and z coordinates, color can be defined for each pixel. This allows for better object recognition through improved reference comparison.

[0083] The method for determining three-dimensional digital images according to the present invention allows for a suitable, non-labor-intensive process because two-dimensional images in opposite directions on a z-stack can be automatically added to an optimal seed focus plane at a certain distance based on accuracy. Furthermore, expanding the optimal seed focus plane reveals a minimized portion of the dataset. Based on the quality control method, image accuracy can be increased by raising the minimum threshold for the sharpness value of each two-dimensional image.

[0084] In another aspect of the invention, a computer-implemented method is disclosed for training machine and deep learning models to analyze at least one digital image of a sample placed on a glass slide. The method includes method steps that may be performed in a specific given order. However, different orders are also possible. It is also possible to perform two or more method steps simultaneously, either entirely or partially. Furthermore, one or more, or even all, of the method steps may be performed once or repeatedly, such as repeated once or multiple times. Additionally, the method may include additional method steps not listed.

[0085] The method includes generating at least one training dataset. Generating the training dataset involves determining a z-stack of digital images by using at least one imaging device of a slide imaging apparatus to determine multiple digital images of a known sample placed on a slide. The known sample has at least one predetermined or predefined feature. The imaging apparatus includes at least one transmission device having a focal length. The z-stack includes multiple digital images determined at at least three different distances between the transmission device and the slide, wherein the distances are defined based on sharpness values ​​determined by using a computer-implemented method for quality control according to the present invention for quality control, as described above or in more detail below, such that the z-stack includes digital images determined when the transmission device is out of focus and focused digital images determined when the transmission device is focused. The method includes applying a machine learning model to the z-stack of the digital images and adjusting the machine and deep learning models.

[0086] As an example, machine or deep learning with the highest focus value defined by the quality control described in this invention can be used to define key features for analysis, which are preferably obtained using z-stacking at z=0. The properties used by the machine or deep learning algorithm can be highlighted in the heatmap. Furthermore, the defined properties can be trained on different z-stackings. Additionally, lower focus values ​​can be included in the algorithm's training by lowering the minimum threshold for the sharpness value. This can allow for improved robustness of the algorithm related to out-of-focus areas.

[0087] For possible definitions, choices, or implementations, please refer to the description of the methods given above.

[0088] As used herein, the term "machine and deep learning" is a broad term and will be given a common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, methods of automatically building machine learning models, particularly predictive models, using artificial intelligence (AI). Control and evaluation apparatus may be configured to execute and / or implement at least one machine and deep learning algorithm. Machine and deep learning models may be based on the results of machine and deep learning algorithms. Machine and deep learning models may be based on, for example, convolutional neural networks and / or conventional neural networks.

[0089] As used herein, the term "training" is a broad term and is given a common and conventional meaning to those skilled in the art, and is not limited to a specific or custom-defined meaning. Specifically, the term may refer to, but is not limited to, the process of determining the parameters of an algorithm for a machine and deep learning model on a training dataset. The training may include at least one optimization or tuning process in which the optimal combination of parameters is determined. As used herein, the term "training dataset" is a broad term and is given a common and conventional meaning to those skilled in the art, and is not limited to a specific or custom-defined meaning. Specifically, the term may refer to, but is not limited to, a dataset on which machine and deep learning models are trained. A training dataset may include a z-stack of multiple digital images. For training machine and deep learning models, digital images of known samples with at least one predetermined or predefined feature may be used. For example, features may include one or more of color, texture, morphology, and topology.

[0090] To determine z-stacking, at least two layers of digital images are added at a time; an upper layer and a lower layer. For details on determining z-stacking, refer to the descriptions in Methods for Determining Focus Quality and Methods for Determining 3D Digital Images. These digital images can be defined during algorithm training to what level the algorithm can actually achieve. For example, quality control values ​​compared to the focus plane can be defined as 95%, 90%, 85%, 80%, and 75%. These values ​​may not be directly defined but can be indirectly estimated by defining thresholds using quality control tools. Therefore, its digital defocus is not initially defined or predefined, but the algorithm learns to handle lower sharpness values.

[0091] As used herein, the term "digital image when the transmission device is out of focus" is a broad term and will be given a common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, digital images with a sharpness value below a predefined threshold. A threshold can be selected such that tissues with lower contrast are also included in the image. For example, the threshold could be 15% of the maximum sharpness value. As used herein, the term "digital image when the transmission device is in focus" is a broad term and will be given a common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. Specifically, the term may refer to, but is not limited to, digital images with a sharpness value above a predefined threshold (such as a threshold described above in this paragraph).

[0092] The method according to the invention can make trained machines and deep learning models more robust, enabling the processing of out-of-focus or less clear areas based on additional training until a certain threshold is reached.

[0093] This document further discloses and proposes a computer program including computer-executable instructions that, when executed on a computer or computer network, are used to perform at least one method according to the method according to the invention in one or more embodiments of the appended embodiments. Specifically, the computer program may be stored on a computer-readable data carrier and / or a computer-readable storage medium.

[0094] As used herein, the terms "computer-readable data carrier" and "computer-readable storage medium" can specifically refer to non-transitory data storage devices, such as hardware storage media having computer-executable instructions stored thereon. Computer-readable data carriers or storage media can specifically be or may include storage media such as random access memory (RAM) and / or read-only memory (ROM).

[0095] Therefore, specifically, one, more than one, or even all of the method steps indicated above can be performed by using a computer or computer network, preferably by using a computer program.

[0096] This document further discloses and proposes a computer program product having program code tools so that, when executed on a computer or computer network, at least one method according to the invention is performed in one or more embodiments of the appended embodiments. Specifically, the program code tools may be stored on a computer-readable data carrier and / or a computer-readable storage medium.

[0097] This document further discloses and proposes a data carrier having a data structure stored thereon, which, after being loaded into a computer or computer network, such as after being loaded into the working memory or main memory of the computer or computer network, can perform at least one of the methods according to one or more embodiments disclosed herein.

[0098] This document further discloses and proposes a computer program product having program code tools stored on a machine-readable medium, so that when the program is executed on a computer or computer network, it performs at least one of the methods according to one or more embodiments disclosed herein. As used herein, a computer program product refers to a program that is a tradable product. The product can generally exist in any format (such as paper format) or on a computer-readable data carrier and / or computer-readable storage medium. Specifically, the computer program product can be distributed on a data network.

[0099] Finally, this document discloses and proposes a modulated data signal containing instructions readable by a computer system or computer network for performing at least one method according to one or more embodiments disclosed herein.

[0100] Referring to the computer implementation aspects of the present invention, one or more method steps, or even all method steps, of the methods according to one or more embodiments disclosed herein can be performed using a computer or computer network. Therefore, in general, any method steps, including providing and / or processing data, can be performed using a computer or computer network. Typically, these method steps may include any method steps other than those requiring manual work (such as providing samples and / or performing certain aspects of actual measurements).

[0101] Specifically, this article further discloses the following:

[0102] - A computer or computer network, the computer or computer network including at least one processor, wherein the processor is adapted to perform at least one method according to an embodiment of the embodiments described in this specification.

[0103] - A computer-loadable data structure, which, when executed on a computer, performs at least one method of an embodiment according to the embodiments described in this specification.

[0104] - A computer program, wherein the computer program is adapted, when executed on a computer, to perform at least one method of an embodiment according to the embodiments described in this specification.

[0105] - A computer program, including program tools, which, when executed on a computer or on a computer network, perform at least one method of an embodiment described in this specification.

[0106] - A computer program, comprising program means according to the foregoing embodiments.

[0107] These program devices are stored on a computer-readable storage medium.

[0108] - A storage medium, wherein a data structure is stored on the storage medium and wherein the data structure is adapted to, after being loaded into the main memory and / or working memory of a computer or computer network, execute at least one method of an embodiment according to the embodiments described in this specification, and

[0109] - A computer program product having program code tools, wherein such program code tools may be stored or stored on a storage medium for performing at least one method of an embodiment of the methods described in this specification when such program code tools are executed on a computer or on a computer network.

[0110] In another aspect of the invention, a slide imaging device is disclosed. This slide imaging device includes...

[0111] - At least one imaging device configured to generate at least one digital image of a sample placed on a glass slide;

[0112] - At least one control and evaluation device;

[0113] The slide imaging device is configured to perform at least one of the methods according to the invention as described above or in more detail below.

[0114] For possible definitions, selections, or embodiments of slide imaging devices, please refer to the description of the methods given above.

[0115] In summary, and without excluding other possible embodiments, the following embodiments are conceivable:

[0116] Example 1. A computer-implemented method for quality control of at least one digital image of a sample placed on a glass slide, wherein the method includes the following steps:

[0117] a) Using at least one imaging device of a slide imaging apparatus to provide at least one digital image of a sample placed on a slide;

[0118] b) Determine the quality of the digital image by using at least one edge detection image filter to determine the sharpness value of a sub-region of at least one region of interest and by comparing the sharpness values ​​within the region of interest, wherein the quality of the region of interest is classified according to the comparison;

[0119] c) Generate at least one indication based on the quality classification.

[0120] Steps b) and c) are executed automatically.

[0121] Example 2. The method according to the foregoing embodiments, wherein the indication is a graphic indication including a heatmap, wherein the heatmap is generated by mapping the sharpness values ​​of sub-regions according to their image coordinates.

[0122] Example 3. The method according to the foregoing embodiments, wherein the generation of the heatmap includes assigning color values ​​in the color space to each sharpness value in the sharpness values.

[0123] Example 4. The method described in the foregoing examples, wherein the color space is the RGB color space, and the color space ranges from 255, representing high-quality white, to 0, representing out-of-focus black.

[0124] Example 5. The method according to any one of the preceding three embodiments, wherein the sub-region corresponds to the pixels of the digital image, wherein the generation of the heatmap includes: selecting a pixel group of the region of interest, determining an average sharpness value for each pixel group in the pixel group of the region of interest, and mapping the average sharpness value according to its image coordinates.

[0125] Example 6. The method according to any one of the foregoing embodiments, wherein the edge detection image

[0126] The filter is a Laplace filter.

[0127] Example 7. The method according to any one of the foregoing embodiments, wherein the sharpness value is color

[0128] Color gradient or intensity gradient.

[0129] Example 8. The method according to any one of the foregoing embodiments, wherein the region of interest is

[0130] The entire digital image or a portion of a digital image.

[0131] Example 9. A method for determining focusing quality to determine the quality of a sample placed on a glass slide.

[0132] A computer-implemented method for displaying a digital image, wherein the method includes the following steps:

[0133] i) Using at least one imaging device of a slide imaging apparatus to determine a z-stack of digital images of a sample placed on a slide, wherein the imaging device includes at least one transmission device having a focal length, and wherein the z-stack includes a plurality of digital images determined at at least three different distances between the transmission device and the slide.

[0134] ii) Determine information regarding the sharpness of multiple image regions of each digital image in a z-stacked digital image by using a computer-implemented method for quality control according to any one of the foregoing embodiments;

[0135] iii) Generate a graphical indication of sharpness information based on the distance to each image region in the image region.

[0136] Example 10. The method according to the foregoing embodiments, wherein the method includes, according to the graphic pointer...

[0137] This demonstrates how to adjust the focusing settings of the transmission device for different distances between the transmission device and the glass slide.

[0138] Example 11. An imaging device for determining at least one imaging apparatus using a slide imaging device.

[0139] A computer-generated method for producing a three-dimensional digital image of a sample placed on a glass slide, wherein the imaging apparatus includes at least one transmission device having a focal length, and the method includes...

[0140] The following steps:

[0141] - Determine the z-stack of two-dimensional digital images, wherein the z-stack is determined by imaging the sample at at least three different distances between the slide and the transport device, starting in a seed focusing plane, wherein the seed focusing plane is a recommended optimal focusing plane determined by a slide imaging device, wherein the first two-dimensional image imaged in the seed focusing plane has a first sharpness value, wherein the distance is defined based on the sharpness value, the sharpness value being determined by using a computer-implemented method for quality control according to any of the foregoing embodiments relating to a method for quality control, such that the sharpness values ​​of the two-dimensional digital images determined at different distances are different from the first sharpness value and are different from each other;

[0142] - Select the focus area of ​​each digital 2D image in the z-stack by using an indication based on quality classification;

[0143] - A three-dimensional digital image is determined by combining information about the seed focus plane and the focus area of ​​each digital two-dimensional image of the z-stack.

[0144] Example 12. The method according to the foregoing embodiments, wherein the method includes using color

[0145] Information is applied to the pixels of a 3D image to color the 3D image.

[0146] Example 13. A computer-implemented method for training machine and deep learning models to analyze at least one digital image of a sample placed on a glass slide.

[0147] The method includes generating at least one training dataset, wherein generating the training dataset includes: determining a z-stack of digital images by using at least one imaging device of a slide imaging apparatus to determine multiple digital images of a known sample placed on a slide, wherein the known sample has at least one predetermined or predefined feature, wherein the imaging device includes at least one transmission device having a focal length, wherein the z-stack includes multiple digital images determined at at least three different distances between the transmission device and the slide, wherein the distances are defined based on sharpness values, which are determined by using a computer-implemented method for quality control according to any of the foregoing embodiments relating to a method for quality control, such that the z-stack includes digital images determined when the transmission device is out of focus and digital images determined when the transmission device is in focus;

[0148] This method involves applying machine and deep learning models to a z-stack of digital images and tuning the machine and deep learning models.

[0149] Example 14. The method according to the foregoing embodiments, wherein the machine and the deep learning model are

[0150] Based on convolutional neural networks and / or conventional neural networks.

[0151] Example 15. A slide imaging device, comprising:

[0152] - At least one imaging device configured to generate at least one digital image of a sample placed on a glass slide;

[0153] - At least one control and evaluation device;

[0154] The slide imaging device is configured to perform at least one of the methods described according to Examples 1 to 14. Attached Figure Description

[0155] Preferably, in conjunction with the dependent claims, other optional features and embodiments will be disclosed in more detail in the following description of embodiments. These optional features, as will be recognized by those skilled in the art, can be implemented individually and in any feasible combination. The scope of the invention is not limited to the preferred embodiments. Embodiments are schematically depicted in the accompanying drawings. In these drawings, the same reference numerals refer to the same or functionally equivalent elements.

[0156] In the attached diagram:

[0157] Figure 1A and Figure 1B A flowchart illustrating an embodiment of the slide imaging apparatus according to the present invention and a preferred embodiment of the method for quality control is shown schematically.

[0158] Figure 2A and Figure 2B The experimental results are shown;

[0159] Figure 3 The experimental results are shown;

[0160] Figure 4 , Figures 4A to 4F The experimental results are shown;

[0161] Figure 5 A flowchart illustrating a preferred embodiment of the method for determining focus quality according to the present invention is shown schematically;

[0162] Figure 6 The experimental results are shown;

[0163] Figure 7A and Figure 7B The experimental results are shown;

[0164] Figure 8 A flowchart illustrating a preferred embodiment of the method for determining a three-dimensional digital image according to the present invention is shown schematically;

[0165] Figures 9A to 9C The experimental results are shown;

[0166] Figure 10 A flowchart illustrating a preferred embodiment of the method for training machine and deep learning models according to the present invention is shown schematically; and

[0167] Figure 11 An overview of a procedure developed for quality control of digital images of tissue samples, including biological materials, is shown. Detailed Implementation

[0168] Figure 1A An embodiment of a slide imaging apparatus 110 according to the present invention is illustrated schematically. The slide imaging apparatus 110 is configured to image a sample 112 placed on a slide 114. The sample 112 may be a biological sample, such as a tissue sample. The sample 112 may be or may include biological materials, such as tissue or smears. However, other types of samples may also be feasible.

[0169] The slide 114 may be a substrate designated for placing the sample 112 on its surface. Specifically, for the purpose of bearing the sample 112 without alteration during processing of the slide 114, the substrate is mechanically stable and can therefore comprise any material providing sufficient mechanical stability. Particularly for the purpose of bearing biological samples, the substrate may preferably have a surface configured to be compatible with biological materials. For example, the slide is a glass slide, as glass is known to provide sufficient mechanical stability on the one hand and high compatibility with biological materials on the other. However, other types of materials for the slide 114 may also be feasible. For the purpose of generating a desired image of the sample, the slide 114 may preferably be a plate having a 2D extension and a certain thickness, wherein the 2D extension of the plate may preferably be rectangular or circular in form, and wherein the thickness of the plate may be small compared to the dimension of the extension, preferably 20%, more preferably 10%, particularly 5%, or less than a measure of the linear range of the 2D extension of the plate.

[0170] The slide imaging apparatus 110 includes at least one imaging device 116. In particular, the imaging device 116 may be selected from a 2D camera 117 or a line scan detector. However, other types of imaging devices 116 may also be feasible. The slide 114 may in particular be of a form that enables imaging of the sample 112 placed on the slide 114.

[0171] Imaging device 116 may include at least one transmission device 119. Transmission device 119 may specifically include one or more of the following: at least one lens, for example, at least one lens selected from the group consisting of at least one refractive lens, at least one adjustable focusing lens, at least one aspherical lens, at least one spherical lens, and at least one Fresnel lens; at least one diffractive optical element; at least one multi-lens system. Transmission device 119 has a focal length. Transmission device 119 may form a coordinate system, where “z” is the coordinate along the optical axis. The coordinate system may be a polar coordinate system, where the optical axis of transmission device 119 forms the z-axis and where the distance from the z-axis and the polar angle can be used as additional coordinates. The coordinate along the z-axis can be considered as the longitudinal coordinate z.

[0172] The slide imaging apparatus 110 may include a storage device 118 that can hold a plurality of slides 114 and is configured to store slides 114. The storage device 118 may be selected from a slide tray or slide holder; however, other types of storage devices may also be feasible. The storage device 118 is preferably manually loaded with at least one slide 114; however, automatic loading of the storage device 118 is also conceivable. The slide imaging apparatus 110 may include a supply device 120 configured to supply at least one slide 114 from the storage device 118 to the imaging apparatus 116. For example, the supply device 120 may include a robotic arm.

[0173] The slide imaging apparatus 110 includes at least one control and evaluation device 122. The at least one control and evaluation device 122 may include at least one data processing device having software code stored thereon, the software code including a plurality of computer commands. The control and evaluation device 122 may provide one or more hardware elements for performing one or more specified operations, and / or may provide software running thereon to one or more processors for performing one or more specified operations. The control and evaluation device 122 may include one or more programmable devices, such as one or more computers, application-specific integrated circuits (ASICs), digital signal processors (DSPs), or field-programmable gate arrays (FPGAs), configured to perform steps b) and c). However, additionally or alternatively, the control and evaluation device 122 may also be implemented entirely or partially in hardware.

[0174] The slide imaging device 110 is configured to perform a computer-implemented method for quality control of at least one digital image of a sample 112 placed on a slide 114. Figure 1B A flowchart illustrating a preferred embodiment of the method for quality control according to the present invention is shown schematically.

[0175] This method may include determining the state of a digital image in relation to its suitability being automatically analyzed. Specifically, the method may include determining an indication of the amount of sharpness or blurriness of the digital image.

[0176] The method includes the following steps:

[0177] a) (indicated by reference numeral 124) using at least one imaging device 116 of slide imaging apparatus 110 to provide at least one digital image of sample 112 placed on slide 114;

[0178] b) (denoted by reference numeral 126) The quality of the digital image is determined by using at least one edge detection image filter to determine the sharpness value of a sub-region of at least one region of interest and by comparing the sharpness values ​​within the region of interest, wherein the quality of the region of interest is classified according to the comparison;

[0179] c) (indicated by reference numeral 128) Generate at least one indication based on the quality classification.

[0180] Steps b) and c) are executed automatically.

[0181] Providing a digital image may include using imaging device 116 to image and / or generate at least one digital image.

[0182] The region of interest (ROI) can be any shape of region or area of ​​the digital image to be analyzed. The ROI can be the entire digital image or a portion of it. The ROI can be a region of the image that includes or is suspected of including at least one feature to be analyzed. The digital image can be a pixelated image comprising multiple pixels, arranged, for example, in a pixel array, such as a rectangular array, having m rows and n columns, where m and n are independently positive integers. The ROI can be or may include a group of pixels, which comprises any number of pixels. A subregion can be any part or element of the ROI that includes at least one pixel or group of pixels, particularly an image element. A subregion can be a square area of ​​the ROI. The ROI can include multiple subregions, such as multiple pixels.

[0183] An edge detection image filter can be, or can include, at least one image processing tool configured to identify edges in a region of interest. Preferably, the edge detection image filter can be a Laplacian filter. The Laplacian filter can be based on the second derivative of intensity and / or color values, and thus can determine local extrema. The edge detection image filter can be configured for edge detection based on local extrema. In particular, the edge detection filter can emphasize strong edges and / or gradients. Specifically, the method can include defining the sharpness of the digital image at the sub-region level, preferably at the pixel level, based on a parameter defined by the Laplacian filter. The edge detection filter can be configured to assign grayscale values ​​to each sub-region within the sub-region according to the corresponding local extrema. Using a Laplacian filter can be advantageous because it eliminates the need to rotate the kernel to obtain the x or y direction, as is done with first-derivative filters such as the Sobel or Canny filters. Figure 2A Exemplary experimental results are shown. Figure 2A The left side shows the scanned digital image, and the right side shows a square area defined by a Laplace filter in grayscale values.

[0184] Sharpness values ​​can be local extrema of intensity and / or color values. Sharpness values ​​can also refer to color gradients or intensity gradients. The comparison of sharpness values ​​in step 126 can include comparing local extrema of sub-regions of the region of interest. The method can include determining the maximum local extremum of the region of interest. The maximum local extremum can be the highest local extremum of a sub-region of the region of interest. The method can include the minimum local extremum of the region of interest. The minimum local extremum can be the lowest local extremum of a sub-region of the region of interest. The method can include sorting the local extrema of the sub-regions in descending order from the maximum local extremum to the minimum local extremum. However, ascending order sorting is also possible. The comparison can include at least one mathematical operation for determining the relationship between the local extrema.

[0185] The quality of the region of interest is classified based on the comparison in step 126. The maximum local extremum can be classified as the highest resolution. The minimum local extremum can be classified as the lowest resolution. The local extrema of sub-regions can be classified in descending order from the maximum local extremum to the minimum local extremum.

[0186] The method may also include setting a defocus threshold. The defocus threshold can define the minimum value of a maximum local extremum. For example, the defocus threshold can be set to a high-quality absolute sharpness that achieves the minimum of a maximum local extremum across 99% of a set of controlled images. If the determined maximum local extremum in the region of interest is below the defocus threshold, the digital image can be rejected and / or classified as low quality. Using a defocus threshold can prevent the identification of completely out-of-focus digital images.

[0187] In step 128, for example, the indication may be a graphical indication that includes a heatmap (also referred to as a focus map). The heatmap can be generated by mapping sharpness values ​​of sub-regions according to their image coordinates. Generating the heatmap may include assigning color values ​​from a color space to each sharpness value in the sharpness values. For example, the color space may be the RGB color space. The RGB color space may be a color space with three color channels: one channel for red (R), one channel for green (G), and one channel for blue (B). The color space can range from 255, representing high-quality white, to 0, representing out-of-focus black. Therefore, the maximum white value of 255 can represent the sharpest part of the digital image. The heatmap may include only one or two colors (such as only red and yellow color values), ranging from 255, representing high-quality white, to 0, representing out-of-focus black.

[0188] For example, a sub-region can correspond to pixels in a digital image. Generating a heatmap may include selecting a group of pixels within a region of interest, determining an average sharpness value for each pixel group within that group, and mapping the average sharpness value to its image coordinates. Specifically, generating a heatmap may include determining the average grayscale value within a pixel group. The average grayscale value can be determined by calculating the sum of all grayscale values ​​and dividing by the number of sub-regions of the image region, specifically the number of pixels. To calculate the average grayscale value, black pixels may be excluded or discarded. Black pixels can be pixels that have no organizational contrast and cannot be sharpened or blurred. Therefore, the term pixel can refer to a descriptive measurement.

[0189] The quality determined based on sharpness values ​​may be directly related to the applicability of classification algorithms used in computational pathology. The generated indicators can allow for the identification of artifacts caused by staining, tissue folding, and blurring.

[0190] The method may include displaying instructions, particularly heatmaps, using at least one user interface 130. The user interface 130 may be a human-machine interface configured to display instructions, such as a monitor, screen, etc.

[0191] In identifying out-of-focus areas and artifacts, fully automated quality determination and fully automated generation of indicators may be superior to controlling digital images with the human eye. Figure 2B It demonstrates a high level of clarity that cannot be defined by visual observation alone. Figure 2B The left side depicts a representation of the sharpness of a digital image in a heatmap according to the invention, based on pixel-level analysis using a Laplace filter. The right side shows... The corresponding thumbnail image taken by the DP 200 slide scanner.

[0192] Figure 3 Further experimental results are shown. Blur can be detected at 40x magnification; see the top section. A heatmap of the pixel-level Laplacian analysis according to the invention is shown on the bottom left, and the right side displays the results from... Thumbnail image taken by DP 200 slide scanner.

[0193] Figures 4A to 4F Other experimental results are shown. Figure 4 Different heatmaps of pixel-level analysis by the Laplace filter according to the present invention are shown. Figure 4A The image shows bubbles detected at 40x magnification. Figure 4B The image shows the blurring detected at 40x magnification due to low contrast caused by a lack of tissue morphology. Figure 4C The image shows a clear signal staining detected at 40x magnification. Due to... Figure 4D and Figure 4E Different staining (these show the same area in the digital image) resulted in lower or higher sharpness being detected at 40x magnification. Figure 4F The image shows a region of low resolution detected at 40x magnification in a digital image.

[0194] Figure 5 A flowchart of a preferred embodiment of a computer implementation method according to the present invention for determining focus quality to determine at least one digital image of a sample 112 placed on a glass slide 114 is illustrated schematically.

[0195] The method includes the following steps:

[0196] i) (indicated by reference numeral 132) using at least one imaging device 116 of a slide imaging apparatus 110 to determine a z-stack of digital images of a sample 112 placed on a slide 114, wherein the imaging device includes at least one transmission device having a focal length, and wherein the z-stack includes a plurality of digital images determined at at least three different distances between the transmission device 119 and the slide 114.

[0197] (ii) (denoted by reference numeral 134) information on the sharpness of multiple image regions of each digital image in a z-stacked digital image is determined by using a computer-implemented method for quality control according to the present invention for quality control as described above or in more detail below;

[0198] iii) (indicated by reference numeral 136) Generate a graphical indication of sharpness information based on the distance to each image region in the image region.

[0199] z-stacking includes at least three digital images imaged at at least three different distances (i.e., at different z-levels) between the transport device 119 and the slide 114. Specifically, two digital images may be defined at a layer above and a layer below the focal plane. The z-stacking digital images may include high-magnification views, such as those with a magnification of 20x or 40x.

[0200] Step i) 132 may include imaging the first digital image using the imaging device 116. For example, the first digital image may be imaged in a so-called seed focus plane, which is a suggested optimal focus plane automatically determined by the slide imaging device 110. For example, the slide imaging device 110 may include at least one database storing information about the seed focus plane.

[0201] To determine the z-stack, two layers are added at a time. For example, two additional digital images can be imaged at a distance ±ε from the layer containing the focal point, where ε is a positive number. Specifically, one digital image can be imaged above the image plane of the first digital image, and another digital image can be imaged below the image plane of the first digital image. The distance ±ε from the layer containing the focal point can be chosen based on accuracy. For example, the z-stack can include multiple digital images determined at a distance ε = 1 μm, which is defined by the depth of field and the size of the known object. However, other values ​​for the distance ε and / or non-equidistant distances are also possible. In the example of z-stack, the distance ε can be 0.1 μm, 0.2 μm, 0.25 μm, or 0.5 μm. Furthermore, 1, 3, 5, 7, 9, 11, 13, and 15 layers can be selected.

[0202] The graphic indication may include sharpness values ​​plotted for the z-level, particularly for a predefined number of samples. The determination of sharpness information and the generation of the graphic indication can be performed using at least one control and evaluation device 122.

[0203] This method may include adjusting the focus settings of the transmission device 119 for different distances between the transmission device 119 and the slide 114 according to graphical indications. The height of the focusing mechanism, including the focusing coil, can be adjusted so that the highest average sharpness is obtained when it begins focusing at z=0. The focusing mechanism assembly including the transmission device 119 can be set to its extreme values ​​without exceeding the specification level. To check the height setting of the nominal focus position of the focusing coil, the focusing coil is deactivated and a line track from the focusing camera is displayed. The line track represents the received signal based on the crenellated optics. The accuracy of the displayed curve can indicate whether the height of the focusing mechanism assembly, including the focusing coil and the transmission device, is correctly adjusted. If the three colors of the upper horizontal bar are separated, the line track is acceptable. In known devices, the focusing mechanism assembly can be fixed with screws and must be adjusted manually. This invention proposes to automatically adjust the height of the focusing mechanism assembly based on quality control as described in this invention. This automation method is particularly useful because testing must be performed precisely and is time-consuming. In contrast to the manual method described by Kohlberger T, Liu Y, Moran M et al. in “Whole-Slide Image Focus Quality: Automatic Assessment and Impact on AI Cancer Detection”, J Pathol Inform, 2019, where numerous tests must be performed before an average quality metric is reached to define z=0 based on a strong “V”-shaped trend and a defocus class for the z-level, the method proposed in this invention is labor-efficient due to its automation.

[0204] The focus settings can be adjusted automatically to adjust the depth of field. Specifically, the determination of z-stack, the determination of information about sharpness, and the generation of graphic indicators can be performed automatically, such as by using at least one control and evaluation device 122.

[0205] The method may also include, for example, displaying graphical indicators by using at least one user interface 130.

[0206] Figure 6 Experimental results are shown. In particular, the generation of a z-stack comprising three digital images is depicted. A digital image above the seed plane is imaged at +7 μm, and a digital image below the seed plane is imaged at -7 μm. The digital images imaged in the seed plane in between are shown. Arrows indicate the transmission device 119, and thin arrows indicate focal length variability.

[0207] Figure 7A and Figure 7B Further experimental results are shown. Figure 7A In the initial default setting, the magnification is 40x. Figure 7B In the image, the adjusted default setting is shown as a magnification of 40x.

[0208] Figure 8 A flowchart illustrating a preferred embodiment of a computer implementation method according to the present invention for determining a three-dimensional digital image of a sample 112 placed on a slide 114 using at least one imaging device 116 of a slide imaging device 110.

[0209] The method includes the following steps:

[0210] - (denoted by reference numeral 138) The z-stack of the two-dimensional digital images is determined by imaging the sample 112 at at least three different distances between the slide 114 and the transport device 119, starting in a seed focusing plane, wherein the seed focusing plane is a recommended optimal focusing plane determined by the slide imaging device 110, wherein the first two-dimensional image imaged in the seed focusing plane has a first sharpness value, wherein the distance is defined based on the sharpness value, the sharpness value being determined by using a computer-implemented method for quality control according to the present invention for quality control as described above or in more detail below, such that the sharpness values ​​of the two-dimensional digital images determined at different distances are different from the first sharpness value and are different from each other;

[0211] - (indicated by reference numeral 140) The focus area of ​​each digital 2D image in the z-stack is selected by using an indication based on quality classification;

[0212] - (indicated by reference numeral 142) The three-dimensional digital image is determined by combining information about the focal plane of the seed and the focal area of ​​each digital two-dimensional image of the z-stack.

[0213] The distance between the z-stacked digital images can be defined based on the relative intensity and / or color variation determined in step b) 126 of the method for quality control. Specifically, the distance can be defined such that the sharpness values ​​of the digital images of the two layers can differ in the region of interest. The distance can be defined such that the sharpness values ​​of the digital images of the two layers can differ beyond a predetermined tolerance, such as exceeding at least 5%. The distance ±ε from the layer including the focal point can be selected based on accuracy. For example, z-stack can include multiple digital images determined at a distance ε = 1 μm, which is defined by the depth of field and the size of the known object. However, other values ​​for distance ε and / or non-equidistant distances are also possible. In the example of z-stack, distance ε can be 0.1 μm, 0.2 μm, 0.25 μm, 0.5 μm. Furthermore, 1, 3, 5, 7, 9, 11, 13, and 15 layers can be selected.

[0214] The slide imaging device 110 can define a focal plane for each slide by picking up a seed point (also referred to as a seeding point) near the highest vertical extension on the lower edge of the sample based on z-stacking. Therefore, an optimal focal plane, i.e., the seed plane, can be defined. The slide imaging device 110 can be configured to continuously update the focal plane based on data derived from a dynamic forward-looking focus tracking mechanism during scanning of a digital image. Specifically, the separation mirror and crenellated optics allow a focusing camera adjacent to the imaging camera to scan simultaneously to determine if the signal is in focus. This information is directly used to adjust the output of the dynamic focusing coil in the focusing mechanism assembly, thereby adjusting the height of the transmission device 119.

[0215] The first step of method 138 may include imaging a first two-dimensional digital image in a seed plane and imaging at least two additional two-dimensional digital images in two opposite directions between the slide 114 and the transmission device 119. In each two-dimensional digital image, due to the shallow depth of field, only the regions of the corresponding digital image located on the same z-plane are focused.

[0216] In the next step 140, all two-dimensional digital images can be analyzed, specifically by selecting “sharp” image regions from each two-dimensional image using control and evaluation devices. 3D information can be generated through focus depth. Specifically, a three-dimensional image can be determined by combining the determined focus area with information about the corresponding focus plane. Therefore, by integrating height information, three-dimensional digital image reconstruction is possible.

[0217] The method may also include displaying a three-dimensional digital image. The three-dimensional digital image can be displayed using at least one user interface 130.

[0218] This method can include coloring a 3D digital image by applying color information (i.e., a texture image) to individual voxels, 3D pixels. By texturing using x, y, and z coordinates, color can be defined for each pixel. This allows for better object recognition through improved reference comparison.

[0219] Figures 9A to 9C The experimental results are shown. Figure 9A In the digital image, the focused area from the top left (background) to the bottom right (highest point of the sample) is shown as -7 to +7, with a step size of 0.25 μm and a magnification of 40x. Figure 9B This demonstrates a design intended for semi-quantitative detection of the transmembrane protein HER2. Exemplary two-dimensional analysis of HER2(4B5) antibody. Figure 9C based on Figure 9A The z-stack shown illustrates a design intended for semi-quantitative detection of the transmembrane protein HER2. Three-dimensional image of HER2(4B5) antibody.

[0220] Figure 10 A flowchart illustrating a preferred embodiment of a computer-implemented method according to the invention for training machine and deep learning models to analyze at least one digital image of a sample 112 placed on a glass slide 114 is shown.

[0221] The method includes generating at least one training dataset (denoted by reference numeral 144). Generating the training dataset involves determining a z-stack of digital images using an imaging device 116 to determine multiple digital images of a known sample 112 placed on a slide. The known sample 112 has at least one predetermined or predefined feature. The z-stack includes multiple digital images determined at at least three different distances between the transport device 119 and the slide 114, wherein said distances are defined based on sharpness values ​​determined using a computer-implemented method for quality control according to the present invention for quality control, as described above or in more detail below, such that the z-stack includes digital images determined when the transport device is out of focus and focused digital images determined when the transport device is in focus.

[0222] Machine and deep learning models can be based on, for example, convolutional neural networks and / or conventional neural networks.

[0223] Training may include the process of determining the parameters of an algorithm for a machine and deep learning model on a training dataset. The training may include at least one optimization or tuning process in which an optimal combination of parameters is determined. The training dataset may include a z-stack of multiple digital images. For training the machine and deep learning models, digital images of known samples with at least one pre-determined or predefined feature may be used. For example, features may include one or more of color, texture, morphology, and topology.

[0224] To determine z-stacking, at least two layers of digital images are added at a time; an upper layer and a lower layer. For details on determining z-stacking, refer to the descriptions in Methods for Determining Focus Quality and Methods for Determining 3D Digital Images. These digital images can be defined during algorithm training to what level the algorithm can actually achieve. For example, quality control values ​​compared to the focus plane can be defined as 95%, 90%, 85%, 80%, and 75%. These values ​​may not be directly defined but can be indirectly estimated by defining thresholds using quality control tools. Therefore, its digital defocus is not initially defined or predefined, but the algorithm learns to handle lower sharpness values.

[0225] The method includes (indicated by reference numeral 146) applying a machine learning model to a z-stack of digital images and tuning the machine and deep learning models.

[0226] This method allows trained machines and deep learning models to become more robust, enabling the processing of out-of-focus or less sharp areas based on additional training until a specific threshold is reached.

[0227] As an example, machine or deep learning with the highest focus value defined by the quality control described in this invention can be used to define key features for analysis, which are preferably obtained using z-stacking at z=0. The properties used by the machine or deep learning algorithm can be highlighted in the heatmap. Furthermore, the defined properties can be trained on different z-stackings. Additionally, lower focus values ​​can be included in the algorithm's training by lowering the minimum threshold for the sharpness value. This can allow for improved robustness of the algorithm related to out-of-focus areas.

[0228] Figure 11An overview of a procedure developed according to the present invention for quality control of digital images of tissue samples comprising biological material is shown. The tissue image represented comprises a width of 77,824 pixels and a height of 115,712 pixels. For scanning, a magnification of 40x (0.2500 mpp) was selected. Focusing was autofocus, focus quality was general, and the scanning mode was conventional. To define focus quality, the quality control described in this invention was applied. Subplot "A" indicates the "defocus sub-region," which will be shown in red in the thermal image. Subplot "B" shows the Laplacian defocus threshold based on the red channel of the RGB color space, where the maximum white value of 255 represents the highest sharpness value, and black 0 represents defocus. Subplot "C" shows the percentage of defocused pixels in the entire slide image.

[0229] List of reference numerals

[0230] 110 Slide Imaging Equipment

[0231] 112 samples

[0232] 114 glass slides

[0233] 116 Imaging Device

[0234] 117 2D Camera

[0235] 118 Storage device

[0236] 119 Transmission Device

[0237] 120 supply unit

[0238] 122 Control and evaluation device

[0239] 124 Provide at least one digital image

[0240] 126 Determining the quality of digital images

[0241] 128 Generate at least one instruction

[0242] 130 User Interface

[0243] 132 Determining the z-stacking of digital images

[0244] 134 Determine information about sharpness

[0245] 136 Generate Graphic Instructions

[0246] 138 Determining the z-stacking of two-dimensional digital images

[0247] 140 Select focus area

[0248] 142 Determining the 3D Digital Image

[0249] 144 Generate at least one training dataset

[0250] 146 Applications

Claims

1. A computer-implemented method, comprising: Access a z-stack of multiple digital images containing a sample, wherein each of the multiple digital images is a two-dimensional image depicting the sample placed on a slide and corresponding to different focusing distances between the slide and the imaging device of the slide imaging apparatus; For each digital image in the z-stack, the sharpness value of each sub-region of at least one region of interest in the digital image is determined by using at least one edge detection image filter; The sharpness value is compared among the plurality of digital images for each sub-region of the at least one region of interest. The in-focus area of ​​each digital image in the z-stack is selected based on the comparison. as well as A three-dimensional digital image is generated based on the selected area of ​​focus.

2. The method of claim 1, further comprising a graphical indication including a heatmap, wherein the heatmap is generated by mapping the sharpness values ​​of the sub-regions according to their image coordinates.

3. The method of claim 2, wherein the generation of the heatmap comprises assigning color values ​​of a color space to each of the sharpness values.

4. The method of claim 3, wherein the color space is an RGB color space, wherein the color space ranges from high-quality white 255 to out-of-focus black 0.

5. The method of claim 2, wherein the sub-region corresponds to a pixel of the digital image, wherein the generation of the heatmap comprises: Select the pixel group of the region of interest, determine the average sharpness value for each pixel group in the pixel group of the region of interest, and map the average sharpness value according to its image coordinates.

6. The method according to claim 1, wherein the edge detection image filter is a Laplacian filter.

7. The method of claim 1, wherein the sharpness value is a color gradient or an intensity gradient.

8. The method of claim 1, wherein the region of interest is the entire digital image or a portion thereof.

9. The method of claim 1, further comprising: Determine z-stack, wherein the imaging apparatus includes at least one transmission device having a focal length, wherein multiple digital images are determined at at least three different distances between the transmission device and the slide; Determine information regarding the sharpness value of the sub-region; as well as A graphical indication of the information regarding the sharpness value is generated based on the distance to each sub-region within the sub-region.

10. The method of claim 9, further comprising adjusting the focusing settings of the at least one transmission device for different distances between the at least one transmission device and the slide according to the graphic indication.

11. The method of claim 9, further comprising determining a seed plane within the z-stack, wherein the seed plane corresponds to a recommended optimal focus plane for the sample determined by the slide imaging device, and wherein the selection of the focused region is made with reference to the seed plane.

12. The method of claim 1, further comprising coloring the three-dimensional digital image by applying color information to the pixels of the three-dimensional digital image.

13. A computer-implemented method for training machine and deep learning models to analyze at least one digital image of a sample placed on a glass slide. The method includes: Access at least one training dataset generated as follows: a z-stack of digital images of a known sample placed on a slide, determined using at least one imaging device of a slide imaging apparatus, wherein the known sample has at least one predetermined or predefined feature, wherein the imaging device includes at least one transmission device having a focal length, and wherein the z-stack comprises multiple digital images determined at at least three different distances between the transmission device and the slide, wherein the distances are defined based on sharpness values ​​determined using the computer-implemented method of claim 1; and The machine and deep learning models are applied to the z-stack of the digital image and the machine and deep learning models are tuned.

14. The method of claim 13, wherein the machine and deep learning model are based on convolutional neural networks and / or conventional neural networks.

15. A slide imaging device, comprising: - At least one imaging device configured to generate at least one digital image of a sample placed on a glass slide; and - At least one control and evaluation device; The slide imaging device is configured to perform at least one of the methods according to any one of claims 1-14.

16. A system comprising: One or more data processors; and A non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform at least one of the methods according to any one of claims 1-14.

17. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform at least one method according to any one of claims 1-14.

Images