System for assisting a user in the image-based analysis of a tissue sample

Abstract

The invention relates to a system for assisting a user in the image-based analysis of a tissue sample (10). The system has an image-providing unit (101), which is designed to provide an image of the tissue sample recorded via a microscope camera (11) connected to a microscope (12), and additionally an image-processing unit (102), which is designed to generate an image of the tissue sample with a virtual colouration from the provided image. The system additionally has an image-outputting unit (103), which is designed to output the generated image to a display device (13) for analysis by the user. The system permits resource-conserving and at the same time particularly careful histological analyses.

Description

[0001] System to support a user in the image-based analysis of a tissue sample

[0002] AREA OF INVENTION

[0003] The invention relates to a system, a method, and a computer program for assisting a user in the image-based analysis of a tissue sample. Furthermore, the invention relates to a device comprising the system for the image-based analysis of a tissue sample and to a corresponding use of the system.

[0004] TECHNOLOGICAL BACKGROUND

[0005] To analyze a tissue sample, it is common practice to divide the sample into individual sections and stain them appropriately before imaging them under a microscope, in order to visualize tissue and cell structures of interest. A variety of different stains are available for this purpose.

[0006] The described, standard procedure for analyzing a tissue sample has several disadvantages. In particular, it is relatively labor-intensive and resource-inefficient. Furthermore, in practice, the analysis is limited to specific questions and is difficult to standardize, as the stains used are tailored to specific markers and the procedures can differ in detail between different analytical laboratories.

[0007] SUMMARY OF THE INVENTION

[0008] The present invention aims to overcome at least some of the disadvantages described above.

[0009] According to a first aspect of the invention, a system for assisting a user in the image-based analysis of a tissue sample is proposed. The system comprises an image provision unit, an image processing unit, and an image output unit. The image provision unit is configured to provide an image of the tissue sample captured by a microscope camera connected to a microscope. The image processing unit is configured to generate an image of the tissue sample with virtual staining from the provided image. The image output unit is configured to output the generated image to a display device for analysis by the user. Thus, instead of actually staining the tissue sample, which could also be referred to as chemical staining, a virtual staining is performed. Virtual staining could also be referred to as digital staining.In other words, it is proposed that instead of staining the tissue sample itself, an image captured of it should be stained. This virtual staining is preferably performed using a model trained using machine learning. For example, one of the techniques presented in the following article can be applied: “Deep learning-enabled virtual histological staining of biological samples” by B. Bai et al., Science & Applications (2023)12:57. Analyzing tissue samples using virtual rather than actual staining eliminates the need for staining agents and the associated equipment and personnel costs. Furthermore, virtual staining can be standardized. The time required for staining can also be significantly reduced through its virtual implementation.While the actual staining of a tissue section typically takes about 45 minutes, the virtual staining of a corresponding image can take only about 5 to 10 minutes. The time advantage of virtual staining over actual staining is further amplified if the need for additional staining arises only during the analysis. For example, while further virtual staining of an image simply requires the user to select the appropriate image processing method, actual staining would first necessitate the creation of an additional tissue section, for instance, from a specialized laboratory. Only this additional tissue section could then be stained.Virtual staining thus makes it relatively easy to stain one and the same cell or tissue structure in different ways. This overcomes the further disadvantage of actual, i.e., chemical, staining, namely that it is a partially destructive process, making the reuse of the same sample for a different staining method sometimes impossible or only possible with considerable additional effort and limitations. Virtual staining, on the other hand, allows for the analysis of tissue sections without restricting the possibilities for additional or different staining of the identical tissue section. This can be particularly relevant for tissue samples such as very small needle samples, which do not allow, or only with difficulty allow, several thin, adjacent actual sections. However, there is also a general risk for other tissue samples that relevant cells / features might be missing in actually produced subsequent sections.

[0010] Furthermore, it is specifically proposed that the image to be virtually stained be an image acquired using a microscope camera attached to a microscope. This allows the proposed system to utilize the advantages of virtual staining even in contexts where tissue samples are analyzed using a microscope. This is particularly relevant in histopathology. The underlying principle is that sufficiently reliable virtual staining is possible not only for images of tissue samples acquired using complex imaging devices, but also for those acquired with the relatively simple equipment of a microscope and an attached microscope camera.For example, the use of a whole-slide scanner, which is preferred in scientific histology, to capture the images to be virtually stained is considered unnecessary.

[0011] In particular, any variance in the image properties of images acquired using a microscope is not considered an obstacle to the use of virtual staining for the analysis of tissue samples. Experience has shown that such variance occurs especially when the microscope has a manual focusing mechanism that must be operated by a user to focus on the respective sample. It is therefore particularly surprising that virtual staining of tissue samples, or of the images acquired from them, is reliably possible even when the microscope used to acquire the images has a manual focusing mechanism for manually focusing the respective tissue sample, and the image in question was acquired using this manual focusing mechanism.However, it is also a corresponding embodiment of the present invention in which the microscope has an automatic focusing device for automatically focusing the tissue sample, and the provided image was captured using the automatic focusing device. Such an "autofocus" of a microscope allows for the acquisition of images with more consistent properties and thus even more reliable virtual staining. A virtual stain can be considered reliable if it generates an image from a provided image of a tissue sample that corresponds to the image that would be captured of the tissue sample after it had actually been stained with a dye corresponding to the virtual staining agent.

[0012] Well-known stains include H&E (hematoxylin and eosin), MT (Masson's trichrome), PAS (periodic acid-Schiff), and ICH (immunohistochemical staining). H&E is used to differentiate cell nuclei from the extracellular matrix, MT and PAS are suitable for collagen fibers and glycoproteins, and ICH serves to highlight specific epitopes. In principle, a corresponding virtual stain can be provided for any stain, for example, by training a model used for virtual staining. During training, the model can be trained to map input images to output images of tissue samples. For training purposes, images of unstained tissue samples were used as input images, and images of tissue samples stained with the respective stain were used as output images.In order to also be able to perform virtual staining, instead of the input images of unstained tissue samples, input images of also stained tissue samples can be used for training, whereby the staining of the input images differs from the staining of the initial images used for training.

[0013] It goes without saying that the input images used for training are preferably images of the same type used as input images in the subsequent use of the trained model. This means that the input images used for training are, of course, preferably images of the respective tissue sample used for training, captured using a microscope camera connected to a microscope. The output images used for training would then be images of the respective tissue sample used for training, captured after the tissue sample had been stained or re-stained, i.e., actually stained or re-stained.In one embodiment, the image processing unit can therefore be configured in particular to determine the virtual staining by means of a model that has been trained by machine learning on training images, wherein the training images comprise pairs of a training input image and a training output image, wherein the training input image has been recorded by means of a camera connected to a microscope and shows a tissue sample used for training, and wherein the training output image shows the tissue sample used for training after it has actually been stained or re-stained.

[0014] In any case, the image provided by the image delivery unit can be an image of an unstained tissue sample, or an image of a tissue sample stained with a first stain. If the provided image is an image of a tissue sample stained with a first stain, the virtual staining of the image generated by the image processing unit is a virtual staining with a second staining that differs from the first. Generally, the image processing unit can be configured to generate an image with any second staining from a provided image with no first staining or any first staining. It is understood that the stainings are arbitrary but predefined. Furthermore, it should be noted that the first staining, if present, can be an actual staining or a virtual staining.In other words, the provided image can be an image of a tissue sample stained in a first staining step, or an image that has already been generated by virtual staining. The user can determine which virtual staining is performed by the image provisioning unit. In a preferred embodiment, the image processing unit is configured to select the virtual staining from a predetermined plurality of possible virtual stainings based on user input. A corresponding model can be trained for each of the possible virtual stainings, and the image processing unit is configured to access this model.

[0015] The microscope can also be called a light microscope. The image of the tissue sample is preferably created using light in the visible range, but this is not fundamentally limited to this. The microscope can be, in particular, a transmitted light microscope or a fluorescence microscope. The image provided can therefore be, in particular, a transmitted light microscopic image or a fluorescence microscopic image. A transmitted light microscopic image can be understood as an image that has been acquired using transmitted light microscopy. A fluorescence microscopic image can be understood as an image that has been acquired using fluorescence microscopy.

[0016] Preferably, the microscope has optics with a first optical output for connecting the microscope camera and a second output with an eyepiece for the user to view the imaged tissue sample. The second output may also be omitted. In embodiments with both outputs and an automatic focusing device, the automatic focusing device can be configured to adjust a common part of the microscope's optics, the common part of the optics influencing the image produced by the microscope at both the first and second outputs. The automatic focusing device can thus be configured to focus both the image of the tissue sample projected through the eyepiece and the image of the tissue sample captured by the microscope camera, particularly simultaneously. However, it is also possible for the automatic focusing to be limited to the first output.The microscope may, for example, have a manual focusing device for manually focusing an image of the tissue sample projected through the eyepiece, and also an automatic focusing device configured to focus an image of the tissue sample captured by the microscope camera. The latter automatic focusing device may, in particular, not be configured to focus the image of the tissue sample projected through the eyepiece. Such a focusing device may, for example, be designed as an optical adapter between the first output of the microscope and the camera.

[0017] Typically, the microscope would be controlled by the user while looking through such an eyepiece, i.e., not or not solely based on the images captured by the camera. However, good orientation and correspondingly precise control, especially at high magnifications, generally require that the tissue sample is already stained. Unstained tissue usually does not provide the user looking through the eyepiece with sufficient reference points for orientation and thus control of the microscope. The aforementioned implementation of virtual staining, i.e., generating images in a second stain from images with a different first stain, is therefore particularly relevant in practice. In this way, the user receives the information of the first stain via the eyepiece as usual, but also accesses the information of the second stain via the camera.

[0018] It may be preferred that the provided image and the image generated from it are temporally synchronized moving images, where any movement discernible in the images corresponds to a relative movement of the microscope's optics and the tissue sample with respect to each other, or to a change in the magnification setting of the microscope's optics. Thus, it is possible that while the user controls the microscope—i.e., brings different areas of the tissue sample through the optics for imaging and / or makes magnification adjustments to the optics that result in, for example, an area of ​​the tissue sample currently under the optics being imaged in a larger section—a corresponding moving image, in particular a video, is continuously or quasi-continuously recorded by the camera. The moving image generated by the camera can also be used as a recording, i.e.,especially as a recorded video, of which what is visible through the eyepiece is understood.

[0019] The analysis options for the user can be further improved by a suitable way of displaying one or more virtually colored images generated by the image processing unit.

[0020] In particular, the display unit can be configured to cause the display device to show the provided image and the image generated from it simultaneously in separate display areas. This allows the user to view an image of the tissue sample with the usual information content at the same time as a corresponding image that has virtual colorization and can therefore provide additional information.

[0021] The display unit can also be configured to cause the display device to show the provided image and the image generated from it in the same display area, depending on user input, with the user input determining which image is displayed. Such a display mode can be advantageous to facilitate the user's spatial correlation of the provided image and the image generated from it. In particular, by switching back and forth between the provided and the generated image, the user can recognize how one and the same tissue area, which preferably appears in both images on the same part of the display area, changes in terms of its information content due to the generated virtual coloring.

[0022] In a further embodiment, the display unit is configured to cause the display device to show the provided image in a common display area and, depending on user input, to display a portion of the generated image in a sub-area of ​​the common display area, wherein the displayed portion of the generated image corresponds spatially to the portion of the provided image that would be displayed in that sub-area of ​​the common display. This mode of representation can also be understood as a variant of the aforementioned mode of representation, with the difference that the user input now specifically defines a sub-area in which the generated image is to be displayed instead of the provided image.Because the provided image continues to be displayed in a remaining part of the common display area, this embodiment can also be understood as a variant of the first-mentioned display method, and therefore as a hybrid display. According to this embodiment, the user can freely determine which areas of the tissue sample they wish to display with the respective virtual staining, while simultaneously, and in the correct spatial relationship to the virtually stained image area, other areas are displayed with their original staining or without staining.

[0023] In a modification of the latter embodiment, the display unit is configured to cause the display device to show the image generated from the provided image in the common display area, and to display a portion of the provided image in a sub-area of ​​the common display area depending on a third user input, wherein the displayed portion of the provided image spatially corresponds to the portion of the generated image that would be displayed in the sub-area of ​​the common display. This modification could also be described as an inverted representation compared to the above-mentioned display mode. The two display modes can correspond to user-selectable options.More generally, the display unit can be configured to cause the display device to show a first image of the tissue sample in a common display area, and, depending on a third user input, to show a portion of a second image of the tissue sample in a sub-area of ​​the common display area, wherein the displayed portion of the second image corresponds spatially to the portion of the first image that would be displayed in the sub-area of ​​the common display, where a) the first image is the provided image and the second image is the image generated from it, and / or b) the second image is the provided image and the first image is the image generated from it. Preferably, the second image is not displayed in the sub-area, not even partially. Conversely, the second image is preferably not displayed, not even partially, in a remaining portion of the common display area that differs from the sub-area.The sub-area and the remaining area are therefore preferably mutually exclusive.

[0024] The invention also relates to an intended use of the described system, i.e., to the use of the system for analyzing an image of a tissue sample taken via a microscope camera connected to a microscope. It is understood that the use therefore particularly includes taking the image using the microscope camera.

[0025] Furthermore, the invention relates to a device for image-based analysis of a tissue sample, which, in addition to the system, comprises the microscope with the attached microscope camera and the display device. The device thus includes a microscope with an attached microscope camera for capturing an image of a tissue sample, a system according to one of the embodiments described above for generating a virtually colored image of the tissue sample from the captured image, and a display device for displaying the generated image.

[0026] Furthermore, the invention relates to a method corresponding to the system, i.e., a method for assisting a user in the image-based analysis of a tissue sample, comprising the following steps: providing an image of the tissue sample taken via a microscope camera connected to a microscope, generating a virtually colored image of the tissue sample from the provided image, and outputting the generated image to a display device for analysis by the user.

[0027] The invention also relates to a computer program corresponding to the system and the method, i.e., a computer program to assist a user in the image-based analysis of a tissue sample, comprising instructions whose execution on a computer causes the computer to perform the above-mentioned method.

[0028] It is understood that the system, its use, the setup, the method, and the computer program have similar or identical embodiments. It is therefore understood that the use according to claim 14, the setup according to claim 15, the method according to claim 16, and the computer program according to claim 22 also utilize the particular advantages of the system according to claim 1, as they manifest themselves in the common embodiments according to the dependent claims. For example, the use of the system and / or the method may include manual and / or automatic focusing of the microscope and / or the microscope camera connected to the microscope, i.e., in particular, the use of the manual and / or automatic focusing device.Similarly, the use of the system and / or the method may include, for example, the use of a specific transmitted light microscope as the microscope to which the camera capturing the image is attached. Accordingly, it is also understood that the present disclosure includes, in addition to the use of the system, the use of the apparatus which includes both the system and the microscope.

[0029] In the following, embodiments of the invention are described with reference to the figures mentioned below in order to illustrate the aspects mentioned above as well as other aspects.

[0030] BRIEF DESCRIPTION OF THE FIGURES

[0031] Figure 1 schematically and exemplarily shows a device for image-based analysis of a tissue sample.

[0032] Figure 2 schematically and exemplarily shows a procedure for image-based analysis of a tissue sample.

[0033] Figure 3A schematically and exemplarily shows a first design of a graphical user interface for displaying images of a tissue sample,

[0034] Figure 3B schematically and exemplarily shows a modification of the first design of the graphical user interface,

[0035] Figure 4 schematically and exemplarily shows a second design of the graphical user interface, and

[0036] Figure 5 shows a schematic and exemplary third design of the graphical user interface.

[0037] DETAILED DESCRIPTION OF EXECUTION FORMS

[0038] Figure 1 shows an exemplary schematic representation of a device to assist a user in the image-based analysis of a tissue sample 10. The device comprises a microscope 12 with an attached camera 11 for imaging a suitably prepared tissue sample 10 using the optics of the microscope 12, wherein the image generated by the optics of the microscope 12 is digitized via the camera 11. The microscope 12 can be, in particular, a transmitted light microscope or a fluorescence microscope, and especially also an autofluorescence microscope. As is typical for microscope cameras, the camera 11 does not have adjustable optics.

[0039] The camera 11 does not affect the operation of the microscope 12, but merely allows areas of the tissue sample 10, as imaged by the microscope's optics 12, to be displayed and / or saved as digital images. In particular, the user can still view the tissue sample 10 directly through the optics, i.e., via a suitable eyepiece. The optics therefore have a first output in the form of an eyepiece and a second output for connecting the camera 11.

[0040] The optics of microscope 12 can be adjusted manually, semi-automatically, or fully automatically. If microscope 12 has at least a partially manual focusing device for focusing the tissue sample 10, the sharpness with which the tissue sample is focused depends on...

[0041] The image shown in Figure 10 depends on the user's operation of the focusing device. This can lead to significant variations in the quality of the images captured by camera 1 across different areas of the tissue sample 10, and especially across different tissue samples examined successively. It should be noted that a certain degree of variation is unavoidable even for well-trained users, as the human eye involuntarily corrects the image produced by the optics. On the other hand, controlling the microscope 12 solely via the camera proves to be problematic.

[0042] The use of 11 recorded images proves difficult in practice. It is therefore common practice to manually control the microscope 12, in particular to manually focus the tissue sample 10, using images – involuntarily corrected by the human visual system – that do not correspond to those recorded by the camera 11.

[0043] An adapter, such as a C-mount or TV adapter, can be used to attach the camera 11 to the microscope 12. This adapter may also have its own optics, i.e., optics different from those of the microscope 12. Such optics on the adapter between the camera 11 and the microscope 12 can also be adjustable, for example, manually. However, this feature is typically only used during the initial adjustment of the microscope. During regular operation, i.e., particularly during the analysis of a tissue section, the adapter's optics would remain unchanged. As a result, the image perceived by the user through the eyepiece and the image captured by the camera 11 would generally continue to differ.This circumstance, which has not previously been considered a problem, can be mitigated by providing a fully automatic focusing device for the microscope, or an automatic focusing device that focuses the camera image but not the image at the eyepiece. The latter could, for example, be an autofocus adapter for connecting camera 11 to the microscope.

[0044] The device shown in Figure 1 further comprises a data processing unit 100 for processing and / or storing the images captured by the camera 11. The captured images can thus be stored, for example, on an image memory of the data processing unit 100 for later analysis. An image delivery unit 101 of the data processing unit 100 serves as an interface to the camera 11. The image delivery unit 101 is therefore configured to receive an image of the tissue sample 10 captured by the camera 11 and to store it and / or to provide it to an image processing unit 102 of the data processing unit 100 for processing. The processing by the image processing unit 102 includes generating a virtually colored image from the provided image, which is hereinafter also referred to as the camera image.The generated, virtually stained image shows the same section of tissue sample 10 as the camera image, but with virtual staining. If the camera image shows no staining because tissue sample 10 was not stained, this may mean that tissue structures only become visible in the virtually stained image. If the camera image already shows staining because tissue sample 10 was stained, different tissue structures may become visible than in the camera image if the virtual staining differs.

[0045] To generate the virtually colored image from the camera image, the image processing unit 102 is set up to use one of a plurality of models that have been trained by machine learning or otherwise adapted, parameterized or configured to map images of tissue samples without staining or in an initial staining onto corresponding images of the tissue samples in an initial staining.For example, as shown in Figure 1, the following can be stored in the data processing unit 100 for access by the image processing unit 102: a first model trained to map images of colorless tissue samples to corresponding images of tissue samples with a virtual stain A, a second model trained to map images of colorless tissue samples to corresponding images of tissue samples with a virtual stain B, and a third model trained to map images of tissue samples with stain A to corresponding images of tissue samples with stain B. The training of the models can be carried out using a dedicated model training unit of the data processing unit 100 and camera images previously stored in the image storage unit and declared as training images.However, the training could also have been performed separately, particularly based on images not captured by camera 11, i.e., using data from other sources. It is also possible that the training was not specifically carried out for the intended application, but that the trained model is nevertheless suitable for it. As mentioned above, however, training for the intended application is preferred, especially using the same input images as training input that will later be used as input images when the trained model is deployed. In any case, the architecture of the models, i.e., the structure of the models in their untrained form, is in principle arbitrary. In particular, the architecture of the models is not limited to deep neural networks.

[0046] Advantageously, the trained models have an architecture that allows for real-time image processing, i.e., without significant delay between capturing or providing a camera image and generating a corresponding image with virtual staining. For example, a number of the model parameters can be chosen to be correspondingly low. Real-time image processing can be particularly advantageous when the camera image is a moving image, such as one that might be captured by the video recording function of camera 11 while the user controls the microscope 12 so that its optics and the tissue sample 10 move relative to each other and / or the magnification setting of the optics changes.The different and / or differently sized areas of the tissue sample 10 that then appear in the moving image of the camera 11 over time can also be seen in the virtually colored image generated by the image provision unit 102, which in this case would be a moving image corresponding to the camera image in time.

[0047] The data processing system 100 further comprises an image output unit 103, which is configured to output images generated by the image processing unit 102 to a display device 13 of the device shown in Figure 1 for analysis by the user. The image output unit 103 can therefore have a further interface to the data processing system 100, but in addition, it can be configured in particular to generate and / or control a graphical user interface to be displayed on the display device 13. Figures 3A to 5 show advantageous embodiments of the user interface.

[0048] Figure 2 shows an example flowchart of the steps in a procedure for analyzing tissue sample 10, using the equipment shown in Figure 1. In a first step 201 of the procedure, the tissue sample 10 is taken. This can be done during a surgical procedure and could also be called a biopsy. In a second step 202 of the procedure, the taken tissue sample 10 is prepared for further analysis. In histology, this typically involves fixing the tissue sample 10, embedding it in a support material, sectioning the embedded tissue sample, and arranging the sections on slides for imaging under a microscope, such as microscope 12. Thus, in step 203, a camera image of a tissue section prepared on a slide is provided.In a subsequent step 204, this image is then virtually colored as described above with reference to Figure 1 and subsequently, in a further step 205, displayed on the display device 13.

[0049] Figure 3A shows an exemplary and schematic configuration of the graphical user interface, according to which the camera image and the virtually colored image generated from it are displayed simultaneously in separate display areas. In Figure 3A, the camera image is displayed in a left display area 31 and the virtually colored image in a right display area 32. A selection area 33 is provided between the two display areas 31 and 32, which has user-selectable buttons, each corresponding to a possible virtual color of the camera image. The virtually colored image displayed in display area 32 has the virtual color that corresponds to the button selected by the user.

[0050] Figure 3B shows an exemplary and schematic modified form of the graphical user interface according to Figure 3A. The modification consists in the fact that instead of display area 32, a split display area 34 with separate display sub-areas is provided. In the display sub-areas of display area 34, the camera image is displayed simultaneously in different virtual colors. Which virtual colors are applied or displayed is determined by a selection made by the user in the selection area 33, whereby, unlike in Figure 3A, several buttons can now be selected simultaneously.

[0051] Figure 4 shows an exemplary and schematic embodiment of the graphical user interface displayed by the display device 13, according to which only one display area 41 is provided in which, at any given time, only either the camera image or a virtually colored image is displayed. The user can determine which image is displayed by selecting a corresponding button in a selection area 42. Figure 5 shows an exemplary and schematic embodiment of another embodiment of the graphical user interface. According to this embodiment, a common display area 50 is provided in which, by default, the camera image is displayed, but in which the user can define a sub-area 51 by means of a corresponding user input, in which a corresponding image area with virtual coloring is displayed.The virtual color applied in sub-area 51 can be determined by the user selecting a button in a selection area 53. Alternatively, instead of the camera image, a virtually colored image can be designated as the image to be displayed in display area 50, in which case a section of the image with a different virtual color would be displayed in sub-area 51. It can also be useful to have a virtually colored image designated as the image to be displayed in display area 50, while the image section displayed in sub-area 51 is a corresponding section of the camera image. The user can determine which of these three options is applied by entering a corresponding input.

[0052] The display area 50 could also be understood as a superimposition of the camera image with the virtually colored images generated from it, with each layer of the superimposition corresponding to a different virtual coloring of the tissue sample 10 and in the sub-area 51 a corresponding section of the layer corresponding to the virtual coloring selected by the user is shown.

[0053] Although the advantages of the embodiments described here are particularly evident in histopathology, other fields of application are conceivable. In any case, a tissue sample in the present sense is any sample of tissue, which can be of plant, animal, or human origin. Actual staining of a tissue sample, which could also be referred to as histological staining, is understood here as the marking of biological elements of the tissue sample based on biochemical properties, whereby the choice of staining agent can correspond to the choice of elements to be marked. Virtual staining can be understood as a digital replica of the actual staining. The digital replica is performed on an image of the tissue sample instead of on the tissue sample itself. The term "image" is also intended to encompass image data corresponding to an image.

[0054] Variations of the embodiments described herein will be recognized and implemented by a person skilled in the art after studying the disclosure, including the figures and claims, during the execution of the claimed invention. In the claims, the words "comprising" and "comprising" do not exclude other elements or steps, and the definite article "a" or "an" does not exclude a plurality.

[0055] A single unit or device can perform the functions of several elements listed in the claims. The fact that individual functions and elements are listed in different dependent claims does not preclude the possibility of advantageously using a combination of these functions or elements.

[0056] A computer program can be stored or distributed on a suitable medium, such as an optical storage medium or solid-state storage medium (SSD), which may be provided together with or as part of other hardware, or in other ways, such as via the Internet or other wired or wireless telecommunications systems.

[0057] The reference numerals in the claims are not to be understood as limiting the subject matter and scope of protection of the claims by these reference numerals.

[0058] The invention relates to a system for assisting a user in the image-based analysis of a tissue sample. The system comprises an image provision unit configured to provide an image of the tissue sample captured by a microscope camera connected to a microscope, and an image processing unit configured to generate an image of the tissue sample with virtual staining from the provided image. Furthermore, the system comprises an image output unit configured to output the generated image to a display device for analysis by the user. The system enables resource-efficient and, at the same time, particularly thorough histological analyses.

Claims

Claims 1. System for assisting a user in image-based analysis of a tissue sample (10), comprising: an image provisioning unit (101) configured to provide an image of the tissue sample (10) captured via a microscope camera (11) connected to a microscope (12), an image processing unit (102) configured to generate an image of the tissue sample (10) with virtual staining from the provided image, and an image output unit (103) configured to output the generated image to a display device (13) for analysis by the user.

2. System according to claim 1, wherein the microscope has a manual focusing device for manually focusing the tissue sample (10) and the provided image was taken using the manual focusing device.

3. System according to one of claims 1 or 2, wherein the microscope (12) has an automatic focusing device for automatically focusing the tissue sample (10) and the provided image was captured by means of the automatic focusing device.

4. System according to one of the preceding claims, wherein the provided image is an image of an unstained tissue sample (10).

5. System according to one of claims 1 to 3, wherein the provided image is an image of a tissue sample (10) stained with a first stain and the virtual staining is a virtual staining with a second staining which is different from the first staining.

6. System according to any of the preceding claims, wherein the microscope (12) is a transmitted light microscope.

7. System according to any one of claims 1 to 5, wherein the microscope (12) is a fluorescence microscope.

8. System according to one of the preceding claims, wherein the provided image and the image generated therefrom are temporally corresponding moving images, wherein a movement recognizable in the images corresponds to a relative movement of the optics of the microscope (12) and the tissue sample (10) to each other and / or to a change in a magnification setting of the optics of the microscope (12).

9. System according to one of the preceding claims, wherein the image processing unit (102) is configured to select the virtual coloring from a predetermined plurality of possible virtual colorings depending on a first user input.

10. System according to one of the preceding claims, wherein the display unit (103) is configured to cause the display device (13) to display the provided image and the image generated therefrom simultaneously in separate display areas (31 , 32; 31 , 34).

11. System according to one of the preceding claims, wherein the display unit (103) is configured to cause the display device (13) to display the provided image and the image generated therefrom in the same display area (41) depending on a second user input, wherein the second user input determines which of the images is displayed.

12. System according to one of the preceding claims, wherein the display unit (103) is configured to cause the display device (13) to display a first image of the tissue sample in a common display area (50), and to display a part of a second image of the tissue sample in a sub-area (51) of the common display area (50) depending on a third user input, wherein the displayed part of the second image corresponds spatially to the part of the first image that would be displayed in the sub-area (51) of the common display (50), wherein a) the first image is the provided image and the second image is the image generated therefrom and / or b) the second image is the provided image and the first image is the image generated therefrom.

13. System according to one of the preceding claims, wherein the image processing unit is configured to determine the virtual coloring by means of a model, that has been trained by machine learning using training images, wherein the training images comprise pairs of a training input image and a training output image, wherein the training input image was taken using a camera attached to a microscope and shows a tissue sample used for training, and wherein the training output image shows the tissue sample used for training after it has actually been stained or re-stained.

14. Use of a system according to one of the preceding claims for analyzing an image of a tissue sample (10) taken via a microscope camera (11) connected to a microscope (12).

15. Device for image-based analysis of a tissue sample (10), comprising: a microscope (12) with a microscope camera (11) attached thereto for recording an image of the tissue sample (10), a system according to one of claims 1-13 for generating a virtually colored image of the tissue sample (10) from the recorded image, and a display device (13) for displaying the generated image.

16. Method for assisting a user in image-based analysis of a tissue sample (10), comprising: Providing (203) an image of the tissue sample (10) taken via a microscope camera connected to a microscope (12), Generating (204) a virtually stained image of the tissue sample (10) from the provided image, and Output (205) of the generated image to a display device (13) for analysis by the user.

17. The method of claim 16, wherein providing the image of the tissue sample (10) comprises: Taking the image using the microscope camera connected to the microscope (12).

18. The method of claim 17, wherein capturing the image comprises manually focusing the tissue sample (10).

19. The method of claim 17 or 18, wherein capturing the image comprises automatically focusing the tissue sample (10).

20. Method according to any one of claims 17 to 19, wherein the microscope (12) is a transmitted light microscope.

21. Method according to one of claims 17 to 20, wherein the image capture comprises capturing a moving image in the form of a video, wherein the provided image and the image generated therefrom are temporally corresponding moving images, wherein a movement recognizable in the images corresponds to a relative movement of the optics of the microscope (12) and the tissue sample (10) to each other and / or to a change in a magnification setting of the optics of the microscope (12) that occurred during the capture of the moving image.

22. Computer program for assisting a user in the image-based analysis of a tissue sample (10), comprising instructions the execution of which on a computer causes the computer to execute a method according to claim 16.

Images