Method and apparatus for analyzing histopathology image data

Abstract

The invention provides a method and system for analyzing histopathology image data. The invention provides a method for providing similarity information about different histopathology images of a patient, the method having the steps of: providing first histopathology image data, the first histopathology image data being based on a tissue sample taken from the patient at a first time instance; providing second histopathology image data, the second histopathology image data being based on a tissue sample taken from the patient at a second time instance different from the first time instance; analyzing, by means of an image processing algorithm, the first histopathology image data and the second histopathology image data for a similarity between at least one region in the first histopathology image data indicative of a pathological identification and at least one region in the second histopathology image data indicative of a pathological identification; and determining the similarity information based on the step of analyzing; and providing the similarity information.

Description

Technical Field

[0001] This invention relates to a method and apparatus for analyzing histopathological image data. More particularly, it relates to a method and apparatus for analyzing the similarity of different histopathological image data. Background Technology

[0002] Analyzing tissue samples using histopathological methods is a core element of cancer diagnostics. Here, tissue samples are extracted from areas of the patient's body where pathological changes may be present. Typically, multiple segments or blocks are obtained from the tissue sample and then cut into micron-thin tissue sections. To better identify or quantify potential tissue changes, the tissue sections are stained with histopathological staining agents. The stained tissue sections are then analyzed under a microscope by a female or male pathologist, allowing conclusions to be drawn regarding possible pathological changes in the fine histological structure of the examined tissue.

[0003] Histopathological examination is a very demanding task. In addition to tissue sample extraction itself, the examination requires the preparation of tissue sections, including cutting, fixing, and staining. It should be noted that multiple segments and sections prepared from each tissue sample must typically be analyzed.

[0004] Therefore, to reduce the burden on medical staff, more and more of these processes have been automated and digitized over the past few decades. Consequently, computer-controlled automated preparation and staining machines are commonly used in modern laboratories. Furthermore, stained tissue sections are now typically digitized for further use. This is done using a specialized scanner, known as a slide scanner. The image recorded in this process is also called a "whole slide image." The histopathological image data obtained in this way is then viewed and analyzed by either a female or male pathologist at a digital diagnostic station.

[0005] While many processes in the histopathology workflow can be improved and accelerated through continued digitization, identification itself—albeit performed at a digital diagnostic station—remains a matter that still relies heavily on the judgment of both female and male pathologists. Furthermore, the datasets of histopathological images of tissue samples are already large. Therefore, the individual histopathological images are quite large in size. This is because histopathological images should, on the one hand, provide an overview of the entire tissue section, and on the other hand, must have sufficient resolution to allow observation of individual cells. Moreover, histopathological image datasets typically contain not only single images of tissue sections and segments, but also multiple single images of different tissue sections from different tissue samples, often from different segments and stained with different histopathological stains. Identification is further complicated by the inherently non-uniform nature of histopathological image data, which significantly complicates the quantification and reproducible classification of such data. The problems that must be addressed during identification are similarly complex. In addition to the fundamental question of whether pathological changes exist, it is often necessary to determine, for example, whether new tissue changes have occurred or whether a known disease in the medical history has reappeared.

[0006] To date, the limited scope of effectiveness of all this and the histopathological identification used to treat patients has significantly hampered the effective and comprehensive use of automated identification procedures based on digital image data. This results in pathologists facing an increasing workload due to the continuous automation of upstream processes, yet in actual identification, pathologists largely have to rely on themselves. Furthermore, due to the ever-growing volume of data, incorporating all available information and considering all available information when evaluating cases is becoming increasingly difficult. Summary of the Invention

[0007] Therefore, the object of the present invention is to provide a method and apparatus for assisting users in identifying digital histopathological image data.

[0008] According to the present invention, the proposed objective is achieved by means of a method, apparatus, computer program product, or computer-readable storage medium according to embodiments of the invention. Advantageous improvements are described in subsequent embodiments.

[0009] In the following description, the solution according to the invention to the stated objective is described not only in terms of the claimed device but also in terms of the claimed method. The features, advantages, or alternative embodiments mentioned herein can also be applied to other claimed subjects, and vice versa. In other words, physical embodiments (such as those for devices) can also be improved by incorporating the features described or claimed in connection with the method. Here, the corresponding functional features of the method are constituted by corresponding physical modules.

[0010] Furthermore, the solution according to the invention for the stated objective is described not only in terms of methods and apparatus for visualizing three-dimensional volumes, but also in terms of methods and apparatus for adjusting trained functions. Here, the features and alternative implementations of data structures and / or functions in the methods and apparatus for determining can be applied to similar data structures and / or functions in the methods and apparatus for adjusting. Here, similar data structures can be characterized, in particular, by using the prefix "training". Furthermore, the trained functions used in methods and apparatus for analyzing histopathological image data can be adjusted and / or provided, in particular, by methods and apparatus for adjusting trained functions.

[0011] According to one embodiment of the present invention, a computer-implemented method is provided for providing similarity information about different histopathological image data of a patient. The method comprises several steps. One step involves providing first histopathological image data. The first histopathological image data is based on a tissue sample extracted from the patient at a first time. Another step involves providing second histopathological image data. The second histopathological image data is based on a tissue sample extracted from the patient at a second time, different from the first time. Another step involves determining similarity information based on the first and second histopathological image data using an image processing algorithm. The similarity information includes a description of the similarity between at least one region in the first histopathological image data indicating pathological identification and at least one region in the second histopathological image data indicating pathological identification. A third step involves providing the similarity information.

[0012] The first and second histopathological image data are image datasets, which may contain one or more individual images, particularly two-dimensional ones. Another term for the first histopathological image data is a first histopathological image dataset. Another term for the second histopathological image data is a second histopathological image dataset. A single image or multiple single images may be pixel images, respectively. A single image or multiple single images depict tissue sections prepared from a tissue sample from a patient. If multiple tissue sections are depicted in the histopathological image dataset, all of these tissue sections can be prepared from the same tissue sample. Thus, all image data in the first histopathological image data can be generated based on a single tissue sample, and all image data in the second histopathological image data can be generated based on another / other tissue sample. Both tissue samples can be extracted from the same patient, more precisely, from the same or at least one similar anatomical target region of the patient, but at different times. For example, there may be days, months, or years between said times, as well as various medical treatments of the patient.

[0013] Preparing tissue sections from a tissue sample can include cutting segments from the tissue sample (e.g., using a punching tool) into micron-thin layers, i.e., tissue sections. Another term for a segment is a block or punched section. Here, tissue sections depicted in histopathological image data can be obtained, in particular, from different segments within the same tissue sample. During microscopic observation, a single image of histopathological image data can reveal the fine tissue structure of a tissue sample, and in particular, cellular structures or cells contained within the tissue sample. When observed on a larger length scale, a single image can provide an overview of the tissue structure and density.

[0014] Preparation of tissue sections also includes staining the tissue sections with histopathological staining agents. Here, staining can be used to highlight different structures in the tissue section, such as cell walls or nuclei, or to examine medical indicators, such as cell proliferation levels. Different histopathological staining agents are used for different purposes. Specifically, all individual images included in a histopathological image dataset can depict tissue sections stained with the same histopathological staining agent. Alternatively, individual images included in a histopathological image dataset can depict tissue sections stained with different histopathological staining agents.

[0015] To generate histopathological image data, the stained tissue sections are digitized or scanned. For this purpose, the tissue sections are imaged using a suitable digitizing station, such as a so-called whole-slide scanner, which preferably scans the entire tissue section stretched on a slide and converts it into a pixel image. To obtain the color effect of the histopathological stain, the pixel image is preferably a color pixel image. Because not only the overall impression of the tissue but also the finely resolved cellular structures are important in identification, individual images included in histopathological image data typically have a very high pixel resolution. The data size of a single image can typically be several gigabytes. Alternatively, digitized records of tissue sections can be combined into a histopathological image dataset. Alternatively, individual records can also form histopathological image data. Histopathological image data can be digitally processed and, in particular, archived in a suitable database.

[0016] In addition to image data, histopathological image data may also include metadata, which may include, for example, the time the tissue sample was extracted, the patient identifier, one or more histopathological stains used, pathological identification, and / or the anatomical target region from which the tissue sample originated. Alternatively or additionally, this information may be stored in a database that archives histopathological image data or in a separate database. Such a database may, for example, be part of one or more medical information systems, such as a Hospital Information System (HIS), a Radiology Information System (RIS), a Laboratory Information System (LIS), a Cardiovascular Information Systems (CVIS), and / or a Picture Archiving and Communication System (PACS).

[0017] Therefore, the phrase "based on tissue samples" can generally be interpreted as meaning that the corresponding histopathological image data has image data showing tissue sections that have been prepared from tissue samples and stained with histopathological staining agents.

[0018] Regarding histopathological image data, "provided" can mean that the histopathological image data is provided by a digitization station for further use. Furthermore, "provided" can mean that the histopathological image data is retrieved from a corresponding database, retrieved from a corresponding database, and / or loaded, or can be loaded into a computing unit so that the histopathological image data undergoes one or more processing steps, for example, in a data processing apparatus.

[0019] Image processing algorithms can be understood, in particular, as computer program products configured to determine similarity information by analyzing image data or pixel values ​​of first and second histopathological image data. An image processing algorithm may have program components in the form of one or more instructions for a processor for determining similarity information. Image processing algorithms can be provided, for example, by being stored in a storage device or loaded into the working memory of a suitable data processing device, or generally made available.

[0020] The area indicating pathological identification can specifically display or suggest one or more pathological changes in the depicted tissue. In other words, the area indicating pathological identification can be an area indicating one or more pathological changes. For example, the area indicating pathological identification may have one or more tumor cells or one and / or more pathological tissue structures. The area indicating pathological identification can be automatically and / or identified by the user in the first histopathological image data and the second histopathological image data, respectively. For example, the user can be a female doctor or a male doctor or a female pathologist or a male pathologist.

[0021] Similarity between regions indicating pathological identification can be particularly morphological or structural similarity between suspicious regions. For example, similar regions may have similar tissue structures, similar textures, similar pixel or color values, similar cell densities, similar cell morphologies, similar patterns, and / or other similar features. Image processing algorithms can be configured to automatically extract such and other features from first and second histopathological image data, and to compare the first and second histopathological image data to determine a quantitative measure of similarity (similarity metric).

[0022] Similarity information is created based on similarity analysis. In particular, similarity information can describe the similarity between regions indicating pathological identification. Furthermore, similarity information can describe the similarity between regions indicating pathological identification. In particular, similarity information can describe or be based on a quantitative measure of similarity (similarity measure).

[0023] Providing similarity information can include offering it for any further use. For example, it can be provided to another algorithm for further analysis. Additionally, it can be provided for archiving in a database. Furthermore, it can be made available to users through a user interface.

[0024] By providing similarity information, conclusions are drawn about the degree of similarity between pathological changes in first and second histopathological image data. Therefore, instead of searching for possible similarities indiscriminately across the entire histopathological image data, the search is specifically targeted at areas displaying or predicting pathological tissue changes. The inventors have recognized that this information is particularly relevant to questions such as whether new tissue changes have emerged, i.e., whether a new disease is involved, or whether a recurrence or re-spread of a known disease in the medical record is involved. This phenomenon is also known as relapse. It indicates that tissue changes stemming from a recurrence or re-spread of an underlying disease have similar morphological and / or structural features. The automated assessment of similarity not only enables the discovery of subtle or hidden similarities that may remain invisible to the human eye, but also ensures rapid, systematic, and comprehensive comparisons of the available image data. Given the enormous image size and data volume, comparison is often impractical for users without auxiliary methods. For example, users gain valuable additional information when it comes to decisions such as whether a treatment concept is successful or must be adjusted. Therefore, based on identifying medically relevant parameters and automatically evaluating these parameters in digital measurement data, the inventors have created a method to continuously assist users during medical diagnosis.

[0025] According to one embodiment, the method may further include the step of identifying regions in the first histopathological image data and / or the second histopathological image data that indicate pathological identification.

[0026] In other words, the regions related to the aforementioned problems are automatically or semi-automatically identified, namely, regions in the histopathological image data that display or indicate pathological tissue changes. Here, the regions can be identified not only in the first histopathological image data but also in the second histopathological image data, or only in one of the two histopathological image data. During identification, existing user annotations, such as those compiled during previous identifications, can be utilized. For this purpose, for example, the metadata of the corresponding histopathological image data can be evaluated. Alternatively or additionally, the entire histopathological image data can be analyzed separately. For example, the image processing algorithm can be configured such that regions indicating pathological identification can be identified by applying the image processing algorithm to the first and / or second histopathological image data. As another alternative, user input regarding regions indicating pathological identification can be evaluated. For example, the user can mark one or more regions via the user interface, and these regions should then be used as relevant regions (hereinafter also referred to as regions of interest) as the basis for further processing.

[0027] By identifying regions that indicate pathological identification, similarities between pathological tissue changes can be searched more specifically. Therefore, corresponding conclusions can be provided in a shorter time and with higher reliability. Furthermore, by at least semi-automating the selection of relevant regions, the burden on users in the user task of creating medical diagnoses can be further reduced.

[0028] According to one implementation, the similarity information includes:

[0029] - Description of similar regions in the corresponding first and / or second histopathological image data that indicate pathological identification;

[0030] - An auxiliary image based on first histopathological image data and / or second histopathological image data, in which similar areas for pathological identification are highlighted;

[0031] -Location information of similar regions in the first histopathological image data and / or the second histopathological image data that indicate pathological identification;

[0032] - A quantitative description of the corresponding similarity of similar regions in the corresponding first and second histopathological image data that indicate pathological identification;

[0033] - A statement regarding whether a recurrence relationship exists between the first and second histopathological image data.

[0034] The similarity information provided indicates to the user information relevant to the analysis and identification of histopathological image data. By describing regions with similar pathological changes, the user is specifically shown areas that may indicate the recurrence of the disease. Based on the similar regions, the user can envision possible similarities between pathological changes in successive tissue samples. Here, the region may be characterized by location information, which, for example, can be plotted in a graphical representation in the first and / or second histopathological image data. This location information may include, for example, a description of coordinates.

[0035] Furthermore, according to some implementations, information about the relative proportions of regions indicating pathological identification can be provided in the corresponding histopathological image data. Therefore, the user obtains an explanation of how pathological tissue changes develop over time.

[0036] In addition, one or more auxiliary images may be provided. The auxiliary images may be based on image data from the first histopathological image data and / or the second histopathological image data, or may be presented based on said image data. In the auxiliary images, areas indicating pathological identification may be highlighted, for example, by markings, by using borders or masks, and / or by using color.

[0037] Here, in particular, areas indicating pathological identification, all areas indicating pathological identification, and / or areas indicating pathological identification that are similar in histopathological image data can be separately identified in the auxiliary images. In this way, the user obtains an overview of all tissue changes and, on the other hand, a display of areas indicating recurrence. For example, areas indicating pathological identification can typically be identified by one color, and similar areas can be identified by another color.

[0038] Alternatively or additionally, through optional quantitative descriptions of similarity, the user arrives at conclusions regarding the degree of similarity. Thus, the user can determine which regions have high similarity and focus on those regions in the analysis. Quantitative descriptions can be provided as numerical values ​​or, for example, integrated into auxiliary images (e.g., as numerical indicators or in color-coded form).

[0039] Furthermore, the similarity information may include a description of the recurrence relationship between the first and second histopathological image data. In other words, it is a description of whether a pathological change visible either in the first or second histopathological image data can be identified in a similar manner in the corresponding other histopathological image data. This indicates a conclusion regarding whether the pathological change is a recurrence or further development of an existing pathological change. According to the concept of the invention, such a conclusion can be made based on one or more quantitative descriptions of similarity. For this purpose, for example, the mean, median, or maximum value of the quantitative descriptions of similarity can be evaluated.

[0040] According to one implementation, the step of providing similarity information includes displaying the similarity information to a user via a user interface, thereby allowing the user to directly understand the results of the similarity analysis.

[0041] According to one embodiment, the method further includes the step of filling in a medical report template based on similarity information.

[0042] Automating report template completion can further reduce the burden on users when identifying histopathological image data. For example, the report template could be an electronic medical report. Placeholders for entering case-specific information can be included in the template. During the completion process, one or more placeholders can be entered based on similarity information.

[0043] According to one embodiment, the analysis steps include identifying regions of interest (ROIs) in first histopathological image data that indicate pathological identification, and searching for similar regions in second histopathological image data, each of which is similar to the ROI. Here, the searching step includes applying an image processing algorithm to the second histopathological image data, and the step of determining similarity information is based on the searching step.

[0044] In other words, the region of interest (ROI) represents one or more regions in the first histopathological image data that indicate pathological identification (or pathological changes). The ROI may, in particular, be predefined (e.g., through previous identification) or dynamically predefined (e.g., through user input or automatically). Conversely, similar regions in the second histopathological image data are determined based on the ROI. Similar regions can be understood as regions in the second histopathological image data that indicate pathological identification (or pathological changes). The similarity between the ROI and the similar regions may again include morphological and / or structural similarity. For example, similar regions may have similar tissue structures, similar textures, similar pixel or color values, similar cell densities, similar cell morphologies, similar patterns, and / or other similar features. Image processing algorithms can be configured to automatically extract such features and other features from the ROI and possible similar regions and compare them. Image processing algorithms can also be configured to determine a quantitative measure of similarity (similarity measure) based on the comparison. Then, similar regions may, in particular, be regions where the similarity measure is higher than a predefined or predefined threshold. Here, this threshold may be determined automatically or predefined by the user. In addition, thresholds can be determined semi-automatically by suggesting thresholds to users.

[0045] By identifying regions of interest (ROIs) in the first histopathological image data, regions similar to ROIs can be specifically searched in the second dataset. This allows for more targeted discovery and display of morphological and / or structural similarities in pathological changes in tissue samples from different time periods of the patient's life. Consequently, users can draw informed conclusions about whether, for example, pathological changes in newly extracted tissue samples are recurrences of known pathological changes. This effectively supports users during identification, and especially in making diagnoses or prediagnoses.

[0046] According to one implementation, the region of interest has one or more individual regions defined in the first histopathological image data.

[0047] Here, each region can individually indicate one or more pathological identifications or display one or more diseased tissue changes. For example, each region can have local data from the first histopathological image data and thus also have (pixel) image data. Accordingly, the region of interest can also have (pixel) image data. One or more regions defined in the histopathological image data can also include a single (complete) image of the first histopathological image data or the entire histopathological image data. Correspondingly, the region of interest can also include one or more single images of the first histopathological image data or the entire histopathological image data. The regions defined in the first histopathological image data can have different shapes. For example, the defined regions can be rectangular or circular or have any other boundaries. By considering multiple regions for the region of interest, a larger data base can be provided for searching similar regions. Conversely, if only one region or locality in the first histopathological image data appears relevant, the region of interest can be limited to said only one region or locality in the first histopathological image data. Overall, through adaptive definition of the region of interest, a good adaptation to the corresponding situation can be achieved.

[0048] According to one implementation, the step of identifying the region of interest includes determining the region of interest using an image processing algorithm and / or evaluating a user's annotation that identifies the region of interest. Here, the annotation may be provided, in particular, after the step of providing first histopathological image data via manual input by the user through a user interface.

[0049] In other words, the region of interest (ROI) can be determined automatically and / or manually by the user. This allows for case-specific and flexible determination of the ROI. User annotations can, in particular, identify one or more regions in the first histopathological image data. For this purpose, one or more reference images of the first histopathological image data can be displayed to the user in the user interface. The reference images can be diagrams generated based on the corresponding histopathological image data for display via the user interface. User-inputted annotations can be created, for example, by pointing to relevant regions in the reference images, drawing borders around relevant regions in the reference images, and / or circling relevant regions in the reference images. This can be done, for example, by using a mouse or stylus or by gesture control. Furthermore, one or more previously identified annotations may exist. These annotations can be stored as metadata about the first histopathological image data (e.g., in the first histopathological image data itself or in a separate database). By evaluating the already created annotations, they can be used to define the ROI. It is also particularly feasible to automatically identify other relevant regions in the first histopathological image data based on the user's annotations and associate them with the ROI. Here, regions in the first histopathological image data that are similar to the regions identified by the user's annotations can be searched. Correspondingly, the image processing algorithm can be configured to automatically search for regions of pathological identification indicated in histopathological image data, and, where necessary, consider regions pre-selected by user annotations.

[0050] According to one implementation, the search step includes extracting feature signatures based on regions of interest and obtaining similarity information based on the extracted feature signatures.

[0051] A feature signature may have one or more features, which are extracted from or computed from the region of interest, and in particular from image data of the region of interest. Furthermore, the feature signature may be extracted based on other information (or, where such information is additionally considered), such as, for example, based on the surrounding area around the region of interest, the entire first histopathological image data, and / or metadata about the first histopathological image data. The feature signature may particularly characterize the region of interest. Features of the feature signature may be combined into a feature vector. The feature signature may particularly have such a feature vector. Features may be morphological and / or structural and / or texture-related features and / or pattern-related features. Features may particularly include tissue structure or tissue density. Furthermore, features may include cell density, cell morphology, distribution of histopathological stains, cell size, distribution of one or more specific cell classes, etc.

[0052] Furthermore, image processing algorithms can be configured to obtain similarity information based on feature signatures.

[0053] Obtaining similarity information may include identifying potential similar regions in the second histopathological image data. Furthermore, obtaining similarity information may include extracting feature signatures from each of the potential similar regions. This can be done as with feature signatures extracted based on regions of interest. Additionally, obtaining similarity information may include comparing the feature signatures extracted based on the potential similar regions with the feature signatures extracted based on the regions of interest. Furthermore, obtaining similarity information may include determining a similarity measure based on the comparisons of the potential similar regions and may include obtaining similarity information based on one or more similarity measures.

[0054] The comparison steps can be based, in particular, on determining the intervals of the corresponding feature signatures, calculating the cosine similarity of the feature signatures, and / or calculating the difference or weighted sum of the similarities of the individual features of the feature signatures. In particular, regions in the second histopathological image data whose associated similarity measure is greater than a preset or pre-set threshold can be identified as similar regions.

[0055] By using feature signatures, parameters that are easy to implement and readily transferable for comparing different image data are defined. Furthermore, the features included in the feature signature can be based on higher-level observables derived from the image data, which typically characterize the structure of the mapping better than the image data itself.

[0056] According to one implementation, the image processing algorithm has one or more trained functions.

[0057] A trained function typically maps input data to output data. Here, the output data may be related to one or more parameters of the trained function. One or more parameters of the trained function can be determined and / or adjusted through training. Determining and / or adjusting one or more parameters of the trained function can be based, in particular, on pairs consisting of training input data and associated training output data, wherein the trained function is applied to the training input data to produce training mapping data. Determination and / or adjustment can be based, in particular, on a comparison between the training mapping data and the training output data. Typically, a trainable function, i.e., a function with parameters that are not yet adjusted, is also referred to as a trained function. By training one or more trainable functions optionally included in the image processing algorithm, the image processing algorithm can be configured to perform one or more tasks described in conjunction with the image processing algorithm, such as analyzing the similarity between at least one region indicating pathological identification from the first histopathological image data and at least one region indicating pathological identification from the second histopathological image data, searching for similar regions in the second histopathological image data, identifying regions of interest indicating pathological identification in the first histopathological image data, determining similarity information, extracting feature signatures, and / or obtaining similar regions or similarity information based on the extracted feature signatures. If multiple tasks in the task are implemented by trained functions, the image processing algorithm for each task in the task has a separate trained function. Alternatively or additionally, the trained functions can be configured or trained to complete multiple tasks in the task up to all tasks.

[0058] Other terms used for the function being trained are a training mapping rule, a mapping rule with training parameters, a function with training parameters, an artificial intelligence-based algorithm, and a machine learning algorithm. An example of a trained function is an artificial neural network. The term "neural network" can also be used as an alternative to "neural network." Neural networks are essentially constructed like biological neural networks—such as the human brain. Artificial neural networks specifically include input and output layers. The artificial neural network may also include multiple layers between the input and output layers. Each layer includes at least one, preferably multiple, nodes. Each node can be understood as a biological processing unit, such as a neuron. In other words, each neuron corresponds to an operation applied to the input data. Nodes in one layer can be connected to nodes in other layers via edges or connections, especially directed edges or connections. These edges or connections define the data flow between nodes in the network. Edges or connections are associated with parameters commonly referred to as "weights" or "edge weights." These parameters can adjust the importance of the output of a first node to the input of a second node, where the first and second nodes are connected by an edge. The trained function can also have a deep artificial neural network (the technical term is "deep neural network" or "deep artificial neural network").

[0059] In particular, neural networks can be trained. Specifically, training a neural network is performed using "supervised learning" techniques, based on training input data and associated training output data. This involves feeding known training input data into the neural network and comparing the output data generated by the network with the associated training output data. As long as the output data of the final network layer does not sufficiently correspond to the training output data, the artificial neural network learns and independently adjusts the edge weights used for each node.

[0060] According to one implementation, at least one of the trained functions has a convolutional neural network, and in particular a region-based convolutional neural network.

[0061] The technical term for convolutional neural networks is "convolutional neural network." Convolutional neural networks can be specifically configured as deep convolutional neural networks. Here, the neural network has one or more convolutional layers and one or more deconvolutional layers. The neural network may also include pooling layers. By using convolutional and / or deconvolutional layers, neural networks can be used particularly efficiently for image processing because, despite the multiple connections between nodes, only a small number of edge weights (i.e., edge weights corresponding to the values ​​of the convolutional kernels) must be determined. This allows for improved accuracy of the neural network with the same amount of training data.

[0062] The technical term for region-based convolutional neural networks is "region-based convolutional neural network." Region-based convolutional neural networks can be so-called fast region-based convolutional neural networks (the technical term for this is "fast region-based convolutional neural network") or even faster region-based convolutional neural networks (the technical term for this is "fast region-based convolutional neural network"). A characteristic of region-based convolutional neural networks is that they have the function of defining potentially related image regions, thereby making them suitable for determining similarity by region segmentation according to embodiments of the present invention.

[0063] According to one embodiment, the method further includes the steps of receiving feedback from the user regarding similarity information via a user interface and adjusting the trained function, thereby enabling continuous learning of the trained function during use.

[0064] According to one implementation, the second time step precedes the first time step. In other words, the first histopathological image data relates to data from follow-up examinations. Thus, the user can, for example, define a region of interest in the first histopathological image data, and for this purpose, automatically search for similar regions in the second histopathological image data and provide them to the user. Based on this, the user can then, for example, determine whether the pathological changes in their marked region of interest are a recurrence of pathological changes already visible in the second histopathological image data.

[0065] According to one embodiment, the step of providing second histopathological image data includes accessing a database for histopathological image data and selecting second histopathological image data from histopathological image data stored in the database based on first histopathological image data and / or metadata associated with the first histopathological image data. Alternatively or additionally, according to an embodiment, selection may be based on metadata associated with the second histopathological image data.

[0066] According to one implementation, metadata may have:

[0067] - Patient identification markers used to identify patients

[0068] -Information regarding the following anatomical target regions of the patient: the tissue sample from which the first histopathological image data was extracted, from which the anatomical target regions are based.

[0069] -Information regarding the following anatomical target regions of the patient: the tissue sample from which the second histopathological image data has been extracted;

[0070] -Information regarding one or more histopathological stains used when providing the first histopathological image data.

[0071] -Information regarding one or more histopathological stains used when providing second histopathological image data.

[0072] - Suspected diagnosis or identification based on first histopathological image data

[0073] - Diagnosis or identification based on second histopathological image data

[0074] -Information regarding the second moment,

[0075] and / or

[0076] - Information about the first moment.

[0077] Therefore, not only can second histopathological image data be automatically found, but also particularly suitable second histopathological image data can be provided. This further reduces the burden on the user. Here, it is also possible to provide a comparison based on metadata associated with the second histopathological image data, and especially based on metadata associated with the first histopathological image data and metadata associated with the second histopathological image data. Here, suspected diagnosis or suspected identification can be input by the user through the user interface.

[0078] According to one embodiment, the step of providing first histopathological image data includes a user selecting the first histopathological image data via a user interface. Thus, the user can selectively choose the first histopathological image data they wish to process.

[0079] According to one implementation, the first time step precedes the second time step. In other words, the second histopathological image data relates to data from follow-up examinations. Therefore, for example, the user does not need to first define a region of interest in the histopathological image data, but instead also uses—in this embodiment, the "older"—known regions of interest in the first histopathological image data. Then, based on this, similar regions are automatically searched in the second histopathological image data and provided.

[0080] According to one embodiment, the step of providing first histopathological image data includes accessing a database for histopathological image data and selecting the first histopathological image data from the histopathological image data stored in the database based on second histopathological image data and / or metadata associated with the second histopathological image data.

[0081] According to one implementation, metadata may have:

[0082] -Information regarding the following anatomical target regions of the patient: the tissue sample from which the first histopathological image data was extracted,

[0083] -Information regarding the following anatomical target regions of the patient: the tissue sample on which the second histopathological image data was extracted from said anatomical target regions.

[0084] -Information regarding one or more histopathological stains used when providing the first histopathological image data.

[0085] -Information regarding one or more histopathological stains used when providing second histopathological image data.

[0086] - Suspected diagnosis or identification based on second histopathological image data

[0087] - Diagnosis or identification based on first histopathological image data

[0088] -Information regarding the second moment.

[0089] and / or

[0090] - Information about the first moment.

[0091] Therefore, not only can first histopathological image data be automatically found, but also particularly suitable first histopathological image data can be provided. This further reduces the burden on the user. Here, it is also provided that the search can be based on metadata associated with the first histopathological image data, and especially on a comparison of metadata associated with the first histopathological image data with metadata associated with the second histopathological image data. By considering suspected diagnoses or suspected identifications, first histopathological image data can be found in a targeted manner. For example, second histopathological image data with similar diagnoses or similar identifications can be searched in a targeted manner. According to an embodiment of the invention, a suspected diagnosis or suspected identification can be input by the user through a user interface.

[0092] According to one embodiment, the step of providing second histopathological image data includes a user selecting the second histopathological image data via a user interface. Thus, the user can selectively choose the histopathological image data they wish to process.

[0093] According to one embodiment, the tissue samples on which the first histopathological image data is based and the tissue samples on which the second histopathological image data is based are extracted from the same or at least one similar anatomical target region of the patient. The same anatomical target region may, for example, mean that the tissue samples were extracted from the same organ, the same anatomical structure, or the same tissue region of the patient. The same anatomical target region may also mean that the corresponding extraction sites of the patient's tissue samples have approximately the same coordinates.

[0094] According to another embodiment, a computer-implemented method is provided for providing similarity information about different histopathological image data of a patient. The method has multiple steps. One step involves providing first histopathological image data based on tissue samples extracted from the patient at a first time. Another step involves providing second histopathological image data based on tissue samples extracted from the patient at a second time, different from the first time. Another step involves identifying regions of interest in the first histopathological image data. Another step involves searching for similar regions in the second histopathological image data, said similar regions being similar to the regions of interest. Here, the searching step involves applying an image processing algorithm to the second histopathological image data. Another step involves determining similarity information based on the searching step. Another step involves providing the similarity information.

[0095] According to one embodiment, a system is provided for providing similarity information about different histopathological image data of a patient. The system has an interface and a control device. The interface is configured to receive first histopathological image data and second histopathological image data, wherein the first histopathological image data is based on a tissue sample extracted from the patient at a first time, and the second histopathological image data is based on a tissue sample extracted from the patient at a second time, different from the first time. A computing unit is configured to determine similarity information based on the first and second histopathological image data using an image processing algorithm, the similarity information having a description of the similarity between at least one region from the first histopathological image data indicating pathological identification and at least one region from the second histopathological image data indicating pathological identification. Furthermore, the computing unit is configured to provide the similarity information.

[0096] The control unit can be configured as a centralized or distributed computing unit. The computing unit can have one or more processors. The processors can be configured as a central processing unit (CPU) and / or a graphics processing unit (GPU). Alternatively, the control unit can be implemented as a local or cloud-based processing server.

[0097] Interfaces typically serve as a means of data exchange between control devices and other components. An interface can be implemented as one or more individual data interfaces, which may have hardware and / or software interfaces, such as a PCI bus, USB interface, Fire-Wire interface, ZigBee interface, or Bluetooth interface. The interface may also have an interface to a communication network, which can be a local area network (LAN), such as an intranet or a wide area network (WAN). Correspondingly, one or more data interfaces may have a LAN interface or a wireless LAN interface (WLAN or Wi-Fi).

[0098] The advantages of the proposed device substantially correspond to the advantages of the proposed method. The features, advantages, or alternative embodiments can also be adapted to other claimed subjects, and vice versa.

[0099] According to one embodiment, the system further includes a database for storing multiple histopathological image data and a user interface for user interaction. The interface is data-connected to both the database and the user interface. Furthermore, the control device is configured to select first histopathological image data from the database based on manual user input in the user interface, and to receive the first histopathological image data via the interface. The control device is also configured to select second histopathological image data from the database based on the first histopathological image data and / or based on metadata associated with the first histopathological image data.

[0100] In another aspect, the present invention relates to a computer program product comprising a program that can be directly loaded into the memory of a programmable control device, and the computer program product having program structure, such as libraries and auxiliary functions, so that when the computer program product is executed, a method for providing similarity information, particularly according to the embodiments described above, is performed.

[0101] Furthermore, in another aspect, the present invention relates to a computer-readable storage medium on which readable and executable program segments are stored, so that when the program segments are executed by a control device, all steps of the method for providing similarity information according to the above embodiments are performed.

[0102] Here, the computer program product may include: software having source code, which must also be compiled and linked or only needs to be interpreted; or executable software code that, in order to execute, must also be loaded into a processing unit. The computer program product allows the method to be executed quickly, repeatedly, and robustly. The computer program product is configured such that it can execute the method steps according to the invention by means of a computing unit. Here, the computing unit must have prerequisites, such as, for example, a corresponding working memory, a corresponding processor, a corresponding graphics card, or a corresponding logic unit, to enable efficient execution of the corresponding method steps.

[0103] The computer program product is stored, for example, on a computer-readable storage medium or stored on a network or server, from which it can be loaded into the processor of a corresponding computing unit, which can be directly connected to or constitute part of the computing unit. Furthermore, control information of the computer program product can be stored on a computer-readable storage medium. The control information of the computer-readable storage medium can be configured such that when a data carrier is used in the computing unit, the control information executes the method according to the invention. Examples of computer-readable storage media are DVDs, magnetic tapes, or USB flash drives, on which electronically readable control information, especially software, is stored. When the control information is read from the data carrier and stored in the computing unit, all embodiments of the above-described method according to the invention can be executed. Therefore, the invention can also be based on the computer-readable medium and / or the computer-readable storage medium. The advantages of the proposed computer program product or associated computer-readable medium substantially correspond to the advantages of the proposed method. Attached Figure Description

[0104] Other features and advantages of the invention will become clear from the following description of embodiments with reference to the schematic diagrams. Modifications mentioned in the context can be combined with each other to form new embodiments. In different figures, the same reference numerals are used for the same features.

[0105] The attached diagram shows:

[0106] Figure 1 A schematic diagram illustrating one implementation of a system for providing similarity information based on histopathological image data is shown.

[0107] Figure 2 The flowchart illustrates a method for providing similarity information based on histopathological image data according to one embodiment.

[0108] Figure 3 A flowchart illustrating a method for providing similarity information based on histopathological image data according to another embodiment is shown.

[0109] Figure 4 This diagram illustrates a region of interest extracted from histopathological image data according to one embodiment.

[0110] Figure 5 This diagram illustrates a region of interest extracted from histopathological image data according to another embodiment.

[0111] Figure 6 This diagram illustrates similar regions identified in histopathological image data according to one embodiment.

[0112] Figure 7 A flowchart illustrating a method for providing similarity information based on histopathological image data according to another embodiment is shown.

[0113] Figure 8 A flowchart illustrating a method for determining similar regions in histopathological image data according to one embodiment is shown.

[0114] Figure 9 The diagram illustrates one implementation of an image processing algorithm configured to provide similarity information based on histopathological image data.

[0115] Figure 10 The diagram illustrates another embodiment of an image processing algorithm configured to provide similarity information based on histopathological image data.

[0116] Figure 11 A schematic diagram illustrating another embodiment of an image processing algorithm is shown, the image processing algorithm being configured to provide similarity information based on histopathological image data, and

[0117] Figure 12 A flowchart is shown for a method for providing similarity information based on histopathological image data according to another embodiment. Detailed Implementation

[0118] exist Figure 1 The diagram illustrates a system 1 for providing similarity information (AEI) based on histopathological image data HIS1 and HIS2, according to one embodiment. System 1 has a user interface 10, a computing unit 20, an interface 30, and a storage unit 60. The computing unit 20 is essentially configured to calculate and provide similarity information (AEI) based on the histopathological image data HIS1 and HIS2. The histopathological image data HIS1 and HIS2 can be provided to the computing unit 20 from the storage unit 60 via the interface 30.

[0119] Storage unit 60 can be configured as a centralized or distributed database. Storage unit 60 can be, in particular, part of a server system. Storage unit 60 can be, in particular, part of a medical information system, such as a hospital information system (HIS) and / or a PACS system (PACS stands for picture archiving and communication system) and / or a laboratory information system (LIS).

[0120] Histopathological image data HIS1 and HIS2 are archived in storage unit 60. HIS1 and HIS2 are image data based on tissue samples extracted from the patient at a specific time from an anatomical target region or extraction region. The anatomical extraction region may be, for example, an organ or tissue region identified using imaging modalities such as MR or CT equipment. The tissue samples are used, for example, in biopsies or as preparation for surgery. The tissue is extracted from the patient during a surgical procedure, or excision. Thin sections of tissue, about a few micrometers thick, are produced from the tissue sample. Typically, multiple areas (so-called punched sections or blocks) are punched out of the tissue sample using a punching cylinder, and these areas are then sliced ​​into thin layers. The resulting tissue sections can then be fixed, prepared, and worked using various techniques before being finally stained with histopathological stains. Histopathological stains are used, on the one hand, to increase the contrast of the tissue or cellular structures contained in the sections. On the other hand, histopathological stains can be used selectively to highlight specific features, thereby addressing specific pathological problems. Several different histopathological stains have been developed over the past 120 years. Hematoxylin and eosin (H&E) staining is typically used first as a routine and overview stain. Other histopathological stains, often referred to as special stains, include Congo red, trichrome staining, or Auramin O. Furthermore, immunohistochemical staining can be used, by which proteins or other structures can become visible with the aid of labeled antibodies. Examples of this include Ki67 as a marker of cell proliferation, Her2 as a specific marker for breast cancer, CD8 as a marker for T cells, and PD-L1 as a prognostic marker for successful immunotherapy. In modern laboratories, computer-controlled automated staining machines typically use at least the most common staining agents. Generally, the first tissue section of a block is stained with H&E staining. When needed and depending on the problem, after identifying the H&E-stained tissue sections, other tissue sections from the corresponding block are stained and analyzed using specialized staining agents.

[0121] Today, prepared and stained tissue sections are typically digitized for identification. This is done using a specialized scanner, known as a slide scanner. The image recorded on this slide is also called a "whole slide image." The image data recorded here is usually two-dimensional pixel data, where each pixel is associated with a color value.

[0122] Because multiple cut sections are typically extracted from tissue samples and processed with the same histopathological staining agent, histopathological image data HIS1, HIS2 usually consist of multiple individual images (multiple individual "whole slide images" or multiple individual pixel images).

[0123] Additionally, the histopathological image data HIS1 and HIS2 may have metadata, in which additional information about the corresponding histopathological image data HIS1 and HIS2 may be stored. For example, the metadata may include one or more of the following: the time when the tissue sample on which the corresponding histopathological image data HIS1 and HIS2 are based was extracted from the patient; an electronic identifier identifying the patient, such as, for example, patient ID or name; a description of the histopathological staining agent used for the corresponding histopathological image data HIS1 and HIS2; a description of previous identifications of the histopathological image data HIS1 and HIS2; a marker identifying the user being identified (e.g., name or user ID); a description of the regions in the corresponding histopathological image data HIS1 and HIS2 that indicate pathological identification, such as, for example, region of interest (IB); and / or a description of the anatomically extracted regions from which the corresponding histopathological image data HIS1 and HIS2 are based. Metadata can be stored, for example, in the header of the histopathological image data HIS1, HIS2, or in a separate data container of the histopathological image data HIS1, HIS2, from the actual image data. Alternatively or supplementarily, this metadata can also be stored in the patient's electronic medical record (EMR), i.e., separately from the histopathological image data HIS1, HIS2. This EMR can be archived, for example, in storage device 60 or in a separate storage device, and the computing unit 20 can be connected to the storage device via interface 30.

[0124] User interface 10 has a display unit 11 and an input unit 12. User interface 10 can be configured as a portable computer system, such as, for example, a smartphone, tablet, or laptop. Alternatively, user interface 10 can be configured as a desktop PC. Input unit 12 can be integrated into display unit 11, for example, as a touch-sensitive screen. Alternatively or additionally, input unit 12 can have a keyboard or computer mouse and / or digital pen. Display unit 11 is configured to display single or multiple images from histopathological image data HIS1, HIS2 (hereinafter, the single image shown is also referred to as a "reference image RB"), the obtained similarity information AEI, or auxiliary images AB, which illustrate the similarity information AEI to the user. User interface 10 is also configured to obtain input from the user regarding the region of interest IB related to identification. Here, the user can be a female or male doctor, and especially a female or male pathologist.

[0125] User interface 10 has one or more processors 13 that execute software for manipulating display unit 11 and input unit 12 to provide a graphical user interface that allows a user to select histopathological image data HIS1, HIS2 for identification, input a region of interest (IB), and evaluate the identified similarity information (AEI). The user can activate the software, for example, by downloading it from an app store. Alternatively, the software can be a client-server computer program in the form of a web application running in a browser.

[0126] Interface 30 may have one or more individual data interfaces that ensure data exchange between components 10, 20, and 60 of system 1. The one or more data interfaces may be part of user interface 10, computing unit 20, and / or memory unit 60. The one or more data interfaces may have hardware and / or software interfaces, such as a PCI bus, USB interface, FireWire interface, ZigBee, or Bluetooth interface. The one or more data interfaces may have an interface to a communication network, which may be a local area network (LAN), such as an intranet or wide area network (WAN). Correspondingly, the one or more data interfaces may have a LAN interface or a wireless LAN interface (WLAN or Wi-Fi).

[0127] The computing unit 20 may have a processor. The processor may include a central computing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an image processing processor, integrated (digital or analog) circuitry, or a combination of the above components, and other means for processing histopathological image data HIS1, HIS2 according to embodiments of the present invention. The computing unit 20 may be implemented as a single component or as multiple components operating in parallel or in series. Alternatively, the computing unit 20 may have a real or virtual computer group, such as a cluster or cloud. Such a system may be referred to as a server system. According to embodiments, the computing unit 20 may be configured as a local server or a cloud server. Furthermore, the computing unit 20 may have working memory, such as RAM, for example, to temporarily store the histopathological image data HIS1, HIS2. Alternatively, this working memory may also be located in the user interface 10. The computing unit 20 is configured, for example by computer-readable instructions, by design and / or hardware configuration, such that the computing unit can perform one or more method steps according to embodiments of the present invention. The computing unit 20 may in particular be configured to perform one or more image processing algorithms TF-A, TF-B, TF-C, TF-A', which are further described below.

[0128] The computing unit 20 may have sub-units or modules 21-24, which are configured to provide similarity information (AEI) to the user during continuous human-computer interaction and thereby assist the user in the identification process.

[0129] Module 21 is configured to provide histopathological image data (either histopathological image data represented as HIS1 or HIS2, depending on the time the tissue sample was extracted) to be newly identified. For example, module 21 may be configured to receive such histopathological image data HIS1 or HIS2 from storage unit 60 and load it into computing unit 20 or user interface 10. This may occur, for example, according to instructions input by the user via user interface 10 or be automatically triggered. Furthermore, module 21 may be configured to display, according to instructions, individual images of the histopathological image data HIS1 or HIS2 to be identified as reference images RB to the user via user interface 10. Module 21 may also be configured to determine the region of interest (IB) within the histopathological image data HIS1 or HIS2, which may indicate pathological identification. For this purpose, module 21 may, for example, receive corresponding user input from user interface 10, evaluate annotations present in the histopathological image data HIS1 or HIS2 induced by the user, and / or automatically determine the region of interest (IB). Regarding the last mentioned alternative, module 21 can be configured to apply a suitable image processing algorithm to the histopathological image data HIS1 or HIS2 (e.g., the second image processing algorithm TF-B, which is further described below).

[0130] Module 22 is configured to provide histopathological image data, which can be compared with the histopathological image data to be identified in order to draw conclusions about the progression of the patient's tumor disease (either histopathological image data represented as HIS1 or HIS2, depending on the time of tissue sample extraction). For this purpose, module 22 can be specifically configured to search for the patient's histopathological image data HIS1 or HIS2 from past examinations, which preferably shows tissue from the same or at least one similar anatomically extracted region of the patient. Module 22 can be configured to formulate a suitable search request and, for example, search the storage device 60. Similar to module 21, module 22 can also be configured to identify the region of interest IB in the comparing histopathological image data HIS1 or HIS2—for example, by evaluating previously identified existing annotations or by applying a suitable image processing algorithm.

[0131] Module 23 is configured to determine, in particular morphological and / or structural and / or texture-related similarities between tumor tissues or tumor cells displayed in the histopathological image data HIS1 or HIS2 to be identified and / or from previous examinations of histopathological image data HIS1 or HIS2. For example, such similarity may indicate whether a tumor visible in the newly identified histopathological image data HIS1 or HIS2 is a recurrence of a tumor already mapped in the older histopathological image data HIS1 or HIS2, or whether it involves a newly generated tumor. Module 23 may be configured to provide the identified similarity as similarity information AEI. Module 23 may be particularly configured to apply suitable image processing algorithms to the histopathological image data HIS1, HIS2 to obtain the similarity information AEI. For example, module 23 may be configured to apply one or more of the image processing algorithms TF-A, TF-C, or TF-A' described further below.

[0132] Module 24 can be understood as a visualization module designed to, for example, display the results of the similarity analysis of module 23 to a user via user interface 10. To this end, module 24 may be configured to provide the user with one or more auxiliary images AB based on histopathological image data HIS1, HIS2, in which similarity regions AEB are highlighted graphically and / or by color and / or other means. Here, as mentioned, the similarity region AEB is a region in histopathological image data HIS1 or HIS2 showing large similarities in the structure of tumor tissue / tumor cells from a previous tissue sample to a new tissue sample to be identified. Additionally, module 24 may be configured to archive the results of the similarity analysis of module 23 (e.g., in storage unit 60 or any other storage unit) or provide them to another module or software for further processing.

[0133] The division of computing unit 20 into elements 21-24 here is only for the purpose of more simply explaining how computing unit 20 operates and should not be construed as limiting. Elements 21-24, or their functions, may also be combined in a single element. Here, elements 21-24 may also be understood in particular as computer program products or computer program segments that, when executed in computing unit 20, implement one or more of the method steps described below.

[0134] The computing unit 20 and the processor 13 can together form the control device 40. It should be noted that the illustrated layout of the control device 40, i.e., the described division into the computing unit 20 and the processor 13, is also understood as illustrative only. Therefore, the computing unit 20 can be fully integrated into the processor 13, and vice versa. Method steps can be performed, in particular, by executing a corresponding computer program product (e.g., software installed on the user interface) entirely on the processor 13 of the user interface 10, which then interacts directly with, for example, a storage unit via interface 30. In other words, the computing unit 20 is then identical to the processor 13.

[0135] As already mentioned, according to some implementations, computing unit 20 can alternatively be understood as a server system, such as a local server or a cloud server. In this design, user interface 10 can be referred to as the "frontend" or "client," while computing unit 20 can be understood as the "backend." Communication between user interface 10 and computing unit 20 can then be performed, for example, based on the HTTPS protocol. In this system, computing power can be distributed between the client and the server. In a "thin client" system, the server has most of the computing power, while in a "thick client" system, the client provides more computing power. A similar approach applies to data (here: especially histopathological image data HIS1, HIS2). In a "thin client" system, data is typically stored on the server and only the results are transmitted to the client, while in a "thick client" system, data is also transmitted to the client.

[0136] According to another embodiment, the described functionality can also be provided as a so-called cloud service. The corresponding computing units then constitute a cloud platform. The data to be analyzed, i.e., histopathological image data, can then be uploaded to the cloud platform.

[0137] Figure 2 The diagram illustrates a schematic flowchart of a method for providing similarity information AEI using histopathological image data HIS and HIS2 from tissue samples extracted from patients at time-staggered intervals. The order of the method steps is neither limited by the shown order nor by the selected numbering. Therefore, the order of the steps can optionally be changed, and individual steps can be omitted.

[0138] exist Figure 2In this process, the similarity information AEI is generated based on a comparison of first histopathological image data HIS1 and second histopathological image data HIS2. Here, the first histopathological image data HIS1 is based on tissue samples extracted from the patient at a first time. The second histopathological image data HIS2 is based on tissue samples extracted from the same patient at a second time, different from the first time. Here, the tissue samples are extracted from the same or at least one similar anatomical target region of the patient. Figure 2 This illustrates a general case based only on the different first and second time points. Here, the first histopathological image data HIS1 can be based on a tissue sample extracted from the patient prior to the tissue sample on which the second histopathological image data HIS2 is based—or vice versa. Then, as shown below... Figure 3 and Figure 6 The document further specifies the two options mentioned above.

[0139] exist Figure 2 In general, the first step S10 involves providing first histopathological image data HIS1. Here, providing can be achieved by retrieving the first histopathological image data HIS1 from the storage unit 60 and / or loading the first histopathological image data HIS1 into the computing unit 20.

[0140] The second step S20 involves providing the second histopathological image data HIS2. This can also be achieved by retrieving the second histopathological image data HIS2 from the storage unit 60 and / or loading the second histopathological image data HIS2 into the computing unit 20.

[0141] In the next step S30, similarity information AEI is generated. For this purpose, the first histopathological image data HIS1 and the second histopathological image data HIS2 are input into the first image processing algorithm TF-A. The first image processing algorithm "matches" similar regions in the first histopathological image data HIS1 and the second histopathological image data HIS2 based on a defined similarity metric. Here, the similarity metric can be, in particular, a measure of the morphological and structural similarity of regions. Here, similar regions can be, in particular, regions indicating pathological identification. For example, such regions indicating pathological identification can be regions in the first histopathological image data HIS1 and the second histopathological image data HIS2 that have tumor tissue or tumor cells or general pathological tissue changes. By establishing similarity relationships between similar regions and tissue samples recorded at different times, it is possible to deduce whether the regions indicating pathological identification in the first histopathological image data HIS1 and the second histopathological image data HIS2 are related to each other. For example, if regions similar to one or more regions in the second histopathological image data HIS2 are identified in the first histopathological image data HIS1, then there are, for example, morphologically similar pathological tissue changes. This indicates that the pathological changes in the first histopathological image data HIS1 represent a recurrence of the pathological changes in the second histopathological image data HIS2, or vice versa (depending on which tissue sample in the tissue sample upon which the corresponding histopathological image data HIS1 or HIS2 is based was previously extracted).

[0142] Correspondingly, the similarity information AEI may include information regarding whether a recurrence relationship exists between the first histopathological image data HIS1 and the second histopathological image data HIS2. Furthermore, the similarity information AEI may include location information regarding similar regions in the first histopathological image data HIS1 and / or the second histopathological image data HIS2 that indicate pathological identification; this location information may, for example, include bounded boxes and / or coordinates of the similar regions. Additionally, the similarity information AEI may include a description of the degree of similarity between the similar regions in the first histopathological image data HIS1 and the second histopathological image data HIS2. Furthermore, the similarity information AEI may have one or more auxiliary images AB, which may be based on the first histopathological image data HIS1 and / or the second histopathological image data HIS2, and in these auxiliary images, for example, similar regions indicating pathological identification may be highlighted.

[0143] In another step S40, the similarity information AEI is finally provided. "Provided" generally means that the similarity information AEI is provided for use. For example, the similarity information AEI may be displayed to the user via user interface 10. Alternatively or additionally, the similarity information AEI may be stored in storage unit 60 or input into another algorithm for further processing.

[0144] In an optional step S50, a medical report or identification report is finally automatically created based on the similarity information AEI. This may include pre-filling a suitable template using the similarity information AEI and providing it to the user via the user interface 10 for access and further processing.

[0145] To further illustrate, Figure 3 In the embodiments shown, we now discuss the case where the first histopathological image data HIS1 is based on a tissue sample extracted later than the second histopathological image data HIS2. Therefore, in other words, the second time step is prior to the first time step, and the first histopathological image data HIS1 can be considered a “follow-up” of the second histopathological image data HIS2. The order of the method steps is neither limited by the shown order nor by the chosen numbering. Therefore, the order of the steps can be changed if necessary, and individual steps can be omitted.

[0146] The first step S10' involves providing first histopathological image data HIS1. The first histopathological image data HIS1 can be selected by a user in the user interface 10, for example. The computing unit 20 can then retrieve the first histopathological image data HIS1 from the storage unit 60 and, for example, load it into the working memory of the computing unit 20 or another storage device.

[0147] Then, in the next step S15', the region of interest (ROI) IB in the first histopathological image data HIS1 is determined. The ROI IB is specifically a region in the first histopathological image data HIS1 (or a local area from the first histopathological image data) that displays morphology relevant to pathological identification. In other words, the ROI can be understood as a region (or local area) in the first histopathological image data HIS1 that indicates pathological identification. Here, the ROI IB may have one or more individual regions (or local areas) ROI, ROI1, ROI2, etc. (see...) Figure 4 and Figure 5 Furthermore, depending on some implementation schemes, the region of interest (IB) can also include all first histopathological image data (HIS1). The region of interest (IB) can be determined automatically (step S15A) or manually (step S15B).

[0148] To automatically determine the region of interest (IB) in step S15A, the first histopathological image data HIS1 can be input into the second image processing algorithm TF-B. This second image processing algorithm is configured to identify regions in the histopathological image data HIS1 and HIS2 that are relevant to pathological identification, specifically regions showing changes in pathological tissue. As further explained below, the second image processing algorithm TF-B may in particular have a trained function.

[0149] To manually determine the region of interest (IB) in step S15B, system 1 can be configured such that a user can select one or more reference images (RB) from the first histopathological image data HIS1 and evaluate the one or more reference images in the user interface 10. Furthermore, system 1 and, in particular, the user interface 10 can be configured such that a user can identify the region of interest (IB) by means of annotation, which can be entered by the user through the user interface 10 using user input. For example, system 1 can be configured such that a user can annotate the region of interest (IB) by clicking with a mouse or by drawing a border in one or more reference images (RB) to mark the area associated with the region of interest.

[0150] Furthermore, a semi-automatic determination of the region of interest (IB) is also conceivable, where a second image processing algorithm, TF-B, automatically identifies potentially relevant regions and displays them to the user via user interface 10 for selection. The region confirmed by the user then replaces the IB.

[0151] Then, the next step S20' involves providing second histopathological image data HIS2, which is based on a tissue sample extracted from the patient prior to the extraction of the tissue sample used for the first histopathological image data HIS1. For this purpose, the computing unit 20 can be configured to search for suitable second histopathological image data HIS2 in the storage unit 60 and load the second histopathological image data. According to some embodiments, the second histopathological image data HIS2 found in this way can be displayed to the user for selection. Therefore, the user can select from a pre-selection list the second histopathological image data HIS2 that is most suitable from the user's perspective. According to embodiments of the invention, the considered second histopathological image data HIS2 can be displayed to the user in a timeline format in a graphical user interface for selection, providing the user with a quick and comprehensive overview. Here, various points on the timeline can indicate the available second histopathological image data HIS2. Alternatively, the computing unit 20 can be configured to autonomously select suitable second histopathological image data HIS2. Alternatively, system 1 and especially user interface 10 can be configured to allow the user to select the second histopathological image data HIS2 entirely by himself, for example by independently searching the histopathological image data stored in storage unit 60.

[0152] To suit the acquisition of similarity information (AEI) according to embodiments of the present invention, the second histopathological image data HIS2 should be associated with at least the same patient as the first histopathological image data HIS1. Furthermore, if the second histopathological image data HIS2 is also based on a tissue sample extracted from the same or at least similar anatomical target region from which the first histopathological image data HIS1 was generated, the probability of identifying regions similar to the region of interest (IB) and thus providing a convincing similarity information AEI increases. Additionally, better results can be obtained if the same staining agent is used for both the first and second histopathological image data HIS1. Additionally, a suitable time interval between the first and second moments may be relevant. When providing the second histopathological image data HIS2, this information can be considered as metadata. This metadata can, for example, be automatically extracted from the first histopathological image data HIS1, automatically retrieved from a medical information system, and / or provided by the user. For example, information relating to the patient, anatomical target region, or staining agent can be stored in the header of the first histopathological image data HIS1 and extracted from there. Alternatively, the information can be obtained by retrieving electronic patient records stored in a medical information system. Additionally, users can query metadata, for example, by providing a corresponding input mask via user interface 10. Furthermore, a suitable time window can be pre-allocated for the time interval between the first and second moments.

[0153] According to some implementations, it may also be proposed that, when providing the second histopathological image data HIS2, the region of interest IB identified in step S15 and / or information derived therefrom (e.g., feature signatures – see below) be considered. This is particularly meaningful when such a region of interest IB from previous findings / analysis has also been annotated in the second histopathological image data HIS2.

[0154] The next step, S30', involves obtaining similarity information AEI. To this end, sub-step S30A' proposes searching for a similarity region AEB of the second histopathological image data HIS2. For this purpose, the second histopathological image data HIS2 can be input into a third image processing algorithm TF-C. Optionally, the first histopathological image data HIS1 and / or the region of interest IB can be additionally input into the third image processing algorithm TF-C. The third image processing algorithm TF-C is typically configured to search for regions in the histopathological image data that are similar to predefined image data. In this case, the predefined image data is given by the region of interest IB. To determine whether similarity exists between the image data, the third image processing algorithm TF-C can also be configured to apply a defined similarity metric, as still described below.

[0155] In other words, the similarity region AEB is the region in the second histopathological image data HIS2 that indicates pathological identification. The similarity region AEB can have morphological similarity to the region of interest IB. Furthermore, the similarity region AEB can have a similar pattern or structure to the region of interest IB. Additionally, the similarity region AEB can have a similar feature signature to the region of interest IB. For example, a feature signature can be understood as a set or vector of abstract features that can be extracted from image data and / or metadata. To search for similarity regions, the second histopathological image data HIS2 can be "scanned," and the image data of the second histopathological image data HIS2 can be progressively compared with the image data of the region of interest IB. The similarity region AEB can, in particular, have a similarity to the region of interest IB that is higher than a predetermined similarity level or similarity measure. The similarity level or similarity measure can indicate the degree of consistency between different image data. The similarity compared to a predetermined similarity measure can, for example, be the result of the aforementioned similarity measurement. For example, a predetermined similarity measure can be preset manually or automatically. It should be noted that if no region similar to the region of interest (IB) exists in the second histopathological image data (HIS2), searching for the similarity region (AEB) may also yield negative results.

[0156] Optionally, in step S30A', existing annotations in the second histopathological image data HIS2 may also be considered, which have indicated pathological identification from previous identifications (i.e., the "region of interest" in the second histopathological image data HIS2).

[0157] Then, based on the first sub-step S30A', the similarity information AEI is determined in the second sub-step S30B'. For example, the similarity information AEI may include descriptions of similar regions AEB, such as, for example, the location of the similar region in the second histopathological image data HIS2, the size of the similar region, a quantitative description of the similarity of the similar region, etc. If no similar region AEB is found in step S30B', this can be correspondingly indicated in the similarity information AEI. Furthermore, the similarity information AEI may include visualizations for the user. Here, the visualization may, for example, be based on the second histopathological image data HIS2. In particular, an auxiliary image AB may be generated based on the second histopathological image data HIS2, in which one or more similar regions AEB are highlighted (see...). Figure 6 Here, highlighting can be implemented, for example, by drawing borders and / or by using brightly colored markers for the similarity regions. Regarding brightly colored markers, it is also feasible to display the similarity of each similarity region AEB using color codes, where the color change is associated with a rank in the similarity values ​​calculated for each similarity region AEB. Alternatively or additionally, the similarity information AEI can contain conclusions about the probability that the region of interest IB and the similarity region AEB are in a recurrence relationship. This can indicate that the pathological tissue changes shown in the region of interest IB are a recurrence of the pathological tissue changes shown in the similarity region AEB, which can predict the recurrence of the disease. In addition to the similarity region AEB, the second histopathological image data HIS2 can also have other regions indicating pathological identification, for example, other regions that, while showing pathological tissue changes, are not similar to the regions indicating pathological identification in the first histopathological image data HIS1. These regions can, and preferably are, highlighted in the auxiliary image AB differently from the similarity region AEB—e.g., by highlighting them with a different color.

[0158] In step S40', the similarity information AEI is finally provided. Here, step S40' essentially corresponds to step S40, and the operations described in step S40 can also be performed in step S40'. In particular, the auxiliary image AB can be displayed to the user in step S40' by means of the user interface 10. Alternatively or additionally, the auxiliary image AB can be stored in the storage unit 60.

[0159] The optional step S50' basically corresponds to Figure 2Step S50. In particular, in step S50', the metadata and / or conclusions about the recurrence probability from the auxiliary image AB and / or from the first histopathological image data HIS1 and / or the second histopathological image data HIS2 can be automatically entered into the template or report of the medical appraisal report, and then it can be provided to the user via the user interface 10 or archived in the storage unit 60.

[0160] Optional step S60' is a repetitive step. Step S60' is arranged as follows: Based on the patient's medical history, multiple second histopathological image data HIS2 can be considered for comparison with the "current" first histopathological image data HIS1. As explained in conjunction with step S20', although the considered second histopathological image data HIS2 can be presented for selection, this is not mandatory depending on the implementation. Furthermore, the user may also select multiple second histopathological image data HIS2. In this case, multiple second histopathological image data HIS2 are considered for analysis in subsequent steps. Therefore, it can be proposed in optional step S60' that steps S20', S30', S40', and S50' are repeated for different second histopathological image data HIS2, each processed based on another second histopathological image data HIS2, until all considered second histopathological image data HIS2 have been processed.

[0161] exist Figure 7 Another embodiment of the method for providing similarity information AEI for histopathological image data HIS1, HIS2 is shown. The order of the method steps is neither limited by the shown order nor by the selected numbering. Therefore, the order of the steps can be changed if necessary, and individual steps can be omitted. Figure 3 The implementation shown differs from the one described above, in which the first moment precedes the second moment. In other words, the first histopathological image data HIS1 originates from a tissue sample extracted from the patient prior to the tissue sample from which the second histopathological image data HIS2 originates. Therefore, the second histopathological image data HIS2 is this time the result of a "follow-up" examination, while the first histopathological image data HIS1 is a preliminary examination (the so-called "priors"). Furthermore, as another difference, in the combination... Figure 7 The illustrated implementation assumes that in the first histopathological image data HIS1, relevant regions have been defined by previous findings, which indicate pathological identification and can represent the basis for the region of interest IB.

[0162] In the first step S20", the second histopathological image data HIS2 is first provided. Regarding the technical implementation, step S20" basically corresponds to... Figure 3 Step S10' in the above. Correspondingly, the individual steps, alternatives, explanations and effects described in step S10' can be similarly applied to step S20".

[0163] Then, in step S10", suitable reference data is searched from the patient's previous examinations using the first histopathological image data HIS1. Technically, step S10" essentially corresponds to... Figure 3 Step S20'. The individual steps, alternatives, explanations, and effects described in step S20' for providing the second histopathological image data HIS2 can be similarly applied to providing the first histopathological image data HIS1 according to step S10".

[0164] and Figure 3 The difference lies in step S15, where the region of interest (IB) is not defined in the "Follow-Up" histopathological image data, but rather in the "Priors" data. This is... Figure 7 The illustrated embodiment uses first histopathological image data HIS1. Here, the region of interest (IB) is again the area showing a pattern related to pathological identification. Here, the region of interest IB can also specifically include one or more individual regions ROI, ROI1, ROI2, etc. (see...) Figure 4 and Figure 5 In step S15”, the region of interest (IB) is preferably automatically determined. To automatically determine the IB, the first histopathological image data HIS1 can be input into the previously mentioned second image processing algorithm TF-B, which is configured to identify regions in the histopathological image data HIS1 and HIS2 that are relevant to pathological identification, specifically regions showing pathological tissue changes such as tumor tissue. Alternatively, annotations already existing in the first histopathological image data HIS1 can be automatically evaluated, such as annotations compiled by the user during previous identification processes. For example, these annotations can be saved as tags for the region of interest IB in the first histopathological image data HIS1 or its associated metadata.

[0165] The next step, S30", involves obtaining the similarity information AEI. To this end, sub-step S30A proposes searching for the similarity region AEB of the second histopathological image data HIS2. Therefore, as with... Figure 3The systematic difference in the illustrated implementation is that the similarity region AEB is not searched in the "Priors" HIS1, but rather in the "Follow-Up" data HIS2. Here, regarding the technical implementation, step S30A essentially corresponds to... Figure 3 Step S30A' in the text, and the individual steps, explanations, alternatives and effects described in step S30A' can be similarly transferred to step S30A.

[0166] Then, based on sub-step S30A", the similarity information AEI is determined in the second sub-step S30B. Here, step S30B essentially corresponds to Figure 3 Step S30B' in the text, and the individual steps, explanations, alternatives, and effects described in step S30B' can be similarly transferred to step S30B. In particular, in Figure 7 The illustrated implementation can also generate an auxiliary image AB based on the second histopathological image data HIS2. However, as a systematic difference, in Figure 7 In the embodiments shown, auxiliary image AI is preferably generated based on follow-up examinations.

[0167] Steps S40” and S50” basically correspond to Figure 3 Steps S40' and S50' in the text, and the individual steps, explanations, alternatives and effects described in steps S40' and S50' can be similarly transferred to steps S40' and S50'.

[0168] Optional step S60” is ultimately a repetition of step S60’ – the difference being that step S60” involves the first histopathological image data HIS1. Correspondingly, in optional step S60”, it is proposed that steps S10”, S15”, S30”, S40”, and S50” be repeated for different first histopathological image data HIS1, each proceeding based on another first histopathological image data HIS1, until all considered first histopathological image data HIS1 have been processed.

[0169] It should be noted that Figure 3 and Figure 7The two implementations shown can be combined with each other. Correspondingly, a region of interest (IB) can be defined not only in the first histopathological image data HIS1 but also in the second histopathological image data HIS2, and the similarity of the IIB can then be checked to determine similar regions AEB. Alternatively, the IIB can be omitted entirely, and the first and second histopathological image data HIS1 and HIS2 can, on their own and as a whole, indicate the pathologically identified regions that are similar in time between the first and second time points using a suitable image processing algorithm TF-A.

[0170] exist Figure 8 The diagram illustrates a method for identifying similar regions in histopathological image data HIS1 and HIS2. The order of the method steps is neither limited by the shown order nor by the selected numbering. Therefore, the order of the steps can be changed if necessary, and individual steps can be omitted. The third image processing algorithm TF-C can be specifically configured to achieve [the desired result]. Figure 8 One or more steps are described.

[0171] Figure 8 The implementation shown is based on the existence of a region of interest (IB). Therefore, Figure 8 The steps shown can, for example, follow step S15' or S15'. Based on this, the first step A10 involves extracting the feature signature f based on the region of interest IB. IB Feature signature f IB There can be multiple individual features, which are extracted from the region of interest (IB) and collectively characterize the region of interest (IB). Feature signature f IB It can have so-called feature vectors, in which various features are combined. If the region of interest IB consists of multiple individual regions ROI, ROI1, ROI2, ROI3, then the feature signature f IB The various features can be averaged across different regions. Features may include, for example, patterns, textures, and / or structures within the region of interest (IB). Furthermore, the feature signature f IB The features can include parameters that identify the density of (cell) density and / or histopathological markers in the region of interest (IB). Furthermore, the feature signature f IB One or more features may have parameters indicating color values, grayscale values, or contrast values ​​in the region of interest (IB). Additionally, the feature signature f IBOne or more features may relate to characteristics located outside the region of interest (IB). For example, this could be information about surrounding tissue or information extracted from metadata about the first histopathological image data (HIS1). Feature signatures f can be generated using a separate image processing algorithm. IB The region of interest (IB) is input into the image processing algorithm, and optionally, the first histopathological image data (HIS1) and possible metadata are also input. For this purpose, a so-called texture classification algorithm (see, for example, Hamilton et al., “Fast automated cell phenotype image classification”, BMC Bioinformatics, 8:110, 2007, DOI: 10.1186 / 1471-2105-8-110) or a trained function, such as a convolutional neural network (see below), can be used. The image processing algorithm mentioned above can in particular be implemented as a subroutine of a third image processing algorithm, TF-C.

[0172] In the next step A20, potential similarity regions AEB are identified in the second histopathological image data HIS2. This can be achieved, for example, by systematically scanning the second histopathological image data HIS2. Here, for example, a "moving window" can be moved across the second histopathological image data HIS2, or the second histopathological image data HIS2 can be divided into potential similarity regions AEB using a grid. As an alternative, potential similarity regions AEB can also be determined dynamically, i.e., with variable variables. Here, related regions can be identified based on consistent image values ​​such as grayscale, contrast, density, etc., within the regions. Furthermore, it is feasible to identify potential similarity regions by evaluating image edges (see Zitnick et al., "Edge Boxes: Locating Object Proposals from Edges", Computer Vision–ECCV, 2014, pp. 391–405). In addition, it is feasible to use a trained function, and in particular, a convolutional neural network. As another possibility, prioritization can be performed when identifying potential similarity regions (AEBs), and only regions in the second histopathological image data HIS2 that are considered as similarity regions (AEBs) with a certain probability can be identified as potential similarity regions (AEBs). For this purpose, for example, segmentation can be used to exclude less relevant regions of the second histopathological image data HIS2, such as necrotic tissue regions, from further analysis. Alternatively or additionally, an algorithm similar to the second image processing algorithm TF-B can be applied, which is configured to automatically identify relevant regions in histopathological image data.

[0173] In the next step A30, the feature signature f is extracted from the potentially similar region AEB. IB Feature signature f AEB Here, it can be done essentially as described in step A10.

[0174] In the next step A40, the feature signature f will be extracted from the possible similar regions AEB. AEB Feature signature f of region of interest IB IB A comparison is then made. Specifically, a similarity metric can be determined for each possible similar region AEB, the similarity metric representing the feature signature f extracted from the corresponding possible similar region AEB. AEB Feature signature f of region of interest IB IB A measure of similarity or consistency. For example, a similarity measure can be limited to the interval of feature signatures in a feature space. If feature signatures are understood as feature vectors, then a similarity measure can be limited to, for example, cosine similarity.

[0175] In step A50, similar regions AEB are selected from possible similar regions AEB based on comparison. Here, for example, all possible similar regions AEB whose associated similarity measure indicates a similarity to the region of interest IB above a predetermined threshold can be classified as similar regions AEB. The threshold can be specified automatically or manually.

[0176] Then, step S30B' or S30B" can, for example, follow step A50.

[0177] According to an embodiment of the present invention, Figure 2 , Figure 3 , Figure 7 and Figure 8 The method steps of the embodiments shown are performed by one or more image processing algorithms TF-A, TF-B, TF-C. Figure 9 , Figure 10 and Figure 11 An implementation of the image processing algorithm is shown. Figure 9 The image processing algorithm shown corresponds to the combination Figure 2 The first image processing algorithm, TF-A, is introduced. This algorithm takes first histopathological image data HIS1 and second histopathological image data HIS2 as input data, and outputs similarity regions AEB and / or similarity information AEI as output data. Figure 10 The image processing algorithm shown corresponds to the combination Figure 3 and Figure 7The third image processing algorithm mentioned is TF-C. This algorithm obtains the region of interest (IB) and the second histopathological image data (HIS2) as input data, and outputs the similarity region (AEB) and / or similarity information (AEI) as output data. Figure 11 The diagram illustrates a variant of the first image processing algorithm TF-A, TF-A'. The image processing algorithm TF-A' is characterized in that the second image processing algorithm TF-B and the third image processing algorithm TF-C are implemented as subroutines. As mentioned, the second image processing algorithm TF-B is configured to identify regions of interest (IB) in histopathological image data HIS1 and HIS2. Optionally, at least several parts of the second image processing algorithm TF-B can be implemented in the third image processing algorithm TF-C.

[0178] According to some implementations, image processing algorithms TF-A, TF-B, TF-C, and TF-A' have one or more trained functions. According to implementations, the trained functions may have a neural network. The neural network may have multiple successive layers. Each layer includes at least one, preferably multiple, nodes. Essentially, each node can perform a mathematical operation that associates one or more input values ​​with an output value. Nodes in each layer may be connected to all or only a subset of nodes in previous and / or subsequent layers. Two nodes are "connected" when their inputs and / or outputs are connected. An edge or connection is associated with a parameter commonly referred to as a "weight" or "edge weight". The input values ​​for nodes in a corresponding first layer may, for example, be pixel values ​​of first histopathological image data HIS1 or second histopathological image data HIS2 or a region of interest IB. The corresponding last layer is typically referred to as the output layer. The output values ​​of the nodes in the output layer may, according to the image processing algorithm, be, for example, pixel values ​​or coordinates of a region of interest IB or a similarity region AEB. Furthermore, the output values ​​of the output layer may be similarity information AEI. There are multiple hidden layers (the technical term is "hidden layers") between the input layer and the output layer.

[0179] According to some implementations, the trained function may in particular have a convolutional neural network (CNN) or a deep convolutional neural network. This trained function then has one or more convolutional layers and optionally one or more deconvolutional layers. Furthermore, the trained function may have pooling layers, upsampling layers, and fully connected layers. The convolutional layers convolve the input and pass the result of the input to the next layer by moving an image filter over the input. Convolutional layers can then prove particularly advantageous if similar image regions need to be searched, as in some implementations. Pooling layers reduce the dimensionality of the data by combining the outputs of groups of nodes in the layer to a single node in the next layer. Upsampling and deconvolutional layers reverse the actions of the convolutional and pooling layers. A fully connected layer connects each node in the previous layer to the nodes in the subsequent layer, so that each node essentially gets a “voice”.

[0180] According to some implementations, the trained function employs a so-called "region-based convolutional neural network" (R-CNN). A difficulty in combining this with searching for regions of interest (IBs) or similar regions (AEBs) may arise because these regions appear at different locations in the histopathological image data HIS1 and HIS2 and may have different sizes and shapes. While this problem can theoretically be solved by systematically "scanning" the histopathological image data HIS1 and HIS2 in conjunction with step A20, this is typically only addressed with significant computational and consequently time costs. Essentially, the region-based convolutional neural network first selects some regions from the image data to be analyzed (wherein the techniques described in step A20 can be used). The convolutional neural network then extracts feature signatures from the selected regions, based on which the selected regions can be classified using a classifier. Here, a so-called support vector machine (SVM) or other neural network layer is typically used as the classifier. For further disclosure on region-based convolutional neural networks, see, for example, Girshick et al., “Rich feature hierarchies for accurate object detection and semantic segmentation”, arXiv: 1311.2524.

[0181] Based on the aforementioned basic configuration, several improvements exist, which can also be implemented in the trained function according to embodiments of the present invention, and can also be uniformly referred to as region-based convolutional neural networks. This also includes the other improvements described below when referring to one or more trained functions having a region-based convolutional neural network. One such improvement is called a “fast” region-based convolutional neural network (or simply “fast R-CNN”). Here, feature signatures are calculated separately from the selected regions used for the entire image, and then “pooled” according to the selected regions. This eliminates the need for multiple calculations of feature signatures in cases of overlapping selected regions (see Girshick, “Fast R-CNN”, 2015 IEEE International Conference on Computer Vision (ICCV), DOI: 10.1109 / ICCV.2015.169). Another improvement is called a “faster” region-based convolutional neural network (or simply “faster R-CNN”). Here, the selective selection of regions is replaced by selection using (convolutional) neural networks (see Ren et al., "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", Advances in Neural Information Processing Systems, Vol. 28, 2015).

[0182] According to some embodiments, region-based convolutional neural networks can be implemented, in particular, in automatically determining the region of interest (IB) or searching for similar regions (AEB)—that is, in the second image processing algorithm TF-B and the third image processing algorithm TF-C—where searching for similar regions AEB is performed on the region of interest IB or its feature signature f. IB Classification is performed. Of course, the first image processing algorithm, TF-A, can also include region-based convolutional neural networks.

[0183] As an alternative to region-based convolutional neural networks, a "normal" convolutional neural network can also be trained to have the same functional scope as a region-based convolutional neural network (i.e., provide essentially the same output data). This solution is also known as the YOLO (you only look once) solution (see Redmon et al., "YouOnly Look Once: Unified, Real-Time Object Detection", arXiv: 1506.02640).

[0184] The trained function learns by adjusting the weights or weighted parameters (e.g., edge weights) of various layers and nodes. The trained function can be trained, for example, using supervised learning methods. Backpropagation can be used, for example. During training, the trained function is applied to the training input data to produce corresponding output values, the target values ​​of which are known in the form of the training output data. The difference between the output values ​​and the training output data can be used to introduce a cost function or loss function as a measure of how well or poorly the trained function performs its intended objective. The goal of training is to find the (local) minimum of the cost function by iteratively adjusting the parameters (e.g., edge weights) of the trained function. The trained function eventually provides acceptable results on a (sufficiently) large population of training input data. The optimization problem can be performed using stochastic gradient descent or other schemes known in the field.

[0185] for Figure 9The first image processing algorithm TF-A shown, if having a trained function, provides a training dataset comprising first training histopathological image data HIS1 and second training histopathological image data HIS2, and, according to the configuration of the first image processing algorithm TF-A, associated empirically verified similarity regions AEB or verified similarity information AEI. Here, the first training histopathological image data HIS1 and the second training histopathological image data HIS2 correspond to either the first histopathological image data HIS1 or the second histopathological image data HIS2. Therefore, the first and second training histopathological image data belong to the same patient and are based on tissue samples extracted from the patient at different times but from the same anatomical target region. Here, the verified similarity regions AEB or similarity information AEI can be based on user annotations made by the user based on the analysis or identification of the first and second training histopathological image data HIS1 and HIS2. Corresponding to the terminology used herein, the first histopathological image data HIS1 and the second histopathological image data HIS2 would be the training input data and target value, or the training output data would be the verified similarity region AEB or similarity information AEI. Training the first image processing algorithm TF-A may then include applying the image processing algorithm TF-A to the first training histopathological image data HIS1 and the second training histopathological image data HIS2 to produce output values, and includes comparing the output values ​​with the verified similarity region AEB or similarity information AEI. One or more parameters of the first image processing algorithm TF-A may then be adjusted based on the comparison.

[0186] For the second image processing algorithm TF-B, a suitable training dataset includes training histopathological image data HIS1 and HIS2, and a validated region of interest (IB). Since the purpose of the second image processing algorithm TF-B is to automatically identify regions relevant to pathological identification (i.e., regions indicating pathological identification), the validated IB can be obtained, in particular, through user annotations. The validated IB, for example, indicates tumor cells in the first histopathological image data HIS1. Training the second image processing algorithm TF-B then includes applying the second image processing algorithm TF-B to the training histopathological image data HIS1 and HIS2 to produce output values, and comparing the output values ​​with the validated IB. One or more parameters of the second image processing algorithm TF-B can then be adjusted based on the comparison.

[0187] for Figure 10The third image processing algorithm TF-C shown here has a suitable training dataset that includes a training region of interest (IB) and a second training histopathological image data (HIS2), and, depending on the configuration of the third image processing algorithm TF-C, an associated empirically verified similarity region (AEB) or verified similarity information (AEI). Here, the training region of interest can, in principle, be any region extracted from the histopathological image data. In particular, it can involve any region of interest (IB) from the second histopathological image data (HIS2). However, to better prepare the third image processing algorithm TF-C for the specific circumstances, it is preferable that the training region of interest (IB) indicates pathological identification. This training region of interest (IB) can be annotated, for example, by the user. It is also preferable that the training region of interest (IB) is extracted from the histopathological image data, such as the first histopathological image data (HIS1), which belongs to the same patient (and anatomical target region) as the second training histopathological image data (HIS2), but based on tissue samples extracted at different times. For the verified similarity regions (AEB) and verified similarity information (AEI), it can be performed as described above. However, training the third image processing algorithm TF-C may include applying TF-C to the training region of interest (IB) and the second training histopathological image data (HIS2) to produce an output value, and comparing the output value with a validated similarity region (AEB) or similarity information (AEI). One or more parameters of the third image processing algorithm TF-C can then be adjusted based on the comparison.

[0188] Furthermore, a variant scheme can be implemented based on the third image processing algorithm TF-C, which can identify similar regions AEB within the histopathological image datasets HIS1 and HIS2. In other words, users can, for example, preset regions of interest (IBs) in the histopathological image datasets HIS1 and HIS2, and the image processing algorithm TF-C automatically searches for all similar regions within the same histopathological image datasets HIS1 and HIS2. The corresponding method is... Figure 12 The method steps are shown in the diagram. The order of the steps is neither limited by the shown order nor by the selected numbering. Therefore, the order of the steps can be changed if necessary, and individual steps can be omitted.

[0189] The first step, M10, involves providing histopathological image data. For example, the histopathological image data may correspond to first histopathological image data, HIS1.

[0190] The second step, M20, involves providing the region of interest, IB. Here, step M20 can be designed as in step S15'.

[0191] The third step, M30, involves searching for similarity regions AEB in the histopathological image data HIS1 and HIS2, where each AEB is similar to one or more regions of interest IB. This search step specifically involves applying a (optionally modified) third image processing algorithm, TF-C, to the histopathological image data HIS1 and HIS2. Furthermore, step M30 can be designed as in step S30A'.

[0192] The fourth step, M40, involves providing similarity information (AEI) based on the similarity region (AEB). Here, step M40 can be designed similarly to step S30B'.

[0193] Step M50 involves providing similarity information AEI. Providing similarity information AEI may in particular include displaying or highlighting similar areas in histopathological image data HIS1, HIS2 for the user in the user interface 10.

[0194] If the third image processing algorithm has a trained function, then the third image processing algorithm TF-C can be adjusted based on the provided training dataset. Figure 12 The method involves a training dataset comprising a training region of interest (IB), training histopathological image data HIS1 and HIS2, and associated empirically validated similarity regions (AEB) within the training histopathological image data. Here, the validated similarity regions (AEB) can again be based on annotations made by the user during the analysis or identification of the histopathological image data. Training can then include inputting the training IIB and the training histopathological image data HIS1 and HIS2 into a third image processing algorithm TF-C to produce corresponding output values. These output values ​​are then compared with the validated similarity regions (AEB). The third image processing algorithm TF-C can then be adjusted based on the comparison.

[0195] Even if not explicitly stated, but meaningfully within the scope of the invention, various embodiments, sub-aspects of embodiments, or features may be combined or interchanged with each other without departing from the scope of the invention. Unless explicitly mentioned as adaptable, the advantages described with reference to the embodiments of the invention also apply to other embodiments.

[0196] The following points are also part of this disclosure.

[0197] 1. A computer-implemented method for providing similarity information (AEI) of different histopathological image data (HIS1, HIS2) of a patient, the method comprising the following steps:

[0198] - Provide (S10', S10") first histopathological image data (HIS1), the first histopathological image data (HIS1) being based on a tissue sample extracted from the patient at a first moment;

[0199] - Provide (S20', S20") second histopathological image data (HIS2), the second histopathological image data (HIS2) being based on tissue samples extracted from the patient at a second time different from the first time;

[0200] - Identify the region of interest (IB) (S15', S15”) in the first histopathological image data (HIS1);

[0201] - Search (S30A', S30A”) for similarity regions (AEB) in the second histopathological image data (HIS2), wherein the similarity regions (AEB) are similar to the region of interest (IB), wherein the search (S15', S15”) step includes applying image processing algorithms (TF-A, TF-B, TF-C, TF-A') to the second histopathological image data (HIS2);

[0202] - Determine similarity information (AEI) for (S30B', S30B') based on the steps of searching (S30A', S30A''); and

[0203] - Provide the similarity information (AEI) described in (S40', S40")

[0204] 2. According to the method in 1, where,

[0205] The region of interest (IB) has one or more individual regions (ROI, ROI1, ROI2, ROI3) defined in the first histopathological image data (HIS1), and the one or more individual regions (ROI, ROI1, ROI2, ROI3) specifically indicate pathological identification.

[0206] 3. Based on any one of the methods mentioned above, where,

[0207] The similarity information (AEI) includes:

[0208] - Explanation of the similarity region (AEB);

[0209] - A quantitative description of the corresponding similarity of the similarity regions (AEB);

[0210] -Location information of the similarity region (AEB) in the second histopathological image data (HIS2);

[0211] - An auxiliary image (AB) based on the second histopathological image data (HIS2), in which the similarity region (AEB) is highlighted;

[0212] - The probability of a recurrence relationship between the first histopathological image data and the second histopathological image data (HIS1, HIS2).

[0213] 4. Based on any one of the methods mentioned above, where,

[0214] The first moment is before the second moment.

[0215] 5. According to method 4, where,

[0216] The steps of providing the first histopathological image data (HIS1) include:

[0217] - Access the database for histopathological image data (60);

[0218] - Select the second histopathological image data (HIS2) from the histopathological image data stored in the database (60) based on the first histopathological image data (HIS1) and / or the metadata associated with the first histopathological image data (HIS1).

[0219] 6. According to the method in 5, where,

[0220] The metadata has:

[0221] - Patient identifiers used to identify the patient.

[0222] - Information about the patient's anatomical region, extracting the tissue sample from the anatomical region on which the first histopathological image data (HIS1) is based, and / or

[0223] - Information about the first moment.

[0224] 7. Based on any one of the methods mentioned above, where,

[0225] The second moment occurs before the first moment.

[0226] 8. According to the method in 7, where,

[0227] The step of providing (S20') the second histopathological image data (HIS2) includes:

[0228] - Access the database for histopathological image data (60);

[0229] - Select the second histopathological image data (HIS2) from the histopathological image data stored in the database (60) based on the first histopathological image data (HIS1) and / or the metadata associated with the first histopathological image data (HIS1).

[0230] 9. According to the method in 8, where,

[0231] The metadata has:

[0232] - Patient identifiers used to identify the patient.

[0233] - Information about the anatomical region of the patient, from which the tissue sample on which the first histopathological image data (HIS1) is based is extracted.

[0234] - Information regarding the histopathological staining agents used when providing the first histopathological image data (HIS1), and / or

[0235] - Information about the first moment.

Claims

1. A computer-implemented method for providing similarity information (AEI) of different histopathological image data (HIS1, HIS2) of a patient, the method comprising the following steps: - Provide first histopathological image data (HIS1), which is based on a tissue sample extracted from the patient at a first moment; - Provide second histopathological image data (HIS2), which is based on additional tissue samples extracted from the patient at a second time different from the first time. - Using image processing algorithms (TF-A, TF-B, TF-C, TF-A'), the similarity between the first histopathological image data (HIS1) and the second histopathological image data is analyzed between at least one region from the first histopathological image data (HIS1) and at least one region from the second histopathological image data (HIS2). - Based on the steps of the analysis, similarity information (AEI) is determined, which includes morphological and / or structural similarity and based thereon includes the probability of a recurrence relationship between the first histopathological image data (HIS1) and the second histopathological image data (HIS2), wherein the recurrence relationship represents a conclusion about whether the pathological change is based on a recurrence or further growth of an existing pathological change. as well as - Provide the aforementioned similarity information, The analysis steps include: - Identify (S15', S15'') regions of interest (IB) indicating pathological identification in the first histopathological image data (HIS1); - Search for similarity regions (AEB) of the second histopathological image data (HIS2) (S30A', S30A''), wherein the similarity regions (AEB) are similar to the region of interest (IB), and the step of searching (S30A', S30A'') includes applying the image processing algorithm (TF-A, TF-B, TF-C, TF-A') to the second histopathological image data (HIS2); The similarity information (AEI) is determined based on the search step (S30A', S30A'').

2. The method according to claim 1, wherein, The similarity information (AEI) includes: - Location information of similar regions (IB, AEB) indicating pathological identification in the first histopathological image data (HIS1) and / or the second histopathological image data (HIS2); and - An auxiliary image (AB) based on the first histopathological image data (HIS1) and / or the second histopathological image data (HIS2), in which similar areas for pathological identification are highlighted (IB, AEB).

3. The method according to claim 1 or 2, wherein the step of providing the similarity information (AEI) includes displaying the similarity information (AEI) to a user via a user interface (10).

4. The method according to claim 1 or 2, further comprising the following steps: The medical report template (S50) is filled out based on the similarity information (AEI).

5. The method according to claim 1, wherein, The region of interest (IB) has a single region (ROI, ROI1, ROI2, ROI3) defined in the first histopathological image data (HIS1).

6. The method according to claim 1, wherein, The steps for identifying (S15', S15'') the region of interest (IB) include: The region of interest (IB) is determined by the image processing algorithm, and / or Evaluate the annotations of the user (S15B', S15B''), which identify the region of interest (IB).

7. The method according to claim 1, wherein, The steps for searching (S30A', S30A'') include: Extract (A10) feature signatures based on the region of interest (IB); The similarity information (AEI) is obtained (A20-A40) based on the extracted feature signature.

8. The method according to claim 1 or 2, wherein the image processing algorithm (TF-A, TF-B, TF-C, TF-A') has a trained function.

9. The method of claim 8, wherein the trained function has a convolutional neural network.

10. The method according to claim 1 or 2, wherein, The second moment occurs before the first moment.

11. The method of claim 10, wherein The steps for providing the second histopathological image data (HIS2) include: - Access the database (60) used for histopathological image data; as well as - Select the second histopathological image data (HIS2) from the histopathological image data stored in the database (60) based on the first histopathological image data (HIS1) and / or the metadata associated with the first histopathological image data (HIS1).

12. The method of claim 11, wherein The metadata has: - Patient identifiers used to identify the patient. - Information regarding the anatomical extraction region of the patient, from which the tissue sample upon which the first histopathological image data (HIS1) is based is extracted. - Information regarding the histopathological staining agents used when providing the first histopathological image data (HIS1), and / or - Information about the first moment.

13. The method of claim 1, wherein The region of interest (IB) has multiple individual regions (ROI, ROI1, ROI2, ROI3) defined in the first histopathological image data (HIS1), each of which indicates a pathological identification.

14. The method of claim 6, wherein The annotation is provided by the user through manual input via the user interface (10) after the step of providing the first histopathological image data (HIS1).

15. The method of claim 8, wherein The trained function has a region-based convolutional neural network.

16. A system (1) for providing similarity information (AEI) of different histopathological image data (HIS1, HIS2) of a patient, said system (1) comprising: - Interface and control device (40). - The interface is configured to receive first histopathological image data (HIS1) and second histopathological image data (HIS2), wherein the first histopathological image data (HIS1) is based on a tissue sample extracted from the patient at a first time, and the second histopathological image data (HIS2) is based on a separate tissue sample extracted from the patient at a second time, different from the first time; and The control device (40) described therein is also configured to: Similarity information (AEI) is determined based on the first histopathological image data (HIS1) and the second histopathological image data (HIS2) using image processing algorithms (TF-A, TF-B, TF-C, TF-A'). The similarity information (AEI) has a description of the similarity between at least one region from the first histopathological image data (HIS1) and at least one region from the second histopathological image data (HIS2) that indicates pathological identification. as well as The similarity information (AEI) is provided, which includes morphological and / or structural similarity and, based on this, includes the probability of a recurrence relationship between the first histopathological image data (HIS1) and the second histopathological image data (HIS2), wherein the recurrence relationship represents a conclusion regarding whether the pathological changes are a recurrence or further growth of an existing pathological change. Identify (S15', S15'') regions of interest (IB) in the first histopathological image data (HIS1); Searching for similarity regions (AEB) in the second histopathological image data (HIS2) (S30A', S30A''), wherein the similarity regions (AEB) are similar to the region of interest (IB), and the step of searching (S30A', S30A'') includes applying the image processing algorithm (TF-A, TF-B, TF-C, TF-A') to the second histopathological image data (HIS2); as well as The similarity information (AEI) is determined based on the search steps (S30A', S30A'').

17. The system according to claim 16, further comprising: A database (60) for storing multiple histopathological image data; and User interface (10) for user interaction; wherein The interface is connected to the database (60) and the user interface (10) for data connection, and in addition, the control device is configured to: Based on the user's manual user input in the user interface (10), the first histopathological image data (HIS1) is selected from the database (60) and received via the interface; and The second histopathological image data (HIS2) is selected from the database (60) based on the first histopathological image data (HIS1) and / or the metadata associated with the first histopathological image data (HIS1).

18. A computer program product comprising a program and capable of being directly loaded into the memory of a programmable computing unit of a control device (40), the computer program product having a program mechanism for performing the method according to any one of claims 1 to 15 when the program is run in the control device (40).

19. A computer-readable storage medium storing readable and executable program segments so as to perform all steps of the method according to any one of claims 1 to 15 when the program segments are executed by a control device (40).

Images