Systems and methods for matching a block histology sample and a section histology sample

Abstract

Features for imaging a block sample and a slide sample using an imaging system are disclosed. The imaging system can link images by identifiers associated with the block sample and the slide sample. The imaging system can train a machine learning algorithm based on correctly linked images. In some embodiments, the trained machine learning algorithm can include an image analysis module or a convolutional neural network. The imaging system can use the trained machine learning algorithm to determine a confidence score of a match between the block sample and the slide sample. The trained machine learning algorithm can use features of the block sample and the slide sample, such as shape and tissue morphology, to determine whether the samples match. In some embodiments, the imaging system can alert a user that the samples can not match when the confidence score is below a certain threshold.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 132345, filed December 30, 2020, entitled SYSTEM AND METHOD FOR MATCHINGOFBLOCK AND SLICE HISTOLOGICAL SAMPLES, which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to imaging systems such as those used to capture images of blocks and / or slices of histological samples. Background Technology

[0004] Imaging systems can be used to capture images of a desired scene. These images can then be used for a variety of purposes, including, for example, image analysis. For instance, an imaging system can capture an image and perform image analysis to determine specific image characteristics. Examples of imaging systems include, but are not limited to, cameras, scanners, microscopes, mobile devices, tablet computers, laptop computers, and / or wearable electronics. Summary of the Invention

[0005] One aspect of this disclosure is an image analysis apparatus. The image analysis apparatus may include a first imaging device configured to scan a tissue sample block and generate block image data based on the scanned tissue sample block. The tissue sample block may be configured to be sliced ​​to form one or more slices of the tissue sample block. The image analysis apparatus may further include a second imaging device configured to scan slices from one or more slices of the tissue sample block and generate slice image data based on the scanned slices. The image analysis apparatus may further include a computing device configured to obtain first block image data from the first imaging device and first slice image data from the second imaging device. The computing device is further configured to link the first block image data and the first slice image data together. The computing device is further configured to provide the first block image data, the first slice image data, and information indicating that the first block image data and the first slice image data are correctly linked together as input to a machine learning algorithm. The computing device is further configured to train a machine learning algorithm using the input to the machine learning algorithm. The computing device is also configured to obtain second block image data from the first imaging device and second slice image data from the second imaging device, the second image data and the second slice image data being linked together. The computing device is further configured to perform image analysis on the second image data and the second slice image data using a machine learning algorithm, based on the fact that the second image data and the second slice image data are linked together. The computing device is also configured to determine a confidence value indicating whether the second image data and the second slice image data are correctly linked together, based on the output of the machine learning algorithm.

[0006] In another aspect of this disclosure, the computing device is further configured to obtain a confidence threshold associated with a user. The computing device is also configured to compare a confidence value and a confidence threshold. The computing device is further configured to generate a recommendation for the user, indicating whether the second block of image data and the second slice of image data are correctly linked, based at least in part on the comparison of the confidence value and the confidence threshold.

[0007] In another aspect of this disclosure, the computing device is further configured to perform image analysis using a machine learning algorithm by providing a second block of image data and a second slice of image data to a trained convolutional neural network. The trained convolutional neural network is configured to perform image analysis.

[0008] In another aspect of this disclosure, the computing device is further configured to extract a first plurality of features from block image data. The first plurality of features includes one or more of a plurality of filters or patterns. The computing device is also configured to extract a second plurality of features from slice image data. The second plurality of features includes one or more of a plurality of filters or patterns. Performing image analysis using a machine learning algorithm includes comparing the first plurality of features and the second plurality of features.

[0009] In another aspect of this disclosure, the computing device is configured to perform image analysis by using one or more of the following algorithms: image difference algorithm, spatial analysis algorithm, pattern recognition algorithm, shape comparison algorithm, color distribution algorithm, blob detection algorithm, template matching algorithm, SURF feature extraction algorithm, edge detection algorithm, key point matching algorithm, histogram comparison algorithm, or semantic texel forest algorithm.

[0010] In another aspect of this disclosure, the image analysis apparatus further includes a coverslipper and a staining machine. The staining machine stains sections of tissue sample blocks to generate stained sections. The coverslipper generates glass slides of the stained sections. The second imaging device is configured to scan the stained sections.

[0011] In another aspect of this disclosure, the first imaging device and the second imaging device may be different imaging devices. In another aspect of this disclosure, the first imaging device and the second imaging device may be the same imaging device.

[0012] In another aspect of this disclosure, a tissue sample block is associated with a first identifier, and a slice of the tissue sample block is associated with a second identifier. The linked block image data and the first slice image data are at least partially based on the first identifier corresponding to the second identifier.

[0013] In another aspect of this disclosure, the tissue sample block slices and the tissue sample blocks are each associated with at least one of a tag, RFID tag, Bluetooth tag, identifier, barcode, mark, sign or stamp.

[0014] In another aspect of this disclosure, the tissue sample block includes one or more of the following: paraffin-embedded tissue sample block, OCT-embedded tissue sample block, frozen tissue sample block, or fresh tissue sample block.

[0015] In another aspect of this disclosure, the image analysis apparatus further includes a third imaging device. The third imaging device is configured to scan a second slice of one or more slices of a tissue sample block to generate third slice image data. The computing device is further configured to obtain the third slice image data from the third imaging device and to obtain third block image data from the first imaging device. The computing device is also configured to link the third block image data and the third slice image data together. The computing device is further configured to provide the third block image data, the third slice image data, and information indicating that the third image data and the third slice image data are correctly linked together as additional input to a machine learning algorithm.

[0016] In another aspect of this disclosure, the image analysis apparatus further includes a slicer configured to slice a tissue sample block to generate one or more slices of the tissue sample block.

[0017] In another aspect of this disclosure, the first imaging device is a microtome. The second imaging device includes one or more of a microtome, a coverslipper, a case folder imaging station, a single slide imaging station, a dedicated low-resolution imaging device, or a digital pathology scanner.

[0018] In another aspect of this disclosure, the computing device is also configured to transmit recommendations to a user computing device associated with the user.

[0019] In another aspect of this disclosure, the computing device is also configured to cause recommendations to be displayed via a user computing device associated with the user.

[0020] In another aspect of this disclosure, the computing device is further configured to obtain a response to a recommendation, the response including acceptance of the recommendation. The computing device is also configured to adjust a confidence threshold based on the response corresponding to acceptance of the recommendation.

[0021] In another aspect of this disclosure, the computing device is further configured to obtain a response to a recommendation, the response including a rejection of the recommendation. The computing device is also configured to adjust a confidence threshold based on the response corresponding to the rejection of the recommendation.

[0022] In another aspect of this disclosure, the computing device is further configured to obtain a response to the recommendation. The computing device is also configured to adjust a machine learning algorithm based on the response to the recommendation.

[0023] In another aspect of this disclosure, the recommendations include similarity scores.

[0024] In another aspect of this disclosure, the confidence threshold is associated with multiple users. The computing device also determines the confidence threshold based on multiple characteristics associated with the multiple users.

[0025] In another aspect of this disclosure, the computing device is configured to provide recommendations via an application programming interface.

[0026] The foregoing overview is illustrative only and is not intended to be limiting. Other aspects, features, and advantages of the systems, apparatuses, and methods and / or other subjects described in this application will become apparent from the teachings set forth below. The overview is provided to introduce some of the ideas selected for this disclosure. The overview is not intended to identify key or essential features of any subject matter described herein. Attached Figure Description

[0027] Various examples are depicted in the accompanying drawings for illustrative purposes and should in no way be construed as limiting the scope of the examples. Various features of different disclosed examples may be combined to form additional examples that are part of this disclosure.

[0028] Figure 1 A schematic diagram of an exemplary networking environment according to some implementation schemes is depicted.

[0029] Figure 2 An exemplary workflow for generating image data from tissue sample blocks, according to some implementation schemes, is described.

[0030] Figure 3A An exemplary organization block sample according to some implementation schemes is illustrated.

[0031] Figure 3B Exemplary tissue block samples and exemplary tissue slice samples attached to a glass slide according to some embodiments are illustrated.

[0032] Figure 4 An imaging system for capturing images of histological samples, according to some implementation schemes, is described.

[0033] Figure 5 A schematic diagram of an image analysis module comprising multiple layers of a neural network according to aspects of this disclosure is depicted.

[0034] Figure 6 A schematic diagram depicts an image analysis module comprising multiple convolutional networks according to aspects of this disclosure.

[0035] Figure 7 This is a flowchart of an exemplary routine for performing image analysis on image data from tissue sample blocks and slices of tissue sample blocks attached to a glass slide, according to some implementation schemes.

[0036] Figure 8An exemplary computing apparatus is shown that can be used to implement aspects of this disclosure. Detailed Implementation

[0037] Generally, this disclosure relates to an imaging system that can receive a first image of a histological sample (e.g., a tissue block) and determine the probability of a match between the first image of the histological sample and a second image of the histological sample (e.g., a slice of the tissue block attached to a glass slide). Based on the probability of a match between the first and second images (e.g., the probability that the histological sample in the first image and the histological sample in the second image correspond to the same histological sample), the imaging system can perform various operations, such as generating recommendations and providing recommendations to a user via a user computing device.

[0038] To determine the probability of a match between a first image and a second image, the imaging system can implement an image analysis module (e.g., a convolutional neural network, machine learning algorithm, etc.) that analyzes each image. As described herein, using an image analysis module within such an imaging system can increase the accuracy of the imaging process. For example, the imaging system can provide a more accurate indication of the match between the first and second images. By using the image analysis module, the imaging system can efficiently and accurately determine the probability of a match. Furthermore, the use of the image analysis module can reduce the number of false matches or mismatches and can reduce the number of matches provided for secondary analysis (e.g., reducing the number of matches provided to the user for verification).

[0039] As used herein, the term "imaging system" can refer to any electronic device or component capable of performing an imaging process. For example, an "imaging system" can include scanners, cameras, etc. In some embodiments, the imaging system may not perform imaging, but rather receive image data and perform image analysis on the image data.

[0040] As described herein, an imaging system can be used to perform image analysis on received image data (e.g., image data corresponding to a histological sample). The imaging system can acquire (e.g., via imaging performed by the imaging system or via imaging performed by an imaging device) image data of a first histological sample and image data of a second histological sample. Each histological sample can be associated with a specific tissue block and / or a segment of a specific tissue block, and to ensure accurate medical diagnosis, it is important to ensure that the histological sample image corresponding to a specific tissue block matches the histological sample image corresponding to the same tissue block (e.g., the image corresponding to the same tissue block is linked to that tissue block).

[0041] Each histological sample can be associated with an identifier, and based on these identifiers, the imaging system can compare images of the histological samples to verify the accuracy of the links between them. The imaging system can implement an image analysis module to compare the images of the histological samples. The imaging system can feed images of the histological samples into the image analysis module to determine the probability of a match between the images of the histological samples. The image analysis module can be trained on a predetermined set of training image data to identify matches. Based on this training, the image analysis module can determine the probability of an image match between the histological samples. For example, the image analysis module can determine that the images of the histological samples are extremely unlikely to match. Based on this probability, the imaging system can generate a recommendation and provide the recommendation to the user of the imaging system.

[0042] In many conventional situations, implementing a generic imaging system to perform an imaging procedure may not provide satisfactory results in specific circumstances or for specific users. Such generic imaging systems may determine whether images of histological samples match based on user input. For example, the imaging system may receive indications that two or more images match, and the system may be unable to verify this match. Such generic imaging systems may lead to images being incorrectly matched based on user input. For example, due to user error, the user may incorrectly match image data or incorrectly determine that image data do not match. Since image data corresponds to histological samples (e.g., tissue blocks), determining accurate matches of histological samples can be crucial. Incorrect matches between images corresponding to different histological samples and / or failure to identify images corresponding to the same histological sample can lead to misdiagnosis. Such misdiagnosis can result in additional adverse consequences. Furthermore, the imaging procedure may suffer from performance problems due to the requirement for user input for each pair of images. For example, the imaging procedure of a generic imaging system may be slow, inefficient, and ineffective. Therefore, conventional imaging systems may not be adequate in the aforementioned situations.

[0043] With the proliferation of imaging systems, the demand for faster and more efficient image processing and analysis has also increased. This disclosure provides a system for analyzing image data that offers significant advantages over existing implementations. This disclosure provides systems and methods capable of increasing the accuracy of match recognition between image data relative to conventional imaging systems without significantly impacting speed or efficiency. These advantages are provided by the embodiments discussed herein, and specifically by implementing an image analysis module to analyze image data and determine the likelihood of matches between image data. Furthermore, the use of sample identifiers allows verification of the convolutional neural network's determination of whether a given image is a match or not. The use of the image analysis module also allows the imaging system to determine matches between images based on previously determined matches, thereby increasing the accuracy and efficiency of imaging processing according to the methods described above.

[0044] Some aspects of this disclosure relate to training and using an image analysis module configured to receive image data from one or more histological samples to determine the probability that the image data corresponds to additional image data. The image analysis module described herein can provide improved image analysis accuracy by using a trained module (e.g., a machine learning model, convolutional neural network) trained to identify similarities between image data. Image analysis modules using such trained modules can provide increased precision and recall without significantly impacting the computational speed offered by conventional image analysis systems. In some embodiments, image analysis can be based on additional data. For example, image analysis can be based at least in part on identifiers associated with the image data. Identifiers can include labels, identifiers, or other indicators corresponding to specific image data.

[0045] Identifiers can provide identification information for specific image data. For example, an identifier can identify the origin of a histological sample, the patient associated with the histological sample, a unique code, or any other identifying information. Based on the identifier, the image analysis system can determine that first image data (e.g., an image associated with a tissue block) should be compared with second image data (e.g., an image associated with a tissue slice). For example, the image analysis system can determine that if the first and second image data share the same identifier, then the first and second image data should be compared. The identifier can indicate a user-identifiable match between the image data, and the image analysis system can determine the likelihood that this indication is accurate. In some embodiments, the image analysis system may not use identifiers to identify possible matches. Instead, the image analysis system can compare the first image data with multiple image data stored by the image analysis system. The image analysis system can further use identifiers to weight the matches or non-matches identified by the image analysis system.

[0046] As described herein, the image analysis module may include any machine learning model (e.g., a computing system, computing device, etc.) and / or any convolutional neural network. Furthermore, the image analysis module may implement one or more image analysis algorithms, which may include, for example, image differencing algorithms, spatial analysis algorithms, pattern recognition algorithms, shape comparison algorithms, color distribution algorithms, blob detection algorithms, template matching algorithms, SURF feature extraction algorithms, edge detection algorithms, keypoint matching algorithms, histogram comparison algorithms, semantic texel forest algorithms, and / or any other type of image analysis algorithm. The image analysis module may implement one or more of these algorithms to analyze image data.

[0047] As described herein, based on image analysis performed by the image analysis module, the image analysis system can determine an output indicating the likelihood of a match between first image data and second image data. For example, the image analysis system can determine that the first image data and second image data likely correspond to the same histological sample. Furthermore, based on the output, the image analysis system can determine a confidence value indicating that the first image data and second image data correspond to the same histological sample. For example, if the output indicates that the first image data and second image data may not match, the image analysis system can provide a low confidence value indicating that the first image data and second image data are a match. Furthermore, if the output indicates that the first image data and second image data may match, the image analysis system can provide a higher confidence value indicating that the first image data and second image data are a match. The confidence value can correspond to a rating regarding the level of confidence. For example, the confidence value can be a letter, number, alphanumeric, or symbolic rating (e.g., a confidence value could be 70%, indicating a 70% confidence level regarding the matching of the image data).

[0048] As described herein, an image analysis system can determine a confidence threshold for a specific user associated with the system. For example, a user could be someone requesting the analysis of image data. Furthermore, the user can provide a confidence threshold indicating the desired confidence value for a match. For instance, a confidence threshold might indicate that more than 70% of confidence values ​​can be identified as a match, and that 70% or less of confidence values ​​should be verified by the user. The image analysis system can also compare the confidence values ​​corresponding to the image data comparison with the confidence threshold. Based on the comparison of the confidence values ​​and the confidence threshold, the image analysis system can determine a recommendation and provide it to the user. It should be noted that the approximately 70% confidence threshold level is for illustrative purposes, and the confidence threshold level can be lower or higher depending on the specific application, patient data, user preferences, etc.

[0049] In the following description, various examples will be described. Specific configurations and details are stated for illustrative purposes to provide a thorough understanding of the examples. However, those skilled in the art will also understand that the examples can be practiced without the stated specific details. Furthermore, well-known features may be omitted or simplified to avoid obscuring the described examples.

[0050] System Overview

[0051] Figure 1An exemplary environment 100 is illustrated in which a user and / or system, according to some embodiments, may implement an image analysis system 104. The image analysis system 104 can perform image analysis on received image data. The image analysis system 104 can perform image analysis to determine the probability that first image data in the received image data matches second image data in the received image data, or the probability that the first image data and the second image data are correctly linked or otherwise correlated with each other (e.g., the first image data and the second image data correspond to the same tissue sample block, the first image data and the second image data correspond to the same patient, etc.). Based on the determined probabilities, the image analysis system 104 can generate recommendations.

[0052] Image analysis system 104 can use image analysis module ( Figure 1 Image analysis is performed (not shown in the image). The image analysis system 104 can receive image data from the imaging device 102 and transmit the data to the user computing device 106 for processing. Although some examples herein refer to specific types of devices as imaging device 102, image analysis system 104, or user computing device 106, these examples are merely illustrative and are not intended to be limiting, essential, or exhaustive. The image analysis system 104 can be any type of computing device (e.g., server, node, router, network host, etc.). Furthermore, the imaging device 102 can be any type of imaging device (e.g., camera, scanner, mobile device, laptop computer, etc.). In some embodiments, the imaging device 102 may include multiple imaging devices. Furthermore, the user computing device 106 can be any type of computing device (e.g., mobile device, laptop computer, etc.).

[0053] Imaging device 102 can capture and / or generate image data for analysis. Imaging device 102 may include one or more of a lens, an image sensor, a processor, or a memory. Imaging device 102 can receive user interaction. User interaction may be a request to capture image data. Based on the user interaction, imaging device 102 can capture image data. In some embodiments, imaging device 102 may capture image data periodically (e.g., every 10 minutes, 20 minutes, or 30 minutes). In other embodiments, imaging device 102 may determine that an object has been placed in the field of view of imaging device 102 (e.g., a histological sample has been placed on a table and / or platform associated with imaging device 102), and based on this determination, capture image data corresponding to said object. Imaging device 102 may also receive image data from additional imaging devices. For example, imaging device 102 may be a node that routes image data from other imaging devices to image analysis system 104. In some embodiments, imaging device 102 may be located within image analysis system 104. For example, imaging device 102 may be a component of image analysis system 104. Furthermore, the image analysis system 104 can perform imaging functions. In other embodiments, the imaging device 102 and the image analysis system 104 can be connected (e.g., wireless or wired connection). For example, the imaging device 102 and the image analysis system 104 can communicate on network 108. Additionally, the imaging device 102 and the image analysis system 104 can communicate over a wired connection. In one embodiment, the image analysis system 104 may include a docking station that enables the imaging device 102 to dock with the image analysis system 104. Electrical contacts of the image analysis system 104 can connect to electrical contacts of the imaging device 102. The image analysis system 104 can be configured to determine when the imaging device 102 has been connected to the image analysis system 104, at least in part, based on the electrical contacts of the image analysis system 104. In some embodiments, the image analysis system 104 can use one or more other sensors (e.g., proximity sensors) to determine that the imaging device 102 has been connected to the image analysis system 104. In some embodiments, the image analysis system 104 can be connected to (via wired or wireless connection) multiple imaging devices.

[0054] Image analysis system 104 may include various components for providing the features described herein. In some embodiments, image analysis system 104 may include one or more image analysis modules to perform image analysis on image data received from imaging device 102. The image analysis modules may use the image data to perform one or more imaging algorithms.

[0055] Image analysis system 104 can be connected to user computing device 106. Image analysis system 104 can connect (via wireless or wired connection) to user computing device 106 to provide recommendations for a set of image data. Image analysis system 104 can transmit the recommendations to user computing device 106 via network 108. In some embodiments, image analysis system 104 and user computing device 106 can be configured to connect such that user computing device 106 can engage and disengage from image analysis system 104 to receive the recommendations. For example, user computing device 106 can communicate with image analysis system 104 when it determines that image analysis system 104 has generated recommendations for user computing device 106. Furthermore, based on image analysis performed by image analysis system 104 on image data corresponding to a specific user computing device 106, a specific user computing device 106 can be connected to image analysis system 104. For example, a user can associate multiple histological samples. When a specific histological sample is determined to be associated with a specific user and the corresponding user computing device 106, the image analysis system 104 can transmit recommendations for the histological sample to the specific user computing device 106. In some embodiments, the user computing device 106 may interface with the image analysis system 104 to receive the recommendations.

[0056] In some implementations, the imaging device 102, image analysis system 104, and / or user computing device 106 can communicate wirelessly. For example, the imaging device 102, image analysis system 104, and / or user computing device 106 can communicate over a network 108. The network 108 can include any feasible communication technology, such as wired and / or wireless modes and / or technologies. The network can include any combination of: Personal Area Network (“LAN”), Local Area Network (“LAN”), Campus Network (“CAN”), Metropolitan Area Network (“MAN”), Extranet, Intranet, Internet, Short-Range Wireless Communication Network (e.g., ZigBee, Bluetooth, etc.), Wide Area Network (“WAN”) (centralized and / or distributed) and / or any combination, arrangement, and / or aggregation thereof. The network 108 can include the Internet, and / or may or may not have Internet access. The imaging device 102 and image analysis system 104 can transmit image data. For example, the imaging device 102 can transmit image data associated with a histological sample to the image analysis system 104 via the network 108 for analysis. The image analysis system 104 and the user computing device 106 can transmit recommendations corresponding to the image data. For example, the image analysis system 104 can transmit recommendations indicating the likelihood of a first image data matching a second image data to the user computing device 106. In some embodiments, the imaging device 102 and the image analysis system 104 can communicate via a first network, and the image analysis system 104 and the user computing device 106 can communicate via a second network. In other embodiments, the imaging device 102, the image analysis system 104, and the user computing device 106 can communicate on the same network.

[0057] Referring to the illustrative embodiment, at [A], the imaging device 102 can acquire block data. To acquire block data, the imaging device 102 can image the tissue block (e.g., scan, capture, record, etc.). The tissue block can be a histological sample. For example, the tissue block can be a biological tissue block that has been removed and prepared for analysis. As will be discussed further below, various histological techniques can be performed on the tissue block to prepare it for analysis. The imaging device 102 can capture images of the tissue block and store the corresponding block data in the imaging device 102. The imaging device 102 can acquire block data based on user interaction. For example, a user can provide input through a user interface (e.g., a graphical user interface (“GUI”)) and request the imaging device 102 to image the tissue block. Furthermore, a user can interact with the imaging device 102 to cause it to image the tissue block. For example, a user can toggle a switch on the imaging device 102, press a button on the imaging device 102, provide a voice command to the imaging device 102, or otherwise interact with the imaging device 102 to cause it to image the tissue block. In some implementations, the imaging device 102 can image the tissue block based on the imaging device 102 detecting that the tissue block has been placed in the viewport of the imaging device 102. For example, the imaging device 102 can determine that the tissue block has been placed in the viewport of the imaging device 102 and determine to image the tissue block based on this.

[0058] At [B], imaging device 102 can acquire slice data. In some embodiments, imaging device 102 can acquire both slice data and block data. In other embodiments, a first imaging device can acquire a slide including slices, and a second imaging device can acquire block data. To acquire slice data, imaging device 102 can image (e.g., scan, capture, record, etc.) a slide of slices from a tissue block. Slices from a tissue block can be attached to a slide. In some embodiments, imaging device 102 can directly image slices from a tissue block. Slices from a tissue block can be slices of a histological sample. For example, a tissue block can be sliced ​​(e.g., segmented) to generate one or more slices of the tissue block. In some embodiments, a portion of a tissue block can be sliced ​​to generate slices of the tissue block, such that a first portion of the tissue block corresponds to the tissue block imaged to obtain block data, and a second portion of the tissue block corresponds to a slice of the tissue block imaged to obtain slice data. As will be discussed in more detail below, various histological techniques can be performed on the tissue block to generate slices of the tissue block and attach the slices to a slide. Imaging device 102 can capture images of a glass slide and store the corresponding slide data in imaging device 102. Imaging device 102 can obtain slide data based on user interaction. For example, a user can provide input through a user interface and request imaging device 102 to image the glass slide. Furthermore, the user can interact with imaging device 102 to cause imaging device 102 to image the glass slide. In some embodiments, imaging device 102 can image the tissue block based on imaging device 102 detecting that the tissue block has been sliced ​​or that the glass slide has been placed in the viewport of imaging device 102.

[0059] At [C], imaging device 102 may transmit signals representing captured image data (e.g., block data and slice data) to image analysis system 104. Imaging device 102 may transmit the captured image data as electronic signals via network 108 to image analysis system 104. The signals may include and / or correspond to pixel representations of the block data and / or slice data. It should be understood that the signals may include and / or correspond to more, fewer, or different image data. For example, the signals may correspond to multiple slices of a tissue block and may represent first slice data and second slice data. Furthermore, the signals enable image analysis system 104 to reconstruct the block data and / or slice data. In some embodiments, imaging device 102 may transmit a first signal corresponding to block data and a second signal corresponding to slice data. In other embodiments, a first imaging device may transmit a signal corresponding to block data, and a second imaging device may transmit a signal corresponding to slice data.

[0060] At [D], the image analysis system 104 can perform image analysis on the block data and slice data provided by the imaging device 102. To perform the image analysis, the image analysis system 104 can utilize one or more image analysis modules capable of performing one or more image processing functions. For example, the image analysis module may include an imaging algorithm, a machine learning model, a convolutional neural network, or any other module for performing image processing functions. Based on the performed image processing functions, the image analysis module can determine the probability that the block data and slice data correspond to the same tissue block. For example, the image processing functions may include edge analysis of the block data and slice data, and based on the edge analysis, determine whether the block data and slice data correspond to the same tissue block. The image analysis system 104 can obtain a confidence threshold from the user computing device 106, the imaging device 102, or any other device. In some embodiments, the image analysis system 104 can determine the confidence threshold based on the user computing device 106's response to a specific recommendation. Furthermore, the confidence threshold may be specific to a user, user group, type of tissue block, location of tissue block, or any other factor. The image analysis system 104 can compare the determined confidence threshold with the image analysis performed by the image analysis module. Based on this comparison, the image analysis system 104 can generate recommendations indicating recommended actions for the user's computing device 106 based on the probability that block data and slice data correspond to the same tissue block.

[0061] At [E], the image analysis system 104 can transmit a signal to the user computing device 106 representing a recommendation indicating the likelihood that block data and slide data correspond to the same tissue block. The image analysis system 104 can transmit the recommendation as an electrical signal to the user computing device 106 via network 108. The signal may include and / or a representation corresponding to the recommendation. Based on receiving the recommendation, the user computing device 106 can determine a diagnosis. In some embodiments, the image analysis system 104 may transmit a series of recommendations corresponding to a set of tissue blocks, slides, and / or slides. The image analysis system 104 may include the user's recommendation action in the recommendations. For example, the recommendations may include recommendations for the user to examine tissue blocks and slides attached to a slide. Furthermore, the recommendations may include recommendations that the user does not need to examine tissue blocks and slides attached to a slide. The recommendations may also include a representation of recommendation strength. For example, the recommendations may include qualifiers indicating recommendation strength, such as percentages, rankings, wording (e.g., uncertain, possible, unlikely, etc.).

[0062] Imaging the prefabricated blocks and pre-supported glass slides

[0063] Figure 2An exemplary workflow 200 for generating image data from a tissue sample block, according to some embodiments, is depicted. The exemplary workflow 200 illustrates a process for generating a preform block and a pre-prepared slide including sections from the tissue block, and for generating a preprocessed image based on the preform block and the pre-prepared slide. The exemplary workflow 200 can be implemented by one or more computing devices. For example, the exemplary workflow 200 can be implemented by a microtome, a coverslipper, a staining machine, and an imaging device. Each computing device can execute a portion of the exemplary workflow. For example, a microtome can cut the tissue block to generate one or more sections of the tissue block. A coverslipper can create a first slide for the tissue block and / or create a second slide for sections of the tissue block, a staining machine can stain each slide, and an imaging device can image each slide.

[0064] Tissue blocks can be obtained from a patient (e.g., a human, animal, etc.). The tissue block may correspond to a segment of tissue from the patient. The tissue block can be surgically removed from the patient for further analysis. For example, the tissue block can be removed to determine if it possesses certain characteristics (e.g., whether it is cancerous). To generate prefabricated block 202, the tissue preparer can use a specific preparation process to prepare the tissue block. For example, the tissue block can be preserved and subsequently embedded in a paraffin block. Alternatively, the tissue block (in a frozen or fresh state) can be embedded in a block. Optimal cutting temperature (“OCT”) compounds can also be used to embed the tissue block. The preparation process may include one or more of paraffin embedding, OCT embedding, or any other embedding of the tissue block. Figure 2 In the example, paraffin embedding was used to embed the tissue block. Additionally, the tissue block was embedded within a paraffin block and mounted on a microscope slide to prepare a pre-formed block.

[0065] A microtome can obtain sections of a tissue block to generate pre-prepared slides 204. The microtome can use one or more blades to section the tissue block and generate sections (e.g., portions) of the tissue block. The microtome can further section the tissue block to generate sections with a preferred thickness level. For example, the sections of the tissue block can be 1 mm. The microtome can provide the sections of the tissue block to a coverslipper. The coverslipper can wrap the sections of the tissue block in a slide to generate pre-prepared slides 204. Pre-prepared slides 204 may include sections mounted in specific locations. Furthermore, during the generation of pre-prepared slides 204, a staining machine can also stain the sections of the tissue block using any staining protocol. Additionally, the staining machine can stain the sections of the tissue block to highlight certain portions of the pre-prepared slides 204 (e.g., areas of interest). In some embodiments, the computing device may include both a coverslipper and a staining machine, and the slides may be stained as part of the slide generation process.

[0066] Pre-prepared block 202 and pre-prepared slide 204 can be provided to an imaging apparatus for imaging. In some embodiments, pre-prepared block 202 and pre-prepared slide 204 can be provided to the same imaging apparatus. In other embodiments, pre-prepared block 202 and pre-prepared slide 204 can be provided to different imaging apparatuses. The imaging apparatus can perform one or more imaging operations on pre-prepared block 202 and pre-prepared slide 204. In some embodiments, the computing device may include one or more of a tissue preparer, microtome, coverslipper, stainer, and / or imaging apparatus.

[0067] The imaging apparatus can capture an image of prefabricated block 202 to generate block image 206. Block image 206 can be a representation of prefabricated block 202. For example, block image 206 can be a representation of prefabricated block 202 viewed from one direction (e.g., from above). The representation of prefabricated block 202 can correspond to the same orientation as a slice of prefabricated slide 204 and / or tissue block. For example, if the tissue block is sliced ​​in a cross-sectional manner to generate slices of the tissue block, then block image 206 can correspond to the same cross-sectional view. To generate block image 206, prefabricated block 202 can be placed in the holder of the imaging apparatus and imaged by the imaging apparatus. Furthermore, block image 206 can include certain characteristics. For example, block image 206 can be a color image with a specific resolution level, sharpness level, scaling level, or any other image characteristics.

[0068] The imaging apparatus can capture an image of a pre-prepared slide 204 to generate a slide image 208. The imaging apparatus can capture images of specific sections of the pre-prepared slide 204. For example, the slide may include any number of sections, and the imaging apparatus can capture images of specific sections within those sections. The slide image 208 can be a representation of the pre-prepared slide 204. Depending on how the tissue block sections are generated, the slide image 208 can correspond to a view of the sections. For example, if sections of the tissue block are generated by cutting across a cross-section of the tissue block, the slide image 208 can correspond to the same cross-sectional view. To generate the slide image 208, the pre-prepared slide 204 can be placed in the holder of the imaging apparatus (e.g., in the viewer of a microscope) and imaged by the imaging apparatus. Furthermore, the slide image 208 can include certain characteristics. For example, the slide image 208 can be a color image with a specific resolution level, sharpness level, zoom level, or any other image characteristics.

[0069] The imaging apparatus can process block image 206 to generate a preprocessed image 210 and slide image 208 to generate a preprocessed image 212. The imaging apparatus can perform one or more image operations on block image 206 and slide image 208 to generate preprocessed image 210 and preprocessed image 212. The one or more image operations can include separating (e.g., focusing) various features of preprocessed image 210 and preprocessed image 212. For example, the one or more image operations can include separating the edges of a slice or tissue block, separating a region of interest within a slice or tissue block, or otherwise modifying (e.g., transforming) block image 206 and / or slide image 208. In some embodiments, the imaging apparatus can perform one or more image operations on either block image 206 or slide image 208. For example, imaging can perform one or more image operations on block image 206. In other embodiments, the imaging apparatus can perform a first image operation on block image 206 and a second image operation on slide image 208. The imaging device can provide preprocessed image 210 and preprocessed image 212 to the image analysis system to determine the probability that preprocessed image 210 and preprocessed image 212 correspond to the same tissue block.

[0070] Slice the tissue block

[0071] Figure 3AAn exemplary prefabricated tissue block 300A according to some embodiments is illustrated. Prefabricated tissue block 300A may include a tissue block 306 preserved in a specific manner (e.g., chemical preservation, fixation, support). To generate prefabricated tissue block 300A, tissue block 306 may be placed in a fixative (e.g., a liquid fixative). For example, tissue block 306 may be placed in a fixative such as a formaldehyde solution. The fixative can penetrate tissue block 306 and preserve tissue block 306. Tissue block 306 may then be separated to allow for further preservation of tissue block 306. Furthermore, tissue block 306 may be immersed in one or more solutions (e.g., an ethanol solution) to replace water within tissue block 306 with one or more solutions. Tissue block 306 may be immersed in one or more intermediate solutions. Additionally, tissue block 306 may be immersed in a final solution (e.g., histological wax). For example, the histological wax may be purified paraffin. After immersion in the final solution, tissue block 306 can be formed into prefabricated tissue block 300A. For example, tissue block 306 can be placed in a mold filled with histological wax. By placing the tissue block in the mold, tissue block 306 can be molded (e.g., encapsulated) in the final solution 304. To generate a pre-prepared tissue block 300A, tissue block 306 in the final solution 304 can be placed on platform 302. Thus, pre-prepared tissue block 300A can be generated. It should be understood that pre-prepared tissue block 300A can be prepared according to any tissue preparation method.

[0072] Figure 3BAn exemplary pre-fabricated tissue block 300A and an exemplary pre-fabricated slide 300B with attached tissue sections are illustrated according to some embodiments. The pre-fabricated tissue block 300A may include a tissue block 306 encased in a final solution 304 and placed on a platform 302. To generate the pre-fabricated slide 300B, the pre-fabricated tissue block 300A can be sectioned using a microtome. The microtome may include one or more blades for sectioning the pre-fabricated tissue block 300A. The microtome may use one or more blades to obtain cross-sectional sections 310 of the pre-fabricated tissue block 300A. The cross-sectional sections 310 of the pre-fabricated tissue block 300A may include sections 310 (e.g., portions) of the tissue block 306 encased in the final solution 304. To preserve the sections 310 of the tissue block 306, the sections 310 of the tissue block 306 may be modified (e.g., washed) to remove the final solution 304 from the sections 310 of the tissue block 306. For example, the final solution 304 can be rinsed and / or separated from the sections 310 of tissue block 306. Furthermore, the sections 310 of tissue block 306 can be stained using a staining machine. In some embodiments, the sections 310 of tissue block 306 may not be stained. The sections 310 of tissue block 306 can then be wrapped in a slide 308 using a coverslip machine to generate a pre-prepared slide 300B. The pre-prepared slide 300B may include an identifier 312 identifying the tissue block 306 corresponding to the pre-prepared slide 300B. Figure 3B As not shown in the diagram, the pre-fabricated tissue block 300A may further include an identifier that identifies the tissue block 306 corresponding to the pre-fabricated tissue block 300A. Since the pre-fabricated tissue block 300A and the pre-fabricated slide 300B correspond to the same tissue block 306, the identifier of the pre-fabricated tissue block 300A and the identifier 312 of the pre-fabricated slide 300B can identify the same tissue block 306.

[0073] Imaging device

[0074] Figure 4An exemplary imaging apparatus 400 according to one embodiment is illustrated. Imaging apparatus 400 may include imaging device 402 (e.g., lens and image sensor) and platform 404. Imaging apparatus 400 may receive pre-prepared tissue blocks and / or pre-prepared tissue slides with attached tissue sections via platform 404. Furthermore, the imaging apparatus may use imaging device 402 to capture image data corresponding to the pre-prepared blocks and / or pre-prepared slides. Imaging apparatus 400 may be one or more of a camera, scanner, medical imaging device, etc. Furthermore, imaging apparatus 400 may use imaging techniques such as microscopy. For example, imaging techniques may include bright-field microscopy, dark-field microscopy, phase-contrast microscopy, differential interference contrast microscopy, fluorescence microscopy, polarizing microscopy, Kohler illumination, oil immersion microscopy, optical microscopy, immunofluorescence microscopy, chromogenic in situ hybridization microscopy, in situ hybridization microscopy, fluorescence in situ hybridization microscopy, or any other type of imaging technique. For example, the imaging device may be a dark-field microscope, bright-field microscope, fluorescence microscope, etc.

[0075] Imaging device 400 can receive one or more pre-prepared tissue blocks and / or pre-prepared slides and capture corresponding image data. In some embodiments, imaging device 400 can capture image data corresponding to multiple pre-prepared tissue slides and / or multiple pre-prepared tissue blocks. Imaging device 400 can also capture representations of pre-prepared tissue slides and / or pre-prepared tissue blocks placed on a platform using the lens of imaging device 402 and the image sensor of imaging device 402. Therefore, imaging device 400 can capture image data so that an image analysis system can compare the image data to determine whether the image data corresponds to the same tissue block.

[0076] Imaging algorithms

[0077] Figure 5 A schematic diagram of an image analysis module 500 comprising multiple layers of a neural network according to aspects of this disclosure is depicted. The image analysis module 500 may be an image analysis system or may be implemented by an image analysis system. The image analysis module may implement one or more imaging algorithms to compare image data to determine whether the image data corresponds to the same tissue block. Furthermore, the image analysis module 500 may correspond to one or more of machine learning models, convolutional neural networks, etc. Figure 4 In the example, the image analysis module 500 corresponds to a convolutional neural network.

[0078] A convolutional neural network may include an input layer 502. The input layer 502 may be an array of pixel values. For example, the input layer may include a 320×320×3 pixel value array. Each value of the input layer 502 may correspond to a specific pixel value. Furthermore, the input layer 502 may obtain pixel values ​​corresponding to an image. Each input to the input layer 502 may be transformed according to one or more computations.

[0079] Furthermore, the values ​​of input layer 502 can be provided to hidden layer 504 of the convolutional neural network. In some embodiments, the convolutional neural network may include one or more hidden layers. A hidden layer may include multiple neurons, each performing a corresponding function. Furthermore, hidden layer 504 may perform one or more additional operations on the values ​​of input layer 502. For example, each neuron of hidden layer 504 may compute a weighted sum of the inputs (e.g., one or more inputs of input layer 502 may be added and weighted). By performing one or more operations, a particular hidden layer 504 may be configured to produce a particular output. For example, a particular hidden layer 504 may be configured to identify edges of tissue samples and / or block samples. Furthermore, a particular hidden layer 504 may be configured to identify edges of tissue samples and / or block samples, and another hidden layer 504 may be configured to identify another feature of tissue samples and / or block samples. Therefore, using multiple hidden layers enables the identification of multiple features of tissue samples and / or block samples. By identifying multiple features, the convolutional neural network can provide more accurate identification of a specific image. Furthermore, the combination of multiple hidden layers enables the convolutional neural network to identify and distinguish specific tissue blocks and / or tissue slices.

[0080] The outputs of one or more hidden layers 504 can be provided to the output layer 506 to identify (e.g., predict) tissue blocks associated with an image. The convolutional neural network can further identify the likelihood that the provided image is associated with a specific tissue block. Furthermore, when first image data and second image data are provided to the convolutional neural network, the network can determine the likelihood that the first image data and second image data correspond to the same tissue block. In some embodiments, the convolutional neural network may include pooling layers and / or fully connected layers.

[0081] To identify tissue blocks associated with a specific image, an image analysis module 500 can be trained to recognize tissue blocks. Through such training, the trained image analysis module 500 is trained to recognize differences and / or similarities between images. Advantageously, the trained image analysis module 500 is able to generate indications of the probability that a particular set of image data corresponds to the same scene (e.g., the same tissue block).

[0082] Block training data associated with tissue blocks can be provided to the image analysis module 500, or the image analysis module 500 can access the block training data (e.g., from a scanner, from a data storage device, from a database, from a memory, etc.) for training. Predetermined block training data may include tissue block data that has been previously identified (e.g., verified to correspond to a specific tissue block). Furthermore, slice training data associated with the same training block can be provided to the image analysis module 500, or the image analysis module 500 can access the slice training data (e.g., from a scanner, from a data storage device, from a database, from a memory, etc.) for training. Predetermined slice training data may include tissue slices that have been previously identified (e.g., verified to correspond to the same tissue block). Predetermined slice training data and predetermined block training data may be linked (e.g., in a data storage device, in a memory, etc.).

[0083] Based on block training data and slice training data, the image analysis module 500 generates a tissue block training dataset for training. Furthermore, the image analysis module 500 is trained using the tissue block training dataset. The image analysis module 500 can be trained to recognize the similarity level between first image data and second image data. The image analysis module 500 can generate output including a representation of the similarity between the first image data and the second image data (e.g., letters, numbers, alphanumeric, or symbol representations).

[0084] In some implementations, training the image analysis module 500 may include training a machine learning model, such as a neural network, to determine relationships between different image data. The resulting trained machine learning model may include a set of weights or other parameters, and different subsets of the weights may correspond to different input vectors. For example, the weights may be an encoded representation of the pixels of an image. Furthermore, the image analysis system may provide the trained image analysis module 500 for image processing. In some implementations, the process may be repeated, wherein different image analysis modules 500 are generated and trained for different data domains, different users, etc. For example, a separate image analysis module 500 may be trained for each of the multiple data domains in which the image analysis system is configured to operate.

[0085] Illustratively, an image analysis system may include and implement one or more imaging algorithms. For example, the one or more imaging algorithms may include one or more of the following algorithms: image differencing algorithms, spatial analysis algorithms, pattern recognition algorithms, shape comparison algorithms, color distribution algorithms, blob detection algorithms, template matching algorithms, SURF feature extraction algorithms, edge detection algorithms, keypoint matching algorithms, histogram comparison algorithms, or semantic texel forest algorithms. Image differencing algorithms can identify one or more differences between first image data and second image data. Image differencing algorithms can identify differences between first image data and second image data by identifying differences between each pixel of each image. Spatial analysis algorithms can identify one or more topological or spatial differences between first image data and second image data. Spatial analysis algorithms can identify topological or spatial differences by identifying differences in spatial features associated with first image data and second image data. Pattern recognition algorithms can identify pattern differences between first image data and second image data. Pattern recognition algorithms can identify differences in the patterns of first image data and patterns of second image data. Shape comparison algorithms can analyze one or more shapes of first image data and one or more shapes of second image data and determine whether the shapes match. Shape comparison algorithms can further identify differences in shapes.

[0086] Color distribution algorithms can identify differences in color distribution across first and second image data. Blob detection algorithms can identify regions in the first image data that differ from corresponding regions in the second image data in terms of image properties (e.g., brightness, color). Template matching algorithms can identify portions of the second image data that match a template (e.g., the first image data). SURF feature extraction algorithms can extract features from the first and second image data and compare the features. Features can be extracted based at least in part on the specific importance of the features. Edge detection algorithms can identify the boundaries of objects within the first and second image data. The boundaries of objects in the first image data can be compared with the boundaries of objects in the second image data. Keypoint matching algorithms can extract specific keypoints from the first and second image data and compare the keypoints to identify differences. Histogram comparison algorithms can identify differences between the color histograms associated with the first and second image data. Semantic texel forest algorithms can compare the semantic representations of the first and second image data to identify differences. It should be understood that image analysis systems can implement more, fewer, or different imaging algorithms. Furthermore, the image analysis system can implement any imaging algorithm to identify differences between the first and second image data. Based on the identified differences, the image analysis system can determine the likelihood that the first and second image data are a match (e.g., corresponding to the same tissue block).

[0087] Figure 6 A schematic diagram of an image analysis module 600 comprising multiple neural networks according to aspects of this disclosure is depicted. The image analysis module 600 may be a Siamese network capable of measuring the similarity between the feature space (e.g., embedding) of block image data and the feature space of slice image data. The image analysis module 600 may include a first convolutional neural network 604A for processing block image data 602A and a second convolutional neural network 604B for processing slice image data 602B. In some embodiments, the first convolutional neural network 604A and the second convolutional neural network 604B may be the same Siamese architecture. The image analysis module 600 may include a third convolutional neural network 606 for processing the outputs of the first convolutional neural network 604A and the second convolutional neural network 604B. In some embodiments, one or more of the convolutional neural networks may be generative adversarial networks. It should be understood that the image analysis module 600 may include more, fewer, or different convolutional neural networks.

[0088] The block image data 602A and the slide image data 602B can differ based on rotation, flipping, color, contour, etc. For example, a tissue slide attached to a glass slide can be rotated or flipped relative to the tissue block. Furthermore, the tissue slide can be stained with one or more colors different from the tissue block. In other examples, the tissue slide may have a different perimeter than the tissue block. To identify the level of similarity between the block image data 602A and the slide image data 602B, the image analysis module 600 can perform image analysis.

[0089] Image analysis module 600 can perform multi-faceted image analysis. Furthermore, image analysis module 600 can divide the image analysis into multiple aspects. One or more aspects of the image analysis may include preprocessing block image data 602A and / or slice image data 602B. In some embodiments, image analysis module 600 may include one or more convolutional neural networks to perform preprocessing. Image analysis module 600 may include image registration algorithms to modify (e.g., rotate, flip, etc.) one or more of block image data 602A and / or slice image data 602B. By modifying one or more of block image data 602A and / or slice image data 602B, the image registration algorithm can generate modified block image data and / or modified slice image data rotated and / or flipped in the same manner. Image analysis module 600 may include image segmentation algorithms to extract tissue contours of block image data 602A and / or slice image data 602B. In some embodiments, a U-net convolutional neural network can perform the image segmentation algorithm.

[0090] A first convolutional neural network 604A can learn the feature space of block image data 602A, and a second convolutional neural network 604B can learn the feature space of slice image data 602B. The first convolutional neural network 604A and the second convolutional neural network 604B can use one or more imaging algorithms to learn the feature space. The first convolutional neural network 604A and the second convolutional neural network 604B can share one or more weights to learn the feature space. In some embodiments, one or more of the first convolutional neural network 604A or the second convolutional neural network 604B can perform at least a portion of preprocessing. The first convolutional neural network 604A and the second convolutional neural network 604B can output the feature space to a third convolutional neural network 606 for additional processing. The third convolutional neural network 606 can compare the feature spaces of block image data 602A and slice image data 602B for similarity. Based at least in part on this comparison, the third convolutional neural network 606 can determine the probability that a slide of a slice of a tissue block corresponds to the tissue block (e.g., whether the slice can be generated from the tissue block).

[0091] Analyze block image data and slice image data

[0092] Figure 7 Method 700, performed by an image analysis system, is illustrated according to some examples of the disclosed techniques. The image analysis system may be similar to, for example, image analysis system 104, and may include an image analysis module executing one or more image analysis algorithms, a slicer, a coverslipper, a staining machine, one or more imaging devices, etc. It will be understood that method 700 may be performed by different means (e.g., a computing device). Process 700 may begin at block 701. Process 700 may begin automatically upon receiving image data.

[0093] In block 702, the image analysis system acquires first block image data from a first imaging device and first slice image data from a second imaging device. The first imaging device can scan a tissue sample block and generate block image data based on the scan. The tissue sample block can be further sliced ​​to generate one or more slices of the tissue sample block. One or more slices of the tissue sample block can be attached to one or more slides. The second imaging device can scan one or more slides corresponding to slices from one or more slices of the tissue sample block and generate slice image data based on the scan of the slides. The tissue sample block may include one or more of paraffin-embedded tissue sample blocks, OCT-embedded tissue sample blocks, frozen tissue sample blocks, or fresh tissue sample blocks. The first block image data and the first slice image data may correspond to the tissue sample block. The image analysis system may include a coverslip machine and / or a staining machine. The staining machine can stain the slices of the tissue sample block to generate stained slices. The coverslip machine can display the stained slices in the slides. In some embodiments, the first imaging device and the second imaging device may be different imaging devices. In other embodiments, the first imaging device and the second imaging device may be the same imaging device. One or more of the first imaging device or the second imaging device may be a slicer, a coverslipper, a case folder imaging station, a single slide imaging station, a dedicated low-resolution imaging device, or a direct part (“DP”) scanner.

[0094] In block 704, the image analysis system links first block image data and first slice image data to a tissue sample block. The image analysis system may link the first block image data and first slice image data based on determining that the first block image data and first slice image data correspond to a tissue sample block. To determine that the first block image data and first slice image data correspond to a tissue sample block, one or more of the tissue sample block and slice may be associated with an identifier. The linking of the first block image data and first slice image data may be based at least in part on determining that the identifier of the tissue sample block corresponds to the identifier of the slice. The identifier may be one or more of a tag, radio frequency identification (“RFID”) tag, Bluetooth tag, identifier, barcode, mark, sign, or stamp. In some embodiments, the image analysis system may receive input indicating that the first block image data and first slice image data are correctly linked together (e.g., from a data storage device, from a memory, etc.). The image analysis system may link the first block image data and first slice image data and may store information identifying the first block image data and first slice image data (e.g., in a data storage device, in a memory, etc.).

[0095] In box 706, the image analysis system provides first block image data, first slice image data, and information indicating that the first block image data and the first slice image data are correctly linked together as input to a machine learning algorithm. While providing the first block image data, the first slice image data, and the information indicating that the first block image data and the first slice image data are correctly linked together, the image analysis system can train the machine learning algorithm to generate a trained machine learning algorithm.

[0096] In box 708, the image analysis system uses inputs to train a machine learning algorithm. Based on the trained machine learning algorithm, the image analysis system can generate a trained machine learning algorithm. Furthermore, the training of the machine learning algorithm can be at least partially based on multiple user data sets. These multiple user data sets can include one or more responses from one or more users to one or more recommendations. For example, if a particular user rejects a recommendation to link specific patch image data and specific slice image data, the image analysis system may not recommend linking the patch image data and slice image data together for subsequent users. Furthermore, the image analysis system can modify subsequent recommendations for the user based on the rejection (e.g., the image analysis may be less likely to make a recommendation to link the patch image data and slice image data). The machine learning algorithm can include one or more of the following algorithms: image differencing algorithms, spatial analysis algorithms, pattern recognition algorithms, shape comparison algorithms, color distribution algorithms, blob detection algorithms, template matching algorithms, SURF feature extraction algorithms, edge detection algorithms, keypoint matching algorithms, histogram comparison algorithms, or semantic veneer forest algorithms. In some embodiments, the trained machine learning algorithm can be a trained convolutional neural network. The machine learning algorithm may include extracting a first plurality of features from a second block of image data and extracting a second plurality of features from a second slice of image data. The first plurality of features and the second plurality of features may each include one or more of a plurality of filters or a plurality of patterns. Furthermore, the machine learning algorithm may compare the first plurality of features and the second plurality of features.

[0097] In box 710, the image analysis system obtains second block image data from the first imaging device and second slice image data from the second imaging device. The second block image data and the second slice image data can be linked together.

[0098] In block 712, the image analysis system performs image analysis on the second block image data and the second slice image data using a trained machine learning algorithm. In some embodiments, the image analysis system may perform image analysis on the third slice image data and the second block image data. In other embodiments, the image analysis system may perform image analysis on the third slice image data and the second slice image data. The image analysis system may perform image analysis on any combination of slice image data and block image data.

[0099] In block 714, the image analysis system determines a confidence value indicating whether the second block of image data and the second slice of image data are correctly linked. In some embodiments, the image analysis system may obtain a confidence threshold associated with a user and compare the confidence value with the confidence threshold. Based on this comparison, the image analysis system may generate a recommendation for the user. The recommendation may include a similarity score, ranking, etc. Furthermore, the image analysis system may transmit the recommendation to a user computing device associated with the user (e.g., via an application programming interface) for presentation to the user. In some embodiments, the image analysis system may cause the recommendation to be displayed via the user computing device. Furthermore, the image analysis system may obtain a response to the recommendation indicating that the user has accepted it and adjust the confidence threshold based on the response. In some embodiments, the response may indicate that the user has rejected the recommendation, and the image analysis system may adjust the confidence threshold based on the response. Furthermore, the image analysis system may adjust a machine learning algorithm based on the response. In some embodiments, the confidence threshold may be associated with multiple users. Furthermore, the image analysis system may determine the confidence threshold based on multiple characteristics associated with multiple users.

[0100] Figure 8 An exemplary computing system 800 configured to perform the described process and implement the features described above is illustrated. In some embodiments, the computing system 800 may include: one or more computer processors 802, such as a physical central processing unit (“CPU”); one or more network interfaces 804, such as a network interface card (“NIC”); one or more computer-readable media drives 806, such as a high-density disk (“HDD”), a solid-state drive (“SDD”), a flash memory drive, and / or other persistent non-transitory computer-readable media; an input / output device interface 808, such as an input / output (“IO”) interface for communicating with one or more microphones; and one or more computer-readable storage devices 810, such as random access memory (“RAM”) and / or other volatile non-transitory computer-readable media.

[0101] Network interface 804 can provide connectivity to one or more networks or computing systems. Computer processor 802 can receive information and instructions from other computing systems or services via network interface 804. Network interface 804 can also directly store data into computer-readable storage device 810. Computer processor 802 can communicate with computer-readable storage device 810, execute instructions, and process data in computer-readable storage device 810, etc.

[0102] Computer-readable storage device 810 may include computer program instructions that a computer processor 802 executes to implement one or more embodiments. Computer-readable storage device 810 may store an operating system 812 that provides computer program instructions for the computer processor 802 to use for the general management and operation of computing system 800. Computer-readable storage device 810 may also include computer program instructions and other information for implementing aspects of this disclosure. For example, in one embodiment, computer-readable storage device 810 may include a machine learning model 814. As another example, computer-readable storage device 810 may include image data 816. In some embodiments, multiple computing systems 800 may communicate with each other via corresponding network interfaces 804 and may implement multiple sessions in parallel (e.g., each computing system 800 may execute a portion of a single instance of process 700), each session having corresponding connection parameters (e.g., each computing system 800 may execute one or more separate instances of process 700).

[0103] certain terms

[0104] Orientational terms such as “top,” “bottom,” “proximal,” “far,” “longitudinal,” “transverse,” and “end” are used in the context of the illustrated examples. However, this disclosure should not be limited to the illustrated orientations. In fact, other orientations are possible and within the scope of this disclosure. Terms relating to circles, such as diameter or radius, as used herein should be understood not to require a perfectly circular structure, but rather to be applicable to any suitable structure having a cross-sectional area measurable from one side to the other. Generally shape-related terms such as “circular,” “cylindrical,” “semi-circular,” or “semi-cylindrical,” or any related or similar terms, do not need to strictly conform to the mathematical definition of a circle, cylinder, or other structure, but may cover reasonably similar structures.

[0105] Unless otherwise expressly stated or otherwise understood in the context of use, conditional languages ​​such as “can,” “may,” “possibly,” or “may” are generally intended to express whether certain examples include or exclude certain features, elements, and / or steps. Therefore, such conditional languages ​​generally do not imply that one or more examples necessarily require features, elements, and / or steps.

[0106] Unless otherwise expressly stated, connective language such as “at least one of X, Y, and Z” is, when used in conjunction with the context, otherwise understood to generally convey that an item, term, etc., can be X, Y, or Z. Therefore, such connective language generally does not imply that certain examples require the presence of at least one of X, at least one of Y, and at least one of Z.

[0107] As used herein, the terms “about,” “approximately,” and “substantially” refer to quantities that are close to the specified quantity and still perform the expected function or achieve the expected result. For example, in some instances, as the context may indicate, the terms “about,” “approximately,” and “substantially” may refer to quantities that are less than or equal to 10% of the specified quantity. The term “generally” as used herein refers to a value, quantity, or characteristic that primarily comprises or leans towards a particular value, quantity, or characteristic. As an example, in some instances, as the context may indicate, the term “generally parallel” may refer to something that deviates from exact parallelism by less than or equal to 20 degrees. All ranges include endpoints.

[0108] Overview

[0109] Several illustrative examples of comparing block histological samples and section histological samples have been disclosed. Although this disclosure has been described with reference to certain illustrative examples and uses, other examples and uses, including those that do not provide all the features and advantages set forth herein, are also within the scope of this disclosure. Components, elements, features, actions, or steps may be arranged or performed differently from those described, and components, elements, features, actions, or steps may be combined, merged, added, or omitted in various examples. All possible combinations and sub-combinations of the elements and components described herein are intended to be included in this disclosure. No single feature or set of features is necessary or indispensable.

[0110] Certain features described in this disclosure in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Additionally, although features may function in certain combinations as described above, one or more features from a claimed combination may be removed from said combination in some cases, and said combination may be claimed as a sub-combination or a variation of a sub-combination.

[0111] Any step, process, structure, and / or device disclosed or illustrated in one example of this disclosure may be combined with or used (or replaced) with any other step, process, structure, and / or device disclosed or illustrated in different examples or flowcharts. The examples described herein are not intended to be separate and isolated from each other. Combinations, variations, and some implementations of the disclosed features are within the scope of this disclosure.

[0112] While operations may be depicted in the accompanying drawings or described in the specification in a specific order, these operations do not need to be performed in the specific order shown or sequentially, or all operations need to be performed to achieve the desired result. Other operations not depicted or described may be incorporated into the exemplary methods and processes. For example, one or more additional operations may be performed before, after, simultaneously with, or between any of the described operations. Additionally, operations may be rearranged or reordered in some implementations. Moreover, the separation of various components in the above implementations should not be construed as requiring this separation in all implementations, and it should be understood that the described components and systems may be integrated substantially together in a single product or packaged into multiple products. Furthermore, some implementations are within the scope of this disclosure.

[0113] Furthermore, while illustrative examples have been described, any examples with equivalent elements, modifications, omissions, and / or combinations are also within the scope of this disclosure. Additionally, although certain aspects, advantages, and novel features are described herein, not all of these advantages can necessarily be achieved based on any particular example. For example, some examples within the scope of this disclosure achieve one or a set of advantages as taught herein, without necessarily achieving other advantages taught or suggested herein. Moreover, some examples may achieve advantages different from those taught or suggested herein.

[0114] Several examples have been described in conjunction with the accompanying drawings. The drawings are drawn to scale and / or shown, but this scale should not be limiting, as dimensions and scales other than those shown are contemplated and within the scope of the invention. Distances, angles, etc., are illustrative only and do not necessarily have an exact relationship to actual dimensions and the layout of the illustrated apparatus. Components may be added, removed, and / or rearranged. Furthermore, any specific feature, aspect, method, property, characteristic, quality, attribute, element, etc., disclosed herein in conjunction with the various examples can be applied to all other examples set forth herein. Additionally, any method described herein can be practiced using any apparatus suitable for performing the described steps.

[0115] For the purpose of summarizing this disclosure, certain aspects, advantages, and features of the invention have been described herein. Not all or any of these advantages must be realized according to any particular example of the invention disclosed herein. No aspect of this disclosure is necessary or essential. In many examples, apparatus, systems, and methods may be configured differently from those illustrated in the accompanying drawings or description herein. For example, various functionalities provided by the illustrated modules may be combined, rearranged, added, or removed. In some implementations, additional or different processors or modules may perform some or all of the functionalities described with reference to the examples illustrated and described in the figures. Many implementation variations are possible. Any feature, structure, step, or process disclosed in this specification may be included in any example.

[0116] In summary, various examples comparing block histological samples and slice histological samples have been disclosed. This disclosure extends beyond the specifically disclosed examples to other alternative examples and / or other uses of the examples, as well as certain modifications and equivalents thereof. Furthermore, this disclosure expressly contemplates that various features and aspects of the disclosed examples can be combined or substituted with each other. Therefore, the scope of this disclosure should not be limited to the specific examples disclosed above, but should be determined only by a fair reading of the claims. In some embodiments, the image analysis system disclosed herein can be used to analyze images of samples other than histological samples.

Claims

1. An image analysis device, the image analysis device comprising: A first scanner is configured to scan a tissue sample block and generate block image data based on the scan of the tissue sample block; A second scanner is configured to scan slices from one or more slices obtained by slicing a tissue sample block, and to generate slice image data based on the scan of the slices. as well as A computing device configured to perform the following operations: The block image data is obtained from the first scanner and the slice image data is obtained from the second scanner, the block image data and the slice image data are linked together and linked to the same histological sample; Based on the block image data and the slice image data being linked together, a machine learning algorithm is used to perform image analysis on the block image data and the slice image data, wherein the machine learning algorithm is trained using predetermined training block image data and predetermined training slice image data that are correctly linked to the predetermined training block image data; as well as A confidence value indicating whether the block image data and the slice image data are correctly linked together is determined based on the output of the machine learning algorithm.

2. The image analysis apparatus of claim 1, wherein, in order to scan the tissue sample block, the first scanner is further configured to scan at least one side of the tissue sample block.

3. The image analysis apparatus of claim 1, wherein the predetermined training block image data and the predetermined training slice image data are obtained from at least one of a data storage device, the first scanner, or the second scanner, and the predetermined training block image data and the predetermined training slice image data are linked together, wherein the predetermined training block image data, the predetermined training slice image data, and information indicating that the predetermined training block image data and the predetermined training slice image data are correctly linked together are used to train the machine learning algorithm.

4. The image analysis device as described in claim 1, wherein, In order to obtain the block image data and the slice image data, the computing device is configured to perform the following operations: The block image data and the slice image data are obtained from the data storage device.

5. The image analysis device of claim 4, wherein the block image data and the slice image data are linked together in the data storage device.

6. The image analysis apparatus of claim 1, wherein the computing device is further configured to perform the following operations: Obtain the confidence threshold associated with the user; Compare the confidence value with the confidence threshold; and Recommendations for the user are generated based at least in part on the comparison between the confidence value and the confidence threshold, indicating that the block image data and the slice image data are correctly linked together.

7. The image analysis apparatus of claim 6, wherein the computing device is further configured to perform the following operations: Obtain a response to the recommendation; and Based on the response to the recommendation, at least one of the machine learning algorithm or the confidence threshold is adjusted.

8. The image analysis apparatus of claim 1, wherein the computing device is configured to perform the image analysis using the machine learning algorithm by providing the block image data and the slice image data to a trained convolutional neural network, wherein the trained convolutional neural network is configured to perform the image analysis.

9. The image analysis apparatus of claim 1, wherein the computing device is further configured to perform the following operations: Extracting a first plurality of features from the block image data, wherein the first plurality of features includes one or more of a first plurality of filters or a first plurality of patterns; and Extract a second plurality of features from the sliced ​​image data, wherein the second plurality of features includes one or more of a second plurality of filters or a second plurality of patterns. The image analysis performed using the machine learning algorithm includes comparing the first plurality of features and the second plurality of features.

10. The image analysis apparatus of claim 1, wherein the computing device is configured to perform the image analysis using one or more of the following: Image difference algorithm; Spatial analysis algorithms; Pattern recognition algorithms; Shape comparison algorithm; Color distribution algorithm; Blob detection algorithm; Template matching algorithm; SURF feature extraction algorithm; Edge detection algorithm; Keypoint matching algorithm; Histogram comparison algorithm; or Semantic text forest algorithm.

11. The image analysis device as claimed in claim 1, further comprising: A staining machine configured to stain the sections of the tissue sample block to generate stained sections; as well as A coverslipper, configured to produce glass slides for the stained sections. The second scanner is configured to scan the stained sections.

12. The image analysis apparatus of claim 1, wherein the tissue sample block is associated with a first identifier, and the slice of the tissue sample block is associated with a second identifier, wherein the link between the block image data and the slice image data is at least partially based on the first identifier corresponding to the second identifier.

13. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform the following operations: Block image data is obtained from a first scanner and slice image data is obtained from a second scanner, the block image data and the slice image data are linked together and linked to the same histological sample, wherein the first scanner is configured to scan a block of tissue sample and generate the block image data based on the scan of the tissue sample block, and wherein the second scanner is configured to scan slices from one or more slices obtained by slicing the tissue sample block and generate the slice image data based on the scan of the slices; Based on the block image data and the slice image data being linked together, a machine learning algorithm is used to perform image analysis on the block image data and the slice image data, wherein the machine learning algorithm is trained using predetermined training block image data and predetermined training slice image data that are correctly linked to the predetermined training block image data; as well as A confidence value indicating whether the block image data and the slice image data are correctly linked together is determined based on the output of the machine learning algorithm.

14. The non-transitory computer-readable medium of claim 13, wherein the tissue sample block comprises one or more of the following: Paraffin-embedded tissue sample blocks; OCT-embedded tissue sample blocks; Frozen tissue sample blocks; or Fresh tissue sample block.

15. The non-transitory computer-readable medium of claim 13, wherein the non-transitory computer-readable medium stores other computer-executable instructions that, when executed by the one or more computing devices, cause the one or more computing devices to perform the following operations: A second slice image data is obtained from a third scanner and a second block image data is obtained from a first scanner, wherein the third scanner is configured to scan a second slice in one or more slices of the tissue sample block and generate second slice image data based on the scan of the second slice; Link the second block of image data and the second slice of image data together; as well as The second block image data, the second slice image data, and information indicating that the second block image data and the second slice image data are correctly linked together are provided as other inputs to the machine learning algorithm.

16. The non-transitory computer-readable medium of claim 13, wherein the first scanner corresponds to a slicer, and wherein the second scanner corresponds to one or more of the following: The slicer; Cover chipping machine; Case study folder imaging station; Single-slide imaging station; Dedicated low-resolution imaging device; or Digital pathology scanner.

17. The non-transitory computer-readable medium of claim 13, wherein the non-transitory computer-readable medium stores other computer-executable instructions that, when executed by the one or more computing devices, cause the one or more computing devices to perform the following operations: Obtain a response to the recommendation; and The machine learning algorithm or the confidence threshold is adjusted based on the response to the recommendation.

18. A computer-implemented method, the computer-implemented method comprising: Block image data is obtained from a first scanner and slice image data is obtained from a second scanner, the block image data and the slice image data are linked together and linked to the same histological sample, wherein the first scanner is configured to scan a block of tissue sample and generate the block image data based on the scan of the tissue sample block, and wherein the second scanner is configured to scan slices from one or more slices obtained by slicing the tissue sample block and generate the slice image data based on the scan of the slices; Based on the block image data and the slice image data being linked together, a machine learning algorithm is used to perform image analysis on the block image data and the slice image data, wherein the machine learning algorithm is trained using predetermined training block image data and predetermined training slice image data that are correctly linked to the predetermined training block image data; as well as A confidence value indicating whether the block image data and the slice image data are correctly linked together is determined based on the output of the machine learning algorithm.

19. The computer-implemented method of claim 18, wherein the tissue sample block comprises one or more of the following: Paraffin-embedded tissue sample blocks; OCT-embedded tissue sample blocks; Frozen tissue sample blocks; or Fresh tissue sample block.

20. The computer-implemented method of claim 18, further comprising: A second slice image data is obtained from a third scanner and a second block image data is obtained from a first scanner, wherein the third scanner is configured to scan a second slice in one or more slices of the tissue sample block and generate second slice image data based on the scan of the second slice; Link the second block of image data and the second slice of image data together; as well as The second block image data, the second slice image data, and information indicating that the second block image data and the second slice image data are correctly linked together are provided as other inputs to the machine learning algorithm.

Images