Microscope image display device, microscope image display method, medical image display device, and medical image display method

Abstract

A microscope image display device 1 comprises: an original image acquisition unit 11 that acquires an original image from a microscope; an interest object inference unit 12 that uses machine learning to infer an object of interest from the original image; an inference result evaluation unit 13 that evaluates reliability of the inference result of the inferred object of interest; and a display unit 14 that displays the position of the inferred object of interest on the original image in a state of being superimposed on the original image in a manner of displaying according to the reliability of the inference result.

Description

Microscope image display device, microscope image display method, medical image display device, and medical image display method

[0001] The present disclosure relates to a microscope image display device, a microscope image display method, a medical image display device, and a medical image display method.

[0002] An image diagnosis support system has been proposed that provides diagnostic hints based on information obtained from tomographic images, thereby supporting users so that even users with no or insufficient medical knowledge can smoothly proceed with diagnosis (see, for example, Patent Document 1). Also, a technology has been proposed that applies a convolutional neural network (CNN) to histological images to identify regions of interest (see, for example, Patent Document 2). Furthermore, a technology has been proposed that generates clinical parameters by acquiring images of germ cell structures and providing them to a neural network (see, for example, Patent Document 3).

[0003] JP 2022-160629, JP 2022-534155, JP 2023-541841

[0004] O. Ronneberger, P. Fischer, and T. Brox: U-net: Convolutional networks for biomedical image segmentation, in International Conference on Medical image computing and computer-assisted intervention, Springer, pp. 234-241 (2015). Gal Y, Ghahramani Z. Dropout as a Bayesian approximation: representing model uncertainty in deep learning. Proc Machine Learning Research 2016; 48 pp. 1050-1059.

[0005] In the fields of medicine, drug discovery, and biological research, users often observe and manipulate cells and other objects based on microscope images.

[0006] One example is ICSI, a type of artificial insemination in which sperm are injected directly into egg cells. Demand for this method has been increasing in recent years, in part because it is also effective in treating male infertility. However, because direct manipulation of cells requires advanced techniques, ICSI is performed by embryologists, who are professionals who handle embryos. However, training skilled embryologists is time-consuming and costly, and the turnover rate among embryologists has been increasing recently, making the shortage of skilled embryologists a major issue in the medical field. However, if even relatively inexperienced beginners can perform the procedure appropriately, it is expected to become a powerful solution to this problem.

[0007] Furthermore, not only in ICSI, but also in fields such as diagnostic surgery that generally use endoscopic images and medical images of lesions such as cancer cells, or areas targeted for treatment or surgery, it is desirable for relatively inexperienced doctors and clinical technicians to be able to carry out appropriate processing.

[0008] The technology disclosed herein has been developed in light of this situation, and its purpose is to provide support technology that enables even inexperienced beginners to perform appropriate procedures using microscopic images.

[0009] Another object of the present disclosure is to provide a support technology that enables even inexperienced beginners to perform appropriate treatments using medical images.

[0010] In order to solve the above problem, a microscope image display device according to one embodiment of the present invention comprises an original image acquisition unit that acquires an original image from a microscope, an object of interest estimation unit that uses machine learning to estimate an object of interest from the original image, an estimation result evaluation unit that evaluates the reliability of the estimation result for the estimated object of interest, and a display unit that displays the position of the estimated object of interest superimposed on the original image in a manner that corresponds to the reliability of the estimation result.

[0011] In one embodiment, the original image may be an image of an embryo used for ICSI, and the object of interest may be a polar body.

[0012] In one embodiment, the reliability of the estimation result is evaluated based on the ambiguity of the estimation result, and the display unit may display the estimated position of the object of interest on the original image by superimposing a marker on the object of interest to indicate the position of the object of interest and changing the transparency of the marker depending on the ambiguity.

[0013] In one embodiment, the transparency may be a function that varies depending on the fuzziness.

[0014] In one embodiment, the function may be an upwardly convex increasing function.

[0015] In one embodiment, the subject of interest estimation unit may extract a portion of the original image as an estimation subject region from the original image and estimate the subject of interest by machine learning.

[0016] Another aspect of the present invention is a microscope image display method, which includes an original image acquisition step of acquiring an original image of a microscope, an object of interest estimation step of estimating an object of interest from the original image using machine learning, an estimation result evaluation step of evaluating the reliability of the estimation result of the estimated object of interest, and a display step of superimposing and displaying the position of the estimated object of interest on the original image in a display manner according to the reliability of the estimation result.

[0017] Yet another aspect of the present invention is a medical image display device comprising: an original image acquisition unit that acquires an original image of a medical image; a subject of interest estimation unit that estimates a subject of interest from the original image using machine learning; an estimation result evaluation unit that evaluates the reliability of the estimation result of the estimated subject of interest; and a display unit that displays the position of the estimated subject of interest on the original image by superimposing it on the original image in a display manner corresponding to the reliability of the estimation result.

[0018] Yet another aspect of the present invention is a medical image display method, comprising: an original image acquisition step of acquiring an original image of a medical image, a subject of interest estimation step of estimating a subject of interest from the original image using machine learning, an estimation result evaluation step of evaluating the reliability of the estimation result of the estimated subject of interest, and a display step of superimposing and displaying the position of the estimated subject of interest on the original image in a display manner according to the reliability of the estimation result.

[0019] Any combination of the above components, and conversion of the present disclosure into a method, device, system, recording medium, computer program, etc., are also valid aspects of the present disclosure.

[0020] According to the present disclosure, it is possible to provide a support technique that enables even inexperienced beginners to perform appropriate treatment using microscopic images.

[0021] 10 is a schematic diagram showing the procedure of ICSI. FIG. 11 is a schematic diagram showing the gaze point of an experienced operator at each stage when performing ICSI. FIG. 12 is a functional block diagram of a microscopic image display device according to a first embodiment. FIG. 13 is a diagram showing the structure of U-Net used by the microscopic image display device of FIG. 3. FIG. 14 is a graph showing the relationship between ambiguity U and transparency α. FIG. 15 is a diagram showing the pre-processing procedure of the object-of-interest estimation unit of the microscopic image display device according to the third embodiment. FIG. 16 is a flowchart showing the processing procedure of a microscopic image display method according to a sixth embodiment. FIG. 17 is a diagram showing the results of an evaluation experiment. FIG. 18 is a photograph of the pancreatic region extracted from surgical footage of laparoscopic gastrectomy.

[0022] Preferred embodiments will be described below with reference to the drawings. The same or equivalent components, parts, and processes shown in each drawing will be designated by the same reference numerals, and redundant descriptions will be omitted where appropriate. Furthermore, the embodiments are merely examples and do not limit the invention, and all features and combinations thereof described in the embodiments are not necessarily essential to the invention.

[0023] In addition, the dimensions (thickness, length, width, etc.) of each member shown in the drawings may be enlarged or reduced as appropriate for ease of understanding. Furthermore, the dimensions of multiple members do not necessarily represent the relative size of each other, and even if a member A is depicted as being thicker than another member B in the drawings, member A may actually be thinner than member B.

[0024] [Principle of ICSI] Figure 1 is a schematic diagram showing the ICSI procedure. (1) to (6) are microscope images at each stage. Here, the x-axis runs from left to right on the page, the y-axis runs from bottom to top, and the z-axis runs from back to front (the same applies below unless otherwise noted). First, in (1), the operator grasps the embryo with a pipette and places it in the center of the microscope's field of view. Next, in (2), the operator rotates the embryo around the x-axis to search for the polar body. As a result, in (3), the polar body is found. Next, in (4), the operator rotates the embryo around the z-axis. As a result, in (5), the polar body is positioned at either 12 o'clock or 6 o'clock. Finally, in (6), the operator holds the embryo and performs the injection.

[0025] Figure 2 shows the gaze points of an expert operator at each stage when performing ICSI. This represents tacit knowledge of expert skills extracted through gaze analysis and interviews. In Figure 2, the positions indicated by circles or ellipses are gaze points. When rotating the embryo (A), the expert operator gazes at the polar body to find and track it. When holding the embryo (B), the expert operator gazes at the left edge of the cell, as he needs to determine whether the embryo is being held properly based on the deformation of the embryo. Before injection (C), the expert operator gazes at the tips of both pipettes, as they need to be aligned and focused. As such, experts have a specific gaze point for each stage of the operation.

[0026] In contrast, beginner operators are less able to recognize polar bodies and exhibit significant eye movement and variability. Rotating the embryo is particularly important to avoid damaging the spindles near the polar bodies when inserting the pipette, and requires delicate and highly complex manipulation with the pipette. While experts focus their gaze firmly on the area around the polar bodies, beginners tend to frequently glance at the injection pipette, resulting in an unstable gaze. Thus, there is a difference in the degree to which experts and beginners focus their gaze on the polar bodies. Therefore, to enable beginners to perform appropriate operations, it would be beneficial to provide support by estimating the polar body image and displaying it on the microscope screen, thereby guiding the operator's gaze to the polar body's location.

[0027] [Polar Body Image Estimation] To estimate the polar body image in such support, machine learning is effective. For example, semantic segmentation is a technique for dividing an image into regions by classifying it pixel by pixel, and various methods have been proposed. In particular, U-NET is a semantic segmentation model developed for biomedicine (see, for example, Non-Patent Document 1). U-NET is capable of retaining position information through skip connections and can perform image estimation with a small amount of training data, and is therefore expected to be useful for the polar body estimation disclosed herein.

[0028] [Reliability of Estimation] However, estimation using semantic segmentation has the drawback of not being highly accurate for untrained inputs. This drawback is not limited to semantic segmentation, but is a problem that exists to a greater or lesser extent in estimation using machine learning. If such estimation results are simply displayed on a microscope screen, the person receiving assistance will not be able to correctly determine whether the results are reliable. In fields such as medicine and drug discovery, which are directly related to human life and health, particularly high estimation accuracy is required compared to other applications. In these fields, if estimation results with unclear reliability are presented, the person receiving assistance will not know how much to trust the presented assistance, which may ultimately undermine the reliability of the assistance itself.

[0029] In response to this, rather than displaying the estimated position as is, it may be useful to superimpose a marker on the original image in a manner that corresponds to the reliability of the estimation, such as superimposing a marker on the object of interest to indicate its position with low transparency if the reliability of the estimation is high, or with high transparency if the reliability of the estimation is low. By doing so, the subject can, for example, focus their gaze on the estimated position when it is displayed with low transparency with peace of mind, and conversely, when it is displayed with high transparency, focus their gaze on it while keeping in mind that the estimation may be incorrect.

[0030] From the above, it is believed that the key to assisting with microscopic images, particularly those used in the fields of medicine and drug discovery, is to achieve both of the following: (1) estimating the position of the object of interest that should be focused on, and (2) superimposing the estimated position on the original image in a manner that corresponds to the reliability of the estimation.

[0031] [First embodiment] Based on the above findings, the inventors have devised a novel microscopic image display technology. Figure 3 is a functional block diagram of a microscopic image display device 1 according to the first embodiment. The microscopic image display device 1 comprises an original image acquisition unit 11, a subject-of-interest estimation unit 12, an estimation result evaluation unit 13, and a display unit 14.

[0032] The original image acquisition unit 11 acquires a raw microscope image (hereinafter referred to as an "original image"). In the following, an example will be described in which the original image is an image of an embryo that is the object of manipulation by an embryologist, but it goes without saying that the original image is not limited to an embryo.

[0033] The object of interest estimation unit 12 uses machine learning to estimate the polar body, which is the object of interest for the embryologist, from the original image acquired by the original image acquisition unit 11. In the following description, the object of interest estimation unit 12 performs polar body estimation by semantic segmentation using a U-Net. However, it goes without saying that the machine learning method is not limited to semantic segmentation using a U-Net. More specifically, the object of interest estimation unit 12 estimates the polar body using training data created by annotating embryo images into four types: polar body, cytoplasm, zona pellucida, and others. Figure 4 shows the structure of the U-Net used here.

[0034] The estimation result evaluation unit 13 evaluates the reliability of the estimation result of the polar body estimated by the object of interest estimation unit 12. Specifically, the estimation result evaluation unit 13 obtains n models with different structures using Monte Carlo dropout and derives the ambiguity of the estimation from the variance of the n estimation results obtained through these models. More specifically, the estimation result evaluation unit 13 calculates the average value U of the variance of the n outputs for each pixel in the region estimated to be a polar body as the ambiguity of the estimation result. Here, U is expressed by the following formula. where N is the number of pixels estimated to be polar bodies, and x ｉｊ is the value of each pixel (1 if it is a polar body, 0 otherwise). It goes without saying that the specific method for evaluating the reliability of the estimation result is not limited to the above, and any other suitable method may be used.

[0035] The display unit 14 superimposes the position of the polar body estimated by the object of interest estimation unit 12 on the original microscope image (specifically, the embryo) acquired by the original image acquisition unit 11 in a manner that corresponds to the reliability of the estimation result calculated by the estimation result evaluation unit 13.

[0036] As an example, the reliability of the estimation result may be the ambiguity U of the estimation result. In this case, the higher the reliability of the estimation result, the lower the ambiguity U. In this example, the display unit 14 displays the position of the polar body superimposed on the original microscope image in a display manner according to the ambiguity U of the estimation result.

[0037] The display method according to the ambiguity U may be, for example, a method of changing the transparency of the polar body according to the ambiguity U. More specifically, a marker superimposed on the polar body to indicate its position may be superimposed on the current image with low transparency if the value of the ambiguity U is low (i.e., if the reliability of the estimation is high), and conversely, with high transparency if the value of the ambiguity U is high (i.e., if the reliability of the estimation is low). In other words, the transparency α may be a function that changes according to the ambiguity U.

[0038] Alternatively, the display unit 14 may display the polar body position using a method such as changing the color, size, shape, etc. of the polar body position display depending on the ambiguity U, instead of the transparency.

[0039] Figure 4 shows images of an embryo in which the polar body position is displayed using the microscope image display device 1. (A) is an image when the ambiguity U is low (i.e., when the reliability of the estimation is high), and (B) is an image when the ambiguity U is low (i.e., when the reliability of the estimation is high). The polar body position is displayed with a lower transparency in (A) than in (B). When an embryologist sees this, in the case of (A) they can focus their gaze on the polar body position with peace of mind, but in the case of (B) they can focus their gaze there while keeping in mind that the estimated polar body position may be incorrect.

[0040] As described above, according to this embodiment, a target of interest in a microscopic image is estimated by machine learning, and the position of the target of interest is displayed superimposed on the original image in a display manner according to the reliability of the estimation. This allows even inexperienced beginners to perform appropriate treatment using microscopic images.

[0041] [Second embodiment] The display unit 14 of the microscopic image display device 1 of the second embodiment displays the position of the polar body estimated by the object of interest estimation unit 12 by superimposing it on the original image of the embryo acquired by the original image acquisition unit 11, while changing the transparency α according to the ambiguity U of the estimation result calculated by the estimation result evaluation unit 13. In this embodiment, the relationship between the ambiguity U and the transparency α is as follows: FIG. 5 shows the relationship between the ambiguity U and the transparency α.

[0042] As shown in FIG. 5, α increases rapidly near U=0, and then the rate of increase gradually slows down. In other words, the transparency α is an upwardly convex increasing function of the ambiguity U. In addition, in equation (2), α=1 when U>1, which means that the estimation result is not displayed when U>1. By defining the relationship between α and U using equation (2), a viewer of the displayed image can more intuitively recognize the reliability of the estimation.

[0043] According to this embodiment, the reliability of the estimation can be displayed in an intuitive and easy-to-understand manner.

[0044] It goes without saying that the relationship between the ambiguity U and the transparency α is not limited to formula (2). For example, α does not necessarily have to be proportional to the square root of U, as long as it is an upwardly convex increasing function of U. In this case, the same effect as above can be obtained. Alternatively, α may be proportional to U, or α may be a downwardly convex increasing function of U.

[0045] Third Embodiment Generally, estimation using semantic segmentation such as U-Net requires a long calculation time. However, if the input data size can be reduced, the calculation time can be reduced and real-time estimation can be ensured.

[0046] The object-of-interest estimation unit 12 of the microscopic image display device 1 of the third embodiment extracts a portion of the original image as an estimation target region from the original image, performs preprocessing to reduce the resolution of the estimation target region (resizing), and then estimates the object of interest by machine learning. This reduces the data size of the input required to estimate the object of interest.

[0047] Figure 6 shows the preprocessing procedure. (1) is a photograph of the entire original image, with a pixel count of 648 pixels x 488 pixels. From this, a rectangular area is cut out as the estimation target area. (2) is an enlarged photograph of the cut-out estimation target area, with a pixel count of 256 pixels x 256 pixels. (3) is a photograph of the area of ​​interest (2) resized to a pixel count of 80 pixels x 80 pixels. This reduces the calculation time required for estimating the target of interest, enabling real-time estimation at 7 to 10 fps.

[0048] The above numerical values ​​are merely examples and are not limiting.

[0049] According to this embodiment, by reducing the input data size, it is possible to reduce the calculation time required to estimate the subject of interest and ensure real-time performance.

[0050] [Fourth embodiment] When an operator performs multiple operation steps on an original image, the area of ​​interest may move with each step. For example, in the step of rotating an embryo, the operator focuses on the polar body in the embryo, in the step of holding an embryo with a pipette, the operator focuses on the area of ​​the embryo close to the holding pipette, and in the step of injection, the operator focuses on the tips of both pipettes (injection pipette and holding pipette).

[0051] In order to support a series of operations including such multiple operation steps, the microscopic image display device of the embodiment may further include an operation determination unit that determines the operation step to be performed on the original image, as follows. That is, when the operation determination unit detects from the original image that the operator has finished the operation step being performed, it determines whether there is a next operation step. When the operation determination unit determines that there is no next operation step, it ends the processing, and when it determines that there is a next operation step, it causes the object of interest estimation unit to estimate the object of interest that should be focused on in the next operation step.

[0052] According to this embodiment, when an operator performs multiple operation steps on an original image, even if the area of ​​interest moves at each step, the position of the object of interest can be estimated at each step and displayed together with its reliability.

[0053] Fifth Embodiment In addition to the fourth embodiment, the operation determination unit may further include a list of objects of interest for each operation step. In this case, when the operation determination unit determines the operation step being executed, the operation determination unit causes the object of interest estimation unit to estimate the object of interest based on the list of objects of interest.

[0054] According to this embodiment, when the operation step being executed is determined, the subject of interest can be estimated based on a predetermined list.

[0055] In the above embodiment, the display unit may display the position of the subject of interest estimated by the subject of interest estimation unit on the original image using a marker. In this case, the marker is displayed superimposed on the position of the subject of interest on the original image with a transparency according to the level of reliability evaluated by the estimation result evaluation unit.

[0056] According to this embodiment, the reliability of the estimation can be displayed in an intuitive and easy-to-understand manner.

[0057] 7 is a flowchart showing the processing steps of a microscopic image display method according to a sixth embodiment of the present invention, which includes step S11 of acquiring an original image, step S12 of estimating an object of interest, step S13 of evaluating the estimation result, and step S14 of displaying the result.

[0058] In this method, in step S11 of acquiring an original image, an original microscope image is acquired using an original image acquisition unit or the like. Next, in step S12, the method uses machine learning to estimate an object of interest from the original image acquired in step S11 using an object of interest estimation unit or the like. Next, in step S13, the method uses an estimation result evaluation unit or the like to evaluate the reliability of the estimation result of the object of interest estimated in step S12. Finally, in step S14, the method uses a display unit or the like to superimpose and display the position of the object of interest estimated in step S12 on the original microscope image acquired in step S11 in a display manner corresponding to the reliability of the estimation result calculated in step S13.

[0059] According to this embodiment, even an inexperienced beginner can perform appropriate treatment using a microscope image.

[0060] Seventh Embodiment In order to support a series of operations including such multiple operation steps, the microscopic image display method of the embodiment may further include an operation determination step for determining the operation step to be performed on the original image, as follows: That is, the operation determination step determines whether or not there is a next operation step when the operator detects from the original image that the operation step currently being performed has ended. If the operation determination step determines that there is no next operation step, it terminates the processing, and if it determines that there is a next operation step, it causes the object of interest estimation unit to estimate an object of interest that should be focused on in the next operation step.

[0061] According to this embodiment, when an operator performs multiple operation steps on an original image, even if the area of ​​interest moves at each step, the position of the object of interest can be estimated at each step and displayed together with its reliability.

[0062] Eighth Embodiment In contrast to the seventh embodiment, the operation determination step may further include a list of objects of interest for each operation step. In this case, when the operation determination step determines the operation step being executed, the operation determination step causes step S12 to estimate the object of interest based on the list of objects of interest.

[0063] According to this embodiment, when the operation step being executed is determined, the subject of interest can be estimated based on a predetermined list.

[0064] In the above embodiment, the display step may display the position of the subject of interest estimated by the subject of interest estimation step on the original image with a marker. In this case, the marker is displayed superimposed on the position of the subject of interest on the original image with a transparency according to the level of reliability evaluated by the estimation result evaluation step.

[0065] According to this embodiment, the reliability of the estimation can be displayed in an intuitive and easy-to-understand manner.

[0066] In the above explanation, we focused on embryo images used in ICSI as raw images and polar bodies as the target of interest. However, the technology disclosed herein can also be applied to medical procedures using endoscopic images and other medical images. In such cases, the target of interest can be, for example, a lesion site such as cancer cells, or a target site for treatment or surgery.

[0067] Ninth Embodiment A ninth embodiment is a medical image display device that includes an original image acquisition unit that acquires an original image of a medical image, a subject-of-interest estimation unit that estimates a subject-of-interest from the original image using machine learning, an estimation result evaluation unit that evaluates the reliability of the estimation result of the estimated subject-of-interest, and a display unit that displays the position of the estimated subject-of-interest on the original image by superimposing it on the original image in a display manner according to the reliability of the estimation result.

[0068] According to this embodiment, a target of interest in a medical image is estimated by machine learning, and the position of the target of interest is displayed superimposed on the original image in a display manner according to the reliability of the estimation. This allows even inexperienced beginners to perform appropriate treatment using medical images.

[0069] Tenth Embodiment A tenth embodiment is a medical image display method, which includes an original image acquisition step of acquiring an original image of a medical image, a subject-of-interest estimation step of estimating a subject-of-interest from the original image using machine learning, an estimation result evaluation step of evaluating the reliability of an estimation result of the estimated subject-of-interest, and a display step of superimposing and displaying the position of the estimated subject-of-interest on the original image in a display manner according to the reliability of the estimation result.

[0070] According to this embodiment, a target of interest in a medical image is estimated by machine learning, and the position of the target of interest is displayed superimposed on the original image in a display manner according to the reliability of the estimation. This allows even inexperienced beginners to perform appropriate treatment using medical images.

[0071] [Evaluation Experiment] The inventors conducted an evaluation experiment to confirm the effectiveness of the present disclosure. The experiment contents are as follows: Task: Rotational manipulation of an embryo. Specifically, positioning the hidden polar body at the 12 o'clock or 6 o'clock direction. Conditions: (1) Display of estimated image: No (conventional method) (2) Display of estimated image: Yes, display according to the reliability of the estimation: No (conventional method) (3) Display of estimated image: Yes, display according to the reliability of the estimation: Yes (method of the present disclosure) Subjects: Four adults with no experience in fine manipulation Evaluation method: Questionnaire to subjects. Specifically, responses were made to the following two questions on a 7-point scale: (A) Was it easy to find the position of the polar body? (B) Can this system be used reliably? (However, only conditions (2) and (3) were considered.)

[0072] The results of this evaluation experiment are shown in Figure 8. (a) shows the ease of polar body recognition, (b) shows the system reliability, and (c) shows the evaluation results for usability. All of these items were improved by the method of the present disclosure, demonstrating the effectiveness of the present disclosure.

[0073] [Example in Laparoscopic Images] Laparoscopic surgery is minimally invasive and has been used in recent years for surgeries on various organs. However, because such surgeries are more difficult than traditional open surgery, development of computer-assisted surgery systems to support these surgeries is underway. The technology disclosed herein is capable of displaying the position of an object of interest in surgical images of laparoscopic gastrectomy in a manner that corresponds to the reliability of the estimation results, and is therefore expected to make a significant contribution to such computer-assisted surgery systems.

[0074] FIG. 9 shows a photograph of a pancreatic region extracted from a surgical video of a laparoscopic gastrectomy. FIG. 9( a) is a laparoscopic image (original image) input to a medical image display device according to an embodiment. FIGS. 9( b) and 9( c) are output images from the medical image display device according to an embodiment. That is, FIG. 9( b) is an image in which the extracted pancreatic region is superimposed and displayed as an opaque image, and FIG. 9( c) is an image in which the estimated object of interest is displayed with its transparency varied according to the reliability of the estimation result. In particular, in FIG. 9( c), the incorrectly extracted upper right region and the boundary of the pancreatic region are displayed semi-transparently.

[0075] In Figure 9(b), the entire image is displayed with the same transparency, making it difficult for inexperienced surgeons to properly determine where to focus their gaze. In contrast, in Figure 9(c), the object of interest is displayed with varying transparency depending on the reliability of the estimation result. This allows even inexperienced surgeons to easily determine, for example, the extent of ambiguity in the extracted region. This ambiguity information provides a clue for surgeons to use when confirming anatomical structures using a computer-assisted surgery system. Therefore, compared to simply presenting the extraction results as in Figure 9(b), this improves the reliability of the computer-assisted surgery system, enabling more reliable and safer surgery.

[0076] In this example, a Bayesian U-Net (see, for example, Non-Patent Documents 1 and 2) is used to extract the pancreatic region from the laparoscopic image. At this time, the ambiguity (uncertainty) of the inference is estimated, and the pancreatic region extracted based on the ambiguity is superimposed on the laparoscopic image. The Bayesian U-Net has the advantage of being able to obtain multiple different outputs by performing dropout not only during learning but also during inference. However, it goes without saying that any suitable method can be used to extract the object of interest, not limited to the Bayesian U-Net.

[0077] [Each Aspect of the Present Disclosure] Each aspect of the present disclosure will be summarized below. A microscope image display device according to one aspect of the present disclosure includes an original image acquisition unit that acquires an original image of a microscope, a subject of interest estimation unit that estimates a subject of interest from the original image using machine learning, an estimation result evaluation unit that evaluates the reliability of the estimation result of the estimated subject of interest, and a display unit that displays the position of the estimated subject of interest on the original image by superimposing it on the original image in a display manner according to the reliability of the estimation result.

[0078] According to this aspect, even an inexperienced beginner can perform appropriate treatment using a microscope image.

[0079] In one embodiment, the original image may be an image of an embryo used for ICSI, and the object of interest may be a polar body.

[0080] According to this embodiment, even an embryologist with little experience can perform appropriate procedures in ICSI.

[0081] In one aspect, the reliability of the estimation result is evaluated based on the ambiguity of the estimation result, and the display unit may superimpose the estimated position of the object of interest on the original image by changing the transparency of the object of interest depending on the ambiguity.

[0082] According to this aspect, the reliability of the estimation can be displayed in an intuitive and easy-to-understand manner.

[0083] In some embodiments, transparency may be a function that varies depending on the fuzziness.

[0084] According to this aspect, the reliability of the estimation can be displayed in a more intuitive and easy-to-understand manner.

[0085] In one embodiment, the function may be an upwardly convex increasing function.

[0086] According to this aspect, the reliability of the estimation can be displayed in a more intuitive and easy-to-understand manner.

[0087] In one aspect, the subject of interest estimation unit may extract a portion of the original image as an estimation subject region from the original image and estimate the subject of interest by machine learning.

[0088] According to this aspect, it is possible to reduce the data size of the input required to estimate the subject of interest and ensure real-time performance.

[0089] A microscope image display method according to one aspect of the present disclosure includes an original image acquisition step of acquiring an original image from a microscope, an object of interest estimation step of estimating an object of interest from the original image using machine learning, an estimation result evaluation step of evaluating the reliability of the estimation result for the estimated object of interest, and a display step of superimposing the position of the estimated object of interest on the original image in a manner corresponding to the reliability of the estimation result.

[0090] According to this aspect, even an inexperienced beginner can perform appropriate treatment using a microscope image.

[0091] A medical image display device according to one aspect of the present disclosure includes an original image acquisition unit that acquires an original image of a medical image, a subject of interest estimation unit that estimates a subject of interest from the original image using machine learning, an estimation result evaluation unit that evaluates the reliability of the estimation result of the estimated subject of interest, and a display unit that displays the position of the estimated subject of interest superimposed on the original image in a manner that corresponds to the reliability of the estimation result.

[0092] According to this aspect, a target of interest in a medical image is estimated by machine learning, and the position of the target of interest is displayed superimposed on the original image in a manner that corresponds to the reliability of the estimation. This allows even inexperienced beginners to perform appropriate treatment using medical images.

[0093] A medical image display method according to one aspect of the present disclosure includes an original image acquisition step of acquiring an original image of a medical image, a subject of interest estimation step of estimating a subject of interest from the original image using machine learning, an estimation result evaluation step of evaluating the reliability of the estimation result of the estimated subject of interest, and a display step of superimposing and displaying the position of the estimated subject of interest on the original image in a manner corresponding to the reliability of the estimation result.

[0094] According to this aspect, a target of interest in a medical image is estimated by machine learning, and the position of the target of interest is displayed superimposed on the original image in a manner that corresponds to the reliability of the estimation. This allows even inexperienced beginners to perform appropriate treatment using medical images.

[0095] The present invention has been described above based on the embodiments. These embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the respective components and treatment processes, and that such modifications are also within the scope of the present invention.

[0096] Any combination of the above-described embodiments and modifications is also useful as an embodiment of the present disclosure. A new embodiment resulting from the combination has the combined effects of the combined embodiments and modifications.

[0097] When understanding the abstract technical ideas of the embodiments, the technical ideas should not be interpreted as being limited to the contents of the embodiments. The above-described embodiments and variations are merely illustrative examples, and many design modifications, such as changes, additions, and deletions of components, are possible. In the embodiments, the contents in which such design modifications are possible are emphasized by adding the notation "embodiment." However, design modifications are also permitted even in contents without such notation.

[0098] The microscopic image display device and microscopic image display method disclosed herein can be widely used in medical, drug discovery, or bio-related academic fields, such as in vitro fertilization, surgery and examination using microscopic images, new drug development, and biological research.

[0099] 1. Microscope image display device, 11. Original image acquisition unit, 12. Object of interest estimation unit, 13. Estimation result evaluation unit, 14. Display unit, S11. Step of acquiring original image, S12. Step of estimating object of interest, S13. Step of evaluating estimation result, S14. Step of displaying.

Claims

1. A microscope image display device comprising: an original image acquisition unit that acquires an original image of a microscope; an object of interest estimation unit that uses machine learning to estimate an object of interest from the original image; an estimation result evaluation unit that evaluates the reliability of the estimation result of the estimated object of interest; and a display unit that displays the position of the estimated object of interest on the original image by superimposing it on the original image in a display manner corresponding to the reliability of the estimation result.

2. A microscope image display device according to claim 1, wherein the original image is an image of an embryo used in intracytoplasmic sperm injection, and the object of interest is a polar body.

3. A microscope image display device as described in claim 1 or 2, characterized in that the reliability of the estimation result is evaluated by the ambiguity of the estimation result, and the display unit displays the estimated position of the object of interest on the original image by superimposing the position of the object of interest on the original image and changing the transparency of a marker indicating the position of the object of interest depending on the ambiguity.

4. A microscope image display device according to claim 3, wherein the transparency is a function that changes depending on the ambiguity.

5. A microscope image display device according to claim 4, wherein the function is an upwardly convex increasing function.

6. A microscope image display device as described in claim 1 or 2, characterized in that the object of interest estimation unit cuts out a portion of the original image as an estimation target area from the original image and estimates the object of interest using machine learning.

7. A microscope image display method comprising: an original image acquisition step of acquiring an original image of a microscope; an object of interest estimation step of estimating an object of interest from the original image using machine learning; an estimation result evaluation step of evaluating the reliability of the estimation result of the estimated object of interest; and a display step of superimposing and displaying the position of the estimated object of interest on the original image in a manner corresponding to the reliability of the estimation result.

8. A medical image display device comprising: an original image acquisition unit that acquires an original image of a medical image; an object of interest estimation unit that estimates an object of interest from the original image using machine learning; an estimation result evaluation unit that evaluates the reliability of the estimation result of the estimated object of interest; and a display unit that displays the position of the estimated object of interest superimposed on the original image in a display manner corresponding to the reliability of the estimation result.

9. A medical image display method comprising: an original image acquisition step of acquiring an original image of a medical image; an object of interest estimation step of estimating an object of interest from the original image using machine learning; an estimation result evaluation step of evaluating the reliability of the estimation result of the estimated object of interest; and a display step of superimposing and displaying the position of the estimated object of interest on the original image in a manner corresponding to the reliability of the estimation result.

10. A microscope image display device that supports a series of operations including a plurality of operation steps, comprising: an original image acquisition unit that acquires an original image of a microscope; an operation judgment unit that determines which of the plurality of operation steps is currently being performed based on the original image acquired by the original image acquisition unit; an object of interest estimation unit that uses machine learning to estimate, from the original image, the position of an object of interest that should be focused on in the currently being performed operation step determined by the operation judgment unit; an estimation result evaluation unit that evaluates the reliability of the estimation result of the position of the estimated object of interest; and a display unit that displays the position of the estimated object of interest superimposed on the original image in a display manner according to the reliability of the estimation result, wherein, when the operation judgment unit detects the end of the currently being performed operation step, it determines whether there is a next operation step, and when it determines that there is no next operation step, it terminates processing, and when it determines that there is a next operation step, it causes the object of interest estimation unit to estimate the object of interest that should be focused on in the next operation step.

11. The microscope image display device described in claim 10, characterized in that the operation judgment unit has a list of objects of interest that the operator should pay attention to in each of the multiple operation steps, and when determining the step currently being performed from the original image, selects the object of interest in the step currently being performed from the list and causes the object of interest estimation unit to estimate the position of the object of interest.

12. A microscope image display device as described in claim 10 or 11, characterized in that the display unit displays the position of the object of interest estimated by the object of interest estimation unit on the original image with a marker, and the marker is displayed superimposed on the position of the object of interest on the original image with a transparency corresponding to the level of reliability evaluated by the estimation result evaluation unit.

13. A microscope image display method for supporting a series of operations including a plurality of operation steps, comprising: an original image acquisition step for acquiring an original image of a microscope; an operation determination step for determining which of the plurality of operation steps is currently being performed based on the original image acquired by the original image acquisition step; an object of interest estimation step for estimating, from the original image using machine learning, the position of an object of interest to which attention should be paid in the currently being performed operation step determined by the operation determination step; an estimation result evaluation step for evaluating the reliability of the estimation result of the position of the estimated object of interest; and a display step for displaying, on the original image, the position of the estimated object of interest superimposed on the original image in a display manner according to the reliability of the estimation result, wherein the operation determination step, when it detects the end of the currently being performed operation step, determines whether there is a next operation step; and when it determines that there is no next operation step, terminates processing; and when it determines that there is a next operation step, causes the object of interest estimation step to estimate the object of interest to which attention should be paid in the next operation step.

14. The microscope image display method described in claim 13, characterized in that the operation determination step has a list of objects of interest that the operator should pay attention to in each of the multiple operation steps, and when the currently executing step is determined from the original image, the object of interest in the currently executing step is selected from the list, and the position of the object of interest is estimated in the object of interest estimation step.

15. A microscopic image display method as described in claim 13 or 14, characterized in that the display step displays the position of the object of interest estimated by the object of interest estimation step on the original image with a marker, and the marker is displayed superimposed on the position of the object of interest on the original image with a transparency corresponding to the level of reliability evaluated by the estimation result evaluation step.

Images