Method and device for outputting pathology slide images
The method and apparatus enhance digital pathology by overlaying guides on pathology slide images, addressing the transition challenges and ensuring accurate region identification and complete observation.
Patent Information
- Application Number
- JP2024538107
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-03-04
- Filing Date
- 2022-12-13
- Publication Date
- 2024-12-05
AI Technical Summary
The transition from conventional optical microscopy to digital pathology introduces challenges for pathologists accustomed to manual image reading, leading to difficulties in accurately determining regions of interest and adapting to new reading methods.
A method and apparatus that overlay guides on pathology slide images based on user input, allowing for intuitive region selection and highlighting areas of interest, mimicking the experience of optical microscopy.
Facilitates easy adaptation for pathologists from optical to digital microscopy by providing a familiar reading interface, ensuring accurate region identification and complete observation of pathology slide images.
Smart Images

Figure 2024545326000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a method and an apparatus for displaying pathology slide images. [Background technology]
[0002] The field of digital pathology is a field in which whole slide images generated by scanning pathological slide images are used to obtain histological information of a patient and predict prognosis.
[0003] In conventional pathology, a pathologist fixes a glass slide on an optical microscope, adjusts the magnification and focus of the microscope, and reads the slide image while moving the glass slide up and down or left and right. In contrast, in digital pathology, a pathologist outputs the pathology slide image converted into an image file onto a monitor screen, adjusts the magnification with a mouse or keyboard, and reads the slide image while moving the slide.
[0004] Therefore, there is an increasing demand for technology that can eliminate side effects that may occur due to the differences between image reading methods in the conventional pathology field and image reading methods in the digital pathology field. Summary of the Invention [Problem to be solved by the invention]
[0005] The present invention provides a method and device for outputting a pathology slide image. It also provides a computer-readable recording medium having a program for executing the method on a computer. The technical problem to be solved is not limited to the above-mentioned technical problem, and other technical problems may exist. [Means for solving the problem]
[0006] A method for outputting a pathology slide image according to one embodiment includes the steps of receiving user input related to an output manner of at least one region included in a pathology slide image, determining an area of a guide to be output on the pathology slide image and an area of the pathology slide image included within the guide based on the user input, and outputting a pathology slide image with the guide overlaid based on the determined area and area.
[0007] A computer-readable recording medium according to another aspect includes a recording medium having a program recorded thereon for causing a computer to execute the above-described method.
[0008] A computing device according to another aspect includes at least one memory and at least one processor, wherein the processor determines an area of a guide to be output on a pathology slide image and an area of the pathology slide image contained within the guide based on user input related to an output method of at least one area contained in the pathology slide image, and controls a display device to output a pathology slide image with the guide overlaid based on the determined area and area. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a diagram for explaining an example of a system for outputting a pathology slide image according to an embodiment; [Figure 2A] FIG. 1 is a block diagram showing an example of a user terminal 10 according to an embodiment; [Figure 2B] FIG. 1 is a block diagram illustrating an example of a server according to an embodiment. [Diagram 3] A flowchart illustrating an example of a method for outputting a pathology slide image according to an embodiment. [Figure 4A]FIG. 1 is a diagram for explaining an example of outputting a pathology slide image on which a guide is overlaid according to an embodiment; [Figure 4B] FIG. 1 is a diagram for explaining an example of outputting a pathology slide image on which a guide is overlaid according to an embodiment; [Figure 5A] FIG. 11 is a diagram illustrating another example of outputting a pathology slide image on which a guide is overlaid according to an embodiment; [Figure 5B] FIG. 11 is a diagram illustrating another example of outputting a pathology slide image on which a guide is overlaid according to an embodiment; [Figure 6] 10 is a flowchart illustrating another example of a method for outputting a pathology slide image according to an embodiment. [Figure 7A] FIG. 13 is a diagram for explaining another example of outputting a pathology slide image according to an embodiment; [Figure 7B] FIG. 13 is a diagram for explaining another example of outputting a pathology slide image according to an embodiment; [Figure 8] 10 is a flowchart illustrating another example of a method for outputting a pathology slide image according to an embodiment. [Figure 9A] FIG. 1 is a diagram illustrating an example of dividing an observed region into other regions according to a predetermined criterion according to an embodiment; [Figure 9B] FIG. 1 is a diagram illustrating an example of dividing an observed region into other regions according to a predetermined criterion according to an embodiment; [Figure 9C] FIG. 1 is a diagram illustrating an example of dividing an observed region into other regions according to a predetermined criterion according to an embodiment; [Figure 10] FIG. 13 is a diagram illustrating an example of outputting thumbnail images according to an embodiment; [Figure 11] 10 is a flowchart illustrating another example of a method for outputting a pathology slide image according to an embodiment. [Figure 12] FIG. 1 is a diagram for explaining an example of the operation of a processor according to an embodiment; [Figure 13] 10 is a flowchart illustrating another example of a method for outputting a pathology slide image according to an embodiment. [Figure 14] FIG. 1 is a diagram for explaining an example of how a notification signal is generated according to an embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] (Best Mode for Carrying Out the Invention) A method for outputting a pathology slide image according to one embodiment includes the steps of receiving user input related to an output type of at least one region included in a pathology slide image, determining an area of a guide to be output on the pathology slide image and an area of the pathology slide image included within the guide based on the user input, and outputting a pathology slide image with the guide overlaid based on the determined area and area.
[0011] Reference will now be made in detail to the embodiments. Examples of the embodiments are illustrated in the accompanying drawings, in which the same reference numerals refer to the same elements throughout the drawings. In this regard, the embodiments of the present invention may have different forms and should not be construed as being limited to the description set forth herein. Thus, the embodiments are only described below with reference to the drawings to illustrate aspects. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.
[0012] The terms used in the embodiments are generally used as widely as possible, but may vary depending on the intentions or precedents of engineers in the field, the emergence of new technologies, etc. In addition, in certain cases, the applicant may arbitrarily select terms, and in such cases, the meanings of such terms are described in detail in the relevant description. Therefore, the terms used in the specification should be defined based on the meanings of the terms and the overall content of the specification, rather than simply the names of the terms.
[0013] Throughout the specification, when a part "includes" a certain component, it means that it may further include other components, not excluding other components, unless otherwise specified. Furthermore, the terms "... unit", "... module", etc. described in the specification mean a unit that processes at least one function or operation, which may be realized in hardware or software, or a combination of hardware and software.
[0014] Furthermore, terms including ordinal numbers such as "first" or "second" used in the specification may be used to describe various components, but the components should not be limited by the terms. The terms may be used to distinguish one component from another.
[0015] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following, embodiments will be described in detail with reference to the accompanying drawings, however, the embodiments may be embodied in various different forms and should not be construed as being limited to the examples set forth herein.
[0016] FIG. 1 is a diagram for explaining an example of a system for outputting a pathology slide image according to an embodiment.
[0017] 1, a system 1 includes a user terminal 10 and a server 20. For example, the user terminal 10 and the server 20 are connected by wired or wireless communication and can transmit and receive data (e.g., video data, etc.) to and from each other.
[0018] 1 illustrates a system 1 including a user terminal 10 and a server 20, but is not limited thereto. For example, the system 1 may include other external devices (not shown), and the operations of the user terminal 10 and the server 20 described below may be realized by a single device (e.g., the user terminal 10 or the server 20).
[0019] The user terminal 10 outputs a pathology slide image 40, and the user 30 can operate the user terminal 10 in various ways to observe the pathology slide image 40 from multiple angles.
[0020] Here, the pathology slide image 40 may refer to an image obtained by scanning a pathology slide that has been fixed and stained through a series of chemical processing steps in order to observe tissues, etc., taken from a human body under a microscope. As one example, the pathology slide image 40 may refer to a whole slide image including a high-resolution image of the entire slide. As another example, the pathology slide image 40 may refer to a portion of such a high-resolution whole slide image.
[0021] On the other hand, the pathology slide image 40 may refer to a patch area that is divided into patch units in the entire slide image. For example, a patch may have a certain area size. Alternatively, a patch may refer to an area that includes each of the objects contained in the entire slide.
[0022] Pathology slide image 40 may also refer to a digital image of a pathology slide converted into an image file using a digital scanner, and may contain information about cells, tissues, and / or structures within the human body.
[0023] The user terminal 10 can store and output the pathology slide image 40, and can receive user input from the user 30 to overlay a guide on the pathology slide image 40 and output it. For example, the user terminal 10 can enlarge or reduce at least one area of the pathology slide image 40, and can confirm the area observed by the user 30 on the pathology slide image 40. The user terminal 10 can also store the pathology slide image 40 updated by the operation of the user 30. The user 30 can also record and store information regarding the findings observed on the pathology slide image 40 in the user terminal 10.
[0024] The user terminal 10 can transmit and receive various data, including the pathology slide image 40, through communication with the server 20 and / or other external devices (not shown). Furthermore, the user terminal 10 can store the transmitted and received data by itself, or can request the server 20 and / or other external devices (not shown) to store the transmitted and received data.
[0025] The user terminal 10 may be a computing device including a display device and a device for receiving user input (e.g., a keyboard, a mouse, etc.), and including a memory and a processor. For example, the user terminal 10 may be, but is not limited to, a notebook PC, a desktop PC, a laptop, a tablet computer, a smartphone, etc.
[0026] The server 20 may be a device that communicates with an external device (not shown) including the user terminal 10. As an example, the server 20 may be a device that stores various data including the pathology slide image 40. As another example, the server 20 may be a computing device including a memory and a processor and having its own computing power. When the server 20 is a computing device, the server 20 may perform at least a part of the operations of the user terminal 10, which will be described later with reference to FIGS. 1 to 14. For example, the server 20 may be, but is not limited to, a cloud server.
[0027] The user 30 may be an entity that observes the pathology slide image 40 output to the user terminal 10 and operates the user terminal 10 to which the pathology slide image 40 is output, or records the findings of observing the pathology slide image 40. For example, the user 30 may be, but is not limited to, a pathologist.
[0028] In the field of conventional pathology, a pathologist reads an image by touching the objective lens and the drive adjustment knob with his / her hand while looking at the image formed in a circle through the lens of an optical microscope, whereas in the field of digital pathology, a user 30 reads the image using a computer and a mouse while observing a pathology slide image 40 outputted on a display device.
[0029] For pathologists who are accustomed to reading images using existing optical microscopes, it is difficult to adjust the magnification of slide images or to adapt to the method of searching for areas of interest (e.g., lesion areas) due to the interface changed by digital pathology. This can cause problems such as a long time required to read the pathology slide image 40 or inaccurate reading results.
[0030] For example, in the case of breast cancer, the histological grade of breast cancer is determined based on the number of mitoses of cancer cells confirmed in an area 10 times larger than the range observable in a high power field (HPF) of an optical microscope. However, since it is difficult to estimate the HPF range visible under a microscope on a pathology slide image 40, it may not be easy for a pathologist to accurately determine the histological grade based on the pathology slide image 40.
[0031] The user terminal 10 according to one embodiment determines the area of the guide to be output to the pathology slide image 40 and the area included in the guide (i.e., the area of the pathology slide image 40 included in the guide) based on a user input. The user terminal 10 then outputs the pathology slide image on which the guide is overlaid according to the result of the determination. Thus, the user terminal 10 can output an image that can be read in the same way as when an image is read using an optical microscope. This allows a user 30 who is accustomed to the method of reading images using an existing optical microscope to easily perform reading in the field of digital pathology.
[0032] Furthermore, the user terminal 10 according to an embodiment can output the pathology slide image 40 by dividing the area observed by the user from the other areas (i.e., unobserved areas). This allows the user 30 to intuitively check whether or not there is an unobserved area in the pathology slide image 40.
[0033] In addition, the user terminal 10 according to an embodiment determines whether or not the region of interest is included in the region observed by the user in the pathology slide image 40. This allows the user 30 to intuitively confirm whether or not the region of interest has been completely observed.
[0034] Hereinafter, a method in which the user terminal 10 outputs a pathology slide image will be specifically described with reference to FIGS. 2A to 14.
[0035] Meanwhile, for convenience of explanation, it has been described throughout the specification that the user terminal 10 determines the area of the guide to be output on the pathology slide image 40 and the area included in the guide, and outputs the pathology slide image 40 on which the guide is overlaid, but this is not limiting. For example, at least a part of the operations performed by the user terminal 10 may also be performed by the server 20.
[0036] In other words, at least a part of the operations of the user terminal 10 described with reference to Figs. 1 to 11 can also be performed by the server 20. For example, the server 20 can determine the area of the guide to be output to the pathology slide image 40 and the area included in the guide, and transmit the determination result to the user terminal 10. Then, the user terminal 10 can output the pathology slide image 40 on which the guide is overlaid according to the information transmitted from the server 20 (for example, the pathology slide image, information on the area of the guide, information on the area included in the guide, etc.). However, the operations of the server 20 are not limited to those described above.
[0037] FIG. 2A is a diagram illustrating an example of a configuration of a user terminal 10 according to an embodiment.
[0038] Referring to Fig. 2A, the user terminal 100 includes a processor 110, a memory 120, an input / output interface 130, and a communication module 140. For convenience of explanation, only components related to the present invention are shown in Fig. 2A. Therefore, the user terminal 100 may further include other general-purpose components in addition to the components shown in Fig. 2A. Furthermore, it is obvious to a person skilled in the art related to the present invention that the processor 110, the memory 120, the input / output interface 130, and the communication module 140 shown in Fig. 2A may be realized as independent devices.
[0039] The processor 110 can process instructions of a computer program by performing basic arithmetic, logic, and input / output operations, where the instructions can be provided from the memory 120 or an external device (e.g., the server 20, etc.), and can generally control the operation of other components included in the user terminal 100.
[0040] In particular, processor 110 can determine an area of the guide to be output on the pathology slide image and an area of the pathology slide image included within the guide based on a user input related to an output manner of at least one area included in the pathology slide image, and can control the display device to output the pathology slide image overlaid with the guide based on the determined area and area.
[0041] In addition, the processor 110 can control the display device to check the area observed by the user in the pathology slide image according to predetermined criteria, and to output the observed area separately from other areas according to the results of the check.
[0042] The processor 110 may also set at least one region of interest in the pathology slide image and check the region observed by the user in the pathology slide image according to a predetermined criterion. The processor 110 may then determine whether the region observed by the user includes the region of interest. The processor 110 may then generate a notification signal regarding the presence or absence of further observation of the pathology slide image based on the result of the determination when a user input related to the end of observation of the pathology slide image is received.
[0043] The processor 110 may be implemented with an array of logic gates or may be implemented with a combination of a general-purpose microprocessor and a memory in which a program executable by the microprocessor is stored. For example, the processor 110 may include a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, the processor 110 may also include an application specific semiconductor (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. For example, the processor 110 may refer to a combination of processing devices, such as a combination of a digital signal processor (DSP) and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors coupled to a digital signal processor (DSP) core, or any other such configuration.
[0044] A specific example of the operation of the processor 110 according to an embodiment will be described with reference to FIGS.
[0045] The memory 120 may include any non-transitory computer-readable recording medium. As an example, the memory 120 may include a permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), a disk drive, a solid state drive (SSD), or a flash memory. As another example, the permanent mass storage device such as a ROM, an SSD, a flash memory, or a disk drive may be a permanent storage device independent of the memory. In addition, the memory 120 may store an operating system (OS) and at least one program code (e.g., code for the processor 110 to perform the operations described below with reference to FIGS. 3 to 14).
[0046] Such software components can be loaded from a computer-readable recording medium separate from the memory 120. Such a separate computer-readable recording medium may be a recording medium directly connectable to the user terminal 100, and may include, for example, a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, or a memory card. Alternatively, the software components can be loaded into the memory 120 via the communication module 140 rather than a computer-readable recording medium. For example, at least one program can be loaded into the memory 120 based on a computer program (e.g., a computer program for the processor 110 to perform the operations described below with reference to FIGS. 3 to 14) installed by a file provided via the communication module 140 by a developer or a file distribution system that distributes an installation file for an application.
[0047] The input / output interface 130 may be a means for interfacing with a device for input or output (e.g., a keyboard, a mouse, etc.) connected to or included in the user terminal 100. Although the input / output interface 130 is shown in FIG. 2A as an element configured separately from the processor 110, this is not limiting, and the input / output interface 130 may be configured to be included in the processor 110.
[0048] The communication module 140 may provide a configuration or function for the server 20 and the user terminal 100 to communicate with each other via a network. The communication module 140 may also provide a configuration or function for the user terminal 100 to communicate with other external devices. For example, control signals, instructions, data, etc. provided under the control of the processor 110 may be transmitted to the server 20 and / or the external devices via the communication module 140 and the network.
[0049] Meanwhile, although not shown in Fig. 2A, the user terminal 100 may further include a display device. Alternatively, the user terminal 100 may be connected to an independent display device via a wired or wireless communication method, and data may be transmitted and received between the display device and the user terminal 100.
[0050] FIG. 2B is a diagram illustrating an example of a server according to an embodiment.
[0051] Referring to Fig. 2B, the server 200 includes a processor 210, a memory 220, and a communication module 230. For convenience of explanation, only components related to the present invention are shown in Fig. 2B. Therefore, the server 200 may further include other general-purpose components in addition to the components shown in Fig. 2B. It is obvious to a person skilled in the art related to the present invention that the processor 210, the memory 220, and the communication module 230 shown in Fig. 2B may be realized as independent devices.
[0052] The processor 210 can determine the area of the guide to be output on the pathology slide image and the area of the pathology slide image included within the guide based on a user input related to an output manner of at least one area included in the pathology slide image. Then, the processor 210 can transmit related information to the user terminal 10 so that the pathology slide image overlaid with the guide is output to the user terminal 10.
[0053] The processor 210 can also verify areas observed by the user in the pathology slide image according to predetermined criteria.
[0054] The processor 210 can also set at least one region of interest in the pathology slide image and check the region observed by the user in the pathology slide image according to a predetermined criterion. The processor 210 can then determine whether the region observed by the user includes the region of interest. The processor 210 can then generate a notification signal regarding the presence or absence of further observation of the pathology slide image based on the result of the determination when a user input related to the end of observation of the pathology slide image is received.
[0055] 2A can be performed by the processor 210. In this case, the user terminal 10 can output the information transmitted from the server 20 via the display device.
[0056] Meanwhile, the implementation of the processor 210 is similar to the implementation of the processor 110 described above with reference to FIG. 2A, so a detailed description will be omitted.
[0057] The memory 220 can store various data, such as pathology slide images and data generated by the operation of the processor 210. The memory 220 can also store an operating system (OS) and at least one program (e.g., a program necessary for the processor 210 to operate).
[0058] Meanwhile, the implementation of the memory 220 is similar to the implementation of the memory 220 described above with reference to FIG. 2A, so a detailed description thereof will be omitted.
[0059] The communication module 230 may provide a configuration or function for the server 200 and the user terminal 100 to communicate with each other via a network. The communication module 140 may also provide a configuration or function for the server 200 to communicate with other external devices. For example, control signals, instructions, data, etc. provided under the control of the processor 210 may be transmitted to the user terminal 100 and / or the external devices via the communication module 230 and the network.
[0060] FIG. 3 is a flowchart illustrating an example of a method for outputting a pathology slide image according to an embodiment.
[0061] 3, the method for outputting a pathology slide image is composed of steps that are processed in a time series manner by the user terminal 10, 100 or the processor 110 shown in Figures 1 and 2. Therefore, even if the contents are omitted below, the contents described above regarding the user terminal 10, 100 or the processor 110 shown in Figures 1 and 2A can also be applied to the method for outputting a pathology slide image in Figure 3.
[0062] In step 310, the user terminal 10, 100 receives a user input related to an output manner of at least one region included in a pathology slide image.
[0063] For example, the pathology slide image 40 is output to the display device of the user terminal 10, 100, and then the user terminal 10, 100 can receive user input. The user input can be received via another image (an image configured to receive user input) output to the display device, via the movement of a mouse wheel, or via a shortcut key input preset by the user, but is not limited thereto.
[0064] The user input may be an operation on an area that the user 30 wishes to closely observe in the pathology slide image 40. For example, the user 30 may enlarge or reduce a portion of the pathology slide image 40 through the user input.
[0065] In step 320, the processor 110 determines the area of the guide to be output on the pathology slide image 40 and the area of the pathology slide image 40 that is included within the guide based on user input.
[0066] As one example, the processor 110 can enlarge or reduce the area of the guide while keeping the area of the pathology slide image 40 included in the guide fixed based on a magnification factor specified by user input. As another example, the processor 110 can enlarge or reduce the area of the pathology slide image 40 included in the guide while keeping the area of the guide fixed based on a magnification factor specified by user input.
[0067] Here, the guide may be to limit the area that the user 30 wishes to closely observe. When an image reading operation is performed with an optical microscope according to the conventional technology, the user 30 observes the image through an eyepiece. Therefore, the guide may be a shape that appears on the pathology slide image 40 so as to be similar to the area observed by the user 30 through the eyepiece.
[0068] 3, the processor 110 can calculate the scan magnification and the length per pixel of the pathology slide image 40. Therefore, the processor 110 can calculate the area of the guide displayed on the pathology slide image 40 and the area of the pathology slide image 40 included within the guide according to a user input.
[0069] In step 330, the processor 110 outputs a pathology slide image overlaid with the guide based on the determined area and region. For example, the processor 110 can control a display device to output a pathology slide image overlaid with the guide.
[0070] The processor 110 can overlay the guide on the pathology slide image by manipulating (e.g., zooming in or out) the pathology slide image 40 depending on the area of the guide determined in step 320 and the area of the pathology slide image 40 contained within the guide.
[0071] Hereinafter, an example in which a pathology slide image on which a guide is overlaid is output will be described with reference to FIGS. 4A to 5B.
[0072] 4A and 4B are diagrams for explaining an example in which a pathology slide image on which a guide is overlaid according to an embodiment is output.
[0073] 4A, a guide 410 is overlaid on the pathology slide image. The guide 410 is an indication that separates an area different from an area that the user 30 intends to closely observe, and may be specified by the user 30. For example, the guide 410 may be, but is not limited to, a circular shape.
[0074] There may be various examples of the guide 410 being overlaid on the pathology slide image. As one example, the first guide 410 is displayed at a preset position on the pathology slide image (e.g., the center position of the pathology slide image, the position of the region of interest, etc.), and then the position of the guide 410 may be changed or fixed by the operation of the user 30. As another example, the first guide 410 is displayed along a marker (e.g., a mouse cursor, etc.) on the pathology slide image, and then the position of the guide 410 may be fixed by the operation of the user 30. However, examples of the guide 410 displayed on the pathology slide image are not limited to those described above, and are applicable without limitation as long as they are a method that is useful for the user 30 to observe the pathology slide image.
[0075] Meanwhile, a magnification 431 may be displayed in an area adjacent to the guide 410. Here, the magnification refers to the actual magnification that can be realized by an optical microscope. Therefore, through the guide 410 and the magnification 431 displayed on the pathology slide image, the user 30 can have an experience similar to observing an image through an optical microscope.
[0076] In response to a user input received while a pathology slide image is output on the display device, the processor 110 overlays and outputs a guide on the pathology slide image. For example, when a user input to increase or decrease the magnification is received while the pathology slide image shown in FIG. 4A is output on the display device, the processor 110 determines the area of the guide and the region included in the guide corresponding to the user input. Then, the processor 110 overlays the guide on the pathology slide image based on the determined area of the guide and the region included in the guide.
[0077] For example, the processor 110 may enlarge or reduce the area of the guide based on a scale factor specified by user input, while keeping the area contained within the guide fixed.
[0078] In the example of FIG. 4A and FIG. 4B, it is assumed that the user input increases the magnification from 40x to 100x. In this case, the processor 110 can determine that the area of the guide 410 is larger than the area of the guide 420 while the size of the pathology slide image output to the display device is fixed to the same. That is, the size of the guide 420 corresponding to 100x is smaller than the size of the guide 410 corresponding to 40x. In other words, the processor 110 adjusts the size of the guide 420 so that the actual area of the area observed by the optical microscope with a magnification of 40 (i.e., the area of the area in drawing number 410) and the actual area of the area observed by the optical microscope with a magnification of 100 (i.e., the area of the area in drawing number 420) are the same while the size of the pathology slide image output to the display device is the same.
[0079] The processor 110 overlays the guide 420 on the pathology slide image based on the area of the guide 420 and outputs the same. Furthermore, the processor 110 can display the output image (the image overlaid with the guide 420) together with the magnification 432 corresponding to the user input. Here, the magnification 432 means the actual magnification of the optical microscope. That is, the magnification 432 means the magnification set in the optical microscope when the user 30 intends to observe the area in the guide 420 through the optical microscope.
[0080] Meanwhile, although not shown in Figures 4A and 4B, images for setting magnifications 431, 432 may be output to the display device together with the pathology slide image or before the pathology slide image is output. For example, the user 30 can specify the magnifications 431, 432 via the images, and the processor 110 can output guides 410, 420 on the pathology slide image in response to the user input specifying the magnification. Meanwhile, the images for setting the magnifications 431, 432 may be variously configured so that the user 30 can select the magnification he or she desires.
[0081] 5A and 5B are diagrams for explaining another example in which a pathology slide image on which a guide is overlaid according to an embodiment is output.
[0082] In the example of FIGS. 5A and 5B, as in the example of FIGS. 4A and 4B, it is assumed that the user input is to increase the magnification from 40x to 100x.
[0083] 5A is output on the display device, when a user input to increase or decrease the magnification is received, the processor 110 determines the area of the guide and the region included in the guide corresponding to the user input, and then overlays the guide on the pathology slide image and outputs it based on the determined area of the guide and the region included in the guide.
[0084] For example, processor 110 may enlarge or reduce the area contained within the guide while keeping the area of the guide fixed, based on a scale factor specified by user input.
[0085] When a user input to increase the magnification from 40x to 100x is received, the processor 110 can determine that the actual area of the area included in the guide 510 is smaller than the actual area of the area included in the guide 520 when the areas on the screen of the guide 510 and the guide 520 are fixed to be the same. That is, the actual area of the area included in the guide 520 corresponding to 100x is smaller than the actual area of the area included in the guide 510 corresponding to 40x. In other words, the processor 110 adjusts the actual area of the area included in the guide 520 so that the size of the guide corresponding to an optical microscope with a magnification of 40 (i.e., the area on the screen of the guide 510) and the size of the guide corresponding to an optical microscope with a magnification of 100 (i.e., the area on the screen of the guide 520) are the same.
[0086] Meanwhile, the processor 110 displays the magnifications 531 and 532 in the areas adjacent to the guides 510 and 520, so that the user 30 can intuitively know the magnification of the currently displayed guide. Here, the magnifications 531 and 532 refer to the actual magnifications of the optical microscope. That is, the magnifications 531 and 532 refer to the magnifications set in the optical microscope when the user 30 intends to observe the areas in the guides 510 and 520 through the optical microscope.
[0087] On the other hand, although not shown in FIGS. 5A and 5B, images for setting magnifications 531 and 532 may be output to the display device, as described above with reference to FIGS. 4A and 4B.
[0088] As described above with reference to FIGS. 3 to 5B, a user 30 who is accustomed to the method of reading images using an existing optical microscope can easily perform image reading in the field of digital pathology.
[0089] Meanwhile, an example in which a user input to increase the magnification has been described with reference to Figures 4A to 5B, the same method may be applied to a case in which a user input to decrease the magnification is received.
[0090] FIG. 6 is a flowchart illustrating another example of a method for outputting a pathology slide image according to an embodiment.
[0091] 6, steps 610 to 630 are the same as steps 310 to 330 in Fig. 3. Therefore, in the following, a detailed description of steps 610 to 630 will be omitted.
[0092] In step 640, the user terminal 10, 100 outputs an image in which at least a portion of the pathology slide image 40 is enlarged or reduced based on the additionally received user input. For example, the processor 110 can control the display device to output an image in which at least a portion of the pathology slide image 40 is enlarged or reduced.
[0093] Here, the additional user input may be an input for enlarging or reducing the pathology slide image 40 without adjusting the magnification. For example, the pathology slide image 40 may be enlarged or reduced by various additional user inputs, such as the user 30 zooming in or out on the pathology slide image 40 using a mouse wheel, or operating a UI (e.g., a button, a scroll bar, etc.) output on the display device. In this case, the guide overlaid on the pathology slide image 40 may be enlarged or reduced together with the pathology slide image 40, or may be fixed regardless of the enlargement or reduction of the pathology slide image 40. Hereinafter, with reference to Figs. 7A and 7B, an example in which the processor 110 outputs an image based on an additional user input will be described.
[0094] 7A and 7B are diagrams for explaining another example in which a pathology slide image is output according to an embodiment.
[0095] 7A, assume that the display device is outputting a pathology slide image with a guide 710 corresponding to a magnification of 40. Thereafter, depending on the received user input, the processor 110 can enlarge or reduce the pathology slide image.
[0096] For example, referring to Fig. 7B, the guide 720 displayed in Fig. 7B means the same magnification (40x) as the guide 710 displayed in Fig. 7A. That is, in response to a user input to enlarge the pathology slide image, a pathology slide image with a partial area enlarged is output as shown in Fig. 7B, and at the same time, the size of the guide 720 can also be enlarged to the same size.
[0097] Meanwhile, an example in which a user input for enlarging a pathology slide image is received has been described with reference to Figures 7A and 7B, but the same method may be applied to a case in which a user input for reducing a pathology slide image is received.
[0098] FIG. 8 is a flowchart illustrating another example of a method for outputting a pathology slide image according to an embodiment.
[0099] 8, steps 810 to 830 are the same as steps 310 to 330 in Fig. 3. Therefore, in the following, a detailed description of steps 810 to 830 will be omitted.
[0100] In step 840, the processor 110 identifies areas of the pathology slide image 40 viewed by the user according to predetermined criteria.
[0101] Here, the predetermined criteria include a magnification designated by the user input being equal to or greater than a threshold magnification, i.e., the processor 110 can confirm which area in the pathology slide image 40 the user observed at a magnification equal to or greater than the threshold magnification.
[0102] For example, the threshold magnification may be a magnification level at which the user 30 can confirm the morphology of a cell based on an image. The threshold magnification may be set in advance or may be adjusted by the user 30.
[0103] In step 850, the user terminal 10, 100 outputs the observed area and the other area separately according to the result of the confirmation. For example, the processor 110 can control the display device to output the observed area and the other area separately.
[0104] For example, a thumbnail image may be superimposed on a portion of the pathology slide image 40 and output, and the thumbnail image may be an image that represents the observed area and other areas in accordance with a predetermined criterion.
[0105] Hereinafter, with reference to FIGS. 9A to 10, an example in which an observed region and other regions are separated according to a predetermined criterion and output will be described.
[0106] 9A to 9C are diagrams illustrating an example of dividing an observed region and other regions according to a predetermined criterion according to an embodiment.
[0107] In FIG. 9A, an initial pathology slide image (ie, an image with no areas observed according to predefined criteria) is shown.
[0108] As described above with reference to Figures 3 to 7B, the user 30 can observe at least a portion of the first pathology slide image while enlarging or reducing it. In particular, the user 30 can observe the image while enlarging or reducing it according to a magnification desired by the user.
[0109] The processor 110 can check whether the magnification of the area observed by the user 30 is equal to or greater than a threshold magnification. For example, when the processor 110 receives information on the magnification designated by the user 30 in a user input, the processor 110 can check whether the magnification input by the user is equal to or greater than a threshold magnification through a comparison with the threshold magnification. The processor 110 can then control the display device to output the area observed at or greater than the threshold magnification separately from the other areas.
[0110] 9B, an area 910 currently being viewed by a user 30 is displayed. The processor 110 checks whether the magnification corresponding to the area 910 is equal to or greater than a threshold magnification. In this manner, the processor 110 checks an area 920 viewed at a magnification equal to or greater than the threshold magnification and another area 930, and controls the display device to output an image in which the area 920 and the area 930 are separated.
[0111] 9B and 9C, the region 920 may be distinguished from the region 930 by being displayed with a brighter brightness value than the region 930. However, the manner in which the region 920 and the region 930 are distinguished from each other is not limited to the above.
[0112] FIG. 10 is a diagram for explaining an example in which thumbnail images are output according to an embodiment.
[0113] 10 shows an example in which a thumbnail image 1000 is outputted while being superimposed on a part of a pathology slide image. Here, the thumbnail image 1000 may be the image described above with reference to FIGS. 9A to 9C. However, the area in which the thumbnail image 1000 is outputted and its size are not limited to the example shown in FIG.
[0114] FIG. 11 is a flowchart illustrating another example of a method for outputting a pathology slide image according to an embodiment.
[0115] Although not shown in Fig. 11, the example described below with reference to Fig. 11 may be combined in parallel or in time series with at least some of the examples described above with reference to Figs. 3 to 10. Therefore, in the following, a detailed description of the contents described above with reference to Figs. 3 to 10 will be omitted.
[0116] In step 1110, the processor 110 sets at least one region of interest in the pathology slide image.
[0117] Here, the region of interest means a region in the pathology slide image 40 that needs to be observed by the user 30. For example, the region of interest may be, but is not limited to, a region that should be importantly observed in interpreting the pathology of a disease. For example, when a diagnosis of lung cancer is required, a region in the pathology slide image 40 where lung cancer cells exist may be the region of interest, and when mitosis of breast cancer is evaluated, a region in the pathology slide image 40 where mitosis is expected to be high may be the region of interest.
[0118] The processor 110 analyzes the pathology slide image 40 to set at least one region of interest. As an example, the processor 110 can detect the region of interest by analyzing the pathology slide image 40 using a predetermined image processing technique. As another example, the processor 110 can detect the region of interest in the pathology slide image 40 using a machine learning model. In this case, the machine learning model may be trained to detect the region of interest in the reference pathology slide image using training data including a plurality of reference pathology slide images and a plurality of reference label information.
[0119] As an example, the processor 110 can set at least one region of interest by analyzing the entire pathology slide image 40. As another example, the processor 110 can set at least one region of interest by analyzing a portion of the pathology slide image 40. Specifically, the user 30 can specify in advance a portion of the pathology slide image 40 in which the region of interest is to be set, and the processor 110 can set at least one region of interest by analyzing only the portion specified by the user 30.
[0120] In step 1120, the processor 110 identifies areas in the pathology slide image viewed by the user according to predefined criteria.
[0121] Here, the predetermined criteria include a magnification designated by the user input being equal to or greater than a threshold magnification. That is, the processor 110 can confirm which area in the pathology slide image 40 the user observed at a magnification equal to or greater than the threshold magnification.
[0122] For example, the threshold magnification may be a magnification level at which the user 30 can confirm the morphology of a cell based on an image. The threshold magnification may be set in advance or may be adjusted by the user 30.
[0123] In step 1130, the processor 110 determines whether the area viewed by the user includes at least one region of interest.
[0124] In other words, the processor 110 can check whether the area observed by the user includes the area of interest. Through step 1110, the processor 110 can check the area of interest in advance in the pathology slide image 40. Therefore, the processor 110 can check whether the area observed by the user 30 in the pathology slide image 40 includes the area of interest.
[0125] Hereinafter, with reference to FIG. 12, an example in which the processor 110 checks whether or not a region of interest is included in the region observed by the user will be described.
[0126] FIG. 12 is a diagram for explaining an example of the operation of a processor according to an embodiment.
[0127] 12, the processor 110 can distinguish between an area 1210 observed by the user 30 and other areas 1220 in the pathology slide image. Here, the area 1210 may be an area observed to meet a predetermined criterion, as described above with reference to step 1120. Also, the method by which the processor 110 distinguishes between the areas 1210 and 1220 is as described above with reference to FIG. 8. That is, the areas 1210 and 1220 may be determined by the area 1230 currently observed by the user 30.
[0128] The processor 110 determines whether the region 1210 includes the region of interest 1240. For example, the region of interest 1240 may be detected in advance according to a preset criterion, as described above with reference to step 1110. The processor 110 may distinguish the region 1210 from the region 1220 based on the region 1230, and determine whether the region 1210 includes the region of interest 1240.
[0129] Alternatively, the processor 110 may provide a notification to the user 30 depending on whether the region 1210 includes a region of interest 1240 or not, as will be described with reference to FIGS.
[0130] FIG. 13 is a flowchart illustrating another example of a method for outputting a pathology slide image according to an embodiment.
[0131] 13, steps 1310 and 1320 are the same as steps 1110 and 1120 in Fig. 11. Therefore, in the following, detailed descriptions of steps 1310 and 1320 will be omitted.
[0132] In step 1330, the user terminal 10, 100 receives a user input related to the end of viewing the pathology slide image.
[0133] The user input includes various attempts by the user 30 to end the observation of the pathology slide image 40 currently output on the display device. For example, the user input may include an attempt by the user 30 to replace the currently output pathology slide image 40 with another image, or to end a program for outputting and observing the pathology slide image 40. Alternatively, the user input in step 1330 may include a case where no user input is received for a predetermined time. For example, the reception of the user input in step 1330 may also be a case where no user input is received for a predetermined time after the pathology slide image 40 is output.
[0134] In step 1340, the processor 110 determines whether the area viewed by the user includes at least one region of interest.
[0135] Here, the area observed by the user includes the area observed by the user in the pathology slide image 40 according to a predetermined criterion, and the predetermined criterion includes a magnification designated according to a user input being equal to or greater than a threshold magnification. An example in which the processor 110 determines whether the area observed by the user includes the area of interest is as described above with reference to steps 1120 and 1130. In the case where the area of interest is a plurality of areas, the processor 110 can determine whether all of the plurality of areas of interest are included in the area observed by the user.
[0136] On the other hand, examples of the processor 110 determining whether the region observed by the user includes the region of interest are not limited to those described above. For example, if the region observed by the user does not correspond to a certain ratio or more of the total region of interest, the processor 110 can determine that the region observed by the user does not include the region of interest. When evaluating mitosis in breast cancer, a region in the pathology slide image 40 where mitosis is expected to be high may be set as the region of interest. In this case, if the region observed by the user does not include a predetermined ratio (e.g., 50%) or more of the region of interest, the processor 110 can determine that the region observed by the user does not include the region of interest.
[0137] If the area viewed by the user does not include the area of interest, proceed to step 1350; otherwise, end the process.
[0138] In step 1350, the processor 110 generates a notification signal for the absence or presence of further viewing of the pathology slide image 40.
[0139] As described above with reference to steps 1330 and 1340, the observation of the currently output pathology slide image 40 may be terminated in a state where the region of interest is not observed. In this case, the processor 110 may generate and output a notification signal to help the user 30 decide whether or not to further observe the pathology slide image 40.
[0140] On the other hand, the processor 110 can determine whether or not to generate a notification signal depending on whether or not a predetermined condition is satisfied, regardless of whether or not all regions of interest have been observed. As an example, when at least a part of the region of interest is included in the region previously observed by the user 30 in the pathology slide image 40, the processor 110 may not generate a notification signal even if the region of interest exists in a region not observed by the user 30. As another example, the processor 110 can generate a notification signal when the region of interest exists in a region not observed by the user 30, regardless of whether or not the region of interest is included in the region previously observed by the user 30.
[0141] For example, the notification signal may be an image output to a display device, a sound output through a speaker, or a vibration output through a vibration motor, but examples of the notification signal are not limited to those described above.
[0142] FIG. 14 is a diagram for explaining an example in which a notification signal is generated according to an embodiment.
[0143] 14 illustrates a case where the user viewed area 1410 does not include the area of interest 1420. In this situation, when a user input 1430 related to the end of viewing the pathology slide image is received, the processor 110 generates and outputs a notification signal 1440.
[0144] On the other hand, examples of output of the notification signal 1440 are not limited to those shown in Fig. 14. Specifically, the notification signal 1440 may be of any type, such as an image, sound, or vibration, so long as it is for notifying the user that at least a portion of the region of interest 1420 is not being observed. Furthermore, if the notification signal 1440 is an image, it may be an image displaying any message.
[0145] When an attempt to further view the pathology slide image is received from the user 30 after output of the notification signal 1440, the processor 110 can assist the user 30 in viewing by outputting the region of interest 1420 to the display device.
[0146] As described above, the user 30 who is familiar with the method of reading images using an existing optical microscope can easily read images in the field of digital pathology. The user 30 can also intuitively check whether there is an unobserved area in the pathology slide image 40. The user 30 can also intuitively check whether the area of interest included in the pathology slide image 40 has been completely observed.
[0147] Meanwhile, the above-mentioned method can be created by a computer-executable program, and can be realized by a general-purpose digital computer that runs the program using a computer-readable recording medium. Also, the data structure used in the above-mentioned method can be recorded in a computer-readable recording medium through various means. The computer-readable recording medium can include, but is not limited to, a magnetic storage medium (e.g., ROM, RAM, USB, floppy disk, hard disk, etc.), an optically readable medium (e.g., CD-ROM, DVD, etc.), and other storage media.
[0148] It will be understood by those skilled in the art that the present embodiment relates to that it can be realized in modified forms without departing from the essential characteristics of the above description. Therefore, the disclosed method should be considered from an explanatory point of view, not a limiting point of view, and the scope of the right should be interpreted as including all differences within the scope of the claims, not the above description, and equivalents thereto.
[0149] It should be understood that the embodiments described herein should be considered in an illustrative sense only, and not for the purposes of limitation. The description of a feature or aspect in each embodiment should generally be considered applicable to other similar features or aspects in other embodiments. Although one or more embodiments have been described with reference to the drawings, those skilled in the art will recognize that various changes in form and detail can be made therein without departing from the spirit and scope defined in the following claims. [Explanation of symbols]
[0150] 1 System 10 User terminal 20 Servers 30 users 40 Pathology slide images
Claims
1. receiving a user input related to an output manner of at least one region included in a pathology slide image; determining an area of a guide to be output on the pathology slide image and an area of the pathology slide image included within the guide based on the user input; outputting a pathology slide image on which the guide is overlaid based on the determined area and region; Including, A method for outputting pathology slide images.
2. The determining step includes: enlarging or reducing the area while keeping the region fixed, based on a magnification factor designated in response to the user input; The method of claim 1.
3. The determining step includes: enlarging or reducing the region while keeping the area fixed, based on a magnification factor specified in response to the user input; The method of claim 1.
4. and outputting an image in which at least a portion of the pathology slide image is enlarged or reduced based on additionally received user input. The method of claim 1.
5. identifying an area observed by a user in the pathology slide image according to a predetermined criterion; outputting the observed area and other areas according to the result of the confirmation; Further comprising: The method of claim 1.
6. The predetermined criteria include a magnification ratio specified in response to the user input being equal to or greater than a threshold magnification ratio. The method according to claim 5.
7. The outputting step includes: outputting a thumbnail image in which the observed area and other areas are distinguished according to the predetermined criterion; The method according to claim 5.
8. setting at least one region of interest in the pathology slide image; identifying an area observed by a user in the pathology slide image according to a predetermined criterion; determining whether the area viewed by the user includes at least one of the regions of interest; Further comprising: The method of claim 1.
9. receiving a user input associated with completion of viewing of the pathology slide image; generating a notification signal for whether or not the pathology slide image should be further observed based on the result of the determination; Further comprising: The method according to claim 8.
10. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the method according to claim 1.
11. At least one memory; At least one processor; Including, The processor, determining an area of the guide to be output on the pathology slide image and an area of the pathology slide image included in the guide based on a user input related to an output method of at least one area included in the pathology slide image, and controlling a display device to output a pathology slide image on which the guide is overlaid based on the determined area and area; Computing device.
12. The processor, enlarging or reducing the area while keeping the region fixed, based on a magnification factor designated in response to the user input; The computing device of claim 11.
13. The processor, enlarging or reducing the region while keeping the area fixed, based on a magnification factor specified in response to the user input; The computing device of claim 11.
14. The processor, and controlling the display device to output an image in which at least a portion of the pathology slide image is enlarged or reduced based on additionally received user input. The computing device of claim 11.
15. The processor, and controlling the display device to confirm an area observed by a user in the pathological slide image according to a predetermined criterion, and to output the area observed and other areas separately according to a result of the confirmation. The computing device of claim 11.
16. The predetermined criteria include a magnification ratio specified in response to the user input being equal to or greater than a threshold magnification ratio.
16. The computing device of claim 15.
17. The processor, and controlling the display device to output a thumbnail image in which the observed area and other areas are distinguished according to the predetermined criterion.
16. The computing device of claim 15.
18. The processor, setting at least one region of interest in the pathology slide image, checking a region observed by a user in the pathology slide image according to a predetermined criterion, and determining whether the region observed by the user includes at least one of the regions of interest; The computing device of claim 11.
19. The processor, generating a notification signal for indicating whether or not to further view the pathology slide image based on a result of the determination when a user input related to the end of the viewing of the pathology slide image is received; 20. The computing device of claim 18.
