Method and system for inter-magnification linked display of biological cell images

Abstract

The present invention relates to a method and system for displaying images of living cells linked by magnification. By displaying a low-magnification full slide image, receiving a designation input for a specific area within the low-magnification full slide image, and displaying a high-magnification image corresponding to the designated area, the invention improves diagnostic efficiency by allowing a user to immediately view a high-magnification image of a cell of interest when designating the cell within the low-magnification image.

Description

Technology Field

[0001] The present invention relates to a display technology for biological cell images, and more specifically, to a method and system for displaying a high-magnification image corresponding to a user-specified area in conjunction with a low-magnification full slide image. Background Technology

[0002] Whole Slide Imaging (WSI) technology is widely utilized in the fields of pathological diagnosis and research. WSI systems digitize entire microscope slides to provide a virtual microscope environment, enabling users to observe tissue images at various magnifications on a computer screen.

[0003] In the pathological diagnostic process, pathologists typically observe the entire tissue structure at low magnification to identify the region of interest (ROI), and then switch to high magnification to analyze the morphological characteristics of individual cells in detail. For example, in a blood smear examination, the entire slide is scanned at low magnification to detect leukocytes, and then the detailed morphology of each leukocyte is observed at high magnification to classify the cell type.

[0004] However, conventional WSI systems lacked intuitive integration between low-magnification and high-magnification screens, making it difficult for users to immediately view high-magnification images of cells of interest even if they discovered them on a low-magnification screen. Users faced the inconvenience of having to switch the magnification from low to high and search for the region of interest again, which reduced diagnostic efficiency.

[0005] In particular, in multi-optical systems, since the low-magnification and high-magnification optical systems are physically separated, there was a problem where an incorrect high-magnification image could be displayed to the user if accurate mapping between the cell locations in the low-magnification image and the cell locations in the high-magnification image was not achieved. Prior art literature

[0006] U.S. Patent Publication No. US9412162B2 (Aug. 09, 2016) U.S. Patent Publication No. US11009693B2 (May 18, 2021) U.S. Patent Publication No. US11112952B2 (September 07, 2021)

[0007] Rojo et al., “Critical comparison of 31 commercially available slide systems in pathology”, Int J. Surg. Pathol., 2006; 14(4):285-305 The problem to be solved

[0008] The present invention has been devised to solve the above-mentioned problems and aims to provide a magnification-linked display method and system that immediately displays a high-magnification image corresponding to a specific area when a user designates that area on a screen where a low-magnification full slide image is displayed.

[0009] In addition, another objective of the present invention is to provide an accurate high-magnification image corresponding to a user-specified area through accurate position mapping between a low-magnification image and a high-magnification image.

[0010] In addition, another objective of the present invention is to enable users to easily search for and analyze specific types of cells by classifying and managing high-magnification images according to cell type. means of solving the problem

[0011] According to one aspect of the present invention for achieving the above-mentioned purpose, a method for displaying images of living cells in conjunction with magnification levels is provided.

[0012] A method according to the present invention comprises the steps of: displaying a low-magnification full slide image of a biological cell sample; receiving a designation input for a specific region in the low-magnification full slide image; and displaying a high-magnification image corresponding to the designated region. Herein, the high-magnification image is determined based on position mapping information between the low-magnification image and the high-magnification image.

[0013] In one embodiment, the designated input may include at least one of a user's mouse click, touch input, or drag input. In another embodiment, the designated input may include cells of a specific type being designated in bulk as a specific cell type is selected from a cell type selection bar.

[0014] In one embodiment, the position mapping information may be generated based on a feedforward value including distance information between optical systems between a low-magnification optical system and a high-magnification optical system. The feedforward value may be calculated as the sum of a reference point corresponding to a reference point of a slide, the distance between optical systems, and an environment compensation value.

[0015] In one embodiment, the position mapping information may further include a cell compensation value that corrects the error between the actual position and the predicted position of an object detected in a high-magnification image. The cell compensation value may be calculated individually for each object to improve the accuracy of the position mapping.

[0016] In one embodiment, the high-magnification images may be stored by classifying them into individual cell units, and the classification may include classifying them by cell type based on the morphological characteristics of the living cells.

[0017] According to another aspect of the present invention, a magnification-linked display system for biological cell images is provided.

[0018] The above system includes a display unit that displays a low-magnification full slide image of a living cell sample and a high-magnification image corresponding to a designated area in the low-magnification full slide image, an input unit that receives a designated input for a specific area in the low-magnification full slide image, a mapping information storage unit that stores location mapping information between the low-magnification image and the high-magnification image, and a control unit that controls the display unit to determine a corresponding high-magnification image based on the location mapping information in response to the designated input.

[0019] In one embodiment, the system may further include a high-magnification image storage unit that stores high-magnification images by classifying them into individual cell units. The position mapping information may be generated based on feedforward values ​​and cell compensation values, and the display unit may include a first display area for displaying a low-magnification whole slide image and a second display area for displaying a high-magnification image. Effects of the invention

[0020] According to the present invention, when a user designates a specific area in a low-magnification full slide image, a high-magnification image corresponding to that area is immediately displayed, thereby enabling intuitive coordinated observation between low and high magnifications and significantly improving diagnostic efficiency.

[0021] In addition, according to the present invention, by using position mapping that combines feedforward correction and feedback correction, it is possible to provide an accurate high-magnification image corresponding to a designated area in a low-magnification image, thereby having the effect of improving the accuracy of diagnosis.

[0022] In addition, according to the present invention, since high-magnification images are classified and managed by cell type, there is an effect that allows users to selectively search only for specific types of cells or easily check statistical information by cell type. Brief explanation of the drawing

[0023] FIG. 1 is a conceptual diagram of a scale-linked display method according to one embodiment of the present invention. FIG. 2 is a block diagram of a scale-linked display system according to one embodiment of the present invention. FIG. 3 is a drawing showing a user interface screen according to one embodiment of the present invention. FIG. 4 is a flowchart of a scale-linked display method according to one embodiment of the present invention. FIG. 5 is a diagram illustrating a process for generating a low-magnification full slide image according to one embodiment of the present invention, wherein (a) shows a plurality of low-magnification image tiles captured by a low-magnification optical system, and (b) shows a full slide image generated by stitching the plurality of low-magnification images. FIG. 6 is a diagram illustrating an object detection and location information storage process according to one embodiment of the present invention. FIG. 7 is a diagram illustrating the process of generating location mapping information according to one embodiment of the present invention. FIG. 8 is a diagram illustrating the process of calculating a feedforward value according to one embodiment of the present invention, wherein (a) represents a reference point of the slide and (b) represents the inter-optic distance between the low-magnification optical system and the high-magnification optical system. FIG. 9 is a diagram illustrating a feedback correction process according to one embodiment of the present invention. FIG. 10 is a diagram showing the results of cell type-based classification according to one embodiment of the present invention. Specific details for implementing the invention

[0024] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings so that those skilled in the art to which the present invention pertains can easily implement it. However, embodiments of the present invention may be modified in various different forms, and the scope of the present invention is not limited to the embodiments described below. Furthermore, embodiments of the present invention are provided to more completely explain the present invention to those skilled in the art.

[0025] FIG. 1 is a conceptual diagram of a magnification-linked display method according to one embodiment of the present invention. Referring to FIG. 1, the magnification-linked display method according to the present invention is characterized in that when a user specifies a specific area (11) on a screen where a low-magnification full slide image (10) is displayed, a high-magnification image (20) corresponding to the area (11) is immediately displayed.

[0026] For example, when a user (pathologist) observes a low-magnification full slide image (10) during a blood smear test and becomes interested in a specific white blood cell, the user can specify the area of ​​the white blood cell by clicking or touching it with a mouse. In response, the system immediately displays a high-magnification image (20) of the white blood cell, thereby allowing the user to intuitively perform high-magnification observation without the need to manually switch the magnification and re-search the area of ​​interest.

[0027] FIG. 2 is a block diagram of a magnification-linked display system (100) according to one embodiment of the present invention. Referring to FIG. 2, the system (100) according to the present invention may include a display unit (110), an input unit (120), a mapping information storage unit (130), a control unit (140), and a high-magnification image storage unit (150).

[0028] The display unit (110) displays a low-magnification full slide image and a high-magnification image. In one embodiment, the display unit (110) may include a first display area for displaying a low-magnification full slide image and a second display area for displaying a high-magnification image. The first display area and the second display area may be divided and arranged within a single screen or configured as separate screens.

[0029] The input unit (120) receives a designated input for a specific area in a low-magnification full slide image. In one embodiment, the designated input may include at least one of a user's mouse click, touch input, or drag input. In another embodiment, the designated input may include all cells of a specific type being designated collectively as a specific cell type is selected from the cell type selection bar (330). For example, if the user selects lymphocytes (LY) from the cell type selection bar (330), all lymphocytes in the slide may be designated collectively, and the corresponding high-magnification images may be displayed in a gallery format in the second display area.

[0030] The mapping information storage unit (130) stores location mapping information between a low-magnification image and a high-magnification image. The location mapping information is information indicating which location in the high-magnification image corresponds to a specific location in the low-magnification image. The process of generating the location mapping information will be described later.

[0031] The high-magnification image storage unit (150) stores high-magnification images that have been captured in advance. In one embodiment, the high-magnification images may be stored by classifying them into individual cell units.

[0032] The control unit (140) receives a designated input from the input unit (120), determines a high-magnification image corresponding to the designated area based on the location mapping information stored in the mapping information storage unit (130), and controls the display unit (110) to display it.

[0033] In one embodiment, each component of the system (100) may be implemented in hardware, software, or a combination thereof. For example, the control unit (140) may be implemented as a software module executed by various computing means such as a CPU, GPU, NPU, ASIC, FPGA, etc. The mapping information storage unit (130) and the high-magnification image storage unit (150) may be implemented in various storage media such as RAM, ROM, SSD, HDD, flash memory, cloud storage, etc.

[0034] FIG. 3 is a drawing showing a user interface screen according to an embodiment of the present invention. Referring to FIG. 3, the user interface screen may include a cell type selection bar (330), a first display area (310) for displaying a low-magnification full slide image, and a second display area (320) for displaying a high-magnification image.

[0035] A cell type selection bar (330) is positioned at the top of the screen and displays a selection button for each of a plurality of cell types and cell count information for that type. By selecting a specific cell type from the cell type selection bar (330), the user can filter and display only high-magnification images of that type in the second display area (320). In one embodiment, the currently selected cell type may be highlighted in a different color (e.g., sky blue).

[0036] A low-magnification full slide image is displayed in the first display area (310), and the locations of detected cells may be indicated by boxes or markers. In one embodiment, different colored boxes (e.g., red, green) may be indicated depending on the cell type. The user may specify a particular cell by clicking or touching it in the first display area (310), and a high-magnification image corresponding to the specified cell is highlighted in the second display area (320).

[0037] In the second display area (320), a plurality of high-magnification images corresponding to the type selected from the cell type selection bar (330) are displayed in a gallery format. The gallery may display a plurality of high-magnification cell images arranged in a grid format. When a user selects a specific high-magnification image from the gallery in the second display area (320), reverse linkage may be performed so that a low-magnification position corresponding to that high-magnification image is highlighted in the first display area (310).

[0038] In one embodiment, a reduced thumbnail image of the entire slide may be displayed on one side of the first display area (310), and the location of the area currently being displayed in the thumbnail image may be indicated by a box. In addition, meta information of the slide (e.g., date, patient name, specimen information, etc.) may be displayed together.

[0039] FIG. 4 is a flowchart of a method for inter-magnification display according to an embodiment of the present invention. Referring to FIG. 4, the method according to the present invention includes the step of displaying a low-magnification full slide image (S110), the step of receiving a designated input (S120), and the step of displaying a corresponding high-magnification image (S130).

[0040] In step (S110), the display unit (110) displays a low-magnification full slide image of a biological cell sample. The low-magnification full slide image may be generated by stitching together a plurality of low-magnification images previously captured by a low-magnification optical system.

[0041] In step (S120), the input unit (120) receives a designation input for a specific area in a low-magnification full slide image. The designation input may include at least one of a first type and a second type. The designation input of the first type is for the user to individually designate specific cells by directly clicking or touching them in the low-magnification full slide image. The designation input of the second type is for all cells of a specific type to be designated collectively by the user selecting a specific cell type from the cell type selection bar (330). For example, if the user selects "LY" (lymphocyte) from the cell type selection bar (330), all lymphocytes (e.g., 45) within the slide are designated collectively.

[0042] In step (S130), the control unit (140) determines a high-magnification image corresponding to the designated area based on the location mapping information stored in the mapping information storage unit (130) and displays it on the display unit (110). When a plurality of cells are designated collectively by a second type of designated input, all high-magnification images of that type may be displayed in a gallery format on the second display area (320).

[0043] FIGS. 5 and 6 are drawings illustrating a process for generating a low-magnification full slide image and detecting objects according to an embodiment of the present invention. The low-magnification full slide image and position mapping information may be generated in a preliminary preparation step for providing an integrated display to a user. Referring to FIG. 5(a), multiple regions of a biological cell sample are captured through a low-magnification optical system to obtain multiple low-magnification images (e.g., 0.bmp to 83.bmp). Referring to FIG. 5(b), the multiple low-magnification images are stitched together to generate a full slide image. In one embodiment, the stitching may be performed by aligning the overlapping regions between adjacent images by performing template matching based on the shooting position information of each low-magnification image.

[0044] Referring to FIG. 6, objects of interest (e.g., biological cells) are detected in the entire slide image, and location information of each detected object is stored. In one embodiment, the object detection may be performed using at least one of color, shape, and texture features.

[0045] More specifically, the object detection can be performed using traditional image processing techniques (e.g., binarization, edge detection, morphological operations, etc.) or deep learning-based techniques (e.g., CNN, object detection neural network, etc.). However, the object detection techniques are exemplary, and the present invention is not limited to specific object detection techniques and can be replaced with other object detection techniques that perform substantially the same function.

[0046] The location information of each detected object can be expressed as in Equation 1.

[0047] Mathematical formula 1 :

[0048] LowObject i = [Index x , Index y , Pos x , Pos y ]

[0049] Here, Index x and Index y represents the index coordinates of the corresponding low-magnification image tile within the entire slide image, and Pos x and Pos y represents the relative coordinates of the object within the corresponding low-magnification image tile.

[0050] FIGS. 7 to 9 are drawings illustrating a process for generating position mapping information according to an embodiment of the present invention. The position mapping information defines the correspondence between a position in a low-magnification image and a position in a high-magnification image, and is essential for accurate synchronized display.

[0051] Referring to FIGS. 7 and 8, in one embodiment, the position mapping information may be generated based on a feedforward value (FFV). The feedforward value is set once per slide and is applied when moving the stage for high-magnification image capture. The feedforward value may be calculated as shown in Equation 2.

[0052] Mathematical formula 2 :

[0053] FFV = Reference Point + Inter-Optic Distance + Environment Compensation

[0054] Referring to FIG. 8(a), Reference Point represents a physical reference position corresponding to a reference point of the slide (e.g., top-left corner). Referring to FIG. 8(b), Inter-Optic Distance represents the physical distance between the low-power camera and the high-power camera, that is, the distance between the optical axes of the two optical systems. Environment Compensation is a value for compensating for errors caused by environmental variables, such as cassette tolerances, slide errors, and gripper errors.

[0055] Using the above feedforward value, the high-magnification shooting position corresponding to the object position in the low-magnification image can be determined as in Equation 3.

[0056] Mathematical formula 3 : HighObject i = FFV + LowObject i

[0057] Referring to FIG. 9, in one embodiment, the position mapping information can be adjusted more precisely through feedback correction. Feedback correction is intended to correct minute errors at the individual object level that are not corrected by feedforward correction alone, and is a process of calculating a cell compensation value. The feedback correction is performed at the time of capturing a high-magnification image, and the calculated cell compensation value is included in and stored in the position mapping information.

[0058] The calculation process for the above cell compensation value is as follows. When capturing high-magnification images, first, the feed-forward value (FFV) and the object position in the low-magnification image (LowObject i Based on ), the expected high-magnification shooting location (HighObject i ) is calculated. Next, the stage is moved to the expected high-magnification shooting position, and a high-magnification image of the area is captured through the high-magnification optical system. At this time, if the feedforward correction is accurate, the object should be located at the center of the captured high-magnification image.

[0059] However, in reality, due to factors such as motor precision limitations, equipment vibration, and mechanical backlash, the stage sometimes fails to reach the expected position accurately. In such cases, objects appear at a location other than the center of the captured high-magnification video.

[0060] Therefore, objects are re-detected in the captured high-magnification image, and the offset between the actual location of the detected object and its predicted location (image center) is calculated. This offset becomes the cell compensation value for the corresponding object. For example, if an object is offset from the image center by +5 pixels along the X-axis and -3 pixels along the Y-axis, its cell compensation value becomes (+5, -3).

[0061] The above cell compensation value calculation process is performed individually for each object. This is because errors caused by motor precision and equipment vibration vary with each shot. Therefore, if N objects are detected within a slide, N different cell compensation values ​​are calculated and included in the position mapping information of each object.

[0062] In one embodiment, the cell compensation value can be used to define the final mapping relationship between the object location in the low-magnification image and the high-magnification image of the object. Specifically, the final mapping location can be calculated as shown in Equation 4.

[0063] Mathematical formula 4 :

[0064] Final_Pos i = LowObject i + Cell Compensation i

[0065] Here, Final_Pos i is the image coordinates of the corrected i-th object within the entire low-magnification slide image, and LowObject i is the position of the i-th object in the entire low-magnification slide image, and Cell Compensation i is a cell compensation value calculated individually for the i-th object. The above feedforward value (FFV) is a correction value that is set once per slide and applied only when capturing high-magnification images, whereas the above cell compensation value is a correction value that is set individually for each object and applied when displaying low-magnification images and high-magnification images together. That is, in the step of displaying images to the user after shooting is completed, the above Equation 4 defines an accurate mapping relationship with the corresponding high-magnification image by applying the cell compensation value to the object position within the low-magnification image.

[0066] By combining this two-stage correction system, namely slide-unit feedforward correction (applied during high-magnification shooting) and object-unit feedback correction (reflected in mapping information after shooting), it is possible to provide high-magnification images that accurately correspond to a user-specified location in low-magnification images.

[0067] FIG. 10 is a diagram showing the results of cell type-based classification according to one embodiment of the present invention. In one embodiment, high-magnification images may be classified at the individual cell level and stored in a high-magnification image storage unit (150). The classification may include classifying by cell type based on the morphological characteristics of the living cells.

[0068] In one embodiment, blood cells may be classified into various cell types, such as neutrophils (NB, NS), myelocyte lineage (ME, MY, PR), lymphocyte lineage (LY, LR, LA), monocytes (MO), eosinophils (EO), basophils (BA), blast cells (BL), plasma cells (PC), nucleated red blood cells (NR), platelet-associated cells (GP, PA), macrophages (MA), and artifacts (AR). However, the above cell types are exemplary, and the present invention is not limited to specific cell types; other cell types may be added or changed depending on the diagnostic purpose or the subject of analysis.

[0069] The cell type selection bar (330) displays count information for each cell type, allowing the user to grasp the cell distribution within the slide at a glance. For example, "LY 45" indicates that 45 lymphocytes were detected in the slide. Based on the count information by cell type, the user can perform a differential count or quickly identify abnormal cell distribution patterns.

[0070] Although preferred embodiments of the present invention have been described in the detailed description above, the scope of the present invention is not limited thereto, and it will be obvious to those skilled in the art that various modifications and variations are possible within the scope of the technical concept of the present invention as described in the claims. In particular, the components described in this specification may be replaced with other components that perform substantially the same function, and the order of each step described in the present invention may be changed or performed simultaneously within the scope of technical feasibility. Explanation of the symbols

[0071] 10: Low magnification full slide video 11: Designated area 20: High-magnification video 100: Scale-linked display system 110: Display section 120: Input section 130: Mapping information storage unit 140: Control unit 150: High-magnification image storage unit 310: First display area (low-magnification image display) 320: Second display area (high-magnification video gallery) 330: Cell Type Selection Bar

Claims

Claim 1 A method for a magnification-linked display of a biological cell image comprises: a step of displaying a low-magnification full slide image of a biological cell sample captured by a low-magnification optical system; a step of receiving a designation input for a specific area in the low-magnification full slide image; and a step of displaying a high-magnification image captured by a high-magnification optical system corresponding to the designated area, wherein the high-magnification image is determined based on position mapping information between the low-magnification full slide image and the high-magnification image, and the position mapping information includes a feedforward value comprising optical system distance information between the low-magnification optical system and the high-magnification optical system. A method for magnification-linked display of a biological cell image, characterized in that it is generated based on a cell compensation value calculated individually for each object, which corrects the error between the actual position and the expected position of an object detected in the high-magnification image, and the feedforward value is calculated as the sum of an environment compensation value including at least one of a reference point corresponding to a reference point of a slide, an optical system distance corresponding to the physical distance between the low-magnification optical system and the high-magnification optical system, a slide error, a cassette tolerance, and a gripper error. Claim 2 A method for displaying magnification-linked images of biological cells according to claim 1, wherein the low-magnification full slide image is displayed in a first display area, the high-magnification image is displayed in a second display area separated from the first display area, the designated input includes collectively designating cells of a specific type as a specific cell type is selected from a cell type selection bar, and a plurality of high-magnification images corresponding to the collectively designated cells are displayed in a gallery form in the second display area. Claim 3 delete Claim 4 delete Claim 5 A magnification-linked display system for biological cell images comprises: an input unit receiving a designated input for a specific area in a low-magnification full slide image; a display unit displaying a low-magnification full slide image of a biological cell sample captured by a low-magnification optical system and a high-magnification image captured by a high-magnification optical system corresponding to the designated input; a mapping information storage unit storing position mapping information between the low-magnification full slide image and the high-magnification image; and a control unit controlling to determine a corresponding high-magnification image based on the position mapping information in response to the designated input and to display it on the display unit, wherein the position mapping information includes a feedforward value including optical system distance information between the low-magnification optical system and the high-magnification optical system. A magnification-linked display system for biological cell images, characterized in that it corrects the error between the actual position and the expected position of an object detected in the high-magnification image and is generated based on a cell compensation value calculated individually for each object, wherein the feedforward value is calculated as the sum of an environment compensation value including at least one of a reference point corresponding to a reference point of a slide, an optical system distance corresponding to the physical distance between the low-magnification optical system and the high-magnification optical system, a slide error, a cassette tolerance, and a gripper error. Claim 6 A magnification-linked display system for biological cell images according to claim 5, wherein the display unit comprises: a first display area for displaying the entire low-magnification slide image; and a second display area for classifying the high-magnification image by cell type and displaying it in a gallery format.

Images