Information processing device, information processing method, and program

Abstract

To display information on a segment drawn in a tissue image, in a position that not to overlap an area where the segment is drawn.SOLUTION: An information processing apparatus includes a recognition unit, an interest area information acquisition unit, a generation unit, and a display control unit. The recognition unit recognizes an area to be observed, in a diagnostic image. The interest area information acquisition unit acquires first interest area information representing an interest area in the diagnostic image. The generation unit generates second area information representing the interest area, on the basis of the area to be observed and the first interest area information. The display control unit causes a display unit to display the second area information in a position that does not overlap the area to be observed.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The embodiments disclosed in this specification and the drawings relate to an information processing device, an information processing method, and a program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, in the field of pathology and the like, there is known a technique for setting information such as annotations for tissue images obtained by capturing images of multiple slices cut out from a specimen, based on user operations or image processing results.

[0003] However, there are cases where the information displayed on the tissue image overlaps with a slice depicted on the tissue image, making it difficult for a doctor or other person to see the area he or she wants to observe. In addition, when a user manually adjusts the display position so that the information displayed on the tissue image does not overlap with a slice, the workload can be high. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2016-517559 Summary of the Invention [Problem to be solved by the invention]

[0005] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to display information about a slice depicted in a tissue image at a position that does not overlap with the region in which the slice is depicted. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0006] The information processing apparatus according to the embodiment includes a recognition unit, a region of interest information acquisition unit, a generation unit, and a display control unit. The recognition unit recognizes an observation target region in a diagnostic image. The region of interest information acquisition unit acquires information on a region of interest in the diagnostic image. One or more objects in the observation area Areas of Interest Range The generating unit obtains first region of interest information representing the observation target region and the first region of interest information. Corresponding to the first region of interest information, Areas of Interest Range The display control unit generates second region information representing the second region information on the display unit. Position outside the observation area Display it on. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of an overall configuration of an information processing system according to the first embodiment. [Diagram 2] FIG. 2 is a diagram for explaining a flow until a tissue image is captured according to the first embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of a tissue image according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of an abnormal portion according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of a display annotation according to the first embodiment. [Figure 6] FIG. 6 is a flowchart showing an example of the flow of a generation process of a display annotation according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of an instruction annotation according to the second embodiment. [Figure 8] FIG. 8 is a diagram showing an example of a display annotation according to the second embodiment. [Figure 9] FIG. 9 is a diagram showing another example of the instruction annotation and the display annotation according to the second embodiment. [Figure 10] FIG. 10 is a flowchart showing an example of the flow of a generation process of a display annotation according to the second embodiment. [Figure 11]FIG. 11 is a diagram showing an example of an instruction annotation and a display annotation according to the first modification. [Figure 12] FIG. 12 is a diagram showing an example of an instruction annotation and a display annotation according to the second modification. [Figure 13] FIG. 13 is a diagram showing an example of an instruction annotation and a display annotation according to the third modification. [Figure 14] FIG. 14 is a diagram showing an example of a screen display according to the fourth modification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, embodiments of an information processing device, an information processing method, and a program will be described in detail with reference to the drawings.

[0009] (First embodiment) Fig. 1 is a diagram showing an example of the overall configuration of an information processing system S according to the first embodiment. As shown in Fig. 1, the information processing system S includes an information processing device 100, a specimen image storage device 201, and a tissue image storage device 202. The information processing device 100 is communicatively connected to the specimen image storage device 201 and the tissue image storage device 202 via a network 300 such as an in-hospital LAN (Local Area Network).

[0010] The information processing system S may further include a hospital information system (HIS), a laboratory information system (LIS), a radiology information system (RIS), etc. Alternatively, the information processing system S may be a part of the hospital information system. The information processing system S may further include a terminal device such as a PC (Personal Computer) or a tablet terminal.

[0011] The information processing system S is installed in, for example, a medical institution such as a hospital, a research institute such as a university, an examination center, etc. Furthermore, some or all of the devices constituting the information processing system S may be installed in a cloud environment.

[0012] The specimen image storage device 201 is a device that stores specimen images. The specimen image is an image of the entire tissue piece, which is a specimen. The specimen image is a digital image captured by an imaging device such as a digital camera. The specimen image is also called a macro image.

[0013] In this embodiment, the specimen is a portion of tissue collected from a patient's body, an animal, etc. For example, the specimen is the mucosa and submucosa of the digestive tract excised by endoscopic submucosal dissection (ESD), but is not limited thereto, and may be a piece of tissue excised by laparotomy or the like.

[0014] The tissue image storage device 202 is a device that stores tissue images. A tissue image is an image of at least one slice cut out from a tissue piece that is a specimen, captured at a higher magnification than a specimen image. For example, a tissue image is a WSI (Whole Slide Imaging) image in which the entire slice or a part of the slice placed on a glass slide is digitalized with high accuracy. A tissue image may be a micrograph captured by a WSS (Whole Slide Scanner) or a digital microscope, or an image captured by other methods. A tissue image is also called a micro image. A tissue image is an example of a diagnostic image in this embodiment.

[0015] The sample image storage device 201 and the tissue image storage device 202 are, for example, a server device or a PC. The sample image storage device 201 and the tissue image storage device 202 may be collectively referred to as an image storage device. Although the sample image storage device 201 and the tissue image storage device 202 are illustrated as separate devices in Fig. 1, the sample image storage device 201 and the tissue image storage device 202 may be configured as an integrated device. The sample image storage device 201, the tissue image storage device 202, and the information processing device 100 may be configured as an integrated device.

[0016] FIG. 2 is a diagram for explaining a flow until a tissue image according to the first embodiment is captured. For example, a specimen 5 collected from a patient P by a diagnostician is transferred to a pathologist or a laboratory technician for pathological diagnosis. At this time, a pathology number is assigned for each request (order) for pathological diagnosis. Also, a different specimen number is assigned for each specimen to be the subject of pathological diagnosis. Note that, when there are multiple specimens to be diagnosed in one request for pathological diagnosis, multiple specimen numbers are associated with one pathology number. Hereinafter, in this embodiment, a pathologist or a laboratory technician is referred to as a pathologist, etc.

[0017] The processing performed by the information processing device 100 in this embodiment does not necessarily have to be for the purpose of pathological diagnosis. For example, the processing may be for the purpose of research by a research institute or for the purpose of creating a report at an examination center that performs pathological examinations on behalf of a medical institution.

[0018] A specimen image ID is assigned as identification information of the specimen image 51 obtained by capturing an image of the specimen 5 using a digital camera or the like. The specimen image ID may be registered as additional information of the specimen image 51. Information related to the specimen image 51, such as the pathology number of the specimen 5 that is the subject of the image of the specimen image 51, the specimen number of the specimen 5, the order number of the pathological diagnosis related to the specimen 5, the patient ID of the patient P from whom the specimen 5 was obtained, and the image date and time of the specimen image 51, may be registered as additional information of the specimen image 51. The specimen image ID and other information related to the specimen image 51 may be displayed as text information on the specimen image 51.

[0019] A pathologist or the like cuts out slices 60a-60e from specimen 5, and after performing processing such as staining, for example, slices the cross sections of slices 60a-60e and places them on a glass slide 7. Hereinafter, when there is no need to distinguish between individual slices 60a-60e, they will simply be referred to as slices 60.

[0020] In this embodiment, the group of sections placed on the slide glass 7 is referred to as a tissue specimen 6. The tissue specimen 6 includes at least one section 60.

[0021] In addition, in FIG. 2, each section 60 is placed directly on the glass slide 7, but depending on the length of the section 60 or the shape of the cutting position, one section 60 may be divided into multiple pieces and placed on the glass slide 7.

[0022] In FIG. 2, as an example, the slices 60a to 60e are imaged with their cross sections facing the imaging device, but the placement direction is not limited to this. For example, the slice 60 may be imaged in the same direction as the specimen 5 when the specimen image 51 is imaged. In FIG. 2, the slice 60 is placed on the slide glass 7 so that the horizontal direction of the tissue image 61 is the longitudinal direction of the tissue region 62, but the placement direction is not limited to this. For example, the slice 60 may be placed on the slide glass 7 so that the vertical direction of the tissue image 61 is the longitudinal direction of the tissue region 62. Alternatively, the slice 60 may be placed obliquely on the slide glass 7. In this embodiment, the slice 60 is placed by a method determined by the operation facility.

[0023] An image such as a WSI image obtained by capturing an image of the tissue specimen 6 placed on the slide glass 7 is a tissue image 61. In the tissue image 61, the tissue specimen 6 including slices 60a to 60e is depicted.

[0024] A tissue image ID is assigned as identification information of the tissue image 61. The tissue image ID may be registered as additional information of the tissue image 61. Information related to the tissue image 61, such as the pathology number of the specimen 5 from which the slice that is the subject of imaging of the tissue image 61 was obtained, the specimen number of the specimen 5, the order number of the pathological diagnosis related to the specimen 5, the patient ID of the patient P from which the specimen 5 was obtained, and the imaging date and time of the tissue image 61, may be registered as additional information of the tissue image 61. The tissue image ID and other information related to the tissue image 61 may be depicted as text information on the tissue image 61.

[0025] The tissue image 61 includes a background region 70 depicting the glass slide 7, and a plurality of tissue regions 62a-62e depicting the tissue specimen 6. Hereinafter, the tissue regions 62a-62e will be simply referred to as tissue regions 62 when they are not to be distinguished from one another.

[0026] Returning to FIG. 1, the information processing device 100 is, for example, a server device or a PC, and includes a NW interface 110, a storage circuit 120, an input interface 130, a display 140, and a processing circuit 150.

[0027] The NW interface 110 is connected to the processing circuit 150, and controls the transmission and communication of various data between the information processing device 100 and the sample image storage device 201 or the tissue image storage device 202. The NW interface 110 is realized by a network card, a network adapter, a NIC (Network Interface Controller), or the like.

[0028] The memory circuitry 120 pre-stores various types of information used by the processing circuitry 150. The memory circuitry 120 also stores various programs.

[0029] The input interface 130 is realized by a trackball, a switch button, a mouse, a keyboard, a touch pad that performs an input operation by touching the operation surface, a touch screen in which a display screen and a touch pad are integrated, a non-contact input circuit using an optical sensor, a voice input circuit, etc. The input interface 130 is connected to the processing circuit 150, and converts an input operation received from an operator into an electric signal and outputs it to the processing circuit 150. Note that in this specification, the input interface is not limited to an input interface having physical operation parts such as a mouse and a keyboard. For example, an electric signal processing circuit that receives an electric signal corresponding to an input operation from an external input device provided separately from the device and outputs the electric signal to the processing circuit 150 is also included as an example of the input interface 130.

[0030] The display 140 is a liquid crystal display, an organic electro-luminescence (OEL) display, or the like. The input interface 130 and the display 140 may be integrated. For example, the input interface 130 and the display 140 may be realized by a touch panel. The display 140 is an example of a display unit.

[0031] The processing circuitry 150 is a processor that reads out programs from the storage circuitry 120 and executes them to realize functions corresponding to each program. The processing circuitry 150 of this embodiment includes an acquisition function 151, a recognition function 152, an image processing function 153, a generation function 154, a reception function 155, and a display control function 156. The acquisition function 151 is an example of an acquisition unit. The recognition function 152 is an example of a recognition unit. The image processing function 153 is an example of an image processing unit and an interest region information acquisition unit. The generation function 154 is an example of a generation unit. The reception function 155 is an example of a reception unit. The display control function 156 is an example of a display control unit.

[0032] Here, for example, each processing function of the acquisition function 151, the recognition function 152, the image processing function 153, the generation function 154, the reception function 155, and the display control function 156, which are components of the processing circuit 150, are stored in the storage circuit 120 in the form of a program executable by a computer. The processing circuit 150 is a processor. For example, the processing circuit 150 realizes a function corresponding to each program by reading the program from the storage circuit 120 and executing it. In other words, the processing circuit 150 in a state in which each program is read has each function shown in the processing circuit 150 in FIG. 1. Note that, in FIG. 1, it has been described that the processing functions performed by the acquisition function 151, the recognition function 152, the image processing function 153, the generation function 154, the reception function 155, and the display control function 156 are realized by a single processor, but the processing circuit 150 may be configured by combining multiple independent processors, and each processor may realize a function by executing a program. In addition, although FIG. 1 illustrates a single memory circuit 120 storing a program corresponding to each processing function, multiple memory circuits may be distributed and the processing circuit 150 may read out the corresponding program from each individual memory circuit.

[0033] In the above description, an example has been described in which the "processor" reads out and executes a program corresponding to each function from a storage circuit, but the embodiment is not limited to this. The term "processor" refers to circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). When the processor is, for example, a CPU, the processor realizes a function by reading out and executing a program stored in a storage circuit. On the other hand, when the processor is an ASIC, instead of storing a program in the storage circuit 120, the function is directly incorporated as a logic circuit in the circuit of the processor. Note that each processor in this embodiment is not limited to being configured as a single circuit for each processor, and may be configured as one processor by combining multiple independent circuits to realize the function. Furthermore, multiple components in FIG. 1 may be integrated into one processor to realize the functions thereof.

[0034] The acquiring function 151 acquires the tissue image 61 from the tissue image storage device 202. Also, the acquiring function 151 acquires the sample image 51 from the sample image storage device 201. Note that the source of acquisition of the sample image 51 and the tissue image 61 is not limited to the sample image storage device 201 and the tissue image storage device 202. For example, the acquiring function 151 may acquire the tissue image 61 from an imaging device that captured the tissue image 61.

[0035] The specimen image 51 and tissue image 61 to be acquired are designated, for example, by a user operation accepted by the acceptance function 155. The user of the information processing device 100 is a pathologist or the like.

[0036] A user specifies the specimen image 51 or tissue image 61 to be acquired by inputting, for example, a pathology number, a patient ID, a specimen number, an order number, or an imaging date, or a combination of these. Furthermore, one pathology number corresponds to one order number, one pathology number is associated with one or more specimen numbers and specimen image IDs, and one specimen number is associated with one or more tissue image IDs. Therefore, when a user inputs one pathology number to process a request for a pathological diagnosis, one or more specimen images 51 and one or more tissue images 61 are acquired.

[0037] The user may also select the image to be acquired from a list of the sample images 51 and tissue images 61 stored in the sample image storage device 201.

[0038] The sample image 51 and the tissue image 61 may be stored in advance in the memory circuitry 120 and read out by a user operation.

[0039] The recognition function 152 recognizes, through image processing, a plurality of tissue regions 62 in which the tissue specimen 6 is depicted in the tissue image 61 .

[0040] Fig. 3 is a diagram showing an example of a tissue image 61 according to the first embodiment. The left side of Fig. 3 is a tissue image 61 in which slices 60a-60c included in a tissue specimen 6 are depicted. In the tissue image 61, a glass slide 7 reflected as a background and slices 60f-60h are depicted. As shown on the right side of Fig. 3, the recognition function 152 recognizes a background region 70 in which the glass slide 7 is depicted, and tissue regions 62f-62h in which slices 60f-60h are depicted.

[0041] Although the method for recognizing the tissue region 62 from the tissue image 61 is not particularly limited, for example, the recognition function 152 may recognize the boundary between the background region 70 and the tissue region 62 by executing an image segmentation process on the tissue image 61. In addition, for example, the recognition function 152 may employ rule-based image processing such as selective search that selects candidate regions by grouping similar regions at the pixel level.

[0042] Alternatively, image processing using machine learning such as deep learning may be adopted as a method for recognizing the tissue region 62 from the tissue image 61. When deep learning is adopted, the recognition function 152 inputs the tissue image 61 to a trained model stored in the storage circuitry 120, and acquires a recognition result of the tissue region 62 output from the trained model. The trained model used in the recognition process of the tissue region 62 is a model in which the correspondence between the multiple tissue images 61 and the tissue regions 62 in the multiple tissue images 61 is learned by a method such as deep learning. The trained model may be incorporated into the recognition function 152.

[0043] Furthermore, the recognition function 152 groups the recognized tissue regions 62 into groups of one or more tissue regions 62 that correspond to one section 60. For example, one section 60 may be divided into multiple fragments and arranged on the slide glass 7. In such a case, the multiple tissue regions 62 correspond to one section 60. A group of one or more tissue regions 62 derived from one section 60 is called a section region. The method by which the recognition function 152 recognizes the section region is not particularly limited, and for example, image processing using machine learning such as deep learning may be adopted.

[0044] In the example shown in FIG. 3, all of the tissue regions 62f to 62h are obtained by imaging individual slices 60f to 60h, so that the tissue region 62 and the slice region are equal.

[0045] 1, the image processing function 153 acquires first region-of-interest information that represents a region of interest in the tissue image 61. In this embodiment, the term "acquire" includes obtaining information by image processing.

[0046] In this embodiment, an abnormal region in tissue region 62 is an example of a region of interest. Also, in this embodiment, information indicating the position of the abnormal region on tissue image 61 is an example of first region of interest information. The information indicating the position of the abnormal region is, for example, coordinate information of a pixel where the abnormal region is depicted in tissue image 61.

[0047] An abnormal portion is, for example, a portion where a lesion is depicted. Specifically, a region where a tumor such as an adenoma or adenocarcinoma is depicted in the tissue region 62 is an abnormal portion.

[0048] The image processing function 153 analyzes the degree of abnormality, the presence or absence of abnormality, the range of the abnormal area, and the like in the tissue region 62. The image processing method for analyzing the abnormality is not particularly limited, and any known method can be adopted. For example, the image processing function 153 recognizes an abnormal area from the tissue region 62 by image segmentation processing using a trained model generated by a technology such as deep learning. For example, when a tissue image 61 is input, the trained model outputs information indicating the position of the abnormal area.

[0049] When a plurality of tissue regions 62 are included in one slice region, the image processing function 153 may execute image processing to analyze abnormalities for each slice region. Information on the abnormal site analyzed by the image processing function 153 is associated with the slice region in which the abnormal site is detected.

[0050] FIG. 4 is a diagram showing an example of an abnormal region according to the first embodiment. In the example shown in FIG. 4, three abnormal regions 91a to 91c are included in a tissue region 62i included in a tissue image 61. In FIG. 4, the abnormal regions 91a to 91c are surrounded by contour lines to indicate the ranges of the abnormal regions 91a to 91c, but the contour lines do not need to be displayed on an actual image viewer. Hereinafter, when the individual abnormal regions 91a to 91c are not to be distinguished from one another, they are simply referred to as abnormal regions 91. The number of abnormal regions 91 included in one tissue region 62i is not particularly limited.

[0051] The generating function 154 generates second region information representing the region of interest based on the tissue region 62 and the first region of interest information. In this embodiment, the generating function 154 generates a display annotation representing the abnormal region based on the tissue region 62 and information indicating the position of the abnormal region on the tissue image 61.

[0052] In general, an annotation is an image or comment that notes or provides other information about tissue region 62. In this embodiment, the display annotation is an image that indicates the location of an abnormality in tissue region 62.

[0053] Fig. 5 is a diagram showing an example of display annotations 81a to 81c according to the first embodiment. Fig. 5 shows the display annotations 81a to 81c corresponding to the abnormal regions 91a to 91c described in Fig. 4. Hereinafter, when there is no need to distinguish between the individual display annotations 81a to 81c, they will simply be referred to as the display annotation 81.

[0054] The generation function 154 generates display annotations 81a-81c that represent the positions of the abnormal regions 91a-91c and do not overlap the tissue region 62i based on the positions of the abnormal regions 91a-91c. The display annotations 81a-81c are displayed on the background region 70 where the tissue region 62i is not depicted.

[0055] In this embodiment, the display annotations 81a to 81c have a shape that follows the outside of the contour of the tissue region 62i. The coordinates of the left and right ends of the display annotations 81a to 81c in the horizontal direction of the tissue image 61 match the coordinates of the left and right ends of the abnormal sites 91a to 91c.

[0056] In other words, the coordinates of the start and end of the display annotation 81 in the horizontal direction of the tissue image 61 match the coordinates of the start and end of the abnormal region 91. That is, the range of the display annotation 81 in the horizontal direction of the tissue image 61 matches the range of the abnormal region 91. The horizontal direction is an example of the first direction. When the tissue regions 62 are arranged vertically, the first direction may be the vertical direction. Note that the left end may be the start end and the right end may be the end end, or vice versa.

[0057] 5, the horizontal range of the display annotation 81a, like the abnormal region 91a, is from the left end of the tissue image 61 to the line L1. The left ends of the display annotation 81a and the abnormal region 91a are both at the left end of the tissue image 61, and the right ends of the display annotation 81a and the abnormal region 91a are both at the position indicated by the line L1.

[0058] The horizontal ranges of the display annotation 81b and the abnormal region 91b are both from the line L2 to the line L3. The horizontal ranges of the display annotation 81c and the abnormal region 91c are both from the line L4 to the right end of the tissue image 61.

[0059] 5, when the horizontal ranges of multiple display annotations 81 overlap, the generation function 154 moves any one of the display annotations 81 in a direction away from the tissue region 62i to prevent the multiple display annotations 81 from overlapping and becoming invisible. Note that the display position of the display annotation 81 may be determined at the time of generation by the generation function 154, or may be adjusted by the display control function 156 when it is displayed.

[0060] The positions and shapes of the display annotations 81a to 81c are not limited to the example shown in FIG. 5, and may be any positions and shapes that do not overlap with the tissue region 62i.

[0061] 5, the display annotations 81a-81c extend horizontally because they extend in a direction along the mucosal surface of the tissue region 62i. When the tissue region 62i is placed so that the mucosal surface is vertical or diagonal to the image, the display annotations 81a-81c also extend vertically or diagonally. In this case, the first direction is the direction along the mucosal surface, and the coordinates of the end of the display annotation 81 in the first direction match the coordinates of the end of the abnormal region 91 in the first direction.

[0062] Returning to Fig. 1, the reception function 155 receives various operations by the user via the input interface 130. For example, when a tissue image 61 includes a plurality of tissue regions 62, the reception function 155 receives a user operation to select one of the tissue regions 62. The selection method is a click operation using a mouse, but is not limited to this.

[0063] The display control function 156 displays the display annotation 81 on the display 140 at a position that does not overlap with the tissue region 62. In this embodiment, the display control function 156 displays the display annotation 81 on the tissue image 61, as shown in FIG.

[0064] Next, a flow of a process for generating a display annotation executed by the information processing device 100 configured as above will be described.

[0065] FIG. 6 is a flowchart showing an example of the flow of a generation process of a display annotation according to the first embodiment.

[0066] First, the acquisition function 151 acquires the sample image 51 from the sample image storage device 201 (S1).

[0067] Then, the recognition function 152 recognizes, by image processing, a plurality of tissue regions 62 in which the tissue specimen 6 is depicted from the tissue image 61. Furthermore, when a plurality of tissue regions 62 correspond to one slice 60, the recognition function 152 recognizes a slice region that is a group of a plurality of tissue regions 62 corresponding to the same slice 60 by image segmentation processing or the like (S2).

[0068] Next, the image processing function 153 recognizes an abnormal region 91 on the tissue image 61 by image processing (S3). The image processing function 153 specifies the position and range on the tissue image 61 of the recognized abnormal region 91.

[0069] Next, the generation function 154 generates a display annotation 81 based on the position and range of the abnormal site 91 on the tissue image 61 and the position on the tissue image 61 (S4). The generation function 154 stores the display annotation 81 in the memory circuitry 120 in association with the tissue region 62 or slice region including the abnormal site 91.

[0070] Then, the display control function 156 superimposes the display annotation 81 on the tissue image 61 and displays it on the display 140 (S5). The display annotation 81 may be displayed when the user selects the tissue image 61 as the display target, or may be displayed when the user selects the display annotation 81 after the tissue image 61 is displayed. Here, the processing of this flowchart ends.

[0071] In this manner, the information processing device 100 of this embodiment generates a display annotation 81 representing the abnormal region 91 based on information indicating the positions of the tissue region 62 and the abnormal region 91 in the tissue image 61. The information processing device 100 of this embodiment displays the display annotation 81 on the display 140 at a position that does not overlap the tissue region 62. Therefore, according to the information processing device 100 of this embodiment, information about the slice 60 depicted in the tissue image 61 is displayed at a position that does not overlap the tissue region 62 in which the slice 60 is depicted.

[0072] Furthermore, the information processing device 100 of this embodiment acquires information indicating the position of the abnormal region 91 from the tissue image 61, generates a display annotation 81 that indicates the position of the abnormal region 91 and does not overlap with the tissue region 62 based on the information indicating the position of the abnormal region 91, and displays the display annotation 81 on the tissue image 61. Therefore, according to the information processing device 100 of this embodiment, the user can avoid the display annotation 81 from interfering with observation of the tissue region 62.

[0073] Moreover, the display annotation 81 of this embodiment has a shape that follows the outside of the contour of the tissue region 62, and the range of the display annotation 81 in the first direction of the tissue image 61 coincides with the range of the abnormal region 91 in the first direction. Therefore, according to the information processing device 100 of this embodiment, the user can easily grasp the position and range of the abnormal region 91 in the tissue region 62.

[0074] Second Embodiment In the above-described first embodiment, the display annotation is displayed based on the information indicating the position of the abnormal region 91 acquired by image processing. In this second embodiment, the display annotation is displayed based on the designation of a region of interest by the user.

[0075] The reception function 155 of this embodiment receives an input operation of an instruction annotation by a user. The reception function 155 is an example of an interest region information acquisition unit and a reception unit in this embodiment. In this embodiment, "acquire" also includes obtaining information from an input by a user. Also, the instruction annotation is an example of first interest region information in this embodiment.

[0076] 7 is a diagram showing an example of an instruction annotation 91d according to the second embodiment. For example, a user draws the instruction annotation 91d by operating a mouse or the like on a tissue image 61 displayed on a display 140. The tissue image 61 is an example of a diagnostic image.

[0077] The instruction annotation 91d is an image that indicates the position and range of a region of interest in the tissue region 62.

[0078] The range of the instruction annotation 91d in the direction of the mucosal surface of the tissue region 62j coincides with the range of the region of interest in the direction of the mucosal surface of the tissue region 62j. In this embodiment, the region of interest is determined by the user. For example, a region that the user determines may be a lesion site becomes the region of interest. The tissue region 62j is an example of a region to be observed.

[0079] The generating function 154 of this embodiment generates, based on the tissue region 62 and the instruction annotation 91d, a display annotation that does not overlap with the tissue region 62. The display annotation is an example of second region information.

[0080] 8 is a diagram showing an example of a display annotation 81d according to the second embodiment. The position and range of the display annotation 81d correspond to the instruction annotation 91d. The range of the display annotation 81d in the horizontal direction of the tissue image 61 is equal to the range of the instruction annotation 91d in the horizontal direction.

[0081] The horizontal ranges of both the display annotation 81d and the instruction annotation 91d are from the straight line L5 to the straight line L6. The display annotation 81d is a linear image that is arranged outside the tissue region 62j along the contour line of the tissue region 62j. The display annotation 81d does not overlap the tissue region 62j and is displayed on, for example, the background region 70.

[0082] FIG. 9 is a diagram showing another example of the instruction annotations 91e and 91f and the display annotations 81e and 81f according to the second embodiment.

[0083] Furthermore, when the tissue image 61 includes multiple tissue regions 62, the tissue region 62 included in the section region selected by the user becomes the target of the instruction annotations 91e, 91f. When the section region includes only one tissue region 62, selecting the section region is substantially the same as selecting the tissue region 62. When one section region includes multiple tissue regions 62, the selection is made by the user on a section region basis. The user does not need to be aware that the selection is being made on a section region basis, and other tissue regions 62 that belong to the same section region as the tissue region 62 selected by the user may be automatically selected.

[0084] 9, it is assumed that the tissue region 62k is selected by the user. In this case, the generating function 154 associates the display annotations 81e and 81f corresponding to the instruction annotations 91e and 91f with the tissue region 62k. The selection of the tissue region 62 by the user is, for example, but not limited to, clicking with a mouse.

[0085] Although tissue regions 62k, 62l, and 62m are included on the tissue image 61, display annotations 81e and 81f are not set in the tissue regions 62l and 62m that are not selected by the user because they are not the targets of instruction annotations 91e and 91f.

[0086] Furthermore, the display control function 156 of this embodiment causes the display 140 to display the instruction annotations 91e and 91f and the display annotations 81e and 81f corresponding to the tissue region 62 selected by the user.

[0087] Furthermore, the display control function 156 does not display the instruction annotations 91e and 91f corresponding to the tissue regions 62 not selected by the user on the display 140. For example, in the example shown in FIG. 9, when the user cancels the selection of the tissue region 62k and no tissue region 62 is selected, or when the user selects other tissue regions 62l and 62m, the display control function 156 ends the display of the instruction annotations 91e and 91f corresponding to the tissue region 62. Furthermore, the display control function 156 displays the display annotations 81e and 81f regardless of whether or not the user has selected them. Note that the display and non-display of the display annotations 81e and 81f may be switchable by a user operation.

[0088] 9, the user may set a plurality of instruction annotations 91e, 91f in one tissue region 62. When the ranges of the plurality of instruction annotations 91e, 91f overlap, the generation function 154 changes the position or shape of the display annotations 81e, 81f corresponding to the overlapping plurality of instruction annotations 91e, 91f so that they do not overlap. The generation function 154 may also change the color or line state of the plurality of display annotations 81e, 81f. When the user is performing an operation to set a plurality of instruction annotations 91e, 91f in one tissue region, the display control function 156 may hide other instruction annotations and display annotations during the operation.

[0089] FIG. 10 is a flowchart showing an example of the flow of a process for generating a display annotation 81 according to the second embodiment.

[0090] The process from obtaining a tissue image in S1 to recognizing a tissue region 62 and a slice region depicted in a tissue image 61 in S2 is the same as in the first embodiment.

[0091] The reception function 155 receives a selection of a slice region by the user (S21), and acquires instruction annotations 91d to 91f set by the user for the tissue region 62 included in the selected slice region (S22).

[0092] The generating function 154 generates a display annotation 81 that does not overlap with the tissue region 62, based on the tissue region 62 and the instruction annotations 91d to 91f (S23).

[0093] Then, the display control function 156 superimposes the display annotation 81 on the tissue image 61 and displays it on the display 140 (S24). Here, the processing of this flowchart ends.

[0094] In this way, the information processing device 100 of this embodiment can display the display annotation 81 that does not hide the tissue region 62, based on the instruction annotations 91d to 91f set by the user. Therefore, according to the information processing device 100 of this embodiment, when the user draws the instruction annotations 91d to 91f, it is not necessary to take the time to avoid writing in the tissue region 62, and the work efficiency can be improved.

[0095] Furthermore, in this embodiment, the instruction annotations 91d-91f are images indicating the position and range of the region of interest in the tissue region 62, and the display annotations 81d-81f are images arranged outside the tissue region along the contour of the tissue region 62. The positions and ranges of the display annotations 81d-81f correspond to the instruction annotations 91d-91f. Therefore, according to the information processing device 100 of this embodiment, the user can display the display annotations 81d-81f indicating the position and range of the region of interest in a position that does not overlap with the tissue region 62, simply by performing an operation to specify the position and range of the region of interest.

[0096] Furthermore, the information processing device 100 of this embodiment displays the instruction annotations 91d-91f and the display annotations 81d-81f corresponding to the tissue region 62 selected by the user on the display 140. Furthermore, the information processing device 100 does not display the instruction annotations 91d-91f corresponding to the tissue regions 62 not selected by the user on the display 140. Therefore, according to the information processing device 100 of this embodiment, it is possible to manage the display / non-display of the instruction annotations 91d-91f and the display annotations 81d-81f for each tissue region 62 or for each slice region based on the user's selection.

[0097] (Variation 1) The display manner of the instruction annotations 91d to 91f is not limited to that of the second embodiment described above. For example, in the second embodiment, the instruction annotations 91d to 91f are displayed at the positions drawn by the user, but the present invention is not limited to this. FIG. 11 is a diagram showing an example of the instruction annotations 91g, 911g and the display annotation 81g according to the first modified example. As shown in FIG. 11, when the position where the user draws the instruction annotation 91g is on the tissue region 62n, the display control function 156 may move the instruction annotation 91g onto the background region 70 and display it as the instruction annotation 911g.

[0098] (Variation 2) In addition, in the first and second embodiments described above, the display annotation 81 is linear, but the present invention is not limited to this. The display annotation 81 may be displayed not only along the contour of one side of the tissue image 61, but also on both sides of the tissue image 61. Fig. 12 is a diagram showing an example of display annotations 81h, 81i according to the second modification. As shown in Fig. 12, the display annotations 81h, 81i may be provided on both sides of the tissue image 61 along a range including a region of interest.

[0099] (Variation 3) Moreover, Fig. 13 is a diagram showing an example of display annotations 81j and 81k according to Modification Example 3. As shown in Fig. 13, the display annotations 81j and 81k may be figures surrounding a range including a region of interest.

[0100] (Variation 4) In the second embodiment described above, the screen on which the user operates the instruction annotations 91d-91f and the screen on which the display annotations 81d-81f are displayed are the same. In contrast, in this modified example, a window for operation and a window for display are provided separately.

[0101] FIG. 14 is a diagram showing an example of a screen display according to the fourth modification. In the example shown in FIG. 14, the display control function 156 provides an operation window 141 and a display window 142 on the display 140. The same tissue image 61 is displayed in the operation window 141 and the display window 142, but settings such as zooming in and out can be changed for each window. For example, as shown in FIG. 14, the user may enlarge the tissue image 61 displayed in the operation window 141 more than the tissue image 61 displayed in the display window 142. In this case, the user can operate instruction annotations 91x, 91y on the tissue image 61 enlarged and displayed in the display window 142, making it easy to perform fine settings.

[0102] 14, the same tissue image 61 is displayed in the operation window 141 and the display window 142, but the display control function 156 may display images in different formats in the operation window 141 and the display window 142. For example, a three-dimensional image of the tissue image 61 may be displayed in the display window 142, and a two-dimensional tissue image 61 may be displayed in the operation window 141.

[0103] Although the above-described embodiments have been described with respect to linear annotations, the form of the annotation to be displayed is not limited to this. For example, text data or an image may be displayed on the display 140. The display control function 156 may display various types of meta information related to the selected slice region 8, for example, in the background region 70 near the slice region selected by the user. Also, a specific area in which meta information is displayed may be provided on the GUI displayed on the display 140.

[0104] (Variation 5) In the above-described embodiments, the term "obtain" includes obtaining information by image processing and obtaining information from user input. In addition, receiving information from another information processing device is also included in "obtain". For example, receiving first region of interest information representing a region of interest from a hospital information system or a clinical examination system by the acquisition function 151 via the NW interface 110 is also included in "obtain". In this case, the acquisition function 151 is an example of a region of interest information acquisition unit.

[0105] (Variation 6) In addition, in each of the above-mentioned embodiments, the user determines the slice area on which the instruction annotation is to be set by performing an operation such as clicking within the slice area on the tissue image 61, but the association of the instruction annotation is not limited to this method. For example, the user may draw the instruction annotation on the desired slice area. In this case, the generation function 154 generates the display annotation in association with the slice area on which the instruction annotation is superimposed.

[0106] Furthermore, the generating function 154 may determine the association between the instruction annotation and the slice region based on the proximity of the distance between the instruction annotation and the slice region, regardless of whether a selection operation is performed. For example, the generating function 154 may generate a display annotation corresponding to the instruction annotation by associating the instruction annotation with the slice region closest to the position where the user has drawn the instruction annotation.

[0107] (Variation 7) In each of the above-mentioned embodiments, the extension direction of the instruction annotations and display annotations was the horizontal direction of the tissue image 61, but depending on the direction of the mucosal surface of the tissue region 62, the extension direction of the instruction annotations and display annotations may be the vertical or diagonal direction of the tissue image 61.

[0108] (Variation 8) In the above-described embodiments, the information processing device 100 is a server device or a PC, but is not limited thereto. For example, the information processing device 100 may be a computer built into an imaging device that captures the tissue image 61. The imaging device is, for example, a WSS, a digital camera, or a digital microscope, but is not limited thereto.

[0109] (Variation 9) Furthermore, in each of the above-described embodiments, the tissue image 61 is taken as an example of a diagnostic image, but the diagnostic image may be a medical image captured by a modality such as an X-ray diagnostic device.

[0110] The various data handled in this specification are typically digital data.

[0111] According to at least one of the embodiments described above, information about a slice depicted in a tissue image can be displayed at a position that does not overlap with the region in which the slice is depicted.

[0112] Although some embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents as described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0113] 5. Specimens 6 Tissue specimen 7. Slides 51 Sample images 60,60a~60e section 61 Tissue Images 62,62a~62p tissue area 70 Background area 81a~81k,81x, 81y Display annotations 91d~91g, 91x, 91y, 911g, 911x, 911y Instruction annotations 91a~91c Abnormal area 100 Information processing device 110 Network Interface 120 Memory circuit 140 Display 141 Operation Window 142 Display Window 150 Processing circuit 151 Acquisition Function 152 Recognition Function 153 Image Processing Functions 154 Generation function 155 Reception Function 156 Display control function S Information Processing System

Claims

1. a recognition unit for recognizing an observation target region in a diagnostic image; a region-of-interest information acquisition unit that acquires one or more first region-of-interest information pieces that represent ranges of one or more regions of interest within the observation target region in the diagnostic image; a generating unit that generates second region information representing a range of the region of interest corresponding to the first region of interest information based on the observation target region and the first region of interest information; a display control unit that displays the second region information at a position outside the observation target region on a display unit; An information processing device comprising:

2. When a plurality of pieces of first region of interest information are acquired and the ranges of the regions of interest indicated by the plurality of pieces of first region of interest information overlap, the display control unit moves a position of the second region information corresponding to any one of the plurality of pieces of first region of interest information, thereby displaying the plurality of pieces of second region information corresponding to the plurality of pieces of first region of interest information on the display unit without overlapping each other. The information processing device according to claim 1 .

3. The region of interest information acquisition unit acquires one or more first region of interest information representing positions and ranges of one or more regions of interest within the observation target region in the diagnostic image, The generating unit generates second region information representing a position and a range of the region of interest corresponding to the first region of interest information, based on the observation target region and the first region of interest information.

3. The information processing device according to claim 1 or 2.

4. The diagnostic image is a tissue image depicting a tissue specimen including at least one slice; the observation target region is a tissue region in which the tissue specimen is depicted in the tissue image, the region of interest is an abnormal site in the tissue region; the region-of-interest information acquisition unit acquires, from the tissue image by image processing, information indicating a position of the abnormal site as the first region-of-interest information; the generating unit generates, as the second region information, a display annotation that represents a position of the abnormal site and does not overlap the tissue region, based on the first region of interest information; The display control unit causes the display annotation to be displayed on the diagnostic image. The information processing device according to claim 1 .

5. the display annotation has a shape that conforms to an outer contour of the tissue region; a range of the display annotation in a first direction of the tissue image corresponds to a range of the abnormality in the first direction; The information processing device according to claim 4.

6. the first region of interest information is an instruction annotation indicating the region of interest, The interest region information acquisition unit receives an input operation of the instruction annotation by a user. the generating unit generates, based on the observation target area and the instruction annotation, a display annotation that does not overlap with the observation target area as second area information; The information processing device according to claim 1 .

7. The diagnostic image is a tissue image depicting a tissue specimen including at least one slice; the observation target region is a tissue region in which the tissue specimen is depicted in the tissue image, the instruction annotation is an image indicating a location and extent of the region of interest in the tissue region; the display annotation is an image arranged outside the tissue region along a contour line of the tissue region, The position and range of the display annotation correspond to the instruction annotation. The information processing device according to claim 6.

8. The instruction annotation is an image composed of straight lines indicating the location and extent of the region of interest in the tissue region. The information processing device according to claim 7.

9. The display control unit: displaying, on the display unit, the instruction annotation and the display annotation corresponding to a tissue region selected by a user; a display form of the instruction annotation corresponding to a tissue region not selected by a user is made different from a display form of the instruction annotation corresponding to a tissue region selected by a user; The information processing device according to claim 6 .

10. The display control unit is displaying, on the display unit, the instruction annotation and the display annotation corresponding to a tissue region selected by a user; The instruction annotation corresponding to a tissue region not selected by a user is not displayed on the display unit. The information processing device according to claim 8.

11. a recognition step of recognizing an observation target region in a diagnostic image; an acquiring step of acquiring one or more first region of interest information representing ranges of one or more regions of interest within the observation target region in the diagnostic image; a generating step of generating second region information representing a range of the region of interest corresponding to the first region of interest information based on the observation target region and the first region of interest information; a display step of displaying the second region information at a position outside the observation target region on a display unit; An information processing method comprising:

12. a recognition step of recognizing an observation target region in a diagnostic image; an acquiring step of acquiring one or more first region of interest information representing ranges of one or more regions of interest within the observation target region in the diagnostic image; a generating step of generating second region information representing a range of the region of interest corresponding to the first region of interest information based on the observation target region and the first region of interest information; a display step of displaying the second region information at a position outside the observation target region on a display unit; A program that causes a computer to execute the following.

Images