Information processing device, method for processing information and program

Abstract

To provide an information processing device that can determine a candidate of a region of interest in an image using information relating to the image, and to provide a method for processing information and a program.SOLUTION: An information processing device includes: one or more processors; one or more storage devices for storing a command that the one or more processors execute. The one or more processors acquire an image, related information relating to the image, and one or more first candidates of a region of interest which are included in the image, estimates one or more image regions that the related information indicates from the image and the related information, and determines a second candidate of a region of interest from the first candidates of the region of interest on the basis of the estimated image regions.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program, and more particularly to a technique for determining candidates for a region of interest in an image. [Background technology]

[0002] Learning an object detection model requires a lot of correct answer data (e.g., bounding boxes), but manual annotation is costly. Therefore, there is research that improves the performance of object detection models by utilizing information other than correct answer data. Non-Patent Document 1 discloses a learning method that adds data annotated at the image level (e.g., data used in class classification).

[0003] Furthermore, Patent Document 1 discloses a method for learning a model that predicts text information from an image using an image of a region of interest and text associated with the image, such as its position, size, properties, etc., as learning data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7080304 [Non-patent literature]

[0005] [Non-Patent Document 1] Hoffman, Judy, et al. "Detector discovery in the wild: Joint multiple instance and representation learning." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015. Summary of the Invention [Problem to be solved by the invention]

[0006] Medical data often exists as pairs of images and reports on the images, but there has been no attempt to use the images and reports to improve object detection models.

[0007] Furthermore, the method disclosed in Patent Document 1 is based on the premise that regions of interest and text are linked, but if the regions of interest contained in an image are not linked to information about the image, it may be unclear which information is linked to which region of interest.

[0008] The present invention has been made in view of the above circumstances, and has an object to provide an information processing device, an information processing method, and a program capable of determining candidate regions of interest in an image using information about the image. [Means for solving the problem]

[0009] In order to achieve the above-mentioned object, an information processing device according to a first aspect of the present disclosure is an information processing device that includes one or more processors and one or more storage devices in which instructions executed by the one or more processors are stored, wherein the one or more processors acquire an image, related information related to the image, and one or more first candidate region of interest contained in the image, estimate one or more image regions indicated by the related information from the image and the related information, and determine a second candidate region of interest from the first candidate region of interest based on the estimated image regions.

[0010] According to this aspect, it is possible to determine a region of interest candidate (second region of interest candidate) in an image corresponding to the related information by using related information related to an image, and it is possible to link the related information with a region of interest in an image. In addition, it is possible to improve the performance of an object detection model by using information on the region of interest candidate in an image determined by this aspect.

[0011] An information processing device according to a second aspect of the present disclosure is the information processing device according to the first aspect, wherein the related information may include text related to the content of the image.

[0012] An information processing device according to a third aspect of the present disclosure is the information processing device according to the first aspect, wherein the related information includes text written regarding a region of interest included in the image.

[0013] An information processing device according to a fourth aspect of the present disclosure is an information processing device according to the first aspect, wherein the related information includes structured text information including at least one of the size, position, and characteristics of a region of interest contained in the image.

[0014] An information processing device according to a fifth aspect of the present disclosure may be configured such that, in an information processing device according to any of the second to fourth aspects, one or more processors estimate at least one of the position, size, and characteristics indicated by the text.

[0015] An information processing device according to a sixth aspect of the present disclosure may be configured in an information processing device according to any one of the second to fifth aspects, wherein the image is a medical image, and the one or more processors recognize organs contained in the image and estimate an image area from text and organ recognition results.

[0016] An information processing device according to a seventh aspect of the present disclosure is an information processing device according to any of the first to sixth aspects, wherein the one or more first region of interest candidates include at least one of a bounding box, a heat map, and a mask.

[0017] An information processing device according to an eighth aspect of the present disclosure may be configured such that, in an information processing device according to any of the first to seventh aspects, one or more processors are configured to accept input of an image, related information, and one or more first candidate regions of interest.

[0018] An information processing device according to a 9th aspect of the present disclosure may be configured in an information processing device according to any of the first to seventh aspects, wherein one or more processors receive input of an image and related information, and acquire one or more first region of interest candidates by generating one or more first region of interest candidates based on the received image.

[0019] An information processing device according to a tenth aspect of the present disclosure may be configured such that, in the information processing device according to the ninth aspect, one or more processors perform a process of placing multiple bounding boxes as candidates for a first region of interest at regular intervals on an image based on a rule base.

[0020] An information processing device according to an eleventh aspect of the present disclosure may be configured in the information processing device according to the ninth aspect, wherein the one or more processors generate one or more first region of interest candidates from the image using a machine learning model trained to receive an image as input and estimate one or more first region of interest candidates from the image.

[0021] An information processing device according to a twelfth aspect of the present disclosure may be configured in the information processing device according to the ninth aspect, wherein one or more processors use an object detection model to generate one or more first candidate regions of interest from an image.

[0022] An information processing device according to a 13th aspect of the present disclosure is an information processing device according to the 12th aspect, in which the object detection model is a model learned by machine learning using training data including the determined region of interest candidates.

[0023] An information processing device according to a fourteenth aspect of the present disclosure may be configured in an information processing device according to any of the first to thirteenth aspects, wherein one or more processors are configured to determine, from among one or more first region of interest candidates, a first region of interest candidate that is included in an estimated image region as a second region of interest candidate.

[0024] An information processing device according to a fifteenth aspect of the present disclosure may be configured in an information processing device according to any of the first to fourteenth aspects, wherein one or more processors acquire a plurality of first region of interest candidates and determine a second region of interest candidate from among the plurality of first region of interest candidates.

[0025] An information processing device according to a 16th aspect of the present disclosure may be configured in such a way that, in an information processing device according to any of the 1st to 15th aspects, one or more processors calculate the probability that the image area indicated by the related information is on a pixel-by-pixel basis.

[0026] An information processing device according to a 17th aspect of the present disclosure is an information processing device according to any of the first to 16th aspects, wherein the image region estimated by the one or more processors may include at least one of a bounding box, a heat map, and a mask.

[0027] An information processing device according to an 18th aspect of the present disclosure may be configured in an information processing device according to any of the 1st to 17th aspects, wherein one or more processors calculate a certainty factor of a first region of interest candidate, and delete, from the one or more first region of interest candidates, any first region of interest candidate that does not correspond to an estimated image region.

[0028] An information processing device according to a 19th aspect of the present disclosure may be configured in an information processing device according to any of the first to 18th aspects, wherein one or more processors calculate evaluation values ​​of one or more first region of interest candidates from an estimated image region, and determine a second region of interest candidate based on the evaluation values.

[0029] In order to achieve the above object, an information processing method according to a twentieth aspect of the present disclosure is an information processing method executed by one or more processors, the information processing method including the one or more processors acquiring an image, related information related to the image, and one or more first region of interest candidates included in the image, estimating one or more image regions indicated by the related information from the image and the related information, and determining a second region of interest candidate from the first region of interest candidates based on the estimated image region. According to this aspect, the region of interest candidate of the image can be determined using the related information related to the image, and therefore the performance of the object detection model can be improved by the image in which the region of interest candidate has been determined.

[0030] The information processing method according to the twentieth aspect may have a configuration including the same specific aspect as the information processing device according to any one of the second to nineteenth aspects.

[0031] In order to achieve the above object, a program according to a twenty-first aspect of the present disclosure is a program that causes a computer to realize a function of acquiring an image, related information related to the image, and one or more first region of interest candidates included in the image, a function of estimating one or more image regions indicated by the related information from the image and the related information, and a function of determining a second region of interest candidate from the first region of interest candidates based on the estimated image region. According to this aspect, the region of interest candidate of the image can be determined using the related information related to the image, and therefore the performance of the object detection model can be improved by the image in which the region of interest candidate has been determined.

[0032] The program according to the twenty-first aspect may have a configuration including the same specific aspect as the information processing device according to any one of the second to nineteenth aspects.

[0033] The present disclosure also includes a non-transitory computer-readable recording medium, such as a CD-ROM (Compact Disk-Read Only Memory) that stores the program according to the twenty-first aspect. Effect of the Invention

[0034] According to the present invention, relevant information about an image can be used to determine candidate regions of interest for the image, and the information of the candidate regions of interest determined according to the present invention can be used to improve the performance of an object detection model. [Brief description of the drawings]

[0035] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a medical information processing system. [Diagram 2] FIG. 2 is a block diagram showing the electrical configuration of the medical information processing device. [Diagram 3] FIG. 3 is a block diagram showing a functional configuration of the medical information processing device. [Figure 4] FIG. 4 is a flowchart showing the medical information processing method according to the first embodiment. [Diagram 5] FIG. 5 is a diagram for explaining the processing of each step of the medical information processing method. [Figure 6] FIG. 6 is a diagram for explaining the processing of each step of the medical information processing method. [Figure 7] FIG. 7 is a diagram for explaining the processing of each step of the medical information processing method. [Figure 8] FIG. 8 is a diagram for explaining an example of processing by the image region estimation unit. [Figure 9] FIG. 9 is a diagram for explaining another example of the process performed by the image region estimation unit. [Figure 10] FIG. 10 is a block diagram illustrating a schematic functional configuration of an object detection system according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Here, a medical information processing device, a medical information processing method, and a medical information processing program will be described as examples of an information processing device, an information processing method, and a program according to the present invention.

[0037] <Medical information processing system> The medical information processing system according to the present embodiment is a system for determining a region of interest candidate, which is a lesion candidate, from a medical image having related information. The medical image from which a region of interest candidate has been determined is utilized as correct answer data for training a learning model that estimates a region of interest from a medical image, thereby improving the performance of the learning model.

[0038] Fig. 1 is an overall configuration diagram of a medical information processing system 10. As shown in Fig. 1, the medical information processing system 10 includes a medical image inspection device 12, a medical image database 14, a user terminal device 16, an image interpretation report database 18, and a medical information processing device 20.

[0039] The medical image inspection equipment 12, the medical image database 14, the user terminal device 16, the image interpretation report database 18, and the medical information processing device 20 are connected to each other via a network 22 so as to be able to transmit and receive data. The network 22 includes a wired or wireless LAN (Local Area Network) that communicatively connects various devices in the medical institution. The network 22 may also include a WAN (Wide Area Network) that connects the LANs of multiple medical institutions.

[0040] The medical imaging inspection equipment 12 is an imaging device that captures an image of an inspection target part of a subject and generates a medical image. Examples of the medical imaging inspection equipment 12 include an X-ray imaging device, a CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, a PET (Positron Emission Tomography) device, an ultrasound device, a CR (Computed Radiography) device using a flat X-ray detector, and an endoscope device.

[0041] The medical image database 14 is a database that manages medical images taken by the medical image inspection equipment 12. The medical image database 14 is implemented by a computer equipped with a large-capacity storage device for storing medical images. Software that provides the functions of a database management system is installed in the computer.

[0042] The medical image may be a two-dimensional or three-dimensional still image taken by an X-ray device, a CT device, an MRI device, or the like, or may be a moving image taken by an endoscope device.

[0043] The format of the medical image can be in accordance with the Dicom (Digital Imaging and Communications in Medicine) standard. Supplementary information (Dicom tag information) defined in the Dicom standard may be added to the medical image. Note that the term "image" in this specification includes not only the image itself, such as a photograph, but also image data, which is a signal representing an image.

[0044] The user terminal device 16 is a terminal device for a doctor to create and view an interpretation report. For example, a personal computer is applied to the user terminal device 16. The user terminal device 16 may be a workstation or a tablet terminal. The user terminal device 16 includes an input device 16A and a display 16B. The doctor uses the input device 16A to input an instruction to display a medical image. The user terminal device 16 displays the medical image on the display 16B. Furthermore, the doctor interprets the medical image displayed on the display 16B, and uses the input device 16A to create an interpretation report, which is the interpretation result.

[0045] The image interpretation report is related information paired with a medical image. The related information includes text related to the contents of the medical image. The related information may include text described with respect to a region of interest included in the medical image. The related information may include structured text information including at least one of the size, position, and characteristics of the region of interest included in the medical image. The related information does not have to be associated with the region of interest in the medical image.

[0046] The image interpretation report database 18 is a database that manages image interpretation reports generated by doctors on the user terminal device 16. The image interpretation report database 18 is implemented by a computer equipped with a large-capacity storage device for storing image interpretation reports. Software that provides the functions of a database management system is installed in the computer. The medical image database 14 and the image interpretation report database 18 may be configured in a single computer.

[0047] The medical information processing device 20 is a device that determines candidates for a region of interest of a medical image. The medical information processing device 20 can be a personal computer or a workstation (an example of a "computer"). FIG. 2 is a block diagram showing the electrical configuration of the medical information processing device 20. As shown in FIG. 2, the medical information processing device 20 includes a processor 20A, a memory 20B, and a communication interface 20C.

[0048] The processor 20A executes instructions stored in the memory 20B. The hardware structure of the processor 20A is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and acts as various functional units, a GPU (Graphics Processing Unit), which is a processor specialized for image processing, a PLD (Programmable Logic Device), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing specific processing such as an ASIC (Application Specific Integrated Circuit).

[0049] A processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Also, multiple functional units may be configured with one processor. As an example of configuring multiple functional units with one processor, first, as represented by a computer such as a client or a server, there is a form in which one processor is configured with a combination of one or more CPUs and software, and this processor acts as multiple functional units. Second, as represented by a SoC (System On Chip), there is a form in which a processor is used that realizes the functions of the entire system including multiple functional units with one IC (Integrated Circuit) chip. In this way, the various functional units are configured using one or more of the above various processors as a hardware structure.

[0050] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0051] The memory 20B is a storage device in which instructions executed by the processor 20A are stored. The memory 20B may be composed of two or more storage devices. The memory 20B includes a RAM (Random Access Memory) and a ROM (Read Only Memory), not shown. The processor 20A uses the RAM as a working area, executes software using various programs and parameters including a medical information processing program described later and stored in the ROM, and executes various processes of the medical information processing device 20 by using the parameters stored in the ROM, etc.

[0052] The communication interface 20C controls communications with the medical image inspection equipment 12, the medical image database 14, the user terminal device 16, and the image interpretation report database 18 via the network 22 in accordance with a predetermined protocol.

[0053] The medical information processing device 20 may be a cloud server accessible from a plurality of medical institutions via the Internet. The processing performed by the medical information processing device 20 may be a cloud service based on a fee system or a fixed fee system.

[0054] [Functional configuration of medical information processing device] 3 is a block diagram showing a functional configuration of the medical information processing device 20. Each function of the medical information processing device 20 is realized by the processor 20A executing a medical information processing program stored in the memory 20B. As shown in FIG. 3, the medical information processing device 20 includes an acquisition unit 30, an image region estimation unit 40, a second region of interest candidate identification unit 50, and an output unit 60.

[0055] The acquisition unit 30 acquires a medical image, related information related to the medical image, and one or more first candidate regions of interest included in the medical image, the first candidate regions of interest not associated with the related information. The first candidate regions of interest include at least one of a bounding box, a heat map, and a mask.

[0056] The acquiring unit 30 acquires medical images from the medical image database 14. The acquiring unit 30 acquires image interpretation reports paired with medical images as related information related to the medical images from the image interpretation report database 18. The related information is not limited to the entire image interpretation report, but may be a portion broken down (structured) into the size, position, properties, etc. of a lesion.

[0057] The acquisition unit 30 includes a first region of interest candidate generation unit 31. The first region of interest candidate generation unit 31 acquires the first region of interest candidate by generating the first region of interest candidate based on the medical image received by the acquisition unit 30. The first region of interest candidate generation unit 31 may perform a process of arranging a plurality of bounding boxes as the first region of interest candidate at regular intervals on the medical image based on a rule base like an anchor for object detection. The acquisition unit 30 may arrange the first region of interest candidate by a known method such as selective search.

[0058] The acquiring unit 30 may receive input of a medical image, related information, and a first region of interest candidate. The acquiring unit 30 may receive input of a first region of interest candidate stored in a first region of interest candidate storage unit (not shown) provided in the memory 20B.

[0059] The image region estimation unit 40 estimates one or more image regions indicated by the related information from the medical image and the related information acquired by the acquisition unit 30. The image region estimated by the image region estimation unit 40 includes at least one of a bounding box, a heat map, and a mask.

[0060] The image region estimation unit 40 includes an image region estimation model 40A. A neural network (NN) that is trained to estimate the approximate position of a region of interest when an image and text are input is applied to the image region estimation model 40A. The image region estimation model 40A is stored in the memory 20B.

[0061] The second region of interest candidate identifying section 50 determines a second region of interest candidate from among the first region of interest candidates acquired by the acquiring section 30, based on the image region estimated by the image region estimation section 40. The second region of interest candidate identifying section 50 may determine, as the second region of interest candidate, a first region of interest candidate included in the image region estimated by the image region estimation section 40, from among one or more first region of interest candidates acquired by the acquiring section 30. The acquiring section 30 may acquire a plurality of first region of interest candidates, and the second region of interest candidate identifying section 50 may determine a second region of interest candidate from among the plurality of first region of interest candidates.

[0062] The output unit 60 outputs the second region of interest candidate identified by the second region of interest candidate identifying unit 50, and records the second region of interest candidate in a learning database (not shown) in association with the medical image. The output unit 60 may output the medical image with a bounding box, heat map, or mask attached to the position of the second region of interest candidate. The medical image with the bounding box, heat map, or mask attached to the position of the second region of interest candidate can be used as correct answer data when training a learning model that estimates a region of interest from a medical image.

[0063] <Medical information processing method: First embodiment> FIG. 4 is a flowchart showing a medical information processing method using the medical information processing device 20. Also, FIG. 5, FIG. 6, and FIG. 7 are diagrams for explaining the processing of each step of the medical information processing method. The medical information processing method is a method for determining a lesion candidate, which is a region of interest candidate of a medical image, from a medical image and related information related to the medical image. The medical information processing method is realized by the processor 20A executing a medical information processing program stored in the memory 20B. The medical information processing program may be provided by a computer-readable non-transitory storage medium, or may be provided via the Internet.

[0064] In step S1, the acquiring unit 30 receives a medical image, one or more first region of interest candidates in the medical image, and related information related to the medical image via the network 22. The acquiring unit 30 may receive an input of the medical image and the related information, and generate a first region of interest candidate in the first region of interest candidate generating unit 31 based on the received medical image.

[0065] 5 is a diagram showing a medical image I1 received by the acquisition unit 30, a first region of interest candidate C1 arranged in the medical image I1, and related information R1 of the medical image I1. In this example, the first region of interest candidate C1 is a rectangular bounding box, and multiple first region of interest candidates C1 are arranged in the medical image I1. The first region of interest candidate C1 may be a heat map or a mask.

[0066] Related information R1 is an image interpretation report including text describing a region of interest, such as "A protruding mass of approximately 30 mm was observed in the lower pole of the right kidney." In this example, "lower pole of the right kidney" represents the location of the region of interest, "approximately 30 mm" represents the size of the region of interest, and "protruding" represents the nature of the region of interest. Related information may include structured text information.

[0067] In step S2, the image region estimation unit 40 estimates one or more image regions indicated by the related information from the medical image and related information acquired in step S1 using the image region estimation model 40A. F6A in FIG. 6 shows a medical image I1 and related information R1. F6B in FIG. 6 shows the medical image I1 and an estimated image region A1. The image region A1 is a mask capable of identifying at least one of an approximate position and an approximate size of a region of interest in the medical image I1. The image region A1 may be a bounding box or a heat map.

[0068] In step S3, the second region of interest candidate identifying unit 50 determines a second region of interest candidate from among the first region of interest candidates acquired in step S1 based on the image region estimated in step S2. Fig. 7 shows the medical image I1 and the determined second region of interest candidates D1 and D2. The second region of interest candidates D1 and D2 are each selected from a plurality of first region of interest candidates C1 that are included in the image region A1.

[0069] In this way, according to the medical information processing method, it is possible to determine a region of interest candidate of a medical image using related information related to the medical image, and to link the related information with the region of interest candidate in the medical image. In addition, it is possible to improve the performance of an object detection model using information on the region of interest candidate in the medical image determined by the medical information processing method. The medical image and the determined region of interest candidate may be stored in association with each other and used as learning data for the object detection model.

[0070] [Related information] The related information related to an image is information that does not include position coordinate information that identifies the position coordinates of a region of interest in an image. In the first place, if the "related information" associated with an image includes information that can identify the position coordinates of a region of interest in an image, there is no need to estimate a region of interest from the image using the image region estimation unit 40.

[0071] In this embodiment, it is assumed that the image is not associated with position coordinate information of the region of interest, and instead of position coordinate information, text such as a report describing the region of interest in the image is used. That is, it is assumed that the image acquired in this embodiment is not associated with position coordinate information of the region of interest. Alternatively, even if the image is associated with position coordinate information of the region of interest, it is assumed that the information is not used.

[0072] [Generation of the first region of interest candidate] The first region of interest candidates may be arranged randomly on the image, or one or more types of rectangles with predetermined sizes and aspect ratios may be arranged in a grid pattern. The first region of interest candidates may be a bounding box group stored in a memory in advance, or may be input via a user interface. The first region of interest candidates may be adaptively generated based on an input image. For example, the means for generating the first region of interest candidates may be an object detection system using an object detection framework represented by Faster R-CNN (Region-Based Convolutional Neural Networks) or YOLO (You Only Look Once).

[0073] [Modifications] The acquisition unit 30 may include a first region of interest candidate estimation model, and may generate a first region of interest candidate from a medical image using the first region of interest candidate estimation model. The first region of interest candidate estimation model is a machine learning model trained to receive an input of a medical image and estimate one or more first region of interest candidates from the medical image. A neural network may be applied to the first region of interest candidate estimation model. The first region of interest candidate estimation model is stored in the memory 20B.

[0074] The acquisition unit 30 may include an object detection model and generate one or more first region of interest candidates from the medical image using the object detection model. The object detection model is a model trained by machine learning using training data including the determined region of interest candidates, as described below. A neural network may be applied to the object detection model. The object detection model is stored in the memory 20B.

[0075] The image region estimation unit 40 may include an organ recognition unit. The organ recognition unit recognizes an organ included in the medical image acquired by the acquisition unit 30. A neural network may be applied to the organ recognition unit.

[0076] The image region estimation unit 40 may include a position, size, and attribute estimation unit. The position, size, and attribute estimation unit estimates at least one of the position, size, and attribute indicated by the text of the related information acquired by the acquisition unit 30. The image region estimation unit 40 may estimate the image region on a rule basis from the organ recognition result and the position, size, and attribute indicated by the text of the related information.

[0077] The image region estimation unit 40 may include a probability calculation unit and estimate the image region using the probability calculation unit. The probability calculation unit calculates the probability that the image region is the image region indicated by the related information for each pixel of the medical image.

[0078] The second region of interest candidate identifying section 50 may include a certainty factor calculation section. The certainty factor calculation section calculates the certainty factor of the first region of interest candidate acquired by the acquiring section 30. The second region of interest candidate identifying section 50 may update and correct the certainty factor of the image candidate region calculated by the certainty factor calculation section, and delete a first region of interest candidate that does not correspond to the image region estimated by the image region estimating section 40 from among the one or more first region of interest candidates.

[0079] The second region of interest candidate identifying section 50 may include an evaluation value calculation section. The evaluation value calculation section calculates evaluation values ​​of one or more first region of interest candidates from the image region estimated by the image region estimation section 40. The second region of interest candidate identifying section 50 may determine the second region of interest candidate based on the evaluation value calculated by the evaluation value calculation section.

[0080] <Details of image area estimation unit> Fig. 8 is a diagram for explaining an example of processing by the image region estimation unit 40. F8A in Fig. 8 shows a medical image I2 and related information R2 acquired by the acquisition unit 30. The related information R2 includes the text "A protruding mass of about 30 mm is observed in the lower pole of the right kidney." When the medical image I2 and related information R2 are input, the image region estimation model 40A outputs an image region A3 estimated from the medical image I2.

[0081] 8 shows a medical image I2 and an image region A3 output from the image region estimation model 40A. The image region A3 is a mask capable of identifying at least one of an approximate location and an approximate size of a region of interest in the medical image I2.

[0082] In this manner, the image region estimation unit 40 can estimate an image region from a medical image using the image region estimation model 40A.

[0083] Fig. 9 is a diagram for explaining another example of processing by the image region estimation unit 40. Here, an example will be explained in which an organ recognition unit and a position, size, and attribute estimation unit are used instead of the image region estimation model 40A. F9A in Fig. 9 shows a medical image I2 acquired by the acquisition unit 30.

[0084] The organ recognition section of the image region estimation unit 40 recognizes organs included in the medical image I2. F9B in Fig. 9 shows the medical image I2, the organ E1 extracted from the medical image I2 by the organ recognition section, and related information R2 of the medical image I2 acquired by the acquisition section 30. Here, the extracted organ E1 is shown surrounded by a line.

[0085] Furthermore, the image region estimation unit 40 estimates an image region from the extracted organ E1 and related information R2. That is, the position, size, and attribute estimation unit estimates the position "lower pole of right kidney", the size "approximately 30 mm", and the attribute "protruding" from the text of the related information R2. The image region estimation unit 40 estimates the image region for the organ E1 on a rule basis based on the estimated position, size, and attribute. F9C in FIG. 9 shows the medical image I2 and the image region A2 estimated by the image region estimation unit 40. The image region A2 is a bounding box that can specify at least one of the approximate position and approximate size of the region of interest of the medical image I2.

[0086] In this manner, the image region estimation unit 40 can recognize organs from medical images and estimate the image region using a rule-based model.

[0087] <Medical information processing method: Second embodiment> In the second embodiment, an example in which the accuracy of object detection is improved by using the configuration described in the first embodiment will be described.

[0088] 10 is a block diagram illustrating a schematic functional configuration of an object detection system 100 according to the second embodiment. The object detection system 100 has a system configuration in which the configuration described in the first embodiment is incorporated into the framework of a Faster R-CNN 110.

[0089] That is, the object detection system 100 has a configuration in which an image region estimation unit 40 is added to a network structure of a Faster R-CNN 110 including a backbone CNN (Convolutional Neural Networks) 112, a RPN (Region Proposal Network) 114, a ROI (Region of Interest) pooling unit 116, and a classifier 118.

[0090] The backbone CNN 112 is a neural network including multiple convolution layers, and serves as a feature extractor that extracts features of an input image. An existing feature extractor may be applied to the backbone CNN 112.

[0091] The RPN 114 receives the feature map output from the backbone CNN 112 as input and outputs regions that are candidates for object regions (hereinafter referred to as "region candidates"). In the object detection system 100 that handles medical images, an "object region" is, for example, a region of interest such as a lesion region. The region candidate output from the RPN 114 corresponds to the "first region of interest candidate" described in the first embodiment. The RPN 114 places multiple anchor boxes on the feature map and outputs an object-likeness score for each anchor box. The RPN 114 may output bounding boxes of one or more region of interest candidates with different horizontal and vertical sizes and aspect ratios.

[0092] That is, the RPN 114 outputs one or more first candidate regions of interest contained in the medical image I2 based on the feature map output from the backbone CNN 112.

[0093] The ROI pooling unit 116 receives the output of the backbone CNN 112 and the output of the RPN 114 as input, performs pooling processing on the regions of the feature map corresponding to the region candidates output by the RPN 114, and prepares a feature map of a predetermined size before passing it to the classifier 118.

[0094] The classifier 118 is configured using a CNN including multiple convolution layers, and receives a feature map corresponding to each region candidate output from the ROI pooling unit 116, performs class classification for each region candidate, and outputs the class probability for each class. The class probability output by the classifier 118 may typically indicate the object-likeliness of two classes, that is, an "object (foreground)" which is a lesion, or a "background", but may be configured to classify the region estimated by the RPN 114 into a mass class such as a "nodule" or a "cyst", if the type of lesion can be identified to some extent from the description of the related information R2. Furthermore, the classifier 118 may output a bounding box surrounding the detected object (lesion).

[0095] In the case of a general Faster R-CNN, a loss (Loss1) related to the class probability of object-likeness and a loss (Loss2) related to the deviation of the bounding box are calculated for the output of the RPN 114 using a correct bounding box indicating the correct object region, and the parameters of the RPN 114 are updated based on these losses. Similarly, for the classifier 118, a loss (Loss3) related to the class probability of class classification and a loss (Loss4) related to the deviation of the bounding box are calculated for the output of the classifier 118 using the correct class and the correct bounding box, and the parameters of the backbone CNN 112 and the classifier 118 are updated based on these losses. In this way, learning is performed while improving the accuracy of each network model by alternately updating the parameters of the RPN 114 and the entire Faster R-CNN including the backbone CNN 112 and the classifier 118.

[0096] 10, there is no correct bounding box for the medical image I2, and data combining the medical image I2 and related information R2 related to the medical image I2 is used for learning (training). In other words, in the second embodiment, since data without a correct bounding box is utilized, it is not possible to calculate losses (Loss2 and Loss4) between the output of the RPN 114 and the output of the classifier 118 and the correct bounding box.

[0097] In this regard, in the second embodiment, the output from the image region estimation unit 40 is used instead of the correct bounding box to calculate the loss (Loss1) related to the class probability.

[0098] The image area estimation unit 40 receives a medical image I2 and related information R2 related to the medical image I2 as input, estimates the approximate location (image area) of the area of ​​interest in the medical image I2 indicated in the related information R2, and outputs information on the estimated image area. For example, as shown in Fig. 10, when the image area estimation unit 40 receives the medical image I2 and related information R2, it outputs an image area A3. The image area A3 is a mask capable of identifying at least one of the approximate location and approximate size of the area of ​​interest in the medical image I2.

[0099] The RPN 114 is trained so as to increase the class probability of area candidates that overlap with the image area estimated by the image area estimation unit 40, among the area candidates estimated by the RPN 114. This allows the RPN 114 to improve the performance of outputting area candidates where the area of ​​interest exists, even if it cannot output an accurate bounding box corresponding to the area of ​​interest.

[0100] That is, the second region of interest candidate identifying unit 50 (not shown in FIG. 10, see FIG. 3) evaluates the degree of overlap (degree of agreement) between the image region A3 of the medical image I2 estimated by the image region estimation unit 40 and the first region of interest candidate of the medical image I2 output by the RPN 114 for each first region of interest candidate, and identifies the first region of interest candidate that overlaps with the image region A3 estimated by the image region estimation unit 40 as a "second region of interest candidate." During learning, the parameters of the RPN 114 are updated so as to increase the class probability of the second region of interest candidate identified by the second region of interest candidate identifying unit 50. Similarly, for the class probability output by the classifier 118, the loss (Loss3) can be calculated using the output from the image region estimation unit 40.

[0101] When training the entire model of the actual object detection system 100, it is preferable to use data for which a correct bounding box is given in addition to data combining an image and related information R2 related to that image for training. This makes it possible to further improve the sensitivity of the RPN 114 by calculating losses for the position and size of the bounding box for a portion of the training data that has a correct bounding box and updating the network parameters to improve the deviation of the bounding box based on the losses.

[0102] FIG. 10 illustrates an example in which the framework of Faster R-CNN 110 is used; however, the technology of the present disclosure is not limited to a two-stage detector system as shown in FIG. 10 , and can be similarly applied to, for example, a single-stage detector object detection system such as YOLO, and can also be implemented using a confidence score output by a single-stage detector instead of the class probability described in FIG. 10 .

[0103] [Utilization of the image area estimation unit 40 during inference] By using the trained Faster R-CNN 110 trained by the training method described in FIG. 10, it is possible to detect a region of interest in an image from the image alone at the time of inference, even if there is no report related to the image.

[0104] On the other hand, when a report related to the image exists, it is also possible to utilize the image region estimation unit 40 during inference and correct the class probability, which is the inference result of Faster R-CNN 110, based on the output from the image region estimation unit 40.

[0105] <Other> The medical information processing device, medical information processing method, and medical information processing program according to the present embodiment can also be applied to an information processing device, information processing method, and program that use natural images other than medical images. For example, the invention can be applied to a technology that acquires images of social infrastructure facilities such as transportation, electricity, gas, and water, which have related information, and identifies an area of ​​interest in the image. This makes it possible to easily create correct answer data indicating an area of ​​interest, and to train a learning model that estimates an area of ​​interest from an image of infrastructure facilities using the created correct answer data.

[0106] The technical scope of the present invention is not limited to the scope described in the above embodiments. The configurations and the like in each embodiment can be appropriately combined with each other without departing from the spirit of the present invention. [Explanation of symbols]

[0107] 10. Medical information processing system 12. Medical imaging equipment 14. Medical image database 16...User terminal device 16A…Input device 16B…Display 18…Radiography report database 20...Medical information processing device 20A…Processor 20B…Memory 20C…Communication interface 22…Network 30…Acquisition part 31...first region of interest candidate generating unit 40...Image area estimation unit 40A…Image region estimation model 50...Second region of interest candidate identification unit 60...Output section 100...Object detection system 110…Faster R-CNN 112…Backbone CNN 114…RPN 116…ROI Pooling Section 118...Classifier A1…Image area A2…Image area A3…Image area C1: First candidate region of interest D1: Second candidate region of interest D2: Second candidate region of interest E1…Organs I1…Medical imaging I2…Medical imaging R1…Related Information R2…Related Information S1 to S3: Steps in medical information processing

Claims

1. one or more processors; one or more memory devices on which instructions are stored for execution by the one or more processors; The one or more processors: Obtaining an image, associated information about the image, and one or more first candidate regions of interest contained in the image; Estimating one or more image regions indicated by the related information from the image and the related information; determining a second region of interest candidate from among the first region of interest candidates based on the estimated image region; Information processing device.

2. the related information includes text related to the content of the image; The information processing device according to claim 1 .

3. the associated information includes text describing a region of interest contained in the image; The information processing device according to claim 1 .

4. The related information includes structured text information including at least one of a size, a position, and a characteristic of a region of interest included in the image. The information processing device according to claim 1 .

5. The one or more processors: Inferring at least one of a position, a size, and a property of what the text points to; The information processing device according to claim 2 .

6. the image is a medical image; The one or more processors: Recognizing organs included in the image; estimating the image region from the text and the organ recognition result; The information processing device according to claim 2 .

7. the one or more first region of interest candidates include at least one of a bounding box, a heat map, and a mask; The information processing device according to claim 1 .

8. The one or more processors: accepting input of the image, the related information, and the one or more first region of interest candidates; The information processing device according to claim 1 .

9. The one or more processors: Accepting an input of the image and the related information; obtaining the one or more first region of interest candidates by generating the one or more first region of interest candidates based on the received image; The information processing device according to claim 1 .

10. The one or more processors: performing a process of arranging a plurality of bounding boxes as the first region of interest candidates on the image at regular intervals based on a rule base; The information processing device according to claim 9.

11. The one or more processors: generating the one or more first region of interest candidates from the image using a machine learning model trained to receive the image and estimate the one or more first region of interest candidates from the image; The information processing device according to claim 9.

12. The one or more processors: generating the one or more first candidate regions of interest from the image using an object detection model; The information processing device according to claim 9.

13. The object detection model is a model trained by machine learning using training data including the determined region of interest candidate; The information processing device according to claim 12.

14. The one or more processors: determining, as the second region of interest candidate, the first region of interest candidate included in the estimated image region among the one or more first region of interest candidates; The information processing device according to claim 1 .

15. The one or more processors: Obtaining a plurality of first region of interest candidates; determining the second region of interest candidate from among the plurality of first region of interest candidates; The information processing device according to claim 1 .

16. The one or more processors: calculating a probability that the image region is the image region indicated by the related information for each pixel of the image; The information processing device according to claim 1 .

17. the image region estimated by the one or more processors includes at least one of a bounding box, a heat map, and a mask; The information processing device according to claim 1 .

18. The one or more processors: Calculating a confidence level of the first region of interest candidate; removing the first region of interest candidate that does not correspond to the estimated image region from among the one or more first region of interest candidates; The information processing device according to claim 1 .

19. The one or more processors: calculating an evaluation value of the one or more first region of interest candidates from the estimated image region; determining the second region of interest candidate based on the evaluation value; The information processing device according to claim 1 .

20. 1. An information processing method executed by one or more processors, comprising: the one or more processors, Obtaining an image, associated information about the image, and one or more first candidate regions of interest contained in the image; estimating one or more image regions indicated by the related information from the image and the related information; determining a second region of interest candidate from among the first region of interest candidates based on the estimated image region; An information processing method comprising:

21. On the computer, obtaining an image, associated information about the image, and one or more first candidate regions of interest contained in the image; A function of estimating one or more image regions indicated by the related information from the image and the related information; determining a second region of interest candidate from among the first region of interest candidates based on the estimated image region; A program to achieve this.

Images