Image analysis apparatus, image analysis method, and program

Abstract

To provide an image analysis apparatus, an image analysis method, and a program that identify an interested area from an image to which an arrow to the interested area is given.SOLUTION: An image analysis apparatus includes at least one processor, and at least one memory that stores instructions to be executed by the at least one processor to execute. The at least one processor receives an image to which an arrow to an interested area is given, identifies the arrow, disposes at least one interested area candidate which is a candidate of the interested area according to an orientation of the arrow and a distance from the arrow, calculates an interesting degree for each interested area candidate, and identifies the interested area from the interested area candidates based on the interesting degree.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image analysis device, an image analysis method, and a program, and more particularly to a technique for utilizing an image annotated with an arrow in a region of interest for training a learning model. [Background technology]

[0002] To identify a region of interest from an image, it is common to use an object detection model or a segmentation model, which is generated by learning a bounding box or a mask that is an annotation for segmentation that encloses the region of interest of the target.

[0003] Patent Document 1 discloses a technology that uses an R-CNN (Region Based Convolutional Neural Networks) model to extract an arrow from within an inspection sheet and identify the machine zone closest to the head of the arrow. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6689903 Summary of the Invention [Problem to be solved by the invention]

[0005] To learn a bounding box or a mask, a large amount of correct answer data with bounding boxes or masks is required. However, to create the correct answer data, it is necessary to identify the position of the region of interest in the image and add a bounding box or mask to it, which is time-consuming.

[0006] The present invention has been made in consideration of the above circumstances, and aims to provide an image analysis device, an image analysis method, and a program for identifying a region of interest from an image in which an arrow is added to the region of interest. [Means for solving the problem]

[0007] In order to achieve the above object, an image analysis device according to a first aspect of the present disclosure includes at least one processor and at least one memory that stores instructions to be executed by the at least one processor, and the at least one processor receives an image in which an arrow is added to a region of interest, identifies the arrow, arranges one or more region of interest candidates that are candidates for the region of interest according to the direction of the arrow and the distance from the arrow, calculates a degree of interest for each region of interest candidate, and identifies a region of interest from the region of interest candidates based on the degree of interest. According to this aspect, since a region of interest can be identified from an image in which an arrow is added to the region of interest, it is possible to easily create correct answer data indicating the region of interest. Therefore, a learning model that estimates a region of interest from an image can be trained using the created correct answer data.

[0008] An image analyzing device according to a second aspect of the present disclosure is the image analyzing device according to the first aspect, wherein the region of interest candidate has a shape that enables the size and position of the region of interest to be specified.

[0009] An image analyzing device according to a third aspect of the present disclosure is the image analyzing device according to the second aspect, wherein the candidate region of interest is preferably rectangular or circular.

[0010] In an image analysis device according to a fourth aspect of the present disclosure, in an image analysis device according to any of the first to third aspects, it is preferable that at least one processor positions candidate regions of interest according to a first distance from the tip of the arrow, the first distance in a normal direction to the direction of the arrow.

[0011] In an image analysis device according to a fifth aspect of the present disclosure, in an image analysis device according to any of the first to fourth aspects, it is preferable that at least one processor positions candidate regions of interest according to a second distance from the tip of the arrow, the second distance in a direction parallel to the direction of the arrow.

[0012] In an image analysis device according to a sixth aspect of the present disclosure, in an image analysis device according to any of the first to fifth aspects, it is preferable that at least one processor acquires auxiliary information of the image and arranges candidate regions of interest according to the auxiliary information.

[0013] An image analysis device according to a seventh aspect of the present disclosure is the image analysis device according to the sixth aspect, wherein at least one processor is arranged so that the incidental information includes a sentence describing the contents of the image.

[0014] An image analysis device according to an eighth aspect of the present disclosure is an image analysis device according to the sixth or seventh aspect, wherein the image is a medical image, the region of interest is a lesion, and the additional information includes information on the size of the lesion.

[0015] In an image analysis device according to a ninth aspect of the present disclosure, in an image analysis device according to any of the first to eighth aspects, it is preferable that at least one processor calculates a degree of interest for each region of interest candidate using a first level of interest calculation model that outputs a degree of interest for an input region of interest candidate when an image feature and a region of interest are input, or a second level of interest calculation model that outputs a degree of interest for an input region of interest candidate when an image and a region of interest candidate are input.

[0016] An image analysis device according to a tenth aspect of the present disclosure is preferably an image analysis device according to the ninth aspect, in which the first level of interest calculation model and the second level of interest calculation model are trained models in which a plurality of regions are arranged in an image whose region of interest is known, and the levels of interest between the arranged regions and the known region of interest are used as correct answer data.

[0017] In an image analysis device according to an eleventh aspect of the present disclosure, in an image analysis device according to any of the first to tenth aspects, it is preferable that at least one processor arranges a plurality of region of interest candidates and identifies as the region of interest a region of interest candidate having the highest degree of interest among the plurality of region of interest candidates.

[0018] In order to achieve the above object, an image analysis method according to a twelfth aspect of the present disclosure is an image analysis method including: receiving an image in which an arrow is added to a region of interest; identifying the arrow; arranging one or more region of interest candidates that are candidates for the region of interest according to the direction of the arrow and the distance from the arrow; calculating a degree of interest for each region of interest candidate; and identifying a region of interest from the region of interest candidates based on the degree of interest. According to this aspect, since a region of interest can be identified from an image in which an arrow is added to the region of interest, it is possible to easily create correct answer data indicating the region of interest. Therefore, a learning model that estimates a region of interest from an image can be trained using the created correct answer data.

[0019] In order to achieve the above object, a program according to a 13th aspect of the present disclosure is a program for causing a computer to execute the image analysis method according to the 12th aspect. A non-transitory computer-readable recording medium such as a CD-ROM (Compact Disk-Read Only Memory) storing the program according to the 13th aspect is also included in the present disclosure. Effect of the Invention

[0020] According to the present invention, it is possible to specify a region of interest from an image in which an arrow is added to the region of interest. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a medical image analysis system. [Diagram 2] FIG. 2 is a block diagram showing the electrical configuration of the medical image analyzing apparatus. [Diagram 3] FIG. 3 is a block diagram showing the functional configuration of the medical image analyzing apparatus. [Figure 4] FIG. 4 is a flowchart showing the medical image analyzing method according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing a process of identifying a region of interest from a plurality of region of interest candidates. [Figure 6] FIG. 6 is a flowchart showing a medical image analyzing method according to the second embodiment. [Figure 7] FIG. 7 is a flowchart showing a method for learning the interest level calculation model. [Figure 8] FIG. 8 is a diagram showing medical images used for learning the interest level calculation model. [Figure 9] FIG. 9 is a diagram illustrating an example of the interest level calculation model. [Figure 10] FIG. 10 is a diagram showing another example of the interest level calculation model. [Figure 11] FIG. 11 is a diagram showing an example of the relationship between the arrow and the region of interest candidates. [Figure 12] FIG. 12 is a diagram showing an example of the relationship between the arrow and the region of interest candidates. [Figure 13] FIG. 13 is a diagram showing an example in which each side of a candidate region of interest is arranged along the horizontal and vertical directions of an image. [Figure 14] FIG. 14 is a diagram showing a medical image with an arrow attached. [Figure 15] FIG. 15 is a diagram showing another example of the relationship between the arrow and the region of interest candidates. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Here, a medical image analysis device, a medical image analysis method, and a medical image analysis program will be described as examples of an image analysis device, an image analysis method, and a program according to the present invention.

[0023] <Medical image analysis system> The medical image analysis system according to the present embodiment is a system for identifying a region of interest in a medical image from a medical image in which an arrow is added to the region of interest. The medical image in which the region of interest is identified can be used as correct answer data for training a learning model that estimates the region of interest from the medical image.

[0024] Fig. 1 is an overall configuration diagram of a medical image analysis system 10. As shown in Fig. 1, the medical image analysis system 10 is configured to include a medical image inspection device 12, a medical image database 14, a user terminal device 16, an image interpretation report database 18, and a medical image analysis device 20.

[0025] The medical image inspection equipment 12, the medical image database 14, the user terminal device 16, the image interpretation report database 18, and the medical image analysis device 20 are connected to each other via a network 22 so as to be able to transmit and receive data. The network 22 includes a wired or wireless LAN (Local Area Network) that communicatively connects various devices in the medical institution. The network 22 may also include a WAN (Wide Area Network) that connects the LANs of multiple medical institutions.

[0026] The medical imaging inspection equipment 12 is an imaging device that captures an image of an inspection target part of a subject and generates a medical image. Examples of the medical imaging inspection equipment 12 include an X-ray imaging device, a CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, a PET (Positron Emission Tomography) device, an ultrasound device, a CR (Computed Radiography) device using a flat X-ray detector, and an endoscope device.

[0027] The medical image database 14 is a database that manages medical images taken by the medical image inspection equipment 12. The medical image database 14 is implemented by a computer equipped with a large-capacity storage device for storing medical images. Software that provides the functions of a database management system is installed in the computer.

[0028] The medical image may be a two-dimensional or three-dimensional still image taken by an X-ray device, a CT device, an MRI device, or the like, or may be a moving image taken by an endoscope device.

[0029] The format of the medical image can be in accordance with the Dicom (Digital Imaging and Communications in Medicine) standard. Supplementary information (Dicom tag information) defined in the Dicom standard may be added to the medical image. Note that the term "image" in this specification includes not only the image itself, such as a photograph, but also image data, which is a signal representing an image.

[0030] The user terminal device 16 is a terminal device for a doctor to create and view an interpretation report. For example, a personal computer is applied to the user terminal device 16. The user terminal device 16 may be a workstation or a tablet terminal. The user terminal device 16 includes an input device 16A and a display 16B. The doctor uses the input device 16A to input an instruction to display a medical image. The user terminal device 16 displays the medical image on the display 16B. Furthermore, the doctor creates an interpretation report by interpreting the medical image displayed on the display 16B, adding an arrow to a lesion that is an area of ​​interest in the medical image using the input device 16A, and inputting a finding statement that is the interpretation result.

[0031] An arrow is a symbol used to indicate a direction. For example, an upward arrow "↑" is placed below a region of interest in a medical image to indicate that the region of interest is above the arrow. The direction, shape, thickness, and color of the arrow attached to the region of interest are not limited as long as it indicates the region of interest.

[0032] When an arrow is added to a medical image, the user terminal device 16 may overwrite the arrow on the medical image, or may superimpose the arrow on a layer separate from the medical image.

[0033] The image interpretation report database 18 is a database that manages image interpretation reports generated by doctors on the user terminal device 16. The image interpretation report includes medical images with arrows attached. The image interpretation report database 18 is implemented by a computer equipped with a large-capacity storage device for storing image interpretation reports. Software that provides the functions of a database management system is incorporated into the computer. The medical image database 14 and the image interpretation report database 18 may be configured as a single computer.

[0034] The medical image analysis device 20 is a device for identifying a region of interest in a medical image. The medical image analysis device 20 may be a personal computer or a workstation (an example of a "computer"). FIG. 2 is a block diagram showing the electrical configuration of the medical image analysis device 20. As shown in FIG. 2, the medical image analysis device 20 includes a processor 20A, a memory 20B, and a communication interface 20C.

[0035] The processor 20A executes instructions stored in the memory 20B. The hardware structure of the processor 20A is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and acts as various functional units, a GPU (Graphics Processing Unit), which is a processor specialized for image processing, a PLD (Programmable Logic Device), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing specific processing such as an ASIC (Application Specific Integrated Circuit).

[0036] A processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Also, multiple functional units may be configured with one processor. As an example of configuring multiple functional units with one processor, first, as represented by a computer such as a client or a server, there is a form in which one processor is configured with a combination of one or more CPUs and software, and this processor acts as multiple functional units. Second, as represented by a SoC (System On Chip), there is a form in which a processor is used that realizes the functions of the entire system including multiple functional units with one IC (Integrated Circuit) chip. In this way, the various functional units are configured using one or more of the above various processors as a hardware structure.

[0037] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0038] The memory 20B stores instructions to be executed by the processor 20A. The memory 20B includes a RAM (Random Access Memory) and a ROM (Read Only Memory), not shown. The processor 20A uses the RAM as a working area, executes software using various programs and parameters, including a medical image analysis program described later, stored in the ROM, and executes various processes of the medical image analysis device 20 by using the parameters stored in the ROM, etc.

[0039] The communication interface 20C controls communications with the medical image inspection equipment 12, the medical image database 14, the user terminal device 16, and the image interpretation report database 18 via the network 22 in accordance with a predetermined protocol.

[0040] The medical image analysis device 20 may be a cloud server accessible from a plurality of medical institutions via the Internet. The processing performed by the medical image analysis device 20 may be a cloud service that is based on a fee system or a flat rate system.

[0041] [Functional configuration as a medical image analysis device] Fig. 3 is a block diagram showing the functional configuration of the medical image analysis device 20. Each function of the medical image analysis device 20 is realized by the processor 20A executing a medical image analysis program stored in the memory 20B. As shown in Fig. 3, the medical image analysis device 20 includes an image acquisition unit 32, an arrow identification unit 34, an auxiliary information acquisition unit 36, a region of interest candidate arrangement unit 38, a degree of interest calculation unit 40, a region of interest identification unit 42, and an output unit 44.

[0042] The image acquisition unit 32 acquires a medical image with an arrow added to a lesion that is a region of interest from the image interpretation report database 18. The arrow may be overwritten on the medical image, or may be superimposed on a layer separate from the medical image.

[0043] The arrow identifying unit 34 identifies an arrow included in the medical image received by the image acquiring unit 32. Identifying the arrow includes identifying the position and direction of the arrow. The arrow identifying unit 34 includes an arrow detection model 34A. The arrow detection model 34A is a known object detection model or extraction model to which a convolutional neural network (CNN) is applied. The arrow detection model 34A is stored in the memory 20B.

[0044] The incidental information acquisition unit 36 ​​acquires incidental information that is incidental to the medical image acquired by the image acquisition unit 32. The incidental information may include text describing the contents of the medical image. The incidental information includes information on the region of interest. The information on the region of interest may include information on the size of the region of interest, information on the shape of the region of interest, or information on the position of the region of interest. The shape information may include aspect ratio information.

[0045] The incidental information acquisition unit 36 ​​may acquire an image interpretation report of the medical image as incidental information of the medical image acquired by the image acquisition unit 32, and acquire information on the region of interest from the findings in the image interpretation report.

[0046] The region of interest candidate arrangement unit 38 arranges one or more region of interest candidates in the medical image, which are candidates for the region of interest in the medical image. The region of interest candidate arrangement unit 38 may arrange the region of interest candidates according to the direction of the arrow identified by the arrow identification unit 34 and the distance from the arrow. The region of interest candidate has a shape that allows the approximate size and position of the region of interest to be identified, such as a rectangle or a circle. A rectangle is a quadrilateral with all corners at right angles, and is a rectangle or a square. In addition, a circle is not limited to a perfect circle, but includes an ellipse.

[0047] The region of interest candidate arrangement unit 38 may arrange the region of interest candidates according to the incidental information acquired by the incidental information acquisition unit 36. The region of interest candidate arrangement unit 38 may arrange the region of interest candidates according to at least one of the size information, shape information, and position information of the region of interest acquired by the incidental information acquisition unit 36.

[0048] The degree of interest calculation unit 40 calculates the degree of interest for each region of interest candidate arranged by the region of interest candidate arrangement unit 38. The degree of interest may be an index representing the likelihood that the region of interest candidate is a correct region of interest, or may be an index whose value becomes relatively larger as the likelihood becomes relatively higher.

[0049] The degree of interest calculation unit 40 includes a degree of interest calculation model 40A. The degree of interest calculation model 40A is a learning model (an example of a "first degree of interest calculation model") that outputs a degree of interest in an input region of interest candidate when an image feature (an example of an "image feature") and a region of interest candidate are given as input, or a learning model (an example of a "second degree of interest calculation model") that outputs a degree of interest in an input region of interest candidate when an image and a region of interest candidate are given as input. The degree of interest calculation model 40A is a trained model that is trained by arranging a plurality of regions in an image in which the region of interest is known, and learning the degree of interest between the arranged regions and the known region of interest as correct answer data. The degree of interest calculation model 40A may be a trained model to which CNN is applied. The degree of interest calculation model 40A is stored in the memory 20B.

[0050] An image feature is a feature in an image that can be visually recognized by humans, such as an area with relatively high brightness, an area with relatively low brightness, an edge area where the brightness changes suddenly, and the like.

[0051] The region of interest identifying section 42 identifies one of the region of interest candidates as a region of interest based on the degree of interest calculated by the degree of interest calculating section 40. The region of interest identifying section 42 may identify the region of interest candidate having the highest calculated degree of interest, i.e., the highest likelihood that the region of interest is a region of interest, as a region of interest.

[0052] The output unit 44 outputs the region of interest identified by the region of interest identification unit 42, and records the region of interest in a learning database (not shown) in association with the medical image. The output unit 44 may output the medical image with a bounding box or mask attached to the position of the region of interest. The medical image with a bounding box or mask attached to the position of the region of interest can be used as correct answer data when training a learning model that estimates a region of interest from a medical image.

[0053] <Medical image analysis method: First embodiment> 4 is a flowchart showing a medical image analysis method according to the first embodiment using the medical image analysis device 20. The medical image analysis method is a method for identifying a region of interest in a medical image to which an arrow is added. The medical image analysis method is realized by the processor 20A executing a medical image analysis program stored in the memory 20B. The medical image analysis program may be provided by a computer-readable non-transitory storage medium or via the Internet.

[0054] In step S1, the image acquiring unit 32 receives a medical image with an arrow added thereto from the image interpretation report database 18. The image acquiring unit 32 may receive a medical image with an arrow added thereto from a device other than the image interpretation report database 18 via the network 22.

[0055] In step S2, the arrow identifying unit 34 identifies the arrow in the medical image received in step S1 using the arrow detection model 34A.

[0056] In step S3, the region of interest candidate arrangement unit 38 arranges one or more region of interest candidates that are candidates for the region of interest of the medical image received in step S1 according to the direction of the arrow and the distance from the arrow identified in step S2.

[0057] Fig. 5 is a diagram showing a process of identifying a region of interest from a plurality of region of interest candidates. F5A in Fig. 5 shows a medical image I1 to which an arrow AR1 is added, and a plurality of region of interest candidates are arranged with respect to the arrow AR1. In F5A, region of interest candidates C1, C2, C3, and C4, which are four square regions of different sizes, are arranged. The region of interest candidates C1, C2, C3, and C4 are arranged on a line parallel to the direction perpendicular to the arrow direction (an example of a "normal direction of the arrow direction"), with the side closest to the tip of the arrow passing through a position a certain distance away from the tip of the arrow in a direction parallel to the arrow direction.

[0058] Moreover, the region of interest candidates C1, C2, C3, and C4 are each disposed at a position where the intersection of the side closest to the tip of the arrow and the line extending from the arrow is the midpoint of the side closest to the tip of the arrow. In other words, the region of interest candidates C1, C2, C3, and C4 are disposed at positions where they are equally divided by the lines extending from the arrows.

[0059] The distance between the tip of the arrow added to the medical image and the region of interest of the medical image varies depending on the object of the region of interest. For example, when the region of interest is a tumor, the doctor is also interested in the area around the tumor, so the arrow is added at a position relatively far from the region of interest. That is, the distance between the tip of the arrow and the region of interest is relatively large. On the other hand, when it is the boundary of an organ or the like, the arrow is added at a position relatively close to the region of interest. That is, the distance between the tip of the arrow and the region of interest is relatively small.

[0060] Therefore, the distance between the candidate regions of interest C1, C2, C3, and C4 and the tips of the arrows is preferably determined depending on the object of the region of interest.

[0061] In step S4, the degree-of-interest calculation unit 40 calculates a degree of interest for each region of interest candidate arranged in step S3. In the example shown in Fig. 5, the degree-of-interest calculation unit 40 inputs the image features of the medical image I1 and the region of interest candidates C1, C2, C3, and C4 to the degree-of-interest calculation model 40A, and obtains the degree of interest for each of the region of interest candidates C1, C2, C3, and C4.

[0062] In step S5, the region of interest identifying unit 42 identifies a region of interest based on the degree of interest calculated in step S4. For example, the region of interest candidate with the highest degree of interest among the region of interest candidates C1, C2, C3, and C4 is identified as the region of interest. F5B in FIG. 5 shows a medical image I1 in which the region of interest A1 is identified with respect to the arrow AR1. In F5B, the region of interest candidate C1 shown in F5A is identified as the region of interest A1.

[0063] Furthermore, the output unit 44 links the region of interest A1 with the medical image I1 and records it in the learning database. The output unit 44 may output the medical image I1 to which a square indicating the region of interest candidate C1 identified as the region of interest A1 is added. This allows the medical image I1 to be used as correct answer data for a learning model that extracts a region of interest from an image.

[0064] <Medical image analysis method: second embodiment> FIG. 6 is a flowchart showing a medical image analyzing method according to the second embodiment.

[0065] In step S11, the image acquiring unit 32 accepts a medical image with an arrow added thereto from the image interpretation report database 18. In addition, the incidental information acquiring unit 36 ​​accepts an image interpretation report relating to the medical image accepted by the image acquiring unit 32 as incidental information of the medical image.

[0066] In step S12, the arrow identifying unit 34 identifies the arrow in the medical image received in step S11 using the arrow detection model 34A.

[0067] In step S13, the supplementary information acquisition unit 36 ​​acquires information on the lesion from the findings in the image interpretation report received in step S11. The information on the lesion includes at least one of information on the size, shape, and position of the lesion. The information on the size of the lesion is, for example, "8 mm nodule."

[0068] In step S14, the region of interest candidate placement unit 38 places one or more region of interest candidates according to the direction of the arrow and the distance from the arrow identified in step S12 and the information on the lesion acquired in step S13. For example, the information on the size of the lesion, "8 mm", is converted to the size on the image, and the size of the region of interest candidate is set to the size of the lesion on the image. In this way, by using the information on the size of the lesion, a region of interest candidate of a size according to the size of the lesion can be placed. Similarly, by using the information on the shape of the lesion, a region of interest candidate of a shape according to the shape of the lesion can be placed, and by using the information on the position of the lesion, a region of interest candidate can be placed at a position according to the position of the lesion.

[0069] Steps S15 and S16 are similar to steps S4 and S5 in the first embodiment.

[0070] In this way, by positioning the region of interest candidates according to information about the region of interest in addition to the direction of the arrow and the distance from the arrow, the region of interest candidates can be positioned appropriately, and therefore the region of interest can be properly identified.

[0071] <Interest Calculation Model: Third Embodiment> 7 is a flowchart showing a method for learning the degree of interest calculation model 40A. Here, an example will be described in which the degree of interest calculation model 40A is trained in the medical image analysis device 20. Note that the degree of interest calculation model 40A may be trained in a computer different from the medical image analysis system 10.

[0072] In step S21, the processor 20A acquires a medical image with a known region of interest. Fig. 8 is a diagram showing a medical image used for learning the interest level calculation model 40A. F8A in Fig. 8 shows a medical image I2 with a known region of interest A2. Here, it is assumed that the processor 20A acquires the medical image I2.

[0073] In step S22, the processor 20A creates rectangles by randomly moving and transforming the rectangle of the region of interest in the medical image I2. F8B in Fig. 8 shows rectangles R1, R2, R3, and R4, which are four rectangles (examples of "multiple regions") that are created by moving and transforming the rectangle of the region of interest A2 in the medical image I2. Here, four rectangles are created, but the number of rectangles is not limited.

[0074] In step S23, the processor 20A trains a model that predicts the degree of interest between each of the rectangles R1, R2, R3, and R4 created in step S22 and the region of interest A2 that is the correct rectangle. That is, the processor 20A trains a model using the information of the medical image I2 and each of the rectangles R1, R2, R3, and R4 as inputs, and uses the degree of interest between the regions of the rectangles R1, R2, R3, and R4 and the region of interest A2 as correct answer data, thereby creating a degree of interest calculation model 40A. The degree of interest is a value related to the degree of overlap between the correct answer rectangle and the moved and deformed rectangle. The degree of interest may be, for example, IoU (Intersection over Union). Instead of training the IoU value as it is as the correct answer data of the degree of interest, a value (Target Value) obtained by converting the IoU value by a function may be used as the correct answer data.

[0075] The function for converting the IoU value may be a square root, a tanh function (Hyperbolic tangent function), or a step function according to a threshold. In the case of the tanh function, if the IoU value is iou, then Target Value = (tanh(iou x 5 - 2) + 1) / 2 may be used.

[0076] FIG. 9 is a diagram showing an example of the degree of interest calculation model 40A thus trained. When an image feature and a region of interest candidate are given as input, the degree of interest calculation model 40A shown in FIG. 9 outputs the degree of interest of the input region of interest candidate. In the example shown in FIG. 9, the region of interest candidate arranged by the region of interest candidate arrangement unit 38 is input to the degree of interest calculation model 40A. The image feature input to the degree of interest calculation model 40A is input from the image feature extraction model 40B. The image feature extraction model 40B is a learning model that outputs the image feature of the input image when an image is given as input. The image feature extraction model 40B is stored in the memory 20B.

[0077] Fig. 10 is a diagram showing another example of the degree of interest calculation model 40A. When an image and a region of interest candidate are given as input, the degree of interest calculation model 40A shown in Fig. 10 outputs the degree of interest of the input region of interest candidate. The image is input as is to the degree of interest calculation model 40A. Also, similar to the example shown in Fig. 9, the region of interest candidate arranged by the region of interest candidate arrangement unit 38 is input to the degree of interest calculation model 40A.

[0078] The degree of interest calculation unit 40 may include the degree of interest calculation model 40A shown in Fig. 9 and the degree of interest calculation model 40A shown in Fig. 10. In this case, the degree of interest calculation unit 40 may combine the results of both to obtain a final degree of interest. By using such a degree of interest calculation model 40A, the degree of interest calculation unit 40 can appropriately calculate the degree of interest in the candidate region of interest.

[0079] <Method of arranging region of interest candidates: Fourth embodiment> FIG. 11 is a diagram showing an example of the relationship between the arrow specified by the arrow specifying unit 34 and the region of interest candidates arranged by the region of interest candidate arranging unit 38. Here, region of interest candidates C11 and C12, which are square regions of different sizes with respect to the arrow AR2, i.e., regions with an aspect ratio of 1:1, are arranged. The region of interest candidates C11 and C12 are arranged on a line parallel to the direction of the arrow, the side closest to the tip of the arrow being a line passing through a position a distance d2 (an example of a "second distance") away from the tip of the arrow in a direction parallel to the direction of the arrow. The region of interest candidates C11 and C12 are also arranged on a line parallel to the direction of the arrow, the side closest to the tip of the arrow ... being an intersection point of the side closest to the tip of the arrow and a line extending the arrow, the midpoint of the side closest to the tip of the arrow. That is, the region of interest candidates C11 and C12 are arranged at positions equally divided by the lines extending the arrow. The region of interest candidates C11 and C12 are arranged according to the distance from the arrow in the direction perpendicular to the direction of the arrow (an example of a "first distance"). In the example shown in FIG. 11, the distance from the arrow of region of interest candidate C11 is d1A, and the distance from the arrow of region of interest candidate C12 is d1B.

[0080] FIG. 12 is a diagram showing an example of the relationship between the arrows specified by the arrow specifying section 34 and the region of interest candidates arranged by the region of interest candidate arranging section 38. As shown in FIG.

[0081] F12A in FIG. 12 shows a candidate region of interest C13, which is a rectangular region with an aspect ratio of 1:2, and C14, which is a rectangular region with an aspect ratio of 1:0.5.

[0082] The region of interest candidate arrangement unit 38 arranges region of interest candidates C13 and C14 in addition to region of interest candidates C11 and C12 arranged with respect to the arrow AR2. F12B in FIG. 12 shows region of interest candidates C13 and C14 arranged with respect to the arrow AR2. Note that region of interest candidate C12 is omitted from illustration in F12B. Like region of interest candidate C11, region of interest candidates C13 and C14 are arranged overlapping at positions where the sides closest to the tip of the arrow are a certain distance away from the tip of the arrow in a direction parallel to the direction of the arrow. Furthermore, region of interest candidates C13 and C14 are arranged at positions where the arrows are equally divided by lines extending from the arrows. Region of interest candidates C13 and C14 are arranged according to the distance from the arrow in a direction parallel to the direction of the arrow (an example of a "second distance").

[0083] In this way, by arranging candidate regions of interest of various sizes and shapes according to the distance from the arrow, the region of interest can be appropriately specified.

[0084] Up to this point, the sides of the region of interest candidate rectangle have been arranged along a direction perpendicular to and parallel to the direction of the arrow, but the orientation of the region of interest candidate rectangle is not limited to this. Region of interest candidate arrangement unit 38 may arrange each side of the region of interest candidate along the horizontal and vertical directions of the image, or may arrange each side of the region of interest candidate along the horizontal and vertical directions of the screen of the display on which the image is displayed.

[0085] FIG. 13 is a diagram showing another example of the arrow identified by the arrow identification unit 34 and the region of interest candidate arranged by the region of interest candidate arrangement unit 38. Here, an example in which each side of the region of interest candidate is arranged along the horizontal and vertical directions of the image will be described. F13A in FIG. 13 shows a case in which a horizontal arrow AR11 is added. The region of interest candidate C21 arranged for the arrow AR11 is arranged on a straight line parallel to the vertical direction of the image, with the side closest to the tip of the arrow AR11 passing through a position a certain distance away from the tip of the arrow AR11 in the horizontal direction of the image. Moreover, the region of interest candidate C21 is arranged at a position equally divided by a line extending from the arrow AR11.

[0086] F13B in Fig. 13 shows a case where an arrow AR12 is added at an angle of 45 degrees with respect to the horizontal and vertical directions. A region of interest candidate C22 arranged with respect to the arrow AR12 is arranged at a position where the tip of the arrow AR12 is a fixed distance away from the corner closest to the arrow AR12 and where a line extending from the arrow AR12 intersects with the corner closest to the arrow AR12. In addition, the region of interest candidate C22 is arranged at a position equally divided by the lines extending from the arrow AR12.

[0087] F13C in Fig. 13 shows a case where a vertical arrow AR13 is added. A region of interest candidate C23 arranged with respect to the arrow AR13 is arranged on a straight line parallel to the horizontal direction of the image, with the side closest to the tip of the arrow AR13 passing through a position a certain distance away from the tip of the arrow AR13 in the vertical direction of the image. The region of interest candidate C23 is also arranged at a position equally divided by a line extending from the arrow AR13.

[0088] 14 is a diagram showing a medical image I3 to which an arrow AR21 has been added. Here, a case will be described in which a rectangular region of interest candidate C31 is placed in the medical image I3. Here, each side of the region of interest candidate C31 is placed along the horizontal and vertical directions of the medical image I3.

[0089] An xy two-dimensional coordinate system is set with the upper left corner of the medical image I3 as the origin, the horizontal direction to the right as the x direction, and the vertical direction downward as the y direction. Angles in the xy two-dimensional coordinate system are defined as positive in the counterclockwise direction, with the horizontal direction to the right as 0 degrees.

[0090] The sizes of the medical image I3 in the x and y directions are defined as image_size. The length of the arrow AR21 is defined as arrow_length, the angle of the arrow AR21 as degree, and the distance between the tip of the arrow AR21 and the region of interest candidate C31 as margin. If the starting point of the arrow AR21 is the center of the medical image I3, the x-coordinate arrow_x and the y-coordinate arrow_y of the tip of the arrow AR21 are expressed as follows:

[0091] arrow_x = cos_theta * arrow_length + image_size / 2 arrow_y = sin_theta * arrow_length + image_size / 2 Where: cos_theta = cos(degree / 180*π) sin_theta = -sin(degree / 180*π) That is, the arrow AR21 is an arrow pointing from the center (image_size / 2, image_size / 2) of the medical image I3 toward (arrow_x, arrow_y).

[0092] Moreover, the x-coordinate box_x and the y-coordinate box_y of the intersection between the extension line of the arrow AR21 and the side of the region of interest candidate C31 are expressed as follows.

[0093] box_x = cos_theta * (arrow_length + margin) + image_size / 2 box_y = sin_theta * (arrow_length + margin) + image_size / 2 In the example shown in Figure 14, 0 <= degree < 45, so start_ratio = (degree + 45) / 90, the size in the x direction of region of interest candidate C31 is size_x, and the size in the y direction is size_y. Then, the upper left coordinates (x1, y1) and lower right coordinates (x2, y2) of region of interest candidate C31 are expressed as follows:

[0094] (x1,y1)=(box_x, box_y - size_y * start_ratio) (x2,y2)=(box_x + size_x ,y1+size_y) In this way, regardless of the angle degree of the arrow AR21, a candidate region of interest having sides aligned along the horizontal and vertical directions of the image can be positioned.

[0095] In the example shown in FIG. 14, one region of interest candidate is arranged, but region of interest candidates of various sizes and shapes may be arranged according to the distance from the arrow using the above rules.

[0096] FIG. 15 is a diagram showing an example in which candidate regions of interest having sides along the horizontal and vertical directions of the image and of various sizes and shapes are arranged.

[0097] F15A in Fig. 15 shows arrow AR31 arranged on a medical image, with region of interest candidates C41, C42, C43, and C44 arranged relative to arrow AR31 pointing approximately 15° to the upper right from the horizontal direction of the medical image. F15B in Fig. 15 shows arrow AR32 arranged on a medical image, with region of interest candidates C51, C52, C53, and C54 arranged relative to arrow AR32 pointing approximately 45° to the upper right from the horizontal direction of the medical image.

[0098] As shown in Fig. 15, the rectangles of the region of interest candidates C41, C42, C43, C44, C51, C52, C53, and C54 have their sides arranged along the horizontal direction (X direction in Fig. 15) and vertical direction (Y direction in Fig. 15) of the medical image regardless of the direction of the arrow. In the example shown in F15A, the sides of the region of interest candidates C41, C42, C43, and C44 are arranged near the tip of the arrow AR31. Also, in the example shown in F15B, the corners of the region of interest candidates C51, C52, C53, and C54 are arranged near the tip of the arrow AR32.

[0099] Even when the region of interest candidates are arranged in this manner, the region of interest can be appropriately identified because the region of interest candidates are arranged according to the direction of the arrow and the distance from the arrow.

[0100] As described above, according to the medical image analysis method disclosed herein, the arrow in an image to which an arrow is attached is identified, one or more candidate regions of interest that are candidates for the region of interest in the image are positioned according to the direction of the arrow and the distance from the arrow, a degree of interest is calculated for each candidate region of interest, and a region of interest is identified from among the candidate regions of interest based on the degree of interest, thereby making it possible to easily create correct answer data indicating the region of interest.

[0101] <Other> Here, the region of interest candidate arrangement unit 38 arranges multiple region of interest candidates, and the region of interest identification unit 42 identifies one of the multiple region of interest candidates as a region of interest, but the region of interest candidate arrangement unit 38 may arrange one region of interest candidate. In some cases, such as when an arrow points to the edge of an image, it may not be possible to arrange multiple region of interest candidates. When there is one region of interest candidate, the region of interest identification unit 42 does not necessarily need to identify that region of interest candidate as a region of interest. For example, when the degree of interest of that region of interest candidate is lower than a threshold value, the region of interest identification unit 42 may determine that the region of interest candidate does not correspond to a region of interest.

[0102] Furthermore, even if multiple region of interest candidates are arranged, if the degree of interest of any of the region of interest candidates is lower than the threshold value, region of interest identifying section 42 may determine that there is no corresponding region of interest.

[0103] The medical image analysis device, the medical image analysis method, and the medical image analysis program according to the present embodiment can also be applied to an image analysis device, an image analysis method, and a program that use natural images other than medical images. For example, the image can be applied to a technology for acquiring an image of social infrastructure facilities such as transportation, electricity, gas, and water, in which an arrow is added to the area of ​​interest, and identifying the area of ​​interest. This makes it possible to identify the area of ​​interest in an image of infrastructure facilities with an arrow added to the area of ​​interest, so that correct answer data indicating the area of ​​interest can be easily created, and a learning model that estimates the area of ​​interest from an image of infrastructure facilities can be trained using the created correct answer data.

[0104] The technical scope of the present invention is not limited to the scope described in the above embodiments. The configurations and the like in each embodiment can be appropriately combined with each other without departing from the spirit of the present invention. [Explanation of symbols]

[0105] 10. Medical image analysis system 12. Medical imaging equipment 14. Medical image database 16...User terminal device 16A…Input device 16B…Display 18…Radiography report database 20...Medical image analysis device 20A…Processor 20B…Memory 20C…Communication interface 22…Network 32...Image acquisition section 34...Arrow specific section 34A…Arrow detection model 36…Additional information acquisition section 38...Region of interest candidate placement section 40…Interest calculation section 40A…Interest calculation model 40B…Image feature extraction model 42...Region of interest identification unit 44...Output section A1…Area of ​​interest A2…Area of ​​interest AR1…Arrow AR2…Arrow AR3…Arrow AR4…Arrow AR11…Arrow AR12…Arrow AR13…Arrow AR21…Arrow AR31…Arrow AR32…Arrow C1…Candidate area of ​​interest C2: Candidate area of ​​interest C3…Candidate area of ​​interest C11…Candidate area of ​​interest C12…Candidate area of ​​interest C13…Candidate area of ​​interest C14…Candidate area of ​​interest C21…Candidate area of ​​interest C22…Candidate area of ​​interest C23…Candidate area of ​​interest C31…Candidate area of ​​interest C41…Candidate area of ​​interest C42…Candidate area of ​​interest C43…Candidate area of ​​interest C44…Candidate area of ​​interest C51…Candidate area of ​​interest C52…Candidate area of ​​interest C53…Candidate area of ​​interest C54…Candidate region of interest I1…Medical image I2…Medical imaging I3…Medical Imaging R1…Rectangle R2…Rectangle R3…Rectangle R4…Rectangle S1-S5, S11-S16... Steps in medical image analysis method S21 to S23: Steps of the interest calculation model learning method

Claims

1. At least one processor; at least one memory storing instructions for execution by said at least one processor; Equipped with The at least one processor Accept an image with an arrow marking the region of interest; Identifying said arrow, locating one or more candidate regions of interest that are candidates for the region of interest according to a direction of the arrow and a distance from the arrow; Calculating a degree of interest for each of the candidate regions of interest; identifying the region of interest from among the candidate regions of interest based on the degree of interest; Image analysis device.

2. The region of interest candidate has a shape that allows the size and position of the region of interest to be specified. The image analysis device according to claim 1 .

3. The candidate region of interest is rectangular or circular. The image analysis device according to claim 2.

4. The at least one processor positioning the region of interest candidate according to a first distance from a tip of the arrow in a normal direction to a direction of the arrow; The image analysis device according to claim 1 .

5. The at least one processor positioning the region of interest candidate according to a second distance from a tip of the arrow, the second distance being in a direction parallel to a direction of the arrow; The image analysis device according to claim 1 .

6. The at least one processor Acquire supplementary information of the image; arranging the region of interest candidates according to the supplementary information; The image analysis device according to claim 1 .

7. The at least one processor The supplementary information includes a text describing the content of the image. The image analysis device according to claim 6.

8. the image is a medical image; the region of interest is a lesion; The supplementary information includes information on the size of the lesion. The image analysis device according to claim 6.

9. The at least one processor calculating a degree of interest for each of the region of interest candidates using a first degree-of-interest calculation model that outputs a degree of interest for the input region of interest candidate when the image features and the region of interest are input, or a second degree-of-interest calculation model that outputs a degree of interest for the input region of interest candidate when the image and the region of interest candidate are input; The image analysis device according to claim 1 .

10. The first interest level calculation model and the second interest level calculation model are trained models that are trained by arranging a plurality of regions in an image in which the region of interest is known, and learning the interest levels between the arranged regions and the known region of interest as correct answer data. The image analysis device according to claim 9.

11. The at least one processor locating a plurality of said candidate regions of interest; identifying the region of interest candidate having the highest degree of interest as the region of interest among the plurality of region of interest candidates; The image analysis device according to any one of claims 1 to 10.

12. receiving an image having an arrow attached to a region of interest; Identifying the arrow; locating one or more candidate regions of interest that are candidates for the region of interest according to a direction of the arrow and a distance from the arrow; calculating a degree of interest for each of the region of interest candidates; identifying the region of interest from among the candidate regions of interest based on the degree of interest; An image analysis method comprising:

13. A program for causing a computer to execute the image analysis method according to claim 12.

Images