Information processing device, arrhythmia detection device, area specification method and area specification program
The information processing device enhances image analysis accuracy by using a learning model trained on annotated images to detect and correct missed target areas through morphological operations, addressing the issue of inaccurate target area detection.
Patent Information
- Application Number
- JP2023081782
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-17
- Publication Date
- 2024-11-28
- Estimated Expiration
- 2043-05-17
AI Technical Summary
Existing methods for detecting target areas in input images often fail to accurately identify these regions, leading to missed detections.
An information processing device that uses a learning model trained on teacher images with annotated target and aggregate regions to detect and specify missed target areas by employing morphological operations and correction techniques.
Improves the accuracy of detecting target areas by identifying and correcting missed detections, enhancing precision in image analysis.
Smart Images

Figure 2024165520000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing device and the like that detects a target area included in an object from an input image obtained by photographing the object. [Background technology]
[0002] A technology for detecting a detection target region from an input image of an object is widely used. For example, Patent Document 1 discloses an ultrasound diagnostic device suitable for diagnosing or measuring the heart of a fetus. This ultrasound diagnostic device extracts a region corresponding to a part of interest in the heart of a fetus based on echo data for forming a tomographic image acquired by transmitting and receiving ultrasonic waves. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent No. 4206107 Summary of the Invention [Problem to be solved by the invention]
[0004] However, depending on the object and the input image, the detection target region may not always be detected appropriately, and in some cases detection may be missed.
[0005] An aspect of the present invention aims to improve the accuracy of detecting a detection target region from an input image obtained by capturing an object. [Means for solving the problem]
[0006] In order to solve the above problems, an information processing device according to one embodiment of the present invention includes a detection unit that detects each of the target regions and each of the assembly regions from an input image of the target object using a learning model trained using teacher images in which different classifications are annotated for each of the target regions contained in the target object and each of the assembly regions composed of a plurality of the target regions, and an identification unit that identifies missed detection regions, which are target regions that were not detected within the detected assembly regions.
[0007] Furthermore, an area identification method according to one embodiment of the present invention is an area identification method executed by one or more information processing devices, and includes a detection step of detecting the object areas and the aggregate areas from an input image of an object using a learning model trained using teacher images in which each of the object areas included in the object and each of the aggregate areas composed of a plurality of the object areas are annotated with different classifications, and an identification step of identifying missed detection areas, which are the object areas that were not detected within the detected aggregate areas.
[0008] The information processing device according to each aspect of the present invention may be realized by a computer. In this case, the control program of the information processing device that realizes the information processing device on a computer by causing the computer to operate as each part (software element) of the information processing device, and the computer-readable recording medium on which the control program is recorded, also fall within the scope of the present invention. Effect of the Invention
[0009] According to one aspect of the present invention, it is possible to improve the accuracy of detecting a detection target region from an input image obtained by capturing an object. [Brief description of the drawings]
[0010] [Figure 1] 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment of the present invention. [Diagram 2]1 is a block diagram showing an example of a configuration of a main part of a learning device according to an embodiment of the present invention. [Diagram 3] 1 is a block diagram showing an example of a configuration of a main part of an information processing device according to an embodiment of the present invention; [Figure 4] FIG. 13 is a diagram showing an example of a target region in an image of the heart. [Diagram 5] FIG. 13 is a diagram showing an example of a collection region in an image of the heart. [Figure 6] FIG. 13 is a diagram showing an example of a detection missed region in an image of the heart. [Figure 7] 11A and 11B are diagrams illustrating an example of a problem that occurs when a detection missed region is identified based on a detected target region and a group region. [Figure 8] FIG. 13 is a diagram illustrating an example of a method for identifying a detection-missed region based on detected target regions and aggregate regions. [Figure 9] FIG. 11 is a diagram illustrating another example of a method for identifying a detection missed region based on a detected target region and a collection region. [Figure 10] 11 is a diagram for explaining an example of an extension process performed by a specification unit; FIG. [Figure 11] 11 is a flowchart illustrating an example of a flow of a process executed by an information processing device. [Figure 12] FIG. 11 is a block diagram showing an example of a configuration of a main part of an information processing device according to another embodiment of the present invention. [Figure 13] 11 is a flowchart illustrating an example of a flow of a process executed by an information processing device. [Figure 14] FIG. 2 is a diagram for explaining the positional relationship of target regions in an image of a heart. [Figure 15] 1 is a diagram illustrating an example of the configuration of an information processing system including an arrhythmia detection device according to an embodiment of the present invention. [Figure 16] 1 is a diagram for explaining an example of processing performed by an information processing system including an arrhythmia detection device. [Figure 17] 1 is a diagram for explaining an example of a process in which the arrhythmia detection device detects arrhythmia. [Figure 18]11 is a flowchart illustrating an example of a process executed by the information processing system. [Figure 19] FIG. 13 is a diagram showing an example of a target area in an image of a dragonfly's head photographed from the front. [Figure 20] FIG. 13 is a diagram showing an example of a gathering region in an image of a dragonfly's head photographed from the front. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] [Embodiment 1] Hereinafter, one embodiment of the present invention will be described in detail with reference to FIGS.
[0012] (Overview of information processing device 1) An information processing device 1 according to an embodiment of the present invention detects each of the object regions included in an object and an aggregate region composed of a plurality of object regions from an input image obtained by photographing the object. A learning model trained using a teacher image in which the object regions and the aggregate regions are annotated with different classifications is used to detect the object regions and the aggregate regions. Then, the information processing device 1 identifies a detection miss region, which is an object region that was not detected within the detected aggregate region, using the detection results of the aggregate region and the object region.
[0013] When a learning model is used that is trained using a teacher image in which different classifications are annotated only to each of the object regions of the object, it may fail to detect one of the object regions. The inventors of the present invention have found a detection method that uses a learning model trained using a teacher image in which different classifications are annotated not only to each of the object regions but also to the aggregate region. If this detection method is used, even if a detection miss occurs in a target region, it is possible to identify the target region that was not detected within the detected aggregate region, thereby improving the detection accuracy of the target region.
[0014] The object to which the present invention is applicable is not particularly limited as long as it has a plurality of regions that can be target regions. For example, the object may be, for example, an animal organ. The heart of a mammal, including a human, has left and right atria and left and right ventricles, and when evaluating the function of the heart, each atrium and each ventricle can be a detection target (i.e., a target region). In this case, the input image may be a medical image, for example, an ultrasound image (for example, an echo image for forming a tomographic image). The ultrasound image may be a moving image including images of a plurality of frames.
[0015] The target object is not limited to the heart, but may be the lungs, liver, etc. The lungs and liver have multiple lobes, and each lobe may be the target region. In this case, the input image may be an X-ray CT image, etc.
[0016] The target object is not limited to animal organs, but may be any living thing including insects and plants. For example, the head of an insect has multiple sensory organs that can be detected. In addition, groups of cells that constitute characteristic organs and tissues of living things (e.g., stomata and vascular bundles of plants) can also be target regions.
[0017] The target object is not limited to living things, and may be a non-living object. For example, the target object may be an automobile. In this case, each area corresponding to each part constituting the automobile may be the target area. Each part may be a windshield, door glass, side mirror, door, wheel part of the wheel, tires, etc. In this case, a combination of the windshield and door glass, a combination of the side mirror and door, a combination of multiple doors (e.g., front and rear doors), and a combination of the wheel and tires may be a collective area composed of multiple target areas.
[0018] In the following, an example will be described in which the object is a heart and the input image is an ultrasound image of the heart. In the ultrasound image of the heart, each of the two ventricles and each of the two atria of the heart may be captured as a closed space.
[0019] (Configuration of information processing system 100) First, the configuration of an information processing system 100 according to an embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the information processing system 100. The information processing system 100 may include an imaging device 2, an information processing device 1, a display device 3, and a learning device 5.
[0020] The imaging device 2 may be an ultrasonic imaging device capable of non-invasively imaging the subject's organs. The imaging device 2 may be communicably connected to the information processing device 1 as shown in the figure, in which case the information processing device 1 may directly acquire the input image captured by the imaging device 2 from the imaging device 2. The input image captured by the imaging device 2 may be stored in, for example, a portable recording medium, in which case the information processing device 1 may read the input image from the recording medium. Alternatively, the input image captured by the imaging device 2 may be stored in, for example, a storage device (not shown) in association with subject information for each subject, in which case the information processing device 1 may acquire the input image from the storage device.
[0021] The display device 3 is a device capable of displaying various pieces of information output from the information processing device 1. In addition to the various pieces of information output from the information processing device 1, the display device 3 may display an input image, subject information, and the like.
[0022] The information processing device 1 may be installed in a facility (e.g., a medical facility) where the imaging device 2 is installed, or may be installed in a remote location. When installed in a remote location, the information processing device 1 may acquire input images by communication via a communication network such as the Internet.
[0023] Also, a part of the processing performed by the information processing device 1 may be configured to be performed by another computer system. For example, a calculation using a learning model described later is performed to detect the target area and the collection area that are the detection target, and this calculation may be configured to be performed by another computer system. In this case, the information processing device 1 may transmit information required for the calculation to the other computer system by communication via a communication network, and receive the calculation result transmitted from the other computer system.
[0024] The learning device 5 generates a learning model used to detect target regions and aggregate regions from the image of the heart captured by the imaging device 2. The learning model may be generated by known machine learning. For example, the learning device 5 may generate a learning model such as a convolutional neural network. Each of the target regions may constitute at least one aggregate region. By adopting this configuration, all target regions can be targeted for identification as detection missed regions. The target regions and aggregate regions to be detected will be described later with specific examples (see Figs. 4 and 5).
[0025] The learning model generated by the learning device 5 is used by the information processing device 1 to detect each area of the detection target. The location of the learning device 5 may be the same as or different from that of the information processing device 1. The learning device 5 executes a program that causes a computer to function as a learning device. The function of the learning device 5 may be possessed by the information processing device 1, in which case the learning device 5 is omitted from the components of the information processing system 100.
[0026] (Configuration of information processing device 1) Next, the configuration of the information processing device 1 will be described with reference to FIG. 2. FIG. 2 is a block diagram showing an example of the main configuration of the information processing device 1. As shown in the figure, the information processing device 1 includes a processor 10, a memory 11, and a storage device 12. As an example, the information processing device 1 may be a personal computer, a server, or a workstation. The processor 10 loads an information processing program stored in the storage device 12 into the memory 11 and executes it, thereby functioning as each of the units from an image acquisition unit 101 to an output control unit 105, which will be described later.
[0027] The processor 10 can be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or can be realized by software. When realized by software, the processor 10 may be configured, for example, by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a combination of these. In this case, the above software is stored in the storage device 12. The processor 10 then loads the above software into the memory 11 and executes it.
[0028] The memory 11 and the storage device 12 are both storage devices that store various data used by the information processing device 1. The memory 11 is a storage device that can write and read data at a higher speed than the storage device 12. The storage device 12 has a larger data storage capacity than the memory 11. As the memory 11, for example, a high-speed access memory such as a Synchronous Dynamic Random-Access Memory (SDRAM) can be applied. As the storage device 12, for example, a Hard Disk Drive (HDD), a Solid-State Drive (SSD), a Secure Digital (SD) card, or an embedded Multi-Media Controller (eMMC) can be applied.
[0029] The information processing device 1 also includes, as interfaces (IF) with external devices, an imaging device IF unit 13 and a display device IF unit 14. Any interface may be used as these interfaces, and for example, a Universal Serial Bus (USB), a High-Definition Multimedia Interface (HDMI), or the like may be used.
[0030] The photographing device IF unit 13 is an interface for connecting the photographing device 2 and the information processing device 1 for communication. The photographing device IF unit 13 may be for wired communication or for wireless communication. For example, a USB, a LAN (Local-Area Network), a wireless LAN, or the like may be applied as these IF units. It is an interface for acquiring the above-mentioned input image from the photographing device 2. Moreover, the display device IF unit 14 is an interface for transmitting information output by the information processing device 1 to the display device 3.
[0031] The display device IF unit 14 is an interface used when the information processing device 1 displays an image on the display device 3 (not shown). For example, the information processing device 1 can display, via the display device IF unit 14, an image captured by the imaging device 2 and a detection result output from the information processing device 1 on the display device 3. Note that the information processing device 1 may be provided with a display unit capable of functioning as the display device 3.
[0032] The information processing device 1 may further include an interface for communicating with various devices (e.g., storage devices) on a communication network such as the Internet. The information processing device 1 may also include an interface for accessing an external storage device (e.g., a hard disk) in addition to the above-mentioned interfaces.
[0033] As described above, the processor 10 functions as each of the image acquisition unit 101, the detection unit 102, the identification unit 103, and the output control unit 105. Each of these units will be described below.
[0034] The image acquiring unit 101 acquires an input image of the heart T captured by the photographing device 2. The method of acquiring the input image is not particularly limited, and for example, the image acquiring unit 101 may acquire the photographed image from the photographing device 2 via the photographing device IF unit 13.
[0035] The detection unit 102 uses a learning model generated by machine learning to detect each of the target region and the collection region from an input image capturing the heart T. The machine learning for generating the learning model will be described later.
[0036] The identification unit 103 identifies a missed detection region, which is a target region that was not detected, within the detected assembly region. For example, the identification unit 103 may identify, as a missed detection region, a region that is not detected as a target region among overlapping regions in which detected assembly regions overlap. The identification unit 103 may identify a missed detection region even when an overlapping region includes two or more target regions and some of the target regions are detected. Alternatively, the identification unit 103 may identify, as a missed detection region, a region obtained by excluding the detected target region from the detected assembly region.
[0037] The output control unit 105 controls the output of the detection results and identification results by the information processing device 1 to the display device 3.
[0038] According to this configuration, the information processing device 1 can identify the detection missed region by simple image processing such as removing the detected target region from the detected aggregation region, or removing the region detected as the target region from the overlap region where the detected aggregation regions overlap each other. Details of the process by the identification unit 103 to identify the detection missed region will be described later with a specific example.
[0039] (Configuration of learning device 5) Next, the configuration of the learning device 5 that generates a learning model will be described with reference to FIG. 3. FIG. 3 is a block diagram showing an example of the main configuration of the learning device 5. As shown in the figure, the learning device 5 includes a processor 50, a memory 51, and a storage device 52. The learning device 5 may be a personal computer, a server, or a workstation, similar to the information processing device 1. The processor 50 loads a learning program stored in the storage device 52 into the memory 51 and executes it, thereby functioning as each unit from an image acquisition unit 501 to a learning unit 504, which will be described later.
[0040] The processor 50 can be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or can be realized by software. When realized by software, the processor 50 may be configured, for example, by a CPU, a GPU, or a combination of these. In this case, the above software is stored in the storage device 52. The processor 50 then loads the above software into the memory 51 and executes it.
[0041] Both the memory 51 and the storage device 52 are storage devices that store various data used by the learning device 5. The memory 51 is a storage device that can write and read data at high speed compared to the storage device 52. The storage device 52 has a larger data storage capacity compared to the memory 51. A high-speed access memory such as an SDRAM can be used as the memory 51. Also, for example, an HDD, an SSD, an SD card, or an eMMC can be used as the storage device 52.
[0042] Furthermore, learning device 5 is provided with an imaging device IF section 53, an input device IF section 55, and a display device IF section 54 as interfaces (IFs) with external devices. Input device IF section 55 is an interface for receiving input signals from input devices such as a keyboard or a mouse. Input device IF section 55 may be an interface such as a USB. Of the interfaces provided in learning device 5, interfaces with the same names as those provided in information processing device 1 in FIG. 2 have the same functions, and therefore will not be described repeatedly.
[0043] As described above, the processor 50 functions as each of the image acquisition unit 501, the annotation acquisition unit 502, the teacher data generation unit 503, and the learning unit 504. Each unit will be described below.
[0044] The image acquisition unit 501 acquires learning images captured by the image capture device 2. The method of acquiring learning images is not particularly limited, and for example, the image acquisition unit 501 may acquire the learning images from the image capture device 2 via the image capture device IF unit 53. Also, for example, the image acquisition unit 501 may acquire learning images stored in the storage device 12 of the information processing device 1 by communicating with the information processing device 1. Furthermore, for example, when the learning images are stored in a storage device external to the information processing device 1, the image acquisition unit 301 may access the storage device to acquire the captured images.
[0045] The annotation acquisition unit 502 acquires different classifications for each of the target region and the collection region in the heart T captured in the learning image. The classifications may be input via an input device by a medical professional or the like who has checked the learning image.
[0046] <Target areas R1 to R4 and collection areas R5 to R8> Here, the target region and the aggregate region in an image of a heart T will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a diagram showing examples of target regions R1 to R4 in an image of a heart T, and Fig. 5 is a diagram showing examples of aggregate regions R5 to R8 in an image of a heart T.
[0047] The heart T includes four anatomical regions (i.e., two ventricles and two atria). Therefore, as shown in FIG. 4, a target region R1 corresponding to the right ventricle, a target region R2 corresponding to the left ventricle, a target region R3 corresponding to the left atrium, and a target region R4 corresponding to the right atrium may be set in the heart T. Furthermore, as shown in FIG. 5, a collection region R5 corresponding to the right side, a collection region R6 corresponding to the left side, a collection region R7 corresponding to the left and right ventricles, and a collection region R8 corresponding to the left and right atria may be set in the heart T. The collection region R5 includes the target region R1 and the target region R4. The collection region R6 includes the target region R2 and the target region R3. The collection region R7 includes the target region R1 and the target region R2. The collection region R8 includes the target region R3 and the target region R4. In this way, each of the target regions R1 to R4 constitutes at least one of the collection regions R5 to R8.
[0048] In an ultrasound image of a heart T, it is possible to associate different classifications with the target regions R1 to R4 and the collection regions R5 to R8. For example, it is possible to annotate the target regions R1 to R4 with classifications of "right ventricle," "left ventricle," "left atrium," and "right atrium," respectively, and it is possible to annotate the collection regions R5 to R8 with classifications of "right," "left," "ventricle," and "atrium," respectively.
[0049] 5 shows the collection regions R5 to R8 each composed of two adjacent target regions, but is not limited thereto. For example, the collection region may be a collection region composed of the right ventricle and the left atrium, a collection region composed of the left ventricle and the right atrium, or a collection region composed of three target regions.
[0050] The teacher data generating unit 503 generates a teacher image in which different classifications are annotated to each of the target regions R1 to R4 and each of the aggregation regions R5 to R8. The teacher image is teacher data used by the learning unit 504 when machine learning the learning model. The learning model is a model used by the detection unit 102 of the information processing device 1 to detect the target regions and aggregation regions.
[0051] The learning unit 504 generates a learning model by machine learning. As described above, the learning model is generated using the teacher image generated by the teacher data generating unit 503.
[0052] (Processing performed by the identification unit 103) Next, the process performed by the identifying unit 103 will be described with reference to Figs. 6 to 9. In the following, a case will be described in which the detection unit 102 has determined that the region constituting the assembly regions R6 and R7 (i.e., the target region R2 corresponding to the left ventricle) is a detection missed region. The identifying unit 103 identifies a detection missed region within the detected assembly region (e.g., the assembly region R6 and the assembly region R7). Fig. 6 shows a state in which the detection missed region R (e.g., a region corresponding to the target region R2) has been identified within the detected assembly region.
[0053] The identification unit 103 performs the identification by any one of the following processes (1) to (3). (1) The area excluding the detected target area R1 from the detected aggregate area R7 is identified as a detection missed area. (2) The area excluding the detected target area R3 from the detected aggregate area R6 is identified as a detection missed area. (3) The area where the detected aggregate area R6 and the detected aggregate area R7 overlap is identified as a detection missed area.
[0054] The detection results of the object region and the group region by the detection unit 102 may have errors in the outline and size, etc., compared with the actual object region and group region. For example, the size of the detected object region may be smaller than the actual object region, or the size of the detected group region may be larger than the actual group region. In these cases, when the detected object region is simply removed from the detected group region, a part of the group region may remain around the removed object region. FIG. 7 is a diagram showing an example of a defect that occurs when a detection omission region is specified based on the detected object region and group region. When the detected object region R1 is removed as it is from the detected group region R7, as shown in FIG. 7, a long and thin region is specified between the outline of the group region R7 and the outline of the object region R1.
[0055] Therefore, the identification unit 103 may identify an area obtained by further excluding an area adjacent to the periphery of the target area to be excluded from the detected aggregate area as a detection omission area. The area adjacent to the periphery of the target area to be excluded may be an area that has been increased by performing an expansion process using a well-known image processing technique on the target area to be excluded. Note that the area adjacent to the periphery of the target area to be excluded may be an area that has been increased by enlarging the target area to be excluded by a predetermined magnification.
[0056] According to this configuration, the information processing device 1 specifies an area obtained by removing not only the detected target area but also the area adjacent to the detected target area from the detected aggregate area as a detection missed area. This allows the information processing device 1 to reduce the possibility that a part of the aggregate area remains around the removed target area, and improve the accuracy of specifying the detection missed area. FIG. 8 is a diagram showing an example of a method for specifying a detection missed area based on the detected target area and aggregate area. FIG. 8 shows a process of removing an area R1a obtained as a result of performing an expansion process on the target area R1 to be removed from the aggregate area R7.
[0057] The detection results of the object region and the group region by the detection unit 102 may have errors in the outline, size, etc., compared to the actual object region and group region. For example, the size of the detected object region may be smaller than the actual object region, or the size of the detected group region may be larger than the actual group region. Even in these cases, when the detected object region is simply removed from the detected group region, a part of the group region may remain around the removed object region.
[0058] Therefore, the identifying unit 103 may identify, as a detection omission region, a region obtained by shrinking the detected aggregation region and excluding the detected target region.
[0059] According to this configuration, the information processing device 1 specifies an area obtained by excluding the detected target area from an area obtained by shrinking the detected aggregate area as a detection missed area. This allows the information processing device 1 to reduce the possibility that a part of the aggregate area remains around the excluded target area, and improve the accuracy of specifying the detection missed area. FIG. 9 is a diagram showing an example of a method for specifying a detection missed area based on the detected target area and aggregate area. FIG. 9 shows a process of excluding the target area R1 from an area R7a obtained as a result of performing a shrinking process on the aggregate area R7.
[0060] <Examples of expansion and contraction processing> Here, an example of expansion processing and reduction processing by morphological operations that can be executed by the determination unit 103 will be described with reference to Fig. 10. Fig. 10 is a diagram for explaining an example of the expansion processing performed by the determination unit 103. Note that the example shown in Fig. 10 will be described taking as an example a process of expanding a white area (an area with a pixel value of 1) in an image in which pixel values have been binarized for simplification.
[0061] First, a kernel is set to scan each pixel of the image one by one. For example, in the case of a kernel with a size of 3 pixels x 3 pixels, the central pixel can be called the anchor. In the example shown on the left in Figure 10, when the kernel has an anchor at the position of the pixel in the second row and third column (the pixel value before expansion is "0"), the maximum value of the pixels that have a common side with the anchor pixel (i.e., are adjacent) is "1". Therefore, a calculation is performed to change the pixel value of the pixel corresponding to the anchor position from "0" to "1". This process is performed until the kernel has scanned the positions of all pixels. The right in Figure 10 shows the result of performing such expansion process once.
[0062] In the case of a process for shrinking an area made up of pixels whose pixel value is "1" (i.e., a white area), the opposite calculation to the above is performed. That is, when the pixel value where the anchor is located is "1", if the minimum value of the pixels that share a common side with the anchor pixel (i.e., adjacent pixels) is "0", a calculation is performed to change the pixel value of the pixel corresponding to the anchor position to "0".
[0063] The identifying unit 103 may repeatedly execute the above calculation until no part of the aggregate region remains around the removed target region, thereby enabling the information processing device 1 to appropriately identify the detection omission region.
[0064] (Processing executed by information processing device 1) Next, the process executed by the information processing device 1 will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of the flow of the process executed by the information processing device 1. Note that the process of each step below may be executed by one or more information processing devices 1.
[0065] First, the image acquisition unit 101 acquires an input image of the heart T (step S1).
[0066] Next, the detection unit 102 detects a target region and a collection region from the input image using the learning model (step S2: detection step).
[0067] If there is an undetected region in the detection result by the detection unit 102, the identification unit 103 identifies the undetected region in the detected assembly region (step S3: identification step). This allows the information processing device 1 to identify the undetected region based on the detected object region and the detected assembly region even if there is a target region that could not be detected. This allows the information processing device 1 to detect the detection target region with high accuracy from the input image in which the target object is photographed.
[0068] [Embodiment 2] An information processing device 1a according to another embodiment of the present invention will be described with reference to Figures 12 to 14. For ease of explanation, members having the same functions as those described in the above embodiment will be denoted by the same reference numerals, and the description thereof will not be repeated.
[0069] (Information processing device 1a) The information processing device 1a has a function of correcting the detection result of the target area. FIG. 12 is a block diagram showing an example of a main configuration of the information processing device 1a according to this embodiment. The processor 10a included in the information processing device 1a can be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or can be realized by software. When realized by software, the processor 10a may be configured, for example, by a CPU, a GPU, or a combination of these. In this case, the above software is stored in the storage device 12. Then, the processor 10a loads the above software into the memory 11 and executes it.
[0070] The processor 10a functions as an image acquisition unit 101, a detection unit 102, an identification unit 103a, a correction unit 104, and an output control unit 105. The identification unit 103a and the correction unit 104 will be described below.
[0071] The identification unit 103a identifies a missed detection region, which is a target region that was not detected, within the detected assembly region. The identification unit 103a also identifies a missed detection region indicated by a result of correction by the correction unit 104, which will be described later. For example, the identification unit 103a may identify, as a missed detection region, a region that has not been detected as a target region among overlapping regions in which detected assembly regions overlap. The identification unit 103a may also identify a missed detection region when an overlapping region includes two or more target regions and some of the target regions are detected. Alternatively, the identification unit 103a may identify, as a missed detection region, a region obtained by excluding the detected target region from the detected assembly region.
[0072] The correction unit 104 corrects the erroneous detection made by the detection unit 102. Typical examples of erroneous detection made by the detection unit 102 include the following (A) and (B). (A) When the number of target regions included in the object is a predetermined number, more than the predetermined number of target regions are detected. For example, the heart T has four target regions, the left and right ventricles and the left and right atria. Therefore, when more than four target regions are detected, it includes a false positive. (B) When the positional relationship between the target regions included in the object is determined, multiple target regions are detected as the same classification. For example, the relative positional relationship between the four target regions of the left and right ventricles and the left and right atria of the heart T is determined. Therefore, for example, when multiple target regions are detected as the "right ventricle," one of them is a false positive.
[0073] The correction unit 104 may include an integration unit 1041 and a replacement unit 1042 capable of correcting the erroneous detection of (A) above, and / or a correction unit 1043 and a determination unit 1044 capable of correcting the erroneous detection of (B) above.
[0074] The number of object regions included in the object is a predetermined number, and when the number of detected object regions exceeds the predetermined number, the integration unit 1041 integrates, into one object region, the object regions of a combination of two of the detected object regions that have the largest degree of overlap between the object regions and that exceeds a predetermined threshold. Here, the degree of overlap may be a value obtained by dividing the area of the area where the two object regions of the combination overlap by the total area of the two object regions. The predetermined threshold is a threshold that is set in advance so as to increase the effect of correcting erroneous detection.
[0075] According to this configuration, when the information processing device 1a detects target areas that exceed the number of actual target areas included in the object, the information processing device 1a integrates the combination of target areas with a large degree of overlap into one target area. This allows the information processing device 1a to properly correct the detection result when the number of target areas is erroneously detected as more than the actual number. For example, when five target areas are detected from an input image of a heart T, the integration unit 1041 can correct the erroneous detection by integrating the target areas of the combination of two of the detected target areas that have the maximum degree of overlap between the target areas and exceed a predetermined threshold into one target area.
[0076] When the number of detected object regions exceeds a predetermined number and the maximum degree of overlap does not exceed a predetermined threshold, the replacement unit 1042 replaces the object region with the smallest area among the object regions detected as the same classification with a non-detected object region.
[0077] According to this configuration, the information processing device 1a replaces one of the target regions erroneously detected as belonging to the same classification with one that has not been detected, and therefore can correct the detection result.
[0078] For example, when five target regions are detected from an input image of a heart T, the integration unit 1041 replaces the target region with the smallest area among the target regions detected as the same classification with a non-detection. As a result, the target region becomes a detection-missed region and is identified by the identification unit 103a, so that the erroneous detection can be corrected.
[0079] 13 is a flowchart showing an example of the flow of processing executed by the information processing device 1a including the integration unit 1041 and the replacement unit 1042. Note that the processing of each of the following steps may be executed by one or more information processing devices 1a.
[0080] In step S2a, the modifying unit 104a determines whether the number of detected object regions exceeds the number of object regions included in the object (e.g., the heart). If the result in step S2a is YES, the modifying unit 104a determines whether the maximum value of the degree of overlap between the object regions exceeds a predetermined threshold (step S2b).
[0081] If the answer is YES in step S2b, the integration unit 1041 integrates the object regions of the combination with the greatest degree of overlap into one object region (step S2c). On the other hand, if the answer is NO in step S2b, the replacement unit 1042 replaces the object region with the smallest area among the object regions detected as the same classification with a non-detection (step S2d).
[0082] If the result of step S2a is NO, or after steps S2c and S2d, the identification unit 103a identifies an undetected region (step S3: identification step).
[0083] According to this configuration, the information processing device 1a can correct erroneous detection of the detection object, and therefore can improve the detection accuracy from the object.
[0084] Returning to FIG. 12, the correction unit 1043 corrects the classification of at least one of multiple target areas detected as having the same classification, based on the positional relationship between the target areas and the other detected target areas, for which the positional relationship between the target areas is determined.
[0085] For example, the positional relationship of the right ventricle, right atrium, left ventricle, and left atrium is fixed in the heart T. For example, when the heart T is imaged from the ventral side and when it is imaged from the dorsal side, an input image in which the positions of each target region are different is obtained. For example, in one example shown in FIG. 14, the target regions R2, R3, and R4 are arranged in the order of clockwise from the target region R1, while in the other example, the target regions R4, R3, and R2 are arranged in the order of clockwise from the target region R1. However, since the arrangement of each target region in the object is the same, the positional relationship between the target regions in these input images is the same. In other words, at least one of the multiple target regions detected as the same classification is a false detection, and it is possible to correct the classification of at least one of the multiple target regions detected as the same classification based on the positional relationship with the other detected target regions.
[0086] When the positional relationship between the detected target regions is determined, the position of the object of a detected target region is uniquely determined from the positional relationship with the other detected target regions. According to this configuration, the information processing device 1a corrects the classification of at least one of the multiple target regions detected as the same classification based on the positional relationship with the other detected target regions. As a result, the information processing device 1a corrects the classification error corresponding to the detected target region based on the positional relationship with the other detected target regions, and therefore can correct the detection result.
[0087] The determining unit 1044 determines the positional relationship between the target regions when all the target regions included in the target object have been detected. By including the determining unit 1044, the information processing device 1a can correctly determine the positional relationship between the target regions included in the target object.
[0088] [Embodiment 3] An information processing system 100a according to another embodiment of the present invention will be described with reference to Figures 15 to 18. For ease of explanation, the same reference numerals are given to components having the same functions as those described in the above embodiment, and the description thereof will not be repeated.
[0089] (Information processing system 100a) When a moving image (ultrasound moving image) including images of a plurality of frames obtained by photographing the heart T is used as an input image, the information processing device 1 can output a detection result including a detected target region and a specified missed detection region for each of the plurality of frames. As shown in FIG. 15, the information processing system 100a includes an arrhythmia detection device 4 that detects arrhythmia based on time series changes in the area of each of the target region detected by the information processing device 1 and the specified missed detection region. The arrhythmia detection device 4 detects arrhythmia based on time series changes in the area of each of the left atrium, right atrium, left ventricle, and right ventricle detected for each frame of the ultrasound moving image obtained by photographing the heart T by the information processing device 1 or specified as a missed detection region. With this configuration, the arrhythmia detection device 4 can detect arrhythmia with high accuracy.
[0090] (Processing performed by arrhythmia detection device) The process performed by the information processing system 100a including the arrhythmia detection device 4 will be described with reference to Fig. 16. Fig. 16 is a diagram for explaining an example of the process performed by the information processing system 100a.
[0091] The top row of FIG. 16 shows images corresponding to frames 0, 1, 2, 3, . . . of an input image capturing an image of a fetal heart. The middle row of FIG. 16 shows target regions indicating the left atrium, right atrium, left ventricle, and right ventricle, which are detected by the information processing device 1 or identified as a detection missed region, superimposed on each image in each frame. The bottom row of FIG. 16 shows the area of each of the left atrium, right atrium, left ventricle, and right ventricle, which are detected by the information processing device 1 or identified as a detection missed region, determined for each frame. The information processing device 1 executes the processes in the top row to the middle row in FIG. 16, and the arrhythmia detection device 4 executes the processes in the middle row to the bottom row in FIG. 16.
[0092] Next, the process of detecting arrhythmia by the arrhythmia detection device 4 will be described with reference to Fig. 17. Fig. 17 is a diagram for explaining an example of the process of detecting arrhythmia by the arrhythmia detection device 4. Fig. 17 is a graph showing the area (size) of the left atrium, right atrium, left ventricle, and right ventricle detected by the information processing device 1 or identified as a detection missed region, in a time series. The unit of area is mm 2 17. Alternatively, the area may be indicated by the number of pixels included in the regions corresponding to the left atrium, right atrium, left ventricle, and right ventricle in the image. When no arrhythmia occurs, the areas (sizes) of the left atrium, right atrium, left ventricle, and right ventricle all repeat periodic increase and decrease patterns. In contrast, when arrhythmia occurs, the periodic increase and decrease patterns are disrupted as seen in the frame shown in the square frame in FIG. 17. The arrhythmia detection device 4 can detect arrhythmia by detecting the disruption of the periodic increase and decrease patterns of the areas (sizes) of the left atrium, right atrium, left ventricle, and right ventricle.
[0093] (Processing Executed by Information Processing Device and Arrhythmia Detection Device) Next, the processes executed by the information processing device 1 and the arrhythmia detection device 4 will be described with reference to Fig. 18. Fig. 18 is a flowchart showing an example of the processes executed by the information processing system 100a. Note that the processes of the following steps may be executed by one or more information processing devices 1 and one or more arrhythmia detection devices 4.
[0094] The image acquisition unit 101 of the information processing device 1 acquires an ultrasound video image of the heart (step S11). Next, the detection unit 102 of the information processing device 1 detects the left atrium, right atrium, left ventricle, right ventricle, and the following collection regions (i) to (iv) for each frame of the ultrasound video image using a learning model (step S12). (i) The left atrium and left ventricle. (ii) The right atrium and right ventricle. (iii) The left and right atria. (iv) Left and right ventricles.
[0095] Next, the identifying unit 103 identifies a detection failure region in the detected assembly region (step S13).
[0096] Next, the arrhythmia detection device 4 analyzes the time series changes in the areas of the target regions detected by the information processing device 1 and the identified detection missed regions (step S14), and detects arrhythmia (step S15).
[0097] [Embodiment 4] (Area of interest on the dragonfly's head) The case where the object is the head of a dragonfly will be described with reference to Figs. 19 and 20. Fig. 19 is a diagram showing an example of an object region in an image of a dragonfly's head photographed from the front. Fig. 20 is a diagram showing an example of a collection region in an image of a dragonfly's head photographed from the front. The left side of Fig. 19 is an image of the dragonfly's head. The information processing device 1, 1a according to the present invention is capable of detecting an object region in an input image of a dragonfly's head photographed, and identifying an undetected region.
[0098] When the regions corresponding to the right compound eye, left compound eye, and mandible shown on the right in FIG. 19 are to be the target regions, the three regions shown in FIG. 20 may be set as the aggregate regions.
[0099] The information processing device 1, 1a can detect target areas corresponding to the anatomical organs of a dragonfly with high accuracy. In this way, the information processing device 1, 1a can be applied to detecting target areas of a wide range of objects. Therefore, the information processing device 1, 1a can be applied to, for example, creating learning materials about the anatomical structure of living organisms using input images of living organisms as materials.
[0100] [Software implementation example] The functions of the information processing device 1, 1a (hereinafter referred to as the "device") can be realized by a program for causing a computer to function as the device, and a program for causing a computer to function as each control block of the device (particularly each part included in the processor 10, 10a).
[0101] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program to realize each function described in each of the above embodiments.
[0102] The program may be non-transitory and may be recorded in one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be provided to the device via any wired or wireless transmission medium.
[0103] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.
[0104] Furthermore, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI may be executed by the control device or another device (for example, an edge computer or a cloud server).
[0105] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.
[0106] 〔summary〕 An information processing device according to aspect 1 of the present invention includes a detection unit that detects each of the object regions and each of the assembly regions from an input image of the object using a learning model trained using teacher images in which different classifications are annotated for each of the object regions contained in the object and each of the assembly regions composed of a plurality of the object regions, and an identification unit that identifies missed detection regions, which are object regions that were not detected within the detected assembly regions.
[0107] In the information processing device according to aspect 2 of the present invention, in the above-mentioned aspect 1, each of the target regions may constitute at least one of the group regions.
[0108] In an information processing device according to aspect 3 of the present invention, in aspect 1 or 2 above, the identification unit may identify, as the missed detection area, an overlap area in which the detected collection areas overlap, which is not detected as the target area.
[0109] An information processing device according to aspect 4 of the present invention, in any one of aspects 1 to 3 above, may be configured such that the identification unit identifies, as the missed detection area, an area obtained by excluding the detected target area from the detected collection area.
[0110] An information processing device according to aspect 5 of the present invention, in aspect 4 above, may be such that the identification unit identifies, from the detected aggregation area, an area further excluding areas adjacent to the periphery of the excluded target area as the missed detection area.
[0111] In an information processing device according to aspect 6 of the present invention, in aspect 4 or 5 above, the identification unit may identify, as the missed detection area, an area obtained by shrinking the detected collection area and excluding the detected target area.
[0112] An information processing device according to aspect 7 of the present invention may, in any of aspects 1 to 6 above, further include an integration unit that, when the number of target areas contained in the object exceeds the predetermined number, integrates, among two combinations of the detected target areas, the target areas of the combination in which the degree of overlap between the target areas is greatest and exceeds a predetermined threshold, into a single target area, where the number of target areas contained in the object exceeds a predetermined number.
[0113] An information processing device according to aspect 8 of the present invention, in aspect 7 above, may be such that the degree of overlap is a value obtained by dividing the area of the overlapping area between the two target areas of the combination by the sum of the areas of the two target areas.
[0114] An information processing device according to aspect 9 of the present invention may further include a replacement unit that, in aspect 7 or 8 above, replaces the target area with the smallest area among the target areas detected as the same classification with a non-detected area when the number of the detected target areas exceeds the specified number and the maximum degree of overlap does not exceed the specified threshold.
[0115] An information processing device according to aspect 10 of the present invention, in any of aspects 1 to 9 above, may further include a correction unit that, in which the positional relationship between the target areas is determined and which corrects the classification of at least one of a plurality of the target areas detected as having the same classification based on the positional relationship with the other detected target areas.
[0116] An information processing device according to aspect 11 of the present invention may, in aspect 10 above, further include a determination unit that determines the positional relationship between the target areas when all of the target areas contained in the object have been detected.
[0117] The target object is a heart, the target regions are the left atrium, right atrium, left ventricle, and right ventricle, and the input image is an ultrasound moving image, and the arrhythmia detection device of aspect 12 of the present invention detects arrhythmia based on the time-series changes in the area of the target region and the identified detection miss region detected for each frame of the input image by an information processing device described in any of aspects 1 to 11 above.
[0118] The area identification method according to aspect 13 of the present invention is an area identification method executed by one or more information processing devices, and includes a detection step of detecting the object areas and the aggregate areas from an input image of the object using a learning model trained using teacher images in which different classifications are annotated for each of the object areas contained in the object and each of the aggregate areas composed of a plurality of the object areas, and an identification step of identifying missed detection areas, which are the object areas that were not detected within the detected aggregate areas.
[0119] An area specifying program for causing a computer to function as the information processing device according to the first aspect, the area specifying program causing a computer to function as the detection unit and the specification unit. [Explanation of symbols]
[0120] 1, 1a Information processing device 2. Imaging Device 3 Display device 4. Arrhythmia detection device 100, 100a Information Processing Systems 102 Detection unit 103, 103a specific part 1041 Integration Department 1042 Replacement part 1043 Correction section 1044 Decision Section S2 Detection step S3 Identification Step
Claims
1. a detection unit that detects each of the target regions and each of the aggregate regions from an input image of the object by using a learning model that is trained using teacher images in which different classifications are annotated to each of the target regions included in the object and each of the aggregate regions formed by a plurality of the target regions; an identification unit that identifies a detection missed region, which is the target region that was not detected, within the detected aggregation region.
2. The information processing device according to claim 1 , wherein each of the target regions constitutes at least one of the collective regions.
3. The information processing device according to claim 1 , wherein the identifying unit identifies, as the detection missed region, a region that has not been detected as the target region among overlapping regions in which the detected aggregate regions overlap each other.
4. The information processing device according to claim 1 , wherein the identifying unit identifies, as the detection missed region, a region obtained by excluding the detected target region from the detected assembly region.
5. The information processing device according to claim 4 , wherein the identifying unit identifies, from the detected aggregation region, a region further excluding a region adjacent to a periphery of the target region to be excluded, as the detection missed region.
6. The information processing device according to claim 4 , wherein the identifying unit identifies, as the detection missed region, a region obtained by excluding the detected target region from a region obtained by contracting the detected aggregation region.
7. the number of the target regions included in the target object is a predetermined number, 2. The information processing device according to claim 1, further comprising an integration unit that, when the number of the detected target regions exceeds the predetermined number, integrates, into one target region, the target regions of a combination of two of the detected target regions that has the largest degree of overlap between the target regions and exceeds a predetermined threshold.
8. The information processing apparatus according to claim 7 , wherein the degree of overlap is a value obtained by dividing an area of an overlapping region between the two target regions of the combination by a sum of areas of the two target regions.
9. 8. The information processing device according to claim 7, further comprising a replacement unit that replaces the target area having the smallest area among the target areas detected as the same classification with a non-detected area when the number of the detected target areas exceeds the predetermined number and the maximum value of the degree of overlap does not exceed the predetermined threshold.
10. A positional relationship between the target regions is determined, The information processing apparatus according to claim 1 , further comprising a correction unit that corrects the classification of at least one of a plurality of the target regions detected as having the same classification based on a positional relationship between the detected target regions and other of the target regions.
11. The information processing apparatus according to claim 10 , further comprising a determination unit that determines a positional relationship between the target regions in a state in which all the target regions included in the object have been detected.
12. the object is a heart, the target regions are the left atrium, the right atrium, the left ventricle, and the right ventricle; the input image is an ultrasound video image, 12. An arrhythmia detection device that detects arrhythmia based on time-series changes in the areas of the target region detected for each frame of the input image and the identified detection missed region, in the information processing device according to claim 1.
13. 1. A method for identifying an area to be identified by one or more information processing devices, comprising: a detection step of detecting the object regions and the aggregate regions from an input image of the object using a learning model trained using teacher images in which each of the object regions included in the object and each of the aggregate regions formed by the plurality of the object regions are annotated with different classifications; and identifying a missed detection region, which is a target region that was not detected, within the detected aggregate region.
14. 2. An area specifying program for causing a computer to function as the information processing device according to claim 1, the area specifying program causing a computer to function as the detection unit and the specification unit.
Citation Information
Patent Citations
Part detection apparatus, part detection method and program
JP2012113414A
A system and method for quality assurance of medical image segmentation
WO2023057281A1
ultrasound diagnostic equipment
JP4206107B2
