Information processing device, arrhythmia detection device, region identification method, and region identification program

The information processing device employs a learning model with annotated classifications to enhance the detection of target areas and aggregates, addressing the issue of missed detections by using morphological operations, resulting in improved accuracy in image processing and object recognition.

WO2024236882A1PCT designated stage expired Publication Date: 2024-11-21KANADEVIA CORP
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/007721
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-17
Filing Date
2024-03-01
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Existing technologies face challenges in accurately detecting target areas from input images, leading to missed detections, which affects the precision of applications such as medical imaging and object recognition.

Method used

An information processing device uses a learning model trained with teacher images annotated for different classifications to detect target areas and aggregate regions, and a specifying unit identifies undetected areas within detected aggregates, improving detection accuracy by employing morphological operations like expansion and reduction processing.

Benefits of technology

This approach enhances the accuracy of detecting target areas by correctly identifying missed regions within detected aggregates, thereby improving the overall precision of image processing and object recognition systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024007721_21112024_PF_FP_ABST
    Figure JP2024007721_21112024_PF_FP_ABST
Patent Text Reader

Abstract

In order to improve the accuracy of detecting a detection target region from an input image capturing an object, an information processing device (1) is provided with: a detection unit (102) that detects each of a target region and an aggregate region from an input image capturing an object by using a trained model that is trained using training images in which each target region included in an object and each aggregate region composed of a plurality of target regions are annotated with different classifications; and a specification unit (103) that specifies an undetected region, which is a target region that has failed to be detected, in the detected aggregate region.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing device, arrhythmia detection device, region identification method, and region identification program

[0001] The present invention relates to an information processing device and the like that detects a target area included in an object from an input image obtained by photographing the object.

[0002] Techniques for detecting a detection target region from an input image of an object are widely used. For example, Patent Document 1 discloses an ultrasound diagnostic device suitable for diagnosing or measuring the fetal heart. This ultrasound diagnostic device extracts a region corresponding to a portion of interest in the fetal heart based on echo data for forming a tomographic image acquired by transmitting and receiving ultrasound waves.

[0003] Japanese Patent No. 4206107

[0004] However, depending on the object and input image, the detection target area may not always be detected properly, and in some cases, the detection may be missed.

[0005] An aspect of the present invention aims to improve the accuracy of detecting a detection target region from an input image of an object.

[0006] In order to solve the above problem, an information processing device according to one aspect of the present invention includes a detection unit that detects each of the target regions and each of the assembly regions from an input image of the object using a learning model trained using teacher images in which different classifications are annotated for each of the target regions included in the object and for each of the assembly regions made up of a plurality of the target regions, and an identification unit that identifies missed detection regions, which are target regions that were not detected within the detected assembly regions.

[0007] Furthermore, an area identification method according to one aspect of the present invention is an area identification method executed by one or more information processing devices, and includes a detection step of detecting the object areas and the assembly areas from an input image of the object using a learning model trained using teacher images in which different classifications are annotated for each of the object areas included in the object and for each of the assembly areas made up of a plurality of the object areas, and an identification step of identifying missed detection areas, which are the object areas that were not detected within the detected assembly areas.

[0008] The information processing device according to each aspect of the present invention may be realized by a computer. In this case, the control program of the information processing device that realizes the information processing device on a computer by causing the computer to operate as each part (software element) of the information processing device, and the computer-readable recording medium on which it is recorded, also fall within the scope of the present invention.

[0009] According to one aspect of the present invention, it is possible to improve the accuracy of detecting a detection target region from an input image obtained by capturing an object.

[0010] 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating an example of the configuration of the main parts of a learning device according to an embodiment of the present invention. FIG. 3 is a block diagram illustrating an example of the configuration of the main parts of an information processing device according to an embodiment of the present invention. FIG. 4 is a diagram illustrating an example of a target region in an image of an image of a heart. FIG. 5 is a diagram illustrating an example of an aggregate region in an image of an image of a heart. FIG. 6 is a diagram illustrating an example of a detection missed region in an image of an image of a heart. FIG. 7 is a diagram illustrating an example of a defect that occurs when a detection missed region is identified based on a detected target region and aggregate region. FIG. 8 is a diagram illustrating an example of a method for identifying a detection missed region based on a detected target region and aggregate region. FIG. 9 is a diagram illustrating another example of a method for identifying a detection missed region based on a detected target region and aggregate region. FIG. 10 is a diagram illustrating an example of an extension process performed by an identification unit. FIG. 11 is a flowchart illustrating an example of the flow of processing performed by an information processing device. FIG. 12 is a block diagram illustrating an example of the configuration of the main parts of an information processing device according to another embodiment of the present invention. FIG. 13 is a flowchart illustrating an example of the flow of processing performed by an information processing device. FIG. 14 is a diagram illustrating the positional relationship of target regions in an image of an image of a heart. FIG. 15 is a diagram illustrating an example of the configuration of an information processing system including an arrhythmia detection device according to an embodiment of the present invention. FIG. 16 is a diagram illustrating an example of processing performed by an information processing system including an arrhythmia detection device. FIG. 17 is a diagram illustrating an example of processing for detecting arrhythmia by an arrhythmia detection device. 10A and 10B are diagrams showing examples of target regions and aggregation regions in an image of a dragonfly's head photographed from the front, respectively;

[0011] First Embodiment Hereinafter, one embodiment of the present invention will be described in detail with reference to FIGS.

[0012] (Overview of Information Processing Device 1) An information processing device 1 according to one embodiment of the present invention detects, from an input image capturing an object, each of the object regions included in the object and an aggregate region composed of multiple object regions. To detect the object regions and aggregate regions, a learning model trained using teacher images in which the object regions and aggregate regions are annotated with different classifications is used. The information processing device 1 then identifies missed detection regions, which are object regions that were not detected within the detected aggregate regions, using the detection results of the aggregate regions and the object regions.

[0013] When a learning model trained using teacher images in which different classifications are annotated only to each of the object regions of an object is used, one of the object regions may be missed. The inventors of the present invention have discovered a detection method that uses a learning model trained using teacher images in which different classifications are annotated not only to each of the object regions but also to the aggregate region. By using this detection method, even if a target region is missed from detection, it is possible to identify the missed target region within the detected aggregate region, thereby improving the detection accuracy of the target region.

[0014] The object to which the present invention can be applied is not particularly limited as long as it has multiple regions that can be target regions. For example, the object may be, for example, an animal organ. The heart of a mammal, including a human, has left and right atria and left and right ventricles, and when evaluating cardiac function, each atrium and each ventricle can be a detection target (i.e., a target region). In this case, the input image may be a medical image, such as an ultrasound image (e.g., an echo image for forming a tomographic image). The ultrasound image may be a moving image including multiple frame images.

[0015] The target object is not limited to the heart, but may be the lungs, liver, etc. The lungs and liver have multiple lobes, and each lobe may be the target region. In this case, the input image may be an X-ray CT image, etc.

[0016] The target object is not limited to animal organs, but may be any living organism, including insects and plants. For example, the head of an insect has multiple sensory organs that can be detected. Furthermore, groups of cells that make up characteristic organs and tissues of living organisms (e.g., stomata and vascular bundles in plants) can also be target regions.

[0017] The target object is not limited to living things and may be non-living things. For example, the target object may be an automobile. In this case, each area corresponding to each part that makes up the automobile may be a target area. Each part may be a windshield, door glass, side mirrors, doors, wheel parts, tires, etc. In this case, a combination of a windshield and door glass, a combination of a side mirror and doors, a combination of multiple doors (e.g., front and rear doors), and a combination of wheels and tires may be considered as a collective area made up of multiple target areas.

[0018] In the following, an example will be described in which the object is a heart and the input image is an ultrasound image of the heart, in which each of the two ventricles and each of the two atria of the heart can be captured as a closed space.

[0019] (Configuration of Information Processing System 100) First, the configuration of an information processing system 100 according to an embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example configuration of the information processing system 100. The information processing system 100 may include an imaging device 2, an information processing device 1, a display device 3, and a learning device 5.

[0020] The imaging device 2 may be an ultrasound imaging device capable of non-invasively imaging the subject's organs. The imaging device 2 may be communicably connected to the information processing device 1 as shown in the figure. In this case, the information processing device 1 may directly acquire the input image captured by the imaging device 2 from the imaging device 2. The input image captured by the imaging device 2 may be stored in, for example, a portable recording medium. In this case, the information processing device 1 may simply read the input image from the recording medium. Alternatively, the input image captured by the imaging device 2 may be stored in, for example, a storage device (not shown) in association with subject information for each subject. In this case, the information processing device 1 may simply acquire the input image from the storage device.

[0021] The display device 3 is a device capable of displaying various pieces of information output from the information processing device 1. In addition to the various pieces of information output from the information processing device 1, the display device 3 may also display an input image, subject information, and the like.

[0022] The information processing device 1 may be installed in a facility (e.g., a medical facility) where the imaging device 2 is installed, or may be installed in a remote location. When installed in a remote location, the information processing device 1 may acquire input images by communication via a communication network such as the Internet.

[0023] Furthermore, a configuration may be adopted in which some of the processing performed by the information processing device 1 is performed by another computer system. For example, a calculation using a learning model, which will be described later, is performed to detect the target region and the collection region that are the detection target, and this calculation may be performed by another computer system. In this case, the information processing device 1 transmits information required for the calculation to the other computer system by communication via a communication network, and receives the calculation results transmitted from the other computer system.

[0024] The learning device 5 generates a learning model used to detect target regions and aggregation regions from the cardiac images captured by the imaging device 2. The learning model may be generated by known machine learning. For example, the learning device 5 may generate a learning model such as a convolutional neural network. Each target region may constitute at least one aggregation region. By adopting this configuration, all target regions can be targeted for identification as detection omission regions. Target regions and aggregation regions to be detected will be described later with specific examples (see FIGS. 4 and 5).

[0025] The learning model generated by the learning device 5 is used by the information processing device 1 to detect each area of ​​the detection target. The installation location of the learning device 5 may be the same as or different from that of the information processing device 1. The learning device 5 also executes a program that causes a computer to function as a learning device. The functions of the learning device 5 may be possessed by the information processing device 1, in which case the learning device 5 is omitted from the components of the information processing system 100.

[0026] (Configuration of Information Processing Device 1) Next, the configuration of the information processing device 1 will be described with reference to FIG. 2. FIG. 2 is a block diagram showing an example of the configuration of the main parts of the information processing device 1. As shown in the figure, the information processing device 1 includes a processor 10, a memory 11, and a storage device 12. The information processing device 1 may be, for example, a personal computer, a server, or a workstation. The processor 10 loads an information processing program stored in the storage device 12 into the memory 11 and executes it, thereby functioning as each part from an image acquisition unit 101 to an output control unit 105, which will be described later.

[0027] The processor 10 can be realized by a logic circuit (hardware) formed on an integrated circuit (IC chip) or the like, or by software. When realized by software, the processor 10 may be configured, for example, by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a combination of these. In this case, the software is stored in the storage device 12. The processor 10 then loads the software into the memory 11 and executes it.

[0028] The memory 11 and the storage device 12 are both storage devices that store various data used by the information processing device 1. The memory 11 is a storage device that can write and read data at higher speeds than the storage device 12. The storage device 12 has a larger data storage capacity than the memory 11. A high-speed access memory such as an SDRAM (Synchronous Dynamic Random-Access Memory) can be used as the memory 11. Furthermore, a hard disk drive (HDD), a solid-state drive (SSD), a secure digital (SD) card, or an embedded multi-media controller (eMMC) can be used as the storage device 12.

[0029] The information processing device 1 also includes, as interfaces (IF) with external devices, an imaging device IF unit 13 and a display device IF unit 14. Any interface may be used for these interfaces, and for example, a Universal Serial Bus (USB) or a High-Definition Multimedia Interface (HDMI (registered trademark)) may be used.

[0030] The photographing device IF unit 13 is an interface for connecting the photographing device 2 and the information processing device 1 for communication. The photographing device IF unit 13 may be for wired communication or for wireless communication. For example, a USB, a LAN (Local-Area Network), a wireless LAN, etc. can also be applied as these IF units. This is an interface for acquiring the above-mentioned input image from the photographing device 2. The display device IF unit 14 is an interface for transmitting information output by the information processing device 1 to the display device 3.

[0031] The display device IF unit 14 is an interface used by the information processing device 1 when displaying an image on a display device 3 (not shown). For example, the information processing device 1 can also display, via the display device IF unit 14, an image captured by the imaging device 2 and a detection result output from the information processing device 1 on the display device 3. Note that the information processing device 1 may also be provided with a display unit capable of functioning as the display device 3.

[0032] The information processing device 1 may further include an interface for communicating with various devices (e.g., storage devices) on a communication network such as the Internet. In addition to the above-mentioned interfaces, the information processing device 1 may also include an interface for accessing an external storage device (e.g., a hard disk).

[0033] As described above, the processor 10 functions as the image acquisition unit 101, the detection unit 102, the identification unit 103, and the output control unit 105. Each unit will be described below.

[0034] The image acquisition unit 101 acquires an input image of the heart T captured by the imaging device 2. The method of acquiring the input image is not particularly limited, and for example, the image acquisition unit 101 may acquire the captured image from the imaging device 2 via the imaging device IF unit 13.

[0035] The detection unit 102 uses a learning model generated by machine learning to detect each of the target region and the aggregate region from the input image of the heart T. Machine learning for generating the learning model will be described later.

[0036] The identification unit 103 identifies a missed detection region, which is a target region that was not detected within the detected assembly region. For example, the identification unit 103 may identify, as a missed detection region, a region that has not been detected as a target region among overlapping regions where detected assembly regions overlap. The identification unit 103 may also identify a missed detection region when an overlapping region includes two or more target regions and some of the target regions are detected. Alternatively, the identification unit 103 may identify, as a missed detection region, a region obtained by excluding the detected target region from the detected assembly region.

[0037] The output control unit 105 controls the output of the detection results and identification results by the information processing device 1 to the display device 3 .

[0038] According to this configuration, the information processing device 1 can identify a detection missed region by simple image processing, such as excluding a detected target region from a detected aggregation region, or excluding a region detected as a target region from an overlap region where detected aggregation regions overlap. Details of the process by which the identification unit 103 identifies a detection missed region will be described later with a specific example.

[0039] (Configuration of Learning Device 5) Next, the configuration of the learning device 5 that generates a learning model will be described with reference to FIG. 3. FIG. 3 is a block diagram showing an example of the configuration of the main parts of the learning device 5. As shown in the figure, the learning device 5 includes a processor 50, a memory 51, and a storage device 52. Like the information processing device 1, the learning device 5 may be a personal computer, a server, or a workstation. The processor 50 loads a learning program stored in the storage device 52 into the memory 51 and executes it, thereby functioning as each part from an image acquisition unit 501 to a learning unit 504, which will be described later.

[0040] The processor 50 can be realized by a logic circuit (hardware) formed on an integrated circuit (IC chip) or the like, or by software. When realized by software, the processor 50 may be configured, for example, by a CPU, a GPU, or a combination of these. In this case, the software is stored in the storage device 52. The processor 50 then loads the software into the memory 51 and executes it.

[0041] The memory 51 and the storage device 52 are both storage devices that store various data used by the learning device 5. The memory 51 is a storage device that can write and read data at higher speeds than the storage device 52. The storage device 52 has a larger data storage capacity than the memory 51. A high-speed access memory such as an SDRAM can be used as the memory 51. Furthermore, a HDD, SSD, SD card, eMMC, or the like can be used as the storage device 52.

[0042] The learning device 5 also includes an imaging device IF unit 53, an input device IF unit 55, and a display device IF unit 54 as interfaces (IFs) with external devices. The input device IF unit 55 is an interface for receiving input signals from input devices such as a keyboard or mouse. The input device IF unit 55 may be, for example, an interface such as a USB. Of the interfaces included in the learning device 5, interfaces with the same names as those included in the information processing device 1 in Figure 2 have the same functions, so a description thereof will not be repeated.

[0043] As described above, the processor 50 functions as the image acquisition unit 501, the annotation acquisition unit 502, the teacher data generation unit 503, and the learning unit 504. Each unit will be described below.

[0044] The image acquisition unit 501 acquires learning images captured by the imaging device 2. The method of acquiring learning images is not particularly limited, and for example, the image acquisition unit 501 may acquire the learning images described above from the imaging device 2 via the imaging device IF unit 53. Furthermore, for example, the image acquisition unit 501 may acquire learning images stored in the storage device 12 of the information processing device 1 by communicating with the information processing device 1. Furthermore, for example, if the learning images are stored in a storage device external to the information processing device 1, the image acquisition unit 301 may access the storage device to acquire the captured images.

[0045] The annotation acquisition unit 502 acquires different classifications for the target region and the collection region of the heart T shown in the learning image. The classifications can be input via an input device by a medical professional or the like who has checked the learning image.

[0046] <Target regions R1 to R4 and aggregate regions R5 to R8> Here, target regions and aggregate regions including multiple target regions in an image of a heart T will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a diagram showing examples of target regions R1 to R4 in an image of a heart T, and Fig. 5 is a diagram showing examples of aggregate regions R5 to R8 in an image of a heart T.

[0047] The heart T anatomically includes four regions (i.e., two ventricles and two atria). Therefore, as shown in FIG. 4 , a target region R1 corresponding to the right ventricle, a target region R2 corresponding to the left ventricle, a target region R3 corresponding to the left atrium, and a target region R4 corresponding to the right atrium may be set in the heart T. Furthermore, as shown in FIG. 5 , a collection region R5 corresponding to the right side, a collection region R6 corresponding to the left side, a collection region R7 corresponding to the left and right ventricles, and a collection region R8 corresponding to the left and right atria may be set in the heart T. The collection region R5 includes the target regions R1 and R4. The collection region R6 includes the target regions R2 and R3. The collection region R7 includes the target regions R1 and R2. The collection region R8 includes the target regions R3 and R4. Thus, each of the target regions R1 to R4 constitutes at least one of the collection regions R5 to R8.

[0048] In an ultrasound image of a heart T, different classifications can be associated with the target regions R1 to R4 and the collection regions R5 to R8. For example, the target regions R1 to R4 can be annotated with classifications such as "right ventricle," "left ventricle," "left atrium," and "right atrium," respectively, and the collection regions R5 to R8 can be annotated with classifications such as "right," "left," "ventricle," and "atrium," respectively. Furthermore, the collection region consisting of the target regions R1 and R2 and the collection region consisting of the target regions R3 and R4 can be annotated with classifications such as "ventricle" and "atrium," respectively.

[0049] 5 shows the collection regions R5 to R8, each consisting of two adjacent target regions, but is not limited to this. For example, the collection region may be a collection region consisting of the right ventricle and the left atrium, a collection region consisting of the left ventricle and the right atrium, or a collection region consisting of three target regions.

[0050] The training data generation unit 503 generates training images in which different classifications are annotated for each of the target regions R1 to R4 and each of the aggregation regions R5 to R8. The training images are training data used by the learning unit 504 during machine learning of the training model. The training model is a model used by the detection unit 102 of the information processing device 1 to detect the target regions and aggregation regions.

[0051] The learning unit 504 generates a learning model through machine learning. As described above, the learning model is generated using the training image generated by the training data generation unit 503.

[0052] (Processing Performed by Identifying Unit 103) Next, the processing performed by the identifying unit 103 will be described with reference to Figs. 6 to 9. In the following, an example will be described in which the region constituting the aggregation regions R6 and R7 (i.e., the target region R2 corresponding to the left ventricle) is a detection missed region determined by the detecting unit 102. The identifying unit 103 identifies a detection missed region within the detected aggregation region (e.g., the aggregation region R6 and the aggregation region R7). Fig. 6 shows how the detection missed region R (e.g., the region corresponding to the target region R2) is identified within the detected aggregation region.

[0053] The identification unit 103 performs the identification by one of the following processes (1) to (3).

[0054] (1) The region obtained by excluding the detected target region R1 from the detected aggregation region R7 is identified as a detection omission region.

[0055] (2) The region obtained by excluding the detected target region R3 from the detected aggregate region R6 is identified as a detection omission region.

[0056] (3) The area where the detected aggregate area R6 and the detected aggregate area R7 overlap is identified as a detection omission area.

[0057] The detection results of the object region and the aggregation region by the detection unit 102 may contain errors in the contours, size, etc. compared to the actual object region and aggregation region. For example, the size of the detected object region may be smaller than the actual object region, or the size of the detected aggregation region may be larger than the actual aggregation region. In these cases, when the detected object region is simply removed from the detected aggregation region, a portion of the aggregation region may remain around the removed object region. Figure 7 shows an example of a problem that occurs when a detection omission region is identified based on the detected object region and aggregation region. If the detected object region R1 is simply removed from the detected aggregation region R7, as shown in Figure 7, a long, narrow region is identified between the contours of the aggregation region R7 and the contours of the object region R1.

[0058] Therefore, the identification unit 103 may identify, as a detection omission region, a region obtained by further excluding, from the detected aggregation region, a region adjacent to the periphery of the target region to be excluded. The region adjacent to the periphery of the target region to be excluded may be a region that has been increased by performing an expansion process on the target region to be excluded using a well-known image processing technique. Note that the region adjacent to the periphery of the target region to be excluded may be a region that has been increased by enlarging the target region to be excluded by a predetermined magnification.

[0059] According to this configuration, the information processing device 1 identifies an area remaining in the detected aggregate area, excluding not only the detected target area but also the area adjacent to the detected target area, as a detection missed area. This reduces the possibility that part of the aggregate area remains around the excluded target area, improving the accuracy of identifying the detection missed area. Figure 8 is a diagram showing an example of a method for identifying a detection missed area based on the detected target area and aggregate area. Figure 8 shows the process of removing an area R1a obtained as a result of applying a dilation process to the target area R1 to be excluded from the aggregate area R7.

[0060] The detection results of the object region and the group region by the detection unit 102 may contain errors in the outline, size, etc. compared to the actual object region and group region. For example, the size of the detected object region may be smaller than the actual object region, or the size of the detected group region may be larger than the actual group region. Even in these cases, when the detected object region is simply removed from the detected group region, part of the group region may remain around the removed object region.

[0061] Therefore, the identifying unit 103 may identify, as a detection omission region, a region obtained by shrinking the detected aggregation region and excluding the detected target region.

[0062] According to this configuration, the information processing device 1 identifies, as a detection missed region, an area obtained by excluding the detected target region from an area obtained by shrinking the detected aggregate region. This reduces the possibility that part of the aggregate region remains around the excluded target region, thereby improving the accuracy of identifying the detection missed region. Figure 9 is a diagram showing an example of a method for identifying a detection missed region based on the detected target region and aggregate region. Figure 9 shows the process of excluding the target region R1 from a region R7a obtained as a result of performing a shrinking process on the aggregate region R7.

[0063] <Example of Expansion Processing and Reduction Processing> Here, an example of expansion processing and reduction processing using morphological operations that can be executed by the specification unit 103 will be described with reference to Fig. 10. Fig. 10 is a diagram for explaining an example of expansion processing performed by the specification unit 103. Note that the example shown in Fig. 10 will be described taking as an example a process of expanding a white area (an area with a pixel value of 1) in an image in which pixel values ​​have been binarized for simplification.

[0064] First, a kernel is set to scan each pixel of the image one by one. For example, in the case of a kernel with a size of 3 pixels by 3 pixels, the central pixel can be referred to as the anchor. In the example shown on the left in Figure 10, if the kernel has an anchor at the pixel in the second row and third column (the pixel value before expansion is "0"), the maximum value among the pixels that share a common side with the anchor pixel (i.e., are adjacent) is "1". Therefore, an operation is performed to change the pixel value of the pixel corresponding to the anchor position from "0" to "1". This process is repeated until the kernel has scanned all pixel positions. The right in Figure 10 shows the result of performing this expansion process once.

[0065] In the case of a process for reducing an area made up of pixels whose pixel value is "1" (i.e., a white area), the opposite operation to that described above is performed. That is, when the pixel value where the anchor is located is "1," if the minimum value of the pixels that share a common side with the anchor pixel (i.e., that are adjacent) is "0," an operation is performed to change the pixel value of the pixel corresponding to the anchor position to "0."

[0066] The identification unit 103 may repeatedly perform the above calculation until no part of the aggregate region remains around the removed target region, thereby enabling the information processing device 1 to appropriately identify the detection omission region.

[0067] (Processing Executed by Information Processing Device 1) Next, the processing executed by the information processing device 1 will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of the flow of processing executed by the information processing device 1. Note that the processing of each of the following steps may be executed by one or more information processing devices 1.

[0068] First, the image acquisition unit 101 acquires an input image of the heart T (step S1).

[0069] Next, the detection unit 102 detects a target region and a collection region from the input image using the learning model (step S2: detection step).

[0070] If the detection result by the detection unit 102 includes a detection missed area, the identification unit 103 identifies the detection missed area in the detected aggregation area (step S3: identification step). As a result, even if there is a target area that could not be detected, the information processing device 1 can identify the detection missed area based on the detected target area and the detected aggregation area. As a result, the information processing device 1 can detect the detection target area with high accuracy from the input image obtained by photographing the target object.

[0071] [Embodiment 2] An information processing device 1a according to another embodiment of the present invention will be described with reference to Figures 12 to 14. For ease of explanation, components having the same functions as those described in the above embodiment will be denoted by the same reference numerals, and their description will not be repeated.

[0072] (Information Processing Device 1a) The information processing device 1a has a function for correcting the detection result of the target region. FIG. 12 is a block diagram showing an example of the main configuration of the information processing device 1a according to this embodiment. The processor 10a included in the information processing device 1a can be realized by a logic circuit (hardware) formed on an integrated circuit (IC chip) or the like, or by software. When realized by software, the processor 10a may be configured, for example, by a CPU, a GPU, or a combination of these. In this case, the software is stored in the storage device 12. The processor 10a then loads the software into the memory 11 and executes it.

[0073] The processor 10a functions as an image acquisition unit 101, a detection unit 102, an identification unit 103a, a correction unit 104, and an output control unit 105. The identification unit 103a and the correction unit 104 will be described below.

[0074] The identification unit 103a identifies a missed detection region, which is a target region that was not detected, within the detected assembly region. Furthermore, the identification unit 103a identifies a missed detection region indicated by the result of correction by the correction unit 104, which will be described later. For example, the identification unit 103a may identify, as a missed detection region, a region that has not been detected as a target region within an overlap region where detected assembly regions overlap. The identification unit 103a may also identify a missed detection region when an overlap region includes two or more target regions and some of the target regions are detected. Alternatively, the identification unit 103a may identify, as a missed detection region, a region obtained by excluding the detected target region from the detected assembly region.

[0075] The correction unit 104 corrects the erroneous detection made by the detection unit 102. Typical examples of erroneous detection made by the detection unit 102 include the following (A) and (B).

[0076] (A) When the number of target regions included in the object is a predetermined number, more than the predetermined number of target regions are detected. For example, the heart T has four target regions, namely the left and right ventricles and the left and right atria. Therefore, when more than four target regions are detected, it includes erroneous detection.

[0077] (B) When the positional relationship between target regions included in an object is defined, multiple target regions are detected as the same classification. For example, the relative positional relationship between the four target regions of the left and right ventricles and the left and right atria of the heart T is defined. Therefore, for example, if multiple target regions are detected as the "right ventricle," one of them is a false positive.

[0078] The correction unit 104 may include an integration unit 1041 and a replacement unit 1042 capable of correcting the erroneous detection of (A) above, and / or a correction unit 1043 and a determination unit 1044 capable of correcting the erroneous detection of (B) above.

[0079] The integrating unit 1041 integrates, into a single target area, any two combinations of detected target areas that have the greatest degree of overlap and exceed a predetermined threshold value. The degree of overlap may be calculated by dividing the area of ​​the overlapping area of ​​the two target areas by the sum of the areas of the two target areas. The predetermined threshold value is a threshold value that is set in advance to enhance the effectiveness of correcting erroneous detections.

[0080] According to this configuration, when the information processing device 1a detects target areas that exceed the actual number of target areas included in the object, it integrates combinations of target areas with a large degree of overlap into a single target area. This allows the information processing device 1a to appropriately correct the detection result when more target areas than actually exist are erroneously detected. For example, when five target areas are detected from an input image of a heart T, the integration unit 1041 can correct the erroneous detection by integrating, into a single target area, the target areas of the combination of two of the detected target areas that have the largest degree of overlap between them and that exceeds a predetermined threshold.

[0081] If the number of detected target areas exceeds a predetermined number and the maximum degree of overlap does not exceed a predetermined threshold, the replacement unit 1042 replaces the target area with the smallest area among the target areas detected as the same classification with a non-detected area.

[0082] According to this configuration, the information processing device 1a replaces one of the target regions that has been erroneously detected as belonging to the same classification with one that has not been detected, and therefore can correct the detection result.

[0083] For example, if five target regions are detected from an input image of the heart T, the integration unit 1041 replaces the target region with the smallest area among the target regions detected as the same classification with a non-detected region. As a result, the target region becomes a detection omission region and is identified by the identification unit 103a, so that the erroneous detection can be corrected.

[0084] 13 is a flowchart showing an example of the flow of processing executed by the information processing device 1a including the integration unit 1041 and the replacement unit 1042. Note that the processing of each of the following steps may be executed by one or more information processing devices 1a.

[0085] In step S2a, the correction unit 104a determines whether the number of detected target regions exceeds the number of target regions included in the target (e.g., the heart). If the result in step S2a is YES, the correction unit 104a determines whether the maximum value of the degree of overlap between the target regions exceeds a predetermined threshold (step S2b).

[0086] If the result of step S2b is YES, the integrating unit 1041 integrates the target regions of the combination with the greatest degree of overlap into one target region (step S2c). On the other hand, if the result of step S2b is NO, the replacing unit 1042 replaces the target region with the smallest area among the target regions detected as the same classification with a non-detected region (step S2d).

[0087] If the result of step S2a is NO, or after steps S2c and S2d, the detection omission region is identified by the identifying unit 103a (step S3: identifying step).

[0088] According to this configuration, the information processing device 1a can correct erroneous detection of the detection object, thereby improving the detection accuracy from the object.

[0089] Returning to Figure 12, the correction unit 1043 corrects the classification of at least one of multiple target areas that have been detected as having the same classification, based on their positional relationship with other detected target areas, where the positional relationship between the target areas is determined.

[0090] For example, the positional relationships between the right ventricle, right atrium, left ventricle, and left atrium in the heart T are fixed. For example, when imaging the heart T from the ventral side and when imaging it from the dorsal side, input images are obtained in which the positions of each target region are different. For example, in one example shown in FIG. 14 , target regions R2, R3, and R4 are arranged clockwise from target region R1, while in the other example, target regions R4, R3, and R2 are arranged clockwise from target region R1. However, because the arrangement of each target region in the object is the same, the positional relationships between the target regions in these input images are the same. In other words, at least one of the multiple target regions detected as having the same classification is a false positive, and it is possible to correct the classification of at least one of the multiple target regions detected as having the same classification based on their positional relationships with other detected target regions.

[0091] When the positional relationships between detected target areas are determined, the position of a detected target area in the object is uniquely determined based on its positional relationship with the other detected target areas. With this configuration, the information processing device 1a corrects the classification of at least one of multiple target areas detected as having the same classification based on its positional relationship with the other detected target areas. This allows the information processing device 1a to correct classification errors corresponding to a detected target area based on its positional relationship with the other detected target areas, thereby correcting the detection results.

[0092] The determination unit 1044 determines the positional relationship between the target regions when all of the target regions included in the target object have been detected. By including the determination unit 1044, the information processing device 1a can correctly determine the positional relationship between the target regions included in the target object.

[0093] [Embodiment 3] An information processing system 100a according to another embodiment of the present invention will be described with reference to Figures 15 to 18. For ease of explanation, components having the same functions as those described in the above embodiments will be denoted by the same reference numerals, and their description will not be repeated.

[0094] (Information Processing System 100a) When a moving image (ultrasound moving image) including multiple frame images of a heart T is used as an input image, the information processing device 1 can output detection results including detected target regions and identified detection missed regions for each of the multiple frames. As shown in FIG. 15 , the information processing system 100a includes an arrhythmia detection device 4 that detects arrhythmia based on time-series changes in the area of ​​each of the target regions detected by the information processing device 1 and the identified detection missed regions. The arrhythmia detection device 4 detects arrhythmia based on time-series changes in the area of ​​each of the left atrium, right atrium, left ventricle, and right ventricle detected for each frame of the ultrasound moving image of the heart T captured by the information processing device 1 or identified as detection missed regions. With this configuration, the arrhythmia detection device 4 can accurately detect arrhythmia.

[0095] (Processing Performed by Arrhythmia Detection Device) Processing performed by the information processing system 100a including the arrhythmia detection device 4 will be described with reference to Fig. 16. Fig. 16 is a diagram for explaining an example of processing performed by the information processing system 100a.

[0096] The top row of Fig. 16 shows images corresponding to frames 0, 1, 2, 3, ... of an input image of a fetal heart. The middle row of Fig. 16 shows target regions indicating the left atrium, right atrium, left ventricle, and right ventricle, which have been detected by the information processing device 1 or identified as missed detection regions, superimposed on each image in each frame. The bottom row of Fig. 16 shows how the areas of the left atrium, right atrium, left ventricle, and right ventricle, which have been detected by the information processing device 1 or identified as missed detection regions, have been determined for each frame. The information processing device 1 performs the processes in the top to middle rows of Fig. 16, and the arrhythmia detection device 4 performs the processes in the middle to bottom rows of Fig. 16.

[0097] Next, the process of detecting arrhythmia by the arrhythmia detection device 4 will be described with reference to Fig. 17. Fig. 17 is a diagram for explaining an example of the process of detecting arrhythmia by the arrhythmia detection device 4. Fig. 17 is a graph showing the area (size) of each of the left atrium, right atrium, left ventricle, and right ventricle detected by the information processing device 1 or identified as a detection-missed region, plotted over time. The unit of area is mm 2 17 , etc. Alternatively, the areas may be represented by the number of pixels included in regions corresponding to the left atrium, right atrium, left ventricle, and right ventricle in the image. When no arrhythmia occurs, the areas (sizes) of the left atrium, right atrium, left ventricle, and right ventricle all repeat a periodic increase and decrease pattern. In contrast, when arrhythmia occurs, the periodic increase and decrease pattern is disrupted, as can be seen in the frame indicated by the square in FIG. 17 . The arrhythmia detection device 4 can detect arrhythmia by detecting disruption of the periodic increase and decrease pattern of the areas (sizes) of the left atrium, right atrium, left ventricle, and right ventricle.

[0098] (Processing Executed by Information Processing Device and Arrhythmia Detection Device) Next, processing executed by the information processing device 1 and the arrhythmia detection device 4 will be described with reference to Fig. 18. Fig. 18 is a flowchart showing an example of processing executed by the information processing system 100a. Note that the processing of each of the following steps may be executed by one or more information processing devices 1 and one or more arrhythmia detection devices 4.

[0099] The image acquisition unit 101 of the information processing device 1 acquires an ultrasound video image of the heart (step S11). Next, the detection unit 102 of the information processing device 1 uses a learning model to detect the left atrium, right atrium, left ventricle, right ventricle, and the following aggregate regions (i) to (iv) for each frame of the ultrasound video image (step S12).

[0100] (i) Left atrium and left ventricle.

[0101] (ii) Right atrium and right ventricle.

[0102] (iii) Left and right atria.

[0103] (iv) Left and right ventricles.

[0104] Next, the identifying unit 103 identifies a detection omission region in the detected aggregation region (step S13).

[0105] Next, the arrhythmia detection device 4 analyzes the time series changes in the areas of the target region detected by the information processing device 1 and the identified detection-missed region (step S14), and detects arrhythmia (step S15).

[0106] [Embodiment 4] (Target Area in the Head of a Dragonfly) A case where the target object is the head of a dragonfly will be described using Figs. 19 and 20. Fig. 19 is a diagram showing an example of a target area in an image of the head of a dragonfly photographed from the front. Fig. 20 is a diagram showing an example of a collection area in an image of the head of a dragonfly photographed from the front. The left side of Fig. 19 is an image of the head of a dragonfly. The information processing devices 1 and 1a according to the present invention are capable of detecting a target area in an input image of the head of a dragonfly and identifying areas that have not been detected.

[0107] When the regions corresponding to the right compound eye, left compound eye, and mandible shown on the right side of FIG. 19 are used as target regions, the three regions shown in FIG. 20 may be used as aggregate regions.

[0108] The information processing devices 1 and 1a can detect target regions corresponding to the anatomical organs of a dragonfly with high accuracy. Thus, the information processing devices 1 and 1a can be applied to detecting target regions of a wide range of objects. Therefore, the information processing devices 1 and 1a can be applied to, for example, creating learning materials about the anatomical structure of living organisms using input images of the organisms as materials.

[0109] [Example of implementation by software] The functions of the information processing device 1, 1a (hereinafter referred to as "device") can be realized by a program for causing a computer to function as the device, and a program for causing a computer to function as each control block of the device (particularly each part included in the processor 10, 10a).

[0110] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The functions described in each of the above embodiments are realized by executing the program using the control device and storage device.

[0111] The program may be non-transitory and may be recorded on one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

[0112] Furthermore, some or all of the functions of the control blocks can be realized by logic circuits. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of the control blocks can also be realized by, for example, a quantum computer.

[0113] Furthermore, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI ​​may run on the control device or on another device (for example, an edge computer or a cloud server).

[0114] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

[0115] [Summary] An information processing device according to aspect 1 of the present invention includes a detection unit that detects each of the target regions and each of the aggregate regions from an input image of the object using a learning model trained using teacher images in which different classifications are annotated for each of the target regions included in the object and for each of the aggregate regions made up of a plurality of the target regions, and an identification unit that identifies missed detection regions, which are target regions that were not detected within the detected aggregate regions.

[0116] In the information processing device according to Aspect 2 of the present invention, in Aspect 1, each of the target regions may constitute at least one of the group regions.

[0117] In an information processing device according to aspect 3 of the present invention, in the above-mentioned aspect 1 or 2, the identification unit may identify, as the missed detection area, an area among the overlapping areas where the detected aggregate areas overlap each other that has not been detected as the target area.

[0118] In an information processing device according to aspect 4 of the present invention, in any one of aspects 1 to 3 above, the identification unit may identify the area remaining after excluding the detected target area from the detected aggregation area as the missed detection area.

[0119] In an information processing device according to aspect 5 of the present invention, in the above aspect 4, the identification unit may identify, as the missed detection area, an area obtained by further excluding areas adjacent to the periphery of the target area to be excluded from the detected aggregation area.

[0120] In an information processing device according to aspect 6 of the present invention, in the above-mentioned aspects 4 or 5, the identification unit may identify, as the missed detection area, an area obtained by excluding the detected target area from an area obtained by contracting the detected aggregation area.

[0121] An information processing device according to aspect 7 of the present invention may, in any of aspects 1 to 6 above, further include an integration unit that, when the number of target areas included in the object is a predetermined number and the number of detected target areas exceeds the predetermined number, integrates, into a single target area, the target areas of a combination of two of the detected target areas that has the greatest degree of overlap between the target areas and exceeds a predetermined threshold.

[0122] In an information processing device according to aspect 8 of the present invention, in the above aspect 7, the degree of overlap may be a value obtained by dividing the area of ​​the area where the two target areas of the combination overlap by the sum of the areas of the two target areas.

[0123] An information processing device according to aspect 9 of the present invention may further include a replacement unit that, in the above-mentioned aspects 7 or 8, replaces the target area with the smallest area among the target areas detected as the same classification with a non-detected area when the number of detected target areas exceeds the predetermined number and the maximum value of the degree of overlap does not exceed the predetermined threshold.

[0124] An information processing device according to aspect 10 of the present invention may be in any of aspects 1 to 9 above, wherein the positional relationship between the target areas is determined, and further includes a correction unit that corrects the classification of at least one of the target areas detected as having the same classification based on the positional relationship with the other detected target areas.

[0125] An information processing device according to aspect 11 of the present invention may further include, in aspect 10 above, a determination unit that determines the positional relationship between the target areas when all of the target areas contained in the object have been detected.

[0126] The target object is a heart, the target region is the left atrium, right atrium, left ventricle, and right ventricle, and the input image is an ultrasound moving image.An arrhythmia detection device according to aspect 12 of the present invention detects arrhythmia based on the time series changes in the area of ​​the target region detected for each frame of the input image and the identified detection miss region by an information processing device according to any one of aspects 1 to 11 above.

[0127] An area identification method according to aspect 13 of the present invention is an area identification method executed by one or more information processing devices, and includes a detection step of detecting the object areas and the assembly areas from an input image of the object using a learning model trained using teacher images in which different classifications are annotated for each of the object areas included in the object and for each of the assembly areas made up of a plurality of the object areas, and an identification step of identifying missed detection areas, which are the object areas that were not detected within the detected assembly areas.

[0128] An area specifying program for causing a computer to function as the information processing device according to the first aspect, the area specifying program causing a computer to function as the detection unit and the specification unit.

[0129] REFERENCE SIGNS LIST 1, 1a Information processing device 2 Imaging device 3 Display device 4 Arrhythmia detection device 100, 100a Information processing system 102 Detection unit 103, 103a Identification unit 1041 Integration unit 1042 Replacement unit 1043 Correction unit 1044 Determination unit S2 Detection step S3 Identification step

Claims

1. An information processing device comprising: a detection unit that detects each of the object regions and each of the aggregate regions from an input image of an object using a learning model trained using teacher images in which each of the object regions contained in the object and each of the aggregate regions composed of a plurality of the object regions are annotated with different classifications; and an identification unit that identifies missed detection regions, which are object regions that were not detected within the detected aggregate regions.

2. The information processing device according to claim 1, wherein each of said target regions constitutes at least one of said collective regions.

3. The information processing device according to claim 1, wherein the identification unit identifies, as the detection missed region, an area that has not been detected as the target region among overlapping regions in which the detected aggregate regions overlap each other.

4. The information processing device according to claim 1, wherein the identification unit identifies an area remaining after excluding the detected target area from the detected assembly area as the detection missed area.

5. The information processing device according to claim 4, wherein the identification unit identifies, from the detected aggregation region, a region further excluding a region adjacent to the periphery of the excluded target region, as the detection missed region.

6. The information processing device according to claim 4, wherein the identification unit identifies, as the detection omission region, a region obtained by shrinking the detected aggregation region and excluding the detected target region.

7. The information processing device of claim 1, further comprising an integration unit that, when the number of the object regions contained in the object exceeds the predetermined number, integrates, into a single object region, the object regions of a combination of two of the detected object regions that has the greatest degree of overlap between the object regions and exceeds a predetermined threshold value.

8. The information processing device according to claim 7, wherein the degree of overlap is a value obtained by dividing an area of ​​an overlapping region between the two target regions of the combination by a sum of the areas of the two target regions.

9. The information processing device of claim 7, further comprising a replacement unit that replaces the target area having the smallest area among the target areas detected as the same classification with a non-detected area when the number of the detected target areas exceeds the predetermined number and the maximum value of the degree of overlap does not exceed the predetermined threshold.

10. The information processing device of claim 1, further comprising a correction unit that corrects the classification of at least one of a plurality of target areas detected as being of the same classification based on the positional relationship between the target areas and the other detected target areas, wherein the positional relationship between the target areas is determined.

11. The information processing device according to claim 10, further comprising a determination unit that determines a positional relationship between the target regions when all of the target regions included in the object have been detected.

12. An arrhythmia detection device that detects arrhythmia based on time series changes in the area of ​​the target area and the identified detection miss area detected for each frame of the input image, the target object being a heart, the target area being the left atrium, the right atrium, the left ventricle, and the right ventricle, and the input image being an ultrasound moving image, using an information processing device described in any one of claims 1 to 11.

13. A region identification method executed by one or more information processing devices, comprising: a detection step of detecting each of the object regions included in the object and each of the aggregate regions composed of a plurality of the object regions from an input image of the object using a learning model trained using teacher images in which different classifications are annotated for each of the object regions and each of the aggregate regions composed of a plurality of the object regions; and an identification step of identifying missed detection regions, which are object regions that were not detected within the detected aggregate regions.

14. An area identification program for causing a computer to function as the information processing device according to claim 1, the area identification program causing a computer to function as the detection unit and the identification unit.

Citation Information

Patent Citations

  • Part detection apparatus, part detection method and program

    JP2012113414A

  • Image processing apparatus, image processing method, and program

    JP2021086260A

  • ultrasound diagnostic equipment

    JP4206107B2