Information processing device, information processing method, and information processing program

Abstract

The objective is to obtain an information processing device, an information processing method, and an information processing program that can utilize images unsuitable for extracting regions of interest for diagnostic purposes. [Solution] The information processing device 10 includes at least one processor, which acquires multiple images obtained by photographing a subject, extracts a region of interest from each of the multiple images, detects structures included in the region of interest, synthesizes the detection results of the structures, determines whether or not there is a non-conforming region for each of the multiple regions of interest, and if it is determined that there is a non-conforming region, it determines whether or not there is a common non-conforming region among the extracted multiple regions of interest, and repeatedly acquires a new image including a region corresponding to at least a part of the common non-conforming region, extracts a region of interest from the new image, detects structures included in the region of interest, and re-synthesizes the detection results of the structures until it is determined that there is no common non-conforming region.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, based on medical images obtained by an image acquisition apparatus such as a CT (Computed Tomography) apparatus and an MRI (Magnetic Resonance Imaging) apparatus, a doctor generally makes a diagnosis. In addition, a technique (so-called CAD (Computer Aided Detection / Diagnosis)) in which a computer supports the detection and diagnosis of structures such as abnormal shadows and tissues included in medical images is also known. For example, Patent Document 1 describes analyzing a medical image using a discriminator learned by machine learning and specifying the type of tissue or lesion included in the medical image, that is, the type of finding.

[0003] Also, for example, Non-Patent Document 1 describes not performing CAD when at least a part of a medical image is unclear in order to avoid non-detection and misdiagnosis of abnormal shadows. An unclear medical image can be obtained, for example, when it is not taken in an appropriate environment and when the imaging technique of the photographer is low.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

[0006] Incidentally, in some cases, such as in medical institutions in emerging countries, it has been difficult to obtain clear medical images due to the inability to create an appropriate environment for medical image acquisition or the low skill level of the photographers. Therefore, in recent years, there has been a demand for technology that can be used for diagnosis even with blurry images, that is, images that are unsuitable for extracting regions of interest that may contain abnormal shadows and structures such as tissues.

[0007] This disclosure provides an information processing device, an information processing method, and an information processing program that enable the use of images unsuitable for extracting regions of interest for diagnostic purposes. [Means for solving the problem]

[0008] One aspect of the present disclosure is an information processing device comprising at least one processor, the processor acquires a plurality of images obtained by photographing a subject, extracts a region of interest from each of the plurality of images, detects structures contained in each of the extracted plurality of regions of interest, synthesizes the detection results of structures detected from each of the plurality of regions of interest, determines whether or not there is an incompatible region that is unsuitable for extraction for each of the extracted plurality of regions of interest, and if it is determined that an incompatible region exists, determines whether or not there is a common incompatible region that is unsuitable for extraction for all of the extracted plurality of regions of interest, and repeatedly performs the following until it is determined that there is no common incompatible region: acquiring a new image containing a region corresponding to at least a part of the common incompatible region, extracting a region of interest from the new image, detecting structures contained in the region of interest, and resynthesizing the detection results of the structures.

[0009] In the above embodiment, the processor may terminate the iteration if it determines that no common non-conforming areas exist, or if the number of resynthesis attempts exceeds a predetermined limit.

[0010] In the above embodiment, the processor may terminate the iteration if it determines that no common non-conforming areas exist, or if the ratio of common non-conforming areas to any of the extracted areas of interest falls below a predetermined threshold.

[0011] In the above embodiment, if the processor determines that a common non-conforming region exists at the end of the iteration, it may detect the structures included in the common non-conforming region with reduced detection accuracy.

[0012] In the above embodiment, if the processor determines that a non-conforming region exists, it may detect the structure included in the non-conforming region with reduced detection accuracy.

[0013] In the above embodiment, the processor may indicate that the detection accuracy of the structure is low if the structure is detected with reduced detection accuracy.

[0014] In the above embodiment, the processor may perform control to present a non-conforming region in at least one of the multiple images in a manner that distinguishes it from other regions.

[0015] In the above embodiment, the processor may make a determination based on the degree of similarity between the shape of the extracted region of interest and a predetermined reference shape relating to the region of interest.

[0016] In the above embodiment, the processor takes an image obtained by photographing a subject as input and makes a determination using a trained model for determining whether or not there is an incompatible region in the region of interest extracted from the image. The trained model may be a trained model that has been trained using pairs of images obtained by photographing a subject and information indicating whether or not there is an incompatible region in the region of interest extracted from the image as training data.

[0017] In the above embodiment, the processor may perform control to display a string indicating whether or not a non-compliant area exists on a screen where at least one of multiple images is displayed.

[0018] In the above embodiment, if the processor determines that a non-conforming region exists, it may perform control to present the ratio of the non-conforming region to any of the extracted regions of interest.

[0019] In the above embodiment, if the processor determines that a non-compatible region exists, it may re-extract the region of interest with reduced extraction accuracy.

[0020] In the above embodiment, the processor may detect structures in each of the extracted regions of interest from regions other than the non-conforming region, and may not perform structure detection in the non-conforming region.

[0021] In the above embodiment, the processor may perform control to display structures detected from other regions on the image of the source of detection.

[0022] In the above aspect, the region of interest may be a region including at least one of a subject, a part of the tissues included in the subject, and an abnormal part included in the subject or the tissues.

[0023] In the above aspect, the plurality of images may be images obtained by at least one of a radiographic imaging device, a magnetic resonance imaging device, an ultrasonic device, an ophthalmic imaging device, and an endoscope.

[0024] Another aspect of the present disclosure is an information processing method, which includes acquiring a plurality of images obtained by imaging a subject, extracting a region of interest from each of the plurality of images, detecting a structure included in each of the extracted plurality of regions of interest, synthesizing the detection results of the structures detected from each of the plurality of regions of interest, determining whether there is an incompatible region that is not suitable for extraction for each of the extracted plurality of regions of interest, and when it is determined that there is an incompatible region, determining whether there is a common incompatible region that is commonly not suitable for extraction for the extracted plurality of regions of interest, and repeating the acquisition of a new image including a region corresponding to at least a part of the common incompatible region, the extraction of the region of interest from the new image, the detection of the structure included in the region of interest, and the resynthesis of the detection result of the structure until it is determined that there is no common incompatible region, which is executed by a computer.

[0025] Another aspect of the present disclosure is an information processing program that acquires a plurality of images obtained by photographing a subject, extracts a region of interest from each of the plurality of images, detects a structure included in each of the extracted regions of interest, synthesizes the detection results of the structures detected from each of the plurality of regions of interest, determines whether there is an incompatible region that is incompatible with the extraction for each of the extracted regions of interest, and when it is determined that there is an incompatible region, determines whether there is a common incompatible region that is commonly incompatible with the extraction for the plurality of extracted regions of interest, and until it is determined that there is no common incompatible region, acquires a new image including a region corresponding to at least a part of the common incompatible region, extracts a region of interest from the new image, detects a structure included in the region of interest, and re-synthesizes the detection result of the structure, and is for causing a computer to repeatedly execute the process.

Advantages of the Invention

[0026] According to the above aspect, the information processing apparatus, information processing method, and information processing program of the present disclosure can be utilized for diagnosis even with an image that is incompatible with the extraction of the region of interest.

Brief Description of the Drawings

[0027] [Figure 1] It is a schematic configuration diagram of an information processing system. [Figure 2] It is a block diagram showing an example of the hardware configuration of an information processing apparatus. [Figure 3] It is a block diagram showing an example of the functional configuration of an information processing apparatus according to the first embodiment. [Figure 4] It is an example of a clear medical image. [Figure 5] It is an example of a blurry first image. [Figure 6] It is an example of a presented screen. [Figure 7] It is a flowchart showing an example of a determination process. [Figure 8] It is a block diagram showing an example of the functional configuration of an information processing apparatus according to the second and third embodiments. [Figure 9] This is an example of a screen that may be displayed. [Figure 10] This is an example of a blurry second image. [Figure 11] This is an example of a synthesized third image. [Figure 12] This is an example of a screen that may be displayed. [Figure 13] This is a flowchart showing an example of image synthesis processing. [Figure 14] This is an example of a screen that may be displayed. [Figure 15] This flowchart shows an example of the detection result synthesis process. [Figure 16] This figure shows an example of a pseudo-image that is input into a learning model. [Figure 17] This is a flowchart illustrating an example of the learning process. [Modes for carrying out the invention]

[0028] Hereinafter, with reference to the drawings, examples of embodiments for carrying out the technology of this disclosure will be described in detail.

[0029] [First Embodiment] Referring to Figure 1, an example of the configuration of the information processing system 1 according to this embodiment will be described. As shown in Figure 1, the information processing system 1 includes an information processing device 10 and an image acquisition device 2. The information processing device 10 and the image acquisition device 2 are capable of communicating with each other by wired or wireless communication.

[0030] Image acquisition device 2 is a device (so-called modality) that acquires images obtained by photographing a subject. In this embodiment, as a specific example of an image, medical images will be described as being acquired by image acquisition device 2. At least one of the following can be used as image acquisition device 2: radiography device, magnetic resonance imaging device, ultrasound device, fundus imaging device, and endoscope, and these may be used in appropriate combinations.

[0031] For example, in order to clearly photograph the fundus of the eye, the imaging should be performed in a dark place. However, in medical institutions in developing countries, it may not be possible to create a sufficiently dark place, resulting in blurry medical images. Similarly, in order to clearly photograph the breast during mammography, the breast should be sufficiently compressed. However, depending on the positioning technique of the photographer and the shape of the breast, it may not be possible to compress the breast sufficiently, resulting in blurry medical images. In recent years, there has been a demand for technologies that can utilize even such blurry images for diagnosis.

[0032] Therefore, the information processing device 10 according to this embodiment has a function to determine whether or not the medical image contains unclear areas. "Clear" means that the pixel values ​​such as hue, saturation, brightness, and lightness shown by each pixel of the medical image do not meet predetermined reference values. For example, unclear areas may occur when pixels become darker than the reference value due to insufficient light, or when pixels become brighter than the reference value due to ambient light. An example of the configuration of the information processing device 10 according to this embodiment will be described below.

[0033] First, an example of the hardware configuration of the information processing device 10 according to this embodiment will be described with reference to Figure 2. As shown in Figure 2, the information processing device 10 includes a CPU (Central Processing Unit) 21, a non-volatile storage unit 22, and a memory 23 as a temporary storage area. The information processing device 10 also includes a display 24 such as a liquid crystal display, an input unit 25 such as a keyboard and mouse, and a network I / F 26 that performs wired or wireless communication with an image acquisition device 2 and an external network (not shown). The CPU 21, storage unit 22, memory 23, display 24, input unit 25, and network I / F 26 are connected to each other via a bus 28 such as a system bus and a control bus, enabling the exchange of various types of information.

[0034] The storage unit 22 is implemented by a storage medium such as an HDD (Hard Disk Drive), SSD (Solid State Drive), and flash memory. The information processing program 27 according to this embodiment is stored in the storage unit 22. The CPU 21 reads the information processing program 27 from the storage unit 22, expands it into the memory 23, and executes the expanded information processing program 27. The CPU 21 is an example of the processor of this disclosure. Various computers such as the console of the image acquisition device 2, a workstation, a server computer, and a personal computer can be used as the information processing device 10.

[0035] Next, with reference to Figure 3, an example of the functional configuration of the information processing device 10 according to this embodiment will be described. As shown in Figure 3, the information processing device 10 includes an acquisition unit 11, an extraction unit 12, a determination unit 13, a detection unit 14, and a presentation unit 15. The CPU 21 executes the information processing program 27, thereby enabling the acquisition unit 11, extraction unit 12, determination unit 13, detection unit 14, and presentation unit 15 to function.

[0036] The acquisition unit 11 acquires medical images obtained by photographing the subject from the image acquisition device 2. The medical images in this embodiment include a region of interest that includes the subject, some tissue contained in the subject, and at least one abnormal part contained in the subject or tissue, as the structure of which detection and diagnosis are desired. Examples of subjects include the human body and various organs of the human body such as the fundus, lungs, breasts, stomach, liver, heart, and brain. Examples of tissues include elements that make up various organs such as blood vessels, nerves, and muscles. Examples of abnormal parts include lesions and abnormalities such as tumors, injuries, defects, nodules, and inflammation.

[0037] Figure 4 shows medical image G0 as an example of a clear medical image. Medical image G0 is a clear image of the fundus obtained by a fundus imaging device. In fundus images, abnormal areas such as lesions may be included in the entire imaging range, which includes the fovea, macula, optic nerve head, and blood vessels; therefore, the entire imaging range corresponds to the region of interest A0. In the example in Figure 4, abnormal shadows S1 and S2 are included in the region of interest A0 as an example of abnormal areas.

[0038] On the other hand, Figure 5 shows Image 1 G1 as an example of an unclear medical image. Image 1 G1 is a fundus image similar to medical image G0, but part of the imaging range is unclear, and there is an unsuitable region N1 that is unsuitable for extracting the original first region of interest A1 (i.e., the entire imaging range). The unsuitable region N1 originally contains an abnormal shadow S1, but its detection is difficult due to its clarity. The following explanation assumes that the acquisition unit 11 acquired Image 1 G1.

[0039] The extraction unit 12 extracts the first region of interest A1 from the first image G1 acquired by the acquisition unit 11. The extraction method for the first region of interest A1 can appropriately be one of several known methods, including image processing and AI (Artificial Intelligence) technology. For example, the first region of interest A1 may be extracted by binarizing the first image G1, removing the background, and enhancing the edges of each structure to identify the contour of the shooting range. Alternatively, the shooting range (i.e., the first region of interest A1) may be extracted from the first image G1 using a trained model that takes an image of the fundus as input and is trained to extract and output the shooting range.

[0040] The detection unit 14 detects abnormal shadows contained in the first region of interest A1 extracted by the extraction unit 12. In the example of the first image G1 in Figure 5, the detection unit 14 detects abnormal shadow S2, but abnormal shadow S1 is not detected because it is contained in the unclear non-conforming region N1. Note that known CAD techniques can be appropriately applied as abnormal shadow detection methods.

[0041] The determination unit 13 determines whether or not there is an incompatible region N1 that is unsuitable for extraction with respect to the first region of interest A1 extracted by the extraction unit 12. Various methods can be applied as the determination method. For example, the determination unit 13 may make a determination according to the degree of similarity between the shape of the first region of interest A1 extracted by the extraction unit 12 and a predetermined reference shape for the first region of interest A1. In the case of a fundus image, the region of interest is the entire imaging range, so the reference shape can be predetermined as approximately circular. Therefore, if the degree of similarity between the shape of the contour of the first region of interest A1 extracted by the extraction unit 12 and the approximately circular reference shape is below a predetermined threshold (i.e., the deviation is large), it may be determined that an incompatible region N1 exists. As a determination method according to the degree of similarity, known matching techniques such as feature matching and template matching can be appropriately applied.

[0042] Alternatively, for example, AI technology may be applied to the determination, and the determination may be made using a trained model that takes a medical image as input and determines whether or not there is a non-suitable region in the region of interest extracted from the medical image. In this case, the trained model may be an unsupervised learning model, for example, a model that has been trained to cluster medical images according to the presence or absence of non-suitable regions. Alternatively, it may be a supervised learning model, for example, a model that has been trained using pairs of medical images and information indicating whether or not there is a non-suitable region in the region of interest extracted from the medical image as training data.

[0043] Figure 6 shows an example of a screen D1 presented on the display 24 by the presentation unit 15. As shown in Figure 6, the presentation unit 15 presents the abnormal shadow S2 detected by the detection unit 14 by adding annotations M to the first image G1.

[0044] Furthermore, if the determination unit 13 determines that a non-conforming region N1 exists in the first region of interest A1, the presentation unit 15 identifies and presents the non-conforming region N1 in the first image G1. In the example in Figure 6, the presentation unit 15 presents the non-conforming region N1 in the first image G1 by enclosing it with a thick line. Note that the method of presenting the non-conforming region N1 is not limited to this; for example, the non-conforming region N1 can be emphasized in a way that makes it distinguishable by changing the line type (e.g., line thickness, color, solid and dotted lines, etc.) from other regions, or by adding annotations. Conversely, the non-conforming region N1 can also be distinguished by emphasizing regions other than the non-conforming region N1 in the first region of interest A1.

[0045] Furthermore, if the determination unit 13 determines that a non-conforming area N1 exists in the first area of ​​interest A1, the presentation unit 15 may also present the ratio of the non-conforming area N1 to the first area of ​​interest A1. In the example in Figure 6, the non-conforming area N1 is described as an "unimplementable" area where the detection unit 14 cannot perform abnormal shadow detection processing. In addition, areas in the first area of ​​interest A1 other than the non-conforming area N1 are described as "implemented" areas where the detection unit 14 has completed abnormal shadow detection.

[0046] Next, the operation of the information processing device 10 according to this embodiment will be explained with reference to Figure 7. The CPU 21 executes the information processing program 27, which in turn executes the determination process shown in Figure 7. The determination process shown in Figure 7 is executed, for example, when the user issues an instruction to start processing via the input unit 25.

[0047] In step S10, the acquisition unit 11 acquires the first image G1 from the image acquisition device 2. In step S11, the extraction unit 12 extracts the first region of interest A1 from the first image G1 acquired in step S10. In step S12, the detection unit 14 detects abnormal shadows contained in the first region of interest A1 extracted in step S11. In step S13, the determination unit 13 determines whether or not there are incompatible regions N1 in the first region of interest A1 extracted in step S11 that are unsuitable for extraction.

[0048] If a non-conforming region N1 exists (i.e., step S13 is a positive determination), the process proceeds to step S14, where the display unit 15 identifies the non-conforming region N1 in the first image G1 and presents it together with the abnormal shadow detected in step S12. On the other hand, if a non-conforming region N1 does not exist (i.e., step S13 is a negative determination), the process proceeds to step S15, where the display unit 15 presents only the abnormal shadow detected in step S12. Once step S14 or S15 is completed, the determination process ends. After step S14, the image synthesis process according to the second embodiment and / or the detection result synthesis process according to the third embodiment, which will be described later, may be performed.

[0049] As described above, the information processing device 10 according to the first embodiment includes at least one processor, which acquires a first image G1 obtained by photographing a subject, extracts a first region of interest A1 from the first image G1, and determines whether or not there are any incompatible regions N1 that are unsuitable for extraction in the extracted first region of interest A1. In other words, the information processing device 10 determines whether or not there are any unclear regions in the first region of interest A1 that may contain a target structure for which detection and diagnosis of abnormal shadows, etc., is desired. Therefore, even an unclear first image G1 can be used for diagnosis after recognizing the presence of unclear regions in the first image G1.

[0050] In the first embodiment described above, the detection unit 14 may detect only abnormal shadows in areas of the first region of interest A1 other than the non-conforming region N1, and may not detect abnormal shadows in the non-conforming region N1. This is because the possibility of non-detection and false detection of abnormal shadows is higher in the unclear non-conforming region N1 compared to other areas, resulting in lower reliability of the detection results.

[0051] On the other hand, in the first embodiment described above, if the determination unit 13 determines that a non-conforming area N1 exists in the first area of ​​interest A1, the detection unit 14 may detect structures such as abnormal shadows included in the non-conforming area N1 by reducing the detection accuracy. "Detecting by reducing the detection accuracy" means that detection of abnormal shadows is performed even if the reliability is low, while allowing for the possibility of non-detection and false detection of abnormal shadows. In this case, it is preferable for the presentation unit 15 to indicate that the accuracy of abnormal shadow detection is low for the non-conforming area N1. This is especially preferable when the ratio of the non-conforming area N1 to the first area of ​​interest A1 is above a predetermined threshold (for example, 20% or more). This is because if abnormal shadows are not detected in the non-conforming area N1 when the ratio of the non-conforming area N1 is high, the user would have to visually check for abnormal shadows in a large portion of the first image G1, and the advantages of CAD would be lost. With this format, while acknowledging that the detection accuracy is low in unclear areas, the detection results for abnormal shadows across the entire first image G1 can be used for diagnosis.

[0052] Furthermore, although the first embodiment described above describes a configuration in which the detection unit 14 detects abnormal shadows, the information processing device 10 according to this embodiment does not necessarily have the function of the detection unit 14 (i.e., CAD function), and the abnormal shadows may be confirmed by the user visually. With such a configuration, by making the user aware that there is an unclear area in the first image G1, it is possible to suppress the user overlooking abnormal shadows, and thus even an unclear first image G1 can be used for diagnosis by the user.

[0053] Furthermore, in the first embodiment described above, if the determination unit 13 determines that a non-conforming region N1 exists in the first region of interest A1, the extraction unit 12 may re-extract the first region of interest A1 with reduced extraction accuracy. "Re-extracting with reduced extraction accuracy" means allowing other regions (for example, the background portion of Figure 5) to be extracted as the first region of interest A1, and changing the conditions for extracting the first region of interest A1 so that the number of non-conforming regions N1 decreases. These conditions are determined, for example, by the brightness value of the pixels.

[0054] In this case, it is preferable for the display unit 15 to indicate that the accuracy of extracting the first region of interest A1 is low. This is especially preferable when the ratio of the unsuitable region N1 to the first region of interest A1 is above a predetermined threshold (for example, 20% or more). This is because if a large portion of the first image G1 is determined to be the unsuitable region N1, it becomes difficult to complete the unsuitable region N1 (details will be described later). With this configuration, by making the user aware that the accuracy of extracting the first region of interest A1 is low, it is possible to suppress the user overlooking abnormal shadows, and thus even an unclear first image G1 can be used for diagnosis by the user.

[0055] [Second Embodiment] In the first embodiment described above, the presence or absence of a non-conforming region N1 in the first image G1 was determined. The information processing device 10 according to this embodiment has a function to supplement the non-conforming region N1 using a medical image other than the first image G1 when it is determined that a non-conforming region N1 exists in the first image G1. An example of the configuration of the information processing device 10 according to this embodiment will be described below, but redundant explanations of the same configuration and operation as in the first embodiment will be omitted.

[0056] Referring to Figure 8, an example of the functional configuration of the information processing device 10 according to this embodiment will be described. As shown in Figure 8, the information processing device 10 according to this embodiment includes a synthesis unit 16 in addition to the acquisition unit 11, extraction unit 12, determination unit 13, detection unit 14, and presentation unit 15 similar to those in the first embodiment. The CPU 21 executes the information processing program 27, thereby enabling the acquisition unit 11, extraction unit 12, determination unit 13, detection unit 14, presentation unit 15, and synthesis unit 16 to function.

[0057] Figure 9 shows an example of a screen D2 presented on the display 24 by the presentation unit 15. As shown in Figure 9, when the determination unit 13 determines that a non-conforming region N1 exists in the first region of interest A1 of the first image G1, the presentation unit 15 requests a second image G2 that includes a region corresponding to at least a part of the non-conforming region N1.

[0058] The acquisition unit 11 acquires the second image G2 from the image acquisition device 2. Figure 10 shows the second image G2. The second image G2 is a fundus image similar to the medical image G0, and the region N12 (shown by a dashed line) corresponding to the unsuitable region N1 of the first image G1 is clearly visible. On the other hand, part of the imaging range is unclear, and there is an unsuitable region N2 that is unsuitable for extracting the original second region of interest A2 (i.e., the entire imaging range). The unsuitable region N2 originally contains an abnormal shadow S2, but its detection is difficult due to its unclear appearance.

[0059] The synthesis unit 16 generates a third image G3 by combining the first image G1 and the second image G2 acquired by the acquisition unit 11. Specifically, the synthesis unit 16 generates the third image G3 by combining a part of one image with the other image such that the corresponding area in the other image complements any incompatible areas in either the first image G1 or the second image G2. In the third image G3 shown in Figure 11, a part of the second image G2 is combined with the first image G1 such that the area N1 in the first image G1 is complemented by the area N12 in the second image G2 that corresponds to the incompatible area N1.

[0060] The extraction unit 12 extracts the third region of interest A3 from the third image G3 synthesized by the synthesis unit 16, using the same method as when it extracted the first region of interest A1 from the first image G1.

[0061] The determination unit 13 determines whether or not there is a non-conforming region in the third region of interest A3 extracted by the extraction unit 12. For example, if a part of region N12 in the second image G2 that corresponds to the non-conforming region N1 in the first image G1 is unclear, the second image G2 alone cannot completely complete the first image G1. Therefore, the CPU 21 repeatedly requests and acquires a new image that includes at least a part of the region corresponding to the non-conforming region, and recombines the new image with the third image G3, until the determination unit 13 determines that there is no non-conforming region in the third region of interest A3. The requesting, acquisition, and combination of the new image are performed in the same manner as the requesting, acquisition, and combination of the second image G2 described above.

[0062] If the determination unit 13 determines that there are no incompatible regions in the third region of interest A3, the detection unit 14 detects the abnormal shadows in the third region of interest A3 extracted by the extraction unit 12, using the same method as when detecting the abnormal shadows in the first region of interest A1. In the example of the third image G3 in Figure 11, the detection unit 14 can detect both abnormal shadows S1 and S2.

[0063] Figure 12 shows an example of a screen D3 presented on the display 24 by the presentation unit 15. As shown in Figure 12, the presentation unit 15 presents the abnormal shadows S1 and S2 detected by the detection unit 14 by adding annotations M to the third image G3.

[0064] Next, the operation of the information processing device 10 according to this embodiment will be explained with reference to Figure 13. The CPU 21 executes the information processing program 27, thereby executing the image synthesis process shown in Figure 13. The image synthesis process shown in Figure 13 is executed after step S14 in the flowchart of Figure 7. That is, it is executed when the determination process of the first embodiment determines that a non-conforming region N1 exists in the first image G1.

[0065] In step S31, the presentation unit 15 requests a second image G2 that includes a region corresponding to at least a portion of the non-conforming region N1 identified in step S14. In step S32, the acquisition unit 11 acquires the second image G2 from the image acquisition device 2. In step S33, the synthesis unit 16 synthesizes the first image G1 acquired in step S10 and the second image G2 acquired in step S32 to generate a third image G3.

[0066] In step S34, the extraction unit 12 extracts the third region of interest A3 from the third image G3 synthesized in step S33. In step S35, the determination unit 13 determines whether or not there are any incompatible regions in the third region of interest A3 extracted in step S34 that are unsuitable for extraction.

[0067] If a non-conforming region exists (i.e., step S35 is a positive determination), the process proceeds to step S36, where the presentation unit 15 requests a new image that includes a region corresponding to at least a part of the non-conforming region. In step S37, the acquisition unit 11 acquires a new image from the image acquisition device 2. In step S38, the synthesis unit 16 re-synthesizes the new image acquired in step S37 with the third image G3 synthesized in step S33. Once step S38 is complete, the process returns to step S34. That is, steps S34 to S38 are repeated until it is determined in step S35 that there is no non-conforming region in the third region of interest A3.

[0068] On the other hand, if no incompatible regions exist (i.e., step S35 results in a negative determination), the process proceeds to step S39, where the detection unit 14 detects abnormal shadows included in the third region of interest A3 extracted in step S34. In step S40, the presentation unit 15 presents the abnormal shadows detected in step S39, and the image synthesis process ends.

[0069] As described above, the information processing device 10 according to the second embodiment includes at least one processor, and when the processor determines that a non-conforming region N1 exists in the first region of interest A1 of the first image G1, it requests a second image that includes a region corresponding to at least a part of the non-conforming region N1, and synthesizes the first image G1 and the second image G2. In other words, when the information processing device 10 determines that a non-conforming region N1 exists in the first image G1, it uses a second image G2, which is different from the first image G1, to supplement the non-conforming region N1. Therefore, even if the first image G1 and the second image G2 are unclear, they can be used for diagnosis.

[0070] In the second embodiment described above, the synthesis unit 16 complements the incompatible region in either the first image G1 or the second image G2 with the other image, but it is not limited to this. For example, the synthesis unit 16 may generate the third image G3 by selecting and combining the image with better image quality for each of the multiple sections within the first image G1 and the second image G2. A "section" means, for example, a pixel and a block composed of multiple pixels. The quality of the image can be evaluated based on pixel values ​​such as hue, saturation, brightness, and lightness shown by each pixel. Furthermore, by combining these forms, the synthesis unit 16 may complement the incompatible region with the other image, while selecting and combining the image with better image quality for each section in other regions.

[0071] Furthermore, in the second embodiment described above, if the synthesis unit 16 repeats resynthesis based on the new image more than a predetermined number of times (for example, 3 times), the extraction unit 12 may re-extract the third region of interest A3 with reduced extraction accuracy. This is to terminate the process if the improper region cannot be completed even after repeated resynthesis. In this case, it is preferable for the presentation unit 15 to indicate that the extraction accuracy of the third region of interest A3 is low.

[0072] Furthermore, although the second embodiment described above describes a configuration in which the acquisition and synthesis of new images are repeated until there are no more incompatible regions in the third image G3, the invention is not limited to this. The synthesis process may be stopped even if incompatible regions exist after the synthesis by the synthesis unit 16 has been performed at least once. For example, a limit on the number of synthesis attempts (e.g., 3 times) may be set, and if the number of synthesis attempts exceeds the limit, the process may be stopped even if incompatible regions exist. Also, for example, if the ratio of incompatible regions to the third region of interest A3 of the third image G3 falls below a predetermined threshold (e.g., 5% or less), the process may be stopped even if incompatible regions exist. In these cases, the presentation unit 15 may present the ratio of incompatible regions to the third region of interest A3.

[0073] Furthermore, if processing is terminated while non-conforming areas exist, structures such as abnormal shadows included in the third region of interest A3 may be detected with reduced detection accuracy. In this case, it is preferable for the presentation unit 15 to indicate that the detection accuracy of abnormal shadows is low in the non-conforming areas. This is especially preferable when the ratio of non-conforming areas to the third region of interest A3 is above a predetermined threshold (for example, 20% or more). This is because if abnormal shadows are not detected in non-conforming areas when the ratio of non-conforming areas is high, the user would have to visually check for abnormal shadows in many parts of the third image G3, thus negating the advantages of CAD. With this configuration, the detection results of abnormal shadows for the entire third image G3 can be used for diagnosis, after recognizing that the detection accuracy is low in unclear areas.

[0074] Furthermore, although the second embodiment described above describes a configuration in which the detection unit 14 detects abnormal shadows, the information processing device 10 according to this embodiment does not necessarily have the function of the detection unit 14 (i.e., CAD function), and the abnormal shadows may be confirmed by the user visually. Even in such a configuration, the first image G1 and the second image G2, which are both unclear, can be used for diagnosis by the user.

[0075] Furthermore, in the second embodiment described above, an image synthesis process was described in which the image synthesis process is performed when it is determined in the determination process of the first embodiment that a non-conforming region N1 exists in the first image G1, but the embodiment is not limited to this. For example, the image synthesis process according to this embodiment may be performed when the ratio of the non-conforming region N1 to the first region of interest A1 in the first image G1 is above a predetermined threshold (for example, 20% or more).

[0076] [Third Embodiment] In the second embodiment described above, abnormal shadows were detected based on a third image G3, which was obtained by combining the first image G1 and the second image G2. The information processing device 10 according to this embodiment has the function of detecting abnormal shadows based on the first image G1 and the second image G2 respectively, and combining the detection results. An example of the configuration of the information processing device 10 according to this embodiment will be described below, but redundant explanations of the same configuration and operation as in the first and second embodiments will be omitted.

[0077] Similar to the second embodiment, an example of the functional configuration of the information processing device 10 according to this embodiment will be described with reference to Figure 8. As shown in Figure 8, the information processing device 10 according to this embodiment includes a synthesis unit 16 in addition to the acquisition unit 11, extraction unit 12, determination unit 13, detection unit 14, and presentation unit 15 similar to those in the first embodiment. The CPU 21 executes the information processing program 27, thereby enabling the acquisition unit 11, extraction unit 12, determination unit 13, detection unit 14, presentation unit 15, and synthesis unit 16 to function.

[0078] Figure 9 shows an example of a screen D2 presented on the display 24 by the presentation unit 15. As shown in Figure 9, when the determination unit 13 determines that a non-conforming region N1 exists in the first region of interest A1 of the first image G1, the presentation unit 15 requests a second image G2 that includes a region corresponding to at least a part of the non-conforming region N1. The acquisition unit 11 acquires the second image G2 (see Figure 10) from the image acquisition device 2. The description of the second image G2 is the same as in the second embodiment and will be omitted.

[0079] The extraction unit 12 extracts the second region of interest A2 from the second image G2 acquired by the acquisition unit 11 using the same method as when it extracted the first region of interest A1 from the first image G1. The detection unit 14 detects abnormal shadows contained in the first region of interest A1 and the second region of interest A2, respectively, extracted by the extraction unit 12, using the same method as when it detected abnormal shadows contained in the first region of interest A1. In the examples of Figures 5 and 10, the detection unit 14 detects abnormal shadow S2 from the first image G1 and abnormal shadow S1 from the second image G2.

[0080] The synthesis unit 16 synthesizes the detection results of abnormal shadows detected by the detection unit 14 from the first region of interest A1 and the second region of interest A2, respectively. In other words, the synthesis unit 16 combines the detection results of abnormal shadows obtained from multiple different images into the detection result of abnormal shadows for the same subject.

[0081] The determination unit 13 determines whether there is a common incompatible region that is incompatible with the extraction of both the first region of interest A1 and the second region of interest A2 extracted by the extraction unit 12. If a common incompatible region exists, it may lead to the oversight of abnormal shadows. Therefore, the CPU 21 repeatedly requests and acquires a new image containing at least a portion of the common incompatible region, extracts a region of interest from the new image, detects abnormal shadows contained in the region of interest, and recombines the detection result of the abnormal shadows and the detection results of abnormal shadows detected up to that point, until the determination unit 13 determines that no common incompatible region exists. The requesting and acquisition of a new image, the extraction of a region of interest, and the detection of abnormal shadows are performed in the same manner as the requesting, acquisition, extraction of a region of interest, and detection of abnormal shadows of the second image G2 described above.

[0082] Figure 14 shows an example of a screen D4 presented on the display 24 by the presentation unit 15. As shown in Figure 14, the presentation unit 15 presents the detection results of abnormal shadows synthesized by the synthesis unit 16 by adding annotations M to each image. Alternatively, similar to the second embodiment, the synthesis unit 16 may generate an image by combining the first image G1 and the second image G2, and the presentation unit 15 may present the detection results of abnormal shadows synthesized by the synthesis unit 16 by adding annotations M to this image (i.e., a single image).

[0083] Next, the operation of the information processing device 10 according to this embodiment will be explained with reference to Figure 15. The CPU 21 executes the information processing program 27, thereby executing the detection result synthesis process shown in Figure 15. The detection result synthesis process shown in Figure 15 is executed after step S14 in the flowchart of Figure 7. That is, it is executed when it is determined in the determination process of the first embodiment that a non-conforming region N1 exists in the first image G1.

[0084] In step S51, the presentation unit 15 requests a second image G2 that includes a region corresponding to at least a portion of the non-conforming region N1 identified in step S14. In step S52, the acquisition unit 11 acquires the second image G2 from the image acquisition device 2. In step S53, the extraction unit 12 extracts the second region of interest A2 from the second image G2 acquired in step S52. In step S54, the detection unit 14 detects abnormal shadows included in the second region of interest A2 extracted in step S53.

[0085] In step S55, the synthesis unit 16 synthesizes the detection result of abnormal shadows contained in the first region of interest A1 detected in step S12 and the detection result of abnormal shadows contained in the second region of interest A2 detected in step S54. In step S56, the determination unit 13 determines whether or not there is a common incompatible region that is incompatible with the extraction of both the first region of interest A1 extracted in step S11 and the second region of interest A2 extracted in step S53.

[0086] If a common non-conforming region exists (i.e., step S56 is a positive determination), the processing in steps S51 to S56 is performed on the new image that includes a region corresponding to at least a part of the common non-conforming region. In other words, the processing in steps S51 to S56 is repeated until it is determined in step S56 that no common non-conforming region exists.

[0087] On the other hand, if no common non-conforming areas exist (i.e., step S56 results in a negative determination), the process proceeds to step S57, where the presentation unit 15 presents the detection results of the abnormal shadows synthesized in step S55. Once step S57 is completed, the detection result synthesis process ends.

[0088] As described above, the information processing device 10 according to the third embodiment includes at least one processor, which, when it determines that a non-conforming region N1 exists in the first region of interest A1 of the first image G1, requests a second image that includes a region corresponding to at least a part of the non-conforming region N1. It also extracts the second region of interest A2 from the second image G2, detects abnormal shadows contained in the first region of interest A1 and the second region of interest A2, and synthesizes the detection results of the abnormal shadows. In other words, when it determines that a non-conforming region N1 exists in the first image G1, the information processing device 10 uses a second image G2, which is different from the first image G1, to supplement the detection results of the abnormal shadows. Therefore, even if the first image G1 and the second image G2 are unclear, they can be used for diagnosis.

[0089] In the third embodiment described above, if the synthesis unit 16 repeats the resynthesis of the detection results of abnormal shadows in the new image more than a predetermined number of times (for example, 3 times), the extraction unit 12 may extract the region of interest in the new image with reduced extraction accuracy. This is to terminate the process if the common incompatible region does not disappear even after repeated resynthesis. In this case, it is preferable for the presentation unit 15 to indicate that the extraction accuracy of the region of interest is low.

[0090] Furthermore, although the third embodiment described above describes a configuration in which the synthesis of abnormal shadow detection results based on new images is repeated until no common non-conforming regions remain, the invention is not limited to this. The synthesis process may be stopped even if common non-conforming regions remain after the synthesis of detection results by the synthesis unit 16 has been performed at least once. For example, a limit on the number of synthesis cycles (e.g., 3 times) may be set, and if the number of synthesis cycles exceeds the limit, the process may be stopped even if common non-conforming regions remain. Also, for example, if the ratio of common non-conforming regions to the first region of interest A1 or the second region of interest A2 falls below a predetermined threshold (e.g., 5% or less), the process may be stopped even if common non-conforming regions remain.

[0091] Furthermore, if processing is terminated while a common non-conforming area exists, structures such as abnormal shadows included in the common non-conforming area may be detected with reduced detection accuracy. In this case, it is preferable for the presentation unit 15 to indicate that the detection accuracy of abnormal shadows is low for the common non-conforming area. This is especially preferable when the ratio of the common non-conforming area to the first region of interest A1 or the second region of interest A2 is above a predetermined threshold (for example, 20% or more). This is because if abnormal shadows are not detected in the common non-conforming area when the ratio of the common non-conforming area is high, the user would have to visually check for abnormal shadows in many parts of the first image G1 and the second image G2, thus losing the advantages of CAD. With this configuration, the detection results of abnormal shadows for the entire first image G1 and the second image G2 can be used for diagnosis, after recognizing that the detection accuracy is low for unclear areas.

[0092] Furthermore, in the third embodiment described above, a configuration was described in which the detection result synthesis process is performed when it is determined in the determination process of the first embodiment that a non-conforming region N1 exists in the first image G1, but the embodiment is not limited to this. For example, the detection result synthesis process according to this embodiment may be performed when the ratio of the non-conforming region N1 to the first region of interest A1 in the first image G1 is above a predetermined threshold (for example, 20% or more).

[0093] [Fourth Embodiment] In the first to third embodiments described above, the method by which the extraction unit 12 extracts regions of interest from medical images has been explained. As mentioned above, the extraction unit 12 can use a trained model that has been trained to take a medical image as input and extract and output regions of interest. In this case, the trained model is required to be able to extract regions of interest with high accuracy even when blurry medical images such as those shown in Figures 5 and 10 are input.

[0094] One method to improve the accuracy of this trained model is to use various patterns of blurry medical images as training data. However, it was difficult to acquire a sufficient number and patterns of blurry medical images using the image acquisition device 2. Therefore, in this embodiment, the objective is to improve the accuracy of the trained model by intentionally generating blurry medical images and using them as training data.

[0095] As an example, a configuration in which the information processing device 10 according to this embodiment trains the learning model 4 used in the extraction unit 12 by unsupervised learning will be described. The learning model 4 is composed of deep learning models such as CNN (Convolutional Neural Network), FCN (Fully Convolutional Network), and U-Net, and is a model that takes a medical image as input and is trained to extract and output a region of interest. Alternatively, for example, the technologies described in Patent Document 1, Japanese Patent Application Publication No. 2019-088458, and Japanese Patent Application Publication No. 2020-114302 may be applied as such a model.

[0096] The CPU 21 acquires the source image obtained by photographing the subject from the image acquisition device 2. That is, the source image is an image obtained by at least one of the image acquisition devices 2, which are a radiographic imaging device, a magnetic resonance imaging device, an ultrasound device, a fundus imaging device, and an endoscope. The source image also includes a region of interest that includes the subject, some of the tissue contained in the subject, and at least one of structures such as abnormal parts contained in the subject or tissue. Below, an example of a source image using the clear medical image G0 shown in Figure 4 will be described. As medical image G0 has been described above, its explanation will be omitted.

[0097] The CPU 21 generates a pseudo-image by modifying the pixel values ​​of at least some of the medical image G0. Pixel values ​​represent at least one of the hue, saturation, brightness, and lightness of each pixel in the medical image G0. For example, the pixel values ​​of each pixel are changed by altering the brightness and contrast of the medical image G0, or by adding blur and noise. However, the CPU 21 does not change the resolution when generating the pseudo-image.

[0098] Refer to Figure 16 to explain specific examples of pseudo-images. The multiple pseudo-images P1 to P5 shown on the left side of Figure 16 are pseudo-images generated based on medical image G0, and are input into the training model 4 as training data. Pseudo-images P1 and P2 are images of medical image G0 with the brightness increased and decreased, respectively. Pseudo-image P3 is an image of medical image G0 with the contrast reduced. Pseudo-images P4 and P5 are images of medical image G0 with parts of the area darkened, respectively.

[0099] Furthermore, on the right side of Figure 16, the regions of interest extracted by the learning model 4 during training are shown enclosed in thick lines for each of the input pseudo-images. For pseudo-images P1 and P2, the regions of interest have been properly extracted. For pseudo-image P3, an error indicating extraction was not possible has been output. For pseudo-images P4 and P5, some blurry areas that should have been extracted as regions of interest have not been extracted.

[0100] As shown in Figure 16, the pseudo-image may be one in which the pixel values ​​of the entire medical image G0 have been changed, or in which the pixel values ​​of a part of the medical image G0 have been changed. Furthermore, as shown in pseudo-images P4 and P5, it is preferable to use multiple pseudo-images generated by changing the pixel values ​​in different regions of a single medical image G0 as training data. With this configuration, the number of original images can be reduced, so training can be performed efficiently.

[0101] Furthermore, as shown in the pseudo-images P1 to P5, it is preferable that the pseudo-images are images generated by changing the pixel values ​​in at least a portion of the medical image G0 so that the image quality deteriorates. Specifically, "deteriorating the image quality" means processing that makes it difficult to detect structures contained in the medical image G0, for example, processing that weakens the contrast. This is because, considering the operational phase of the learning model, if the input medical image is a fundus image, the contrast may be weaker than that of a clear medical image due to, for example, insufficient light causing the image to become darker or ambient light causing the image to become brighter.

[0102] Furthermore, as shown in Figure 16, it is preferable that the pseudo-image is an image generated by changing the pixel values ​​of at least the region of interest in the medical image G0. This is because whether or not the region outside the region of interest is blurry is not particularly problematic.

[0103] Furthermore, it is preferable that the pseudo-image is an image generated by changing at least some of the pixel values ​​of the medical image G0 while maintaining the presence or absence of structures contained in the medical image G0. This is because, in this learning model 4, the priority is to accurately extract regions of interest even from unclear medical images, and responding to changes in the presence or absence of structures is a lower priority.

[0104] Furthermore, the pseudo-image may be an image generated based on medical image G0 using an image generation model such as GAN (Generative Adversarial Network) or VAE (Variational Autoencoder).

[0105] The CPU 21 trains the learning model 4 by inputting the pseudo-images P1 to P5 generated as described above as training data. Furthermore, as shown in the pseudo-images P3 to P5 in Figure 16, if the learning model 4 is unable to appropriately extract the region of interest from the input pseudo-images, the CPU 21 may retrain the learning model 4 by inputting the pseudo-images again as training data. This configuration can improve the accuracy of the learning model 4.

[0106] Furthermore, during retraining, correct labels may be assigned to the pseudo-images. Specifically, the CPU 21 may input pairs of pseudo-images and information indicating regions of interest contained in those pseudo-images as training data to the training model 4, and then retrain the training model 4. Information indicating regions of interest is, for example, information indicating the location of the region of interest in the pseudo-image.

[0107] Next, the operation of the information processing device 10 according to this embodiment will be explained with reference to Figure 17. The CPU 21 executes the information processing program 27, thereby executing the learning process shown in Figure 17. The learning process shown in Figure 17 is executed, for example, when the user gives an instruction to start the process via the input unit 25.

[0108] In step S71, the CPU 21 acquires the original image obtained by photographing the subject from the image acquisition device 2. In step S72, the CPU 21 modifies at least some of the pixel values ​​of the original image acquired in step S71 to generate a pseudo-image. In step S73, the CPU 21 uses the pseudo-image generated in step S72 as training data to train the learning model 4. In step S74, the CPU 21 determines whether the learning model 4 was able to appropriately extract the region of interest from the input pseudo-image.

[0109] If the learning model 4 is unable to properly extract the region of interest from the input pseudo-image (i.e., step S74 results in a negative judgment), the process returns to step S73, and the CPU 21 re-inputs the pseudo-image to the learning model 4, thereby causing the learning model 4 to retrain. In other words, the retraining process using the same pseudo-image is repeated until the learning model 4 can properly extract the region of interest from the pseudo-image. The learning process ends when the learning model 4 has successfully extracted the region of interest from the input pseudo-image (i.e., when step S74 results in a positive judgment).

[0110] As described above, the information processing device 10 according to the fourth embodiment includes at least one processor, which receives a pseudo-image generated by changing the pixel values ​​of at least some of the original images obtained by photographing a subject as training data, thereby training a learning model for extracting regions of interest from the input images. Therefore, even images unsuitable for extracting regions of interest can have regions of interest appropriately extracted and utilized for diagnosis.

[0111] In the fourth embodiment described above, the information processing device 10 trains the learning model 4 used in the extraction unit 12 by unsupervised learning, but the invention is not limited to this. The information processing device 10 may also train the learning model 4 by supervised learning and semi-supervised learning. Specifically, the CPU 21 may train the learning model 4 by inputting pairs of pseudo-images and information indicating regions of interest contained in the pseudo-images as training data. In this case, if the learning model 4 is unable to extract regions of interest from the input pseudo-images, the CPU 21 may retrain the learning model 4 by inputting the pseudo-images again as training data.

[0112] Although the embodiments described above were explained using medical images, the technology of this disclosure is not limited to medical images. The technology of this disclosure may be applied, for example, to images acquired from equipment, buildings, pipes, welds, etc., in non-destructive testing such as radiographic inspection and ultrasonic testing.

[0113] Furthermore, while the above embodiments have described an information processing system 1 that includes an information processing device 10 and an image acquisition device 2, the system is not limited to this. For example, the information processing system 1 may include a single device that has both the functions of the information processing device 10 and the functions of the image acquisition device 2. Alternatively, for example, the information processing system 1 may include multiple image acquisition devices 2, and the information processing device 10 may acquire medical images from each of the multiple image acquisition devices 2. Alternatively, for example, the information processing device 10 may consist of multiple devices, each with a different function, such as an acquisition unit 11, an extraction unit 12, a determination unit 13, a detection unit 14, a presentation unit 15, and a synthesis unit 16.

[0114] Furthermore, in each of the above embodiments, the hardware structure of the processing unit that performs various processes, such as the acquisition unit 11, extraction unit 12, determination unit 13, detection unit 14, presentation unit 15, and synthesis unit 16, can be the various processors shown below. As mentioned above, these various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as programmable logic devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacturing, and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations specifically designed to perform specific processes.

[0115] A single processing unit may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor.

[0116] Examples of configuring multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software combine to form a single processor, which then functions as multiple processing units, as exemplified by client and server computers. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as exemplified by System on Chip (SoC). Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned processors.

[0117] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits, which are combinations of circuit elements such as semiconductor devices.

[0118] Furthermore, although the above embodiments describe a configuration in which the information processing program 27 is pre-stored (installed) in the storage unit 22, the invention is not limited thereto. The information processing program 27 may be provided in the form of a recording medium such as a CD-ROM (Compact Disc Read Only Memory), DVD-ROM (Digital Versatile Disc Read Only Memory), or USB (Universal Serial Bus) memory. Alternatively, the information processing program 27 may be downloaded from an external device via a network. Moreover, the technology of this disclosure extends not only to the information processing program but also to storage media for non-temporarily storing the information processing program.

[0119] The technology of this disclosure can also be appropriately combined from the above-described embodiments. The descriptions and illustrations shown above are detailed explanations of the parts relating to the technology of this disclosure and are merely examples of the technology of this disclosure. For example, the above-described explanation of the configuration, function, operation, and effect is an explanation of an example of the configuration, function, operation, and effect of the parts relating to the technology of this disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements added, or replaced from the descriptions and illustrations shown above, as long as they do not deviate from the spirit of the technology of this disclosure. [Explanation of symbols]

[0120] 1. Information Processing System 2 Image acquisition device 4. Learning Models 10 Information Processing Devices 11 Acquisition Department 12 Extraction part 13 Judgment section 14 Detection unit 15 Presentation section 16. Synthesis section 21 CPU 22 Memory section 23 memory 24 displays 25 Input section 26 Network Interface 27 Information Processing Programs 28 buses A0 Area of ​​Interest A1 First area of ​​interest A2 Second area of ​​interest A3 Third area of ​​interest D1~D4 screen G0 Medical Images G1 Image 1 G2 Image 2 G3 Image 3 M annotation N1, N2 Nonconformity area N12 area P1~P5 Pseudo image S1, S2 abnormal shadow

Claims

1. Equipped with at least one processor, The aforementioned processor, Multiple images obtained by photographing the subject are acquired, Extract the region of interest from each of the aforementioned multiple images, The structures contained in each of the extracted regions of interest are detected, The detection results of the structure detected from each of the multiple regions of interest are combined, For each of the extracted regions of interest, a determination is made as to whether or not there are any incompatible regions that do not fit the extraction. If it is determined that the aforementioned non-conforming area exists, For the multiple regions of interest that have been extracted, it is determined whether or not there is a common incompatible region that is incompatible with the extraction. Until it is determined that the aforementioned common non-conforming area does not exist, Acquisition of a new image including an area corresponding to at least a portion of the common non-conforming area, Extraction of regions of interest from the new image, Detection of structures included in the region of interest, The detection results for the structure are repeatedly re-synthesized. Information processing device.

2. The aforementioned processor, The iteration is terminated if it is determined that no common non-conforming region exists, or if the number of resynthesis cycles exceeds a predetermined limit. The information processing apparatus according to claim 1.

3. The aforementioned processor, The iteration terminates if it is determined that no common non-conforming region exists, or if the ratio of the common non-conforming region to any of the extracted regions of interest falls below a predetermined threshold. The information processing apparatus according to claim 2.

4. The aforementioned processor, If it is determined that the common non-conforming area exists at the end of the aforementioned repetition, the structure included in the common non-conforming area is detected with reduced detection accuracy. The information processing apparatus according to claim 2 or claim 3.

5. The aforementioned processor, If it is determined that the aforementioned non-conforming area exists, the structure included in the aforementioned non-conforming area is detected with reduced detection accuracy. An information processing apparatus according to any one of claims 1 to 4.

6. The aforementioned processor, When the aforementioned structure is detected with reduced detection accuracy, the system indicates that the detection accuracy of the aforementioned structure is low. The information processing apparatus according to claim 4 or claim 5.

7. The aforementioned processor, The control is performed to present the non-suitable region in at least one of the plurality of images in a manner that can be distinguished from other regions. An information processing apparatus according to any one of claims 1 to 6.

8. The aforementioned processor, The determination is made according to the degree of similarity between the shape of the extracted region of interest and a predetermined reference shape relating to the region of interest. An information processing apparatus according to any one of claims 1 to 7.

9. The aforementioned processor, The determination is made using a trained model that takes an image obtained by photographing a subject as input and determines whether or not the incompatible region exists in the region of interest extracted from the image. The aforementioned trained model is a trained model that has been trained using pairs of data: an image obtained by photographing a subject, and information indicating whether or not the unsuitable region exists in the region of interest extracted from the image. An information processing apparatus according to any one of claims 1 to 8.

10. The aforementioned processor, The system controls the display of a string indicating whether or not the aforementioned non-conforming region exists within the screen where at least one of the multiple images is displayed. An information processing apparatus according to any one of claims 1 to 9.

11. The aforementioned processor, If it is determined that the aforementioned non-conforming region exists, the control system will display the ratio of the non-conforming region to any of the extracted multiple regions of interest. An information processing apparatus according to any one of claims 1 to 10.

12. The aforementioned processor, If it is determined that the aforementioned non-suitable region exists, the region of interest is re-extracted with reduced extraction accuracy. An information processing apparatus according to any one of claims 1 to 11.

13. The aforementioned processor, For each of the extracted regions of interest, the structure is detected from regions other than the non-conforming region, and no detection of the structure is performed in the non-conforming region. An information processing apparatus according to any one of claims 1 to 12.

14. The aforementioned processor, The control is performed to display the structure detected from the other region on the original image. The information processing apparatus according to claim 13.

15. The region of interest is a region that includes the subject, a portion of the tissue contained in the subject, and at least one of the abnormal parts contained in the subject or the tissue. An information processing apparatus according to any one of claims 1 to 14.

16. The aforementioned plurality of images are images obtained by at least one of the following: a radiographic imaging device, a magnetic resonance imaging device, an ultrasound device, a fundus imaging device, and an endoscope. An information processing apparatus according to any one of claims 1 to 15.

17. Multiple images obtained by photographing the subject are acquired, Extract the region of interest from each of the aforementioned multiple images, The structures contained in each of the extracted regions of interest are detected, The detection results of the structure detected from each of the multiple regions of interest are combined, For each of the extracted regions of interest, a determination is made as to whether or not there are any incompatible regions that do not fit the extraction. If it is determined that the aforementioned non-conforming area exists, For the multiple regions of interest that have been extracted, it is determined whether or not there is a common incompatible region that is incompatible with the extraction. Until it is determined that the aforementioned common non-conforming area does not exist, Acquisition of a new image including an area corresponding to at least a portion of the common non-conforming area, Extraction of regions of interest from the new image, Detection of structures included in the region of interest, The detection results for the structure are repeatedly re-synthesized. An information processing method in which a computer performs the processing.

18. Multiple images obtained by photographing the subject are acquired, Extract the region of interest from each of the aforementioned multiple images, The structures contained in each of the extracted regions of interest are detected, The detection results of the structure detected from each of the multiple regions of interest are combined, For each of the extracted regions of interest, a determination is made as to whether or not there are any incompatible regions that do not fit the extraction. If it is determined that the aforementioned non-conforming area exists, For the multiple regions of interest that have been extracted, it is determined whether or not there is a common incompatible region that is incompatible with the extraction. Until it is determined that the aforementioned common non-conforming area does not exist, Acquisition of a new image including an area corresponding to at least a portion of the common non-conforming area, Extraction of regions of interest from the new image, Detection of structures included in the region of interest, The detection results for the structure are repeatedly re-synthesized. An information processing program that causes a computer to perform a task.

Images