Determination system, method for controlling determination system, and control program

Abstract

To provide a determination system that presents a determination result related to the present or future state of a target part of the human body, and a control program.SOLUTION: A determination system comprises: a target image acquisition unit that acquires a target image capturing a target part of the body of a subject; a first generation unit 211 that generates, from the target image, first determination information related to a disease at the target part; a second generation unit 311 that generates, from the target image, characteristic information indicating characteristics related to the state of the body of the subject; and a third generation unit 212 that generates focused area information indicating the position of a focused area focused in the process during which the first determination information is generated. The characteristic information includes identification information related to characteristics detected from the target image and position information indicating the positions of the characteristics in the target image. The second generation unit has, as a characteristic generation model for generating characteristic information, a second neural network that has trained using patient images capturing target parts of a plurality of patients having diseases at the target parts as teacher data.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a determination system that presents a determination result regarding the current or future state of a target part of a human body, a control method for the determination system, and a control program. [Background technology]

[0002] As described in Patent Document 1, a technology has been devised that allows artificial intelligence (AI) to determine current and future events. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2008-36068 [Non-patent literature]

[0004] [Non-Patent Document 1] Selvaraju et al., “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization”, arXiv:1610.02391v4[cs.CV], 2019. Summary of the Invention

[0005] A determination system according to one embodiment of the present disclosure comprises a target image acquisition unit that acquires a target image showing a target part of a subject's body; a first generation unit that generates first determination information related to a disease of the target part from the target image; a second generation unit that generates feature information indicating features related to the subject's physical condition from the target image; and a third generation unit that generates area-of-interest information indicating the position of an area of ​​interest that was focused on during the generation of the first determination information, wherein the feature information includes identification information for each feature detected from the target image and position information indicating the position of each feature in the target image, and the second generation unit has a second neural network that has been trained using patient images of a plurality of patients having a disease in the target part as training data, the patient image being associated with identification information for each feature detected from the patient image and position information indicating the position of each feature in the patient image.

[0006] A control method for a judgment system according to one embodiment of the present disclosure includes a target image acquisition step of acquiring a target image showing a target part of a subject's body; a first generation step of generating first judgment information related to a disease of the target part from the target image; a second generation step of generating feature information indicating features related to the subject's physical condition from the target image; and a third generation step of generating area-of-interest information indicating the position of an area of ​​interest that was noticed during the generation of the first judgment information, wherein the feature information includes identification information for each feature detected from the target image and position information indicating the position of each feature in the target image, and in the second generation step, a second neural network trained using patient images showing the target part of each of a plurality of patients having a disease in the target part as training data is used as a feature generation model for generating the feature information, and the patient image is associated with identification information for each feature detected from the patient image and position information indicating the position of each feature in the patient image.

[0007] The determination system according to each aspect of the present disclosure may be realized by a computer. In this case, the control program of the determination system that causes the computer to operate as each part (software element) of the device included in the determination system, and the computer-readable recording medium on which the control program is recorded, also fall within the scope of the present disclosure. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram illustrating an example of a configuration of a determination system according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a functional block diagram illustrating an example of a configuration of a determination system. [Figure 3] FIG. 2 is a diagram illustrating an example of a data structure of a display information database. [Figure 4] FIG. 10 is a diagram showing an example of first determination information generated from a target image showing the lungs of a subject. [Figure 5] FIG. 10 is a diagram showing an example of first determination information generated from a target image showing a knee joint of a target person. [Figure 6] 10 is a flowchart illustrating an example of a flow of processing executed by an information processing device of the determination system. [Figure 7] FIG. 10 is a diagram illustrating an example of a display screen. [Figure 8] FIG. 10 is a diagram illustrating an example of a display screen. [Figure 9] FIG. 10 is a diagram illustrating an example of a display screen. [Figure 10] FIG. 10 is a diagram illustrating an example of a display screen. [Figure 11] FIG. 10 is a diagram illustrating an example of a display screen. [Figure 12] FIG. 10 is a diagram illustrating an example of a display screen. [Figure 13] FIG. 10 is a diagram illustrating an example of a display screen. [Figure 14] FIG. 10 is a diagram illustrating an example of a display screen. [Figure 15] 10 is a flowchart showing another example of the flow of processing executed by the determination system. [Figure 16] FIG. 1 is a functional block diagram illustrating an example of a configuration of a determination system. [Figure 17] FIG. 2 is a diagram illustrating an example of a data structure of a display information database. [Figure 18] FIG. 1 is a functional block diagram illustrating an example of a configuration of a determination system. [Figure 19] FIG. 1 is a functional block diagram illustrating an example of a configuration of a determination system. [Figure 20] FIG. 2 is a diagram illustrating an example of a data structure of a display information database. [Figure 21] FIG. 10 is a diagram illustrating an example of a display screen. [Figure 22] FIG. 10 is a diagram illustrating an example of a display screen. [Figure 23] FIG. 1 is a functional block diagram illustrating an example of a configuration of a determination system. [Figure 24] 1 is a block diagram illustrating an example of a configuration of a determination system according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0009] [Embodiment 1] An embodiment of the present disclosure will be described in detail below.

[0010] (Outline of Determination System 100) A determination system 100 according to an embodiment of the present disclosure acquires a target image showing a target part of a body of a subject, and generates first determination information indicating the state of the target part from the target image. The determination system 100 also generates feature information indicating features related to the body state of the subject from the target image used to generate the first determination information.

[0011] Furthermore, the determination system 100 displays the first determination information and the feature information on the display device 5.

[0012] The determination system 100 can present to a user first determination information and feature information generated from the same target image that shows a target part of the subject's body. The feature information is useful information for the user to determine the reliability of the first determination information. By referring to the feature information, the user can understand why the first determination information was generated from the target image and determine the validity and reliability of the first determination information.

[0013] In this specification, the target area may be any part of the subject's body, such as the whole body, head, neck, arms, torso, waist, buttocks, legs, and feet.

[0014] The target image may also be a medical image showing a target region of a subject. In one example, the target image may be a medical image of a target region of a subject captured for examining the subject. Here, the medical image may be any of an X-ray image, a CT (Computed Tomography) image, an MRI (Magnetic Resonance Imaging) image, a PET (Positron Emission Tomography) image, an RI (Radio Isotope) image, a mammography image, an ultrasound image, an endoscopic image, and an angiography image of the subject. The target image may also be an image of any of bones, joints, muscles, fat (subcutaneous fat and visceral fat), organs, blood vessels, etc. of the target region.

[0015] The first determination information may be information that determines the state of the target part at a first time point when the target image was captured, or may include information that determines the state of the target part at a second time point when a predetermined period has elapsed since the first time point when the target image was captured (see FIG. 3).

[0016] Here, the first time point may be, for example, the time point when a medical image of the target region of the subject is acquired. The first time point may typically be the time point when a subject image of the current state of the target region of the subject is acquired. That is, the first time point may substantially refer to the present time. Furthermore, the predetermined period may be any period that has elapsed since the first time point, such as six months, one year, five years, ten years, or fifty years. That is, the second time point may substantially refer to any time point in the future. The predetermined period is not limited to one period, but may include multiple periods. That is, the first determination information may be information indicating a determination result obtained by an artificial intelligence (AI, for example, the first determination information generation device 2 described below) included in the determination system 100, or the like, determining the current or future state of the target region of the subject based on the target image.

[0017] The first determination information may be diagnostic information including a diagnosis result as to whether or not the subject has a disease. The diagnostic information may include the presence or absence of a disease, and the progression or severity of the disease. The diagnostic information may also include a numerical value used to determine whether or not the subject has a disease. It may also include the risk of developing a disease in the future, or the risk of developing symptoms such as pain in the future. If there is no disease, the diagnosis result may be normal or no abnormalities. It may also include information indicating a pre-disease state, such as being within the normal range but showing slight signs of disease. It may also include information indicating the need for detailed examination or information that a diagnosis was impossible.

[0018] The feature information may be information indicating features detected from the target image by an artificial intelligence (AI, for example, the feature information generation device 3 described below) included in the determination system 100. The feature information may include identification information including the name of each feature detected from the target image, and position information indicating the position of each feature in the target image. The feature information may be information indicating the presence of any abnormality (symptom) detected in the target image when the target image is compared with an image of a normal target region, and position information indicating the position where the abnormality was detected. The feature information may include features unrelated to the target region of the subject. Furthermore, the feature information may include features unrelated to determining the state of the target region.

[0019] The feature may be any feature on an image obtained by analyzing the target image. In one example, the feature may be at least one of a malignant tumor, a benign tumor, a thrombus, an vascular sclerosis, a ground-glass opacity, a calcified lesion, a melon skin-like finding, a patchy opacity, an interstitial change, a pulmonary cyst, an emphysema, a vascular compression, a nerve compression, an intervertebral disc degeneration, a meniscus injury, a ligament injury, a bone sclerosis, a fracture, a bone deformation, an osteophyte formation, and a narrowing of the joint space.

[0020] The determination system 100 may further generate region-of-interest information indicating a region of interest associated with the first determination information, which is a partial region of the target image. In this case, the determination system 100 displays, on the display device 5, the first determination information, and the feature information and the region-of-interest image generated from the target image used to generate the first determination information.

[0021] The area-of-interest information may be information indicating the position of an area of ​​interest that the first judgment information generation device 2 focused on in the process of generating the first judgment information. In other words, the area of ​​interest is an area that was focused on in the process of generating the first judgment information from the target image, and the area-of-interest information may be information indicating the position of the area of ​​interest in the target image.

[0022] For example, if the target image is an X-ray image of a lung (target region) and the diagnostic information is pneumonia, the region-of-interest information may be information indicating the periphery of a ground-glass opacity.Also, if the target image is an X-ray image of a knee (target region) and the diagnostic information is osteoarthritis, the region-of-interest information may be information indicating the periphery of a narrowed joint space or a hardened bone.

[0023] The determination system 100 can more effectively improve the user's understanding of the first determination information by displaying the region-of-interest information on the display device 5 in addition to the first determination information and feature information. In this specification, a case will be described as an example in which the determination system 100 can generate the first determination information, feature information, and a region-of-interest image. However, a configuration in which the region-of-interest image is also generated in addition to the first determination information and feature information is not essential.

[0024] (Schematic configuration of determination system 100) The following describes the configuration of the determination system 100, taking as an example a medical facility 6 that employs the determination system 100. FIG. 1 is a block diagram showing an example configuration of the determination system 100. In this case, the subject may be a patient who receives examination and treatment for the condition of the target part at the medical facility 6. The target image may also be a medical image of the target part of the subject. The following describes an example in which the target parts are the subject's lungs and knees, but the target parts are not limited to these.

[0025] As shown in Fig. 1, the determination system 100 may include an information processing device 1, a first determination information generation device 2, and a feature information generation device 3. The information processing device 1, the first determination information generation device 2, and the feature information generation device 3 are all computers. The first determination information generation device 2 and the feature information generation device 3 are communicably connected to the information processing device 1. The information processing device 1, the first determination information generation device 2, and the feature information generation device 3 may be connected to a LAN (local area network) of a medical facility 6, as shown in Fig. 1.

[0026] In addition to the determination system 100, the medical facility 6 may also be equipped with a target image management device 4 and one or more display devices 5. The target image management device 4 and the display devices 5 may also be connected to the LAN of the medical facility 6, as shown in FIG.

[0027] The first determination information generation device 2 is a computer capable of generating first determination information from a target image. The first determination information generation device 2 may also be capable of generating region-of-interest information from the target image. The first determination information generation device 2 will be described later.

[0028] The feature information generating device 3 is a computer capable of generating feature information from a target image. The feature information generating device 3 will be described later.

[0029] The information processing device 1 acquires a target image from a target image management device 4. The information processing device 1 transmits the target image to a first judgment information generation device 2 and instructs it to generate first judgment information and area of ​​interest information. The information processing device 1 also transmits the same target image to a feature information generation device 3 and instructs it to generate feature information. The information processing device 1 receives the first judgment information and area of ​​interest information from the first judgment information generation device 2, and receives feature information from the feature information generation device 3.

[0030] The display device 5 includes a display unit 51 (see, for example, FIG. 2) that can receive the first determination information, the region of interest information, and the feature information from the information processing device 1 and display this information. The display device 5 may be a computer used by a medical professional such as a doctor belonging to the medical facility 6.

[0031] The display device 5 may be, for example, a personal computer, a tablet terminal, a smartphone, etc., and may include a communication unit for transmitting and receiving data to and from other devices, an input unit such as a keyboard and a microphone, etc. In this case, the display device 5 can accept input of various instructions from medical personnel.

[0032] 1, the first determination information generating device 2 generates the first determination information and the region-of-interest information, but is not limited to this. For example, the first determination information generating device 2 may generate only the first determination information, and another device (not shown) different from the first determination information generating device 2 may generate the region-of-interest information. In this case, the information processing device 1 may transmit the target image to the other device and instruct it to generate the region-of-interest information.

[0033] The target image management device 4 is a computer that functions as a server for managing target images. In one example, the target image management device 4 may transmit a target image specified in an instruction from a medical professional input to the display device 5 to the information processing device 1. In this case, the instruction from the medical professional may include identification information unique to the subject (subject ID, described below), identification information unique to the target image (image ID, described below), the MAC address of the information processing device 1, etc.

[0034] The electronic medical record management device 9 is a computer that functions as a server for managing electronic medical record information of subjects who have received medical treatment at the medical facility 6. In this case, each device in the determination system 100 may acquire basic information related to the subject from the electronic medical record management device 9. Here, the basic information is information contained in the subject's electronic medical record information, and may include at least one of the subject's gender, age, height, weight, and information indicating the subject's current (first time point) health condition. The basic information may also include a subject ID, which is identification information assigned to each subject and unique to each subject.

[0035] 1 shows an example in which the information processing device 1, the first determination information generating device 2, the feature information generating device 3, the target image management device 4, and the electronic medical record management device 9 are connected to a LAN arranged in a medical facility 6, but this is not limiting. For example, the network in the medical facility 6 may be the Internet, a telephone communication line network, an optical fiber communication network, a cable communication network, a satellite communication network, or the like. The LAN in the medical facility 6 may be communicably connected to an external communication network.

[0036] In the determination system 100, the information processing device 1 and at least one of the display device 5, the first determination information generation device 2, the feature information generation device 3, the target image management device 4, and the electronic medical record management device 9 may be directly connected without going through a LAN. Also, the number of the display device 5, the first determination information generation device 2, the feature information generation device 3, the target image management device 4, and the electronic medical record management device 9 that can communicate with the information processing device 1 may be plural. Furthermore, a plurality of information processing devices 1 may be introduced in the determination system 100.

[0037] (Configuration of Determination System 100) Next, the configurations of the information processing device 1, the first determination information generating device 2, and the feature information generating device 3 of the determination system 100 will be described with reference to Fig. 2. Fig. 2 is a functional block diagram showing an example of the configuration of the determination system 100. Fig. 2 also shows a target image management device 4 and a display device 5 that are communicably connected to the determination system 100.

[0038] The information processing device 1, the first determination information generating device 2, and the feature information generating device 3 will be described below.

[0039] <Information processing device 1> The information processing device 1 includes a control unit 11 that controls all the units of the information processing device 1, a storage unit 12 that stores various data used by the control unit 11, and a communication unit 13 that performs various data communications.

[0040] The control unit 11 includes an acquisition unit 111 , an instruction generation unit 112 , a display information generation unit 113 , and a display control unit 114 .

[0041] 2, the acquiring unit 111 acquires the target images from the target image management device 4. Alternatively, the target images may be acquired from a computer (for example, the display device 5) used by a doctor. The acquiring unit 111 may acquire an image ID, which is identification information unique to each target image and assigned to each target image, together with the target images.

[0042] The acquisition unit 111 may acquire one or more target images corresponding to each of a plurality of subjects. When acquiring a target image corresponding to each of a plurality of subjects, the acquisition unit 111 may acquire a subject ID in addition to the image ID.

[0043] The instruction generation unit 112 acquires the target image and image ID and generates various instructions. The instruction generation unit 112 generates a first generation instruction and a second generation instruction to be transmitted to the first determination information generation device 2, and a third generation instruction to be transmitted to the feature information generation device 3. The first generation instruction is an instruction to cause the first determination information generation device 2 to generate first determination information from the target image, and the second generation instruction is an instruction to cause the feature information generation device 3 to generate feature information from the target image.

[0044] The first generation instruction and the second generation instruction may include the MAC address of the first determination information generation device 2 as the destination and the MAC address of the information processing device 1 as the source, etc. On the other hand, the third generation instruction may include the MAC address of the characteristic information generation device 3 as the destination and the MAC address of the information processing device 1 as the source, etc.

[0045] In one example, the instruction generator 112 may assign a unique instruction ID to each of the generated instructions. In this case, the first generation instruction, the second generation instruction, and the third generation instruction transmitted together with the target image to which the same image ID is assigned may each be assigned an instruction ID associated with each other.

[0046] The first generation instruction and the second generation instruction generated by the instruction generation unit 112 are transmitted to the first determination information generation device 2 via the communication unit 13 together with the instruction ID and the target image. On the other hand, the third generation instruction generated by the instruction generation unit 112 is transmitted to the feature information generation device 3 via the communication unit 13 together with the instruction ID and the target image.

[0047] For each target image, the display information generation unit 113 acquires (1) the first judgment information and region of interest information generated by the first judgment information generation device 2, and (2) the feature information generated by the feature information generation device 3. The display information generation unit 113 generates display information by associating the first judgment information and region of interest information with the feature information, with reference to an instruction ID or the like associated with each piece of acquired information. The display information generation unit 113 may store the generated display information together with the instruction ID in the display information database 121 of the storage unit 12.

[0048] [Display Information Database 121] The display information database 121 will now be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the data structure of the display information database 121.

[0049] The display information database 121 stores the following information in association with each other. ·Instruction ID. The first judgment information and the region-of-interest information generated by the first judgment information generating device 2 in accordance with the first generation instruction and the second generation instruction to which each instruction ID is assigned. The characteristic information generated by the characteristic information generating device 3 in response to the third generation instruction to which each instruction ID is assigned.

[0050] The display information database 121 may further store the following information in association with each other. The image ID of the target image transmitted to the first judgment information generation device 2 and the feature information generation device 3 together with the first generation instruction, the second generation instruction, and the third generation instruction to which each instruction ID is assigned. Electronic medical record information of the subjects appearing in the target images to which each image ID has been assigned (may include the subject ID).

[0051] Returning to FIG. 2, the display control unit 114 uses each piece of information associated with each instruction ID from the display information database 121 to cause the display unit 51 of the display device 5 to display it.

[0052] <First judgment information generation device 2> The first judgment information generating device 2 includes a control unit 21 that controls each part of the first judgment information generating device 2 in an integrated manner, a memory unit 22 that stores various data used by the control unit 21, and a communication unit 23 that performs various data communications.

[0053] The control unit 21 includes a first generating unit 211 that generates first determination information based on the target image, and a third generating unit 212 that generates region-of-interest information.

[0054] [First generation unit 211] The first generation unit 211 generates, from the target image, first determination information related to the target part at a first time point or at a second time point after a predetermined period has elapsed since the first time point. FIG. 4 is a diagram showing an example of first determination information generated from a target image showing the lungs of the subject. FIG. 4 shows a case where a "determination result related to the current lung condition" is generated as an example of the first determination information. FIG. 5 is a diagram showing an example of first determination information generated from a target image showing the knee joint of the subject. FIG. 5 shows a case where a "determination result related to the current knee joint condition" is generated as the first determination information (Example 1), and a case where a "determination result related to the future knee joint condition" is generated as the first determination information (Example 2).

[0055] The first generating unit 211 may have a determination information generation model that can generate first determination information indicating the state of a target part of a subject, using a target image of the subject.

[0056] The determination information generation model may be a trained first neural network trained using patient information on a plurality of patients with a disease in the target region as training data. The patient information may include medical images and diagnostic information indicating a diagnosis result for the condition of the target region of each patient. The patient information may further be associated with information indicating the time point at which the medical images were captured. This trained first neural network may be used as a determination information generation model capable of outputting first determination information from a target image of a subject.

[0057] Here, the disease may be at least one of cancer, heart disease, pneumonia, emphysema, cerebral infarction, dementia, osteoarthritis, spondylosis, fracture, and osteoporosis. The osteoarthritis may be hip osteoarthritis or knee osteoarthritis. The fracture may be a vertebral compression fracture or a proximal femoral fracture. Here, the first assessment information may be the progression or severity of the disease. If the disease is cancer or heart disease, the first assessment information may include a stage classification or the like indicating the progression or severity of the disease. Alternatively, if the disease is osteoarthritis or spondylosis, the first assessment information may include the Kellgren-Lawrence (KL) classification or the like indicating the progression or severity of the disease. Alternatively, if the disease is a proximal femoral fracture, the first assessment information may include the Garden Stage or the like indicating the progression or severity. Here, the first assessment information may be a numerical value used to determine whether or not a patient has a disease. For example, when the disease is osteoporosis, the first determination information may be a value indicating bone density.

[0058] In response to a target image being input to the input layer, the first generation unit 211 performs calculations based on the judgment information generation model and outputs first judgment information from the output layer. As an example, the first generation unit 211 may be configured to extract predetermined feature amounts from the target image and use them as input data. The following well-known algorithms may be applied to extract the feature amounts. ·Convolutional neural network (CNN) Autoencoder Recurrent neural network (RNN) ·LSTM (Long Short-Term Memory).

[0059] For example, when generating a determination information generation model that outputs first determination information regarding a lung disease, the following learning data and teacher data (correct labels) may be used. As an example, by applying CNN and performing machine learning using the backpropagation method, the determination information generation model can output accurate determination information regarding a lung disease for an X-ray image of the lungs of any subject. Training data: X-ray images of lungs from multiple patients. Training data: The presence or absence of lung disease or the name of the disease (no abnormality, lung cancer, pneumonia, etc.) shown in each patient's lung X-ray image.

[0060] For example, when generating a determination information generation model that outputs determination information regarding knee diseases, the following learning data and teacher data (correct labels) can be used. As an example, by applying CNN and performing machine learning using the backpropagation method, the determination information generation model can output accurate determination information regarding knee diseases for X-ray images of the knee of any subject. Training data: X-ray images of knees from multiple patients. Training data: The presence or absence of knee disease shown in each patient's X-ray image, or the name of the disease (no abnormality, osteoarthritis, Kellgren-Laurence classification, etc.).

[0061] [Third generation unit 212] The third generation unit 212 generates region-of-interest information indicating the position of the region of interest that was noticed in the process in which the first generation unit 211 generated the first determination information.

[0062] In the process of outputting the first determination information, the region of interest information may be information indicating the position of a partial region of the target image to which a certain layer of the first neural network reacts strongly.

[0063] The third generation unit 212 extracts a feature map for the entire target image at the stage when the target image is input to the input layer of the first neural network and processing up to the convolutional layer is executed. The third generation unit 212 also performs segmentation on the target image. Here, segmentation is a process of evaluating, for each pixel of the target image, how that pixel affects the first determination information and outputting the evaluation result.

[0064] In one example, the third generation unit 212 may identify a position that is important to the first determination information based on the magnitude of a change in output that occurs when a gradient change is applied to a certain position on the feature map. In this case, a position that has a large effect on the first determination information also has a large change in gradient, and a position that has a small effect on the first determination information also has a small change in gradient.

[0065] The third generating unit 212 outputs, as a region of interest, a region of the target image that corresponds to a position in the feature map that has a large influence on the first determination information, and generates region of interest information that indicates the region of interest.

[0066] To detect the region of interest, any known algorithm can be applied, such as: ·CAM(Class Activation Mapping) ·Grad-CAM(Gradient-based Class Activation Mapping) Attention Branch Network Attention Guided CNN Residual Attention Network The first determination information generated by the first generating unit 211 and the region-of-interest information generated by the third generating unit 212 are transmitted to the information processing device 1 via the communication unit 23 together with the instruction ID.

[0067] <Feature information generation device 3> The feature information generating device 3 includes a control unit 31 that controls each unit of the feature information generating device 3 in an integrated manner, a memory unit 32 that stores various data used by the control unit 31, and a communication unit 33 that performs various data communications.

[0068] The control unit 31 includes a second generation unit 311 that generates feature information based on the target image.

[0069] [Second generation unit 311] The second generating unit 311 generates feature information from the target image. The second generating unit 311 may have a feature generation model capable of generating feature information related to the physical condition of a subject from the target image of the subject.

[0070] The determination information generation model may be a second neural network trained using patient images of a plurality of patients having a disease in the target region as training data. The patient images may be associated with identification information including the name and annotation of each feature detected from the patient image, and position information indicating the position of each feature in the patient image. This trained second neural network may be used as a feature information generation model capable of outputting feature information from a target image of a subject.

[0071] Here, the disease may be at least one of cancer, heart disease, pneumonia, emphysema, cerebral infarction, dementia, osteoarthritis, spondylosis, bone fracture, and osteoporosis.

[0072] In response to a target image being input to the input layer, the second generation unit 311 performs calculations based on a feature information generation model and outputs feature information from the output layer. As an example, the second generation unit 311 may be configured to extract predetermined feature amounts from the target image and use them as input data. The following well-known algorithms may be applied to extract the feature amounts. ·R-CNN(Region-based convolutional neural network) YOLO (You Only Look Once) ·SDD (Single Shot Detector).

[0073] The feature information generated by the second generation unit 311 is transmitted to the information processing device 1 via the communication unit 33 together with the instruction ID.

[0074] The determination system 100 having the above configuration acquires first determination information, region of interest information, and feature information generated from the same target image, and displays this information on the display device 5. This allows the determination system 100 to present the first determination information, as well as the region of interest information and feature information that aid in understanding the first determination information, to a user (e.g., a medical professional). This allows the user to correctly understand the first determination information generated from the target image.

[0075] For example, when generating a feature information generation model that outputs feature information related to X-ray images of lungs, X-ray images of the lungs of multiple patients can be used as training data, and lung features (ground-glass opacities, calcific lesions, etc.) and their positions (annotation information) shown in each patient's X-ray lung image can be used as training data. As an example, by applying R-CNN and performing machine learning using the backpropagation algorithm, the feature information generation model can output accurate features and positions related to lung feature information for an X-ray image of the lungs of any subject.

[0076] For example, when generating a feature information generation model that outputs feature information related to knee X-ray images, training data may be used, and training data may be used, including knee X-ray images of multiple patients and knee features (such as narrowed joint space areas and bone sclerosis) and their positions (annotation information) that appear in each patient's knee X-ray image. As an example, by applying R-CNN and performing machine learning using the backpropagation algorithm, the feature information generation model can output accurate features and positions related to knee feature information for any subject's knee X-ray image.

[0077] A feature information generation model different from the determination information generation model for outputting the first determination information is applied to generate the feature information. Therefore, the feature information may include information highly relevant to the first determination information, as well as information less relevant to the first determination information and information different from the condition of the target region. The determination system 100 presents the feature information to the user in addition to the first determination information and the region-of-interest information generated from the target image. This allows the determination system 100 to inform the user not only of the condition of the target region of the subject, but also of the presence of features related to the subject's physical condition. Therefore, the user can determine the treatment and intervention applicable to the subject after understanding the condition of the target region of the subject and the subject's physical condition not limited to the target region.

[0078] (Processing performed by the determination system 100) Next, the processing executed by the determination system 100 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of the flow of processing executed by the information processing device 1 of the determination system 100.

[0079] First, the information processing device 1 acquires a target image from the target image management device 4 or the display device 5 (step S1: target image acquisition step).

[0080] Next, the information processing device 1 transmits the target image, the first generation instruction, and the second generation instruction to the first determination information generation device 2 (step S2). In response to the first generation instruction and the second generation instruction, the first determination information generation device 2 generates first determination information and region-of-interest information from the target image (first generation step, third generation step).

[0081] The information processing device 1 acquires the first determination information and the region-of-interest information from the first determination information generating device 2 (step S3).

[0082] Meanwhile, the information processing device 1 transmits the target image and a third generation instruction to the feature information generation device 3 (step S4). In response to the third generation instruction, the feature information generation device 3 generates feature information from the target image (second generation step).

[0083] The information processing device 1 acquires the feature information from the feature information generation device 3 (step S5).

[0084] The information processing device 1 transmits the first determination information and the region of interest information acquired in step S3 and the feature information acquired in step S5 to the display device 5, and causes the display device 5 to display them (step S6: display control step). The processing of steps S2 to S3 may be performed before or after the processing of steps S4 to S5.

[0085] (display screen) Next, examples of display screens displayed on the display unit 51 of the display device 5 will be described with reference to Fig. 7 to Fig. 12. Fig. 7 to Fig. 12 are diagrams showing examples of display screens. Here, an example will be described in which an X-ray image of the lungs or knee joint of a subject is used as the target image.

[0086] 7 to 12 show examples of display screens that respectively display the first determination information, the region of interest information, and the feature information.

[0087] The display screens shown in Figures 7 and 8 have an area R1 that displays the subject ID, an area R2 that displays first assessment information, and an area R5 that displays the subject image and image ID used to generate the first assessment information. In the example shown in Figure 7, the first assessment information "Pneumonia is progressing" is displayed in area R2. In the example shown in Figure 8, the first assessment information "This subject is at risk of developing knee osteoarthritis. There is a possibility that knee pain will occur in three years" is displayed in area R2.

[0088] 7 and 8 also have an area R6 for receiving an instruction to transition to a display screen that displays area-of-interest information, and an area R7 for receiving an instruction to transition to a display screen that displays feature information. For example, when a user selects area R6 shown in FIG. 7, the display screen transitions to the display screen shown in FIG. 9. On the other hand, when a user selects area R7 shown in FIG. 7, the display screen transitions to the display screen shown in FIG. 11.

[0089] 9 and 10 have an area R1 that displays a subject ID, an area R3 that displays region-of-interest information, and an area R5 that displays a target image and image ID. In the example shown in Fig. 9, region R3 displays region-of-interest information such as "upper right lung field" and "lower left lung field." In the example shown in Fig. 10, region R3 displays region-of-interest information such as "inside knee joint" and "outside femur surface shape."

[0090] 9 and 10 have an area R8 for receiving an instruction to transition to a display screen that displays first determination information, and an area R7 for receiving an instruction to transition to a display screen that displays feature information. For example, when a user selects area R8 shown in FIG. 9, the display screen transitions to the display screen shown in FIG. 7. On the other hand, when a user selects area R7 shown in FIG. 10, the display screen transitions to the display screen shown in FIG. 11.

[0091] 11 and 12 have an area R1 for displaying a subject ID, an area R4 for displaying feature information, and an area R5 for displaying a subject image and image ID. In the example shown in Fig. 11, area R4 displays feature information such as "upper right lung field: ground-glass opacity" and "lower left lung field: calcification focus opacity." In the example shown in Fig. 12, area R4 displays feature information such as "inside knee joint: bone sclerosis," "outside femoral surface of knee joint: osteophyte," and "weakened femur."

[0092] 11 and 12 have an area R8 for receiving an instruction to transition to a display screen displaying first determination information, and an area R6 for receiving an instruction to transition to a display screen displaying region-of-interest information. For example, when a user selects area R8 shown in FIG. 11, the display screen transitions to the display screen shown in FIG. 7. On the other hand, when a user selects area R6 shown in FIG. 11, the display screen transitions to the display screen shown in FIG. 9.

[0093] [Variation 1] As shown in Figures 7 and 8 to 11 and 12, the first judgment information, the region of interest information, and the feature information may be configured to be displayed separately on the display unit 51, but this is not limiting. For example, the first judgment information, the region of interest information, and the feature information may be displayed all at once on the display unit 51. An example of a display screen that simultaneously displays the first judgment information, the region of interest information, and the feature information will be described with reference to Figures 13 and 14. Figures 13 and 14 are diagrams showing an example of the display screen.

[0094] The display screen shown in Figures 13 and 14 has an area R1 that displays the subject ID, an area R2 that displays the first judgment information, an area R3 that displays the target area information, an area R4 that displays feature information, and an area R5 that displays the subject image and image ID used to generate the first judgment information.

[0095] In the example shown in FIG. 13, first determination information "pneumonia progression observed" is displayed in region R2, and region-of-interest information "upper right lung field" and "lower left lung field" are displayed in region R3. Furthermore, feature information "upper right lung field: ground-glass opacity" and "lower left lung field: calcification focus opacity" are displayed in region R4. In the example shown in FIG. 14, first determination information "potential for knee osteoarthritis" and "possibility of knee joint pain developing in three years" are displayed in region R2. Furthermore, region R3 displays region-of-interest information "medial side of knee joint" and "femoral surface shape on the lateral side of knee joint." Furthermore, feature information "medial side of knee joint: bone sclerosis," "femoral surface on the lateral side of knee joint: osteophyte," "weakened femur," etc. are displayed in region R4.

[0096] If the first determination information, the region of interest information related to the first determination information, and the feature information generated from the same target image of the subject are all presented to the user together, the user does not need to view multiple screens. This configuration can improve the convenience of the determination system 100.

[0097] 13 and 14 show examples in which the first determination information, the region of interest information, and the feature information are simultaneously displayed on one display unit 51, but this is not limiting. For example, the first determination information, the region of interest information, and the feature information may be simultaneously displayed on multiple display units 51 designated by the user.

[0098] [Embodiment 2] The first generation unit 211 may be configured to take into account the feature information generated by the feature information generation device 3 when generating the first determination information from the target image. The flow of processing performed by the determination system 100 having this configuration will be described with reference to FIG. 15. FIG. 15 is a flowchart showing another example of the flow of processing executed by the determination system 100. For ease of explanation, the same reference numerals are used to denote the same processes as those described in FIG. 6.

[0099] First, the information processing device 1 acquires a target image from the target image management device 4 or the display device 5 (step S1: target image acquisition step).

[0100] Next, the information processing device 1 transmits the target image and a third generation instruction to the feature information generation device 3 (step S4). In response to the third generation instruction, the feature information generation device 3 generates feature information from the target image (second generation step).

[0101] The information processing device 1 acquires the feature information from the feature information generation device 3 (step S5).

[0102] Next, the information processing device 1 transmits the target image, the feature information, the first generation instruction, and the second generation instruction to the first determination information generation device 2 (step S2a). In response to the first generation instruction and the second generation instruction, the first determination information generation device 2 generates first determination information and attention area information from the target image and the feature information (first generation step, third generation step).

[0103] The information processing device 1 acquires the first determination information and the region-of-interest information from the first determination information generating device 2 (step S3).

[0104] The information processing device 1 transmits the feature information acquired in step S5 and the first determination information and the region-of-interest information acquired in step S3 to the display device 5, and causes the display device 5 to display them (step S6).

[0105] With this configuration, feature information obtained from the same target image can be used to generate the first determination information, allowing the determination system 100 to generate and output more reliable first determination information.

[0106] [Embodiment 3] The area of ​​interest information may be information indicating the position of an area of ​​interest in the process of generating first judgment information from the target image. Therefore, a configuration may be adopted in which an image indicating the area of ​​interest information is generated. Here, a determination system 100a including a first judgment information generation device 2a that generates an image of interest indicating the area of ​​interest information will be described with reference to FIG. 16. FIG. 16 is a block diagram showing an example of the configuration of the determination system 100a. For convenience of explanation, components having the same functions as those described in the above embodiment will be denoted by the same reference numerals, and their description will not be repeated.

[0107] The determination system 100a includes an information processing device 1a, a first determination information generating device 2a, and a feature information generating device 3.

[0108] The first judgment information generating device 2a includes a control unit 21a that comprehensively controls each part of the first judgment information generating device 2a, a memory unit 22 that stores various data used by the control unit 21a, and a communication unit 23 that performs various data communications.

[0109] The control unit 21a includes a first generating unit 211 that generates first determination information based on the target image, a third generating unit 212 that generates region-of-interest information, and an image-of-interest generating unit 213.

[0110] The image of interest generation unit 213 generates an image of interest in response to the second generation instruction. The image of interest generation unit 213 may generate an image of interest by superimposing the region of interest indicated by the region of interest information generated by the third generation unit 212 on the target image.

[0111] The first judgment information generated by the first generation unit 211, the focus area information generated by the third generation unit 212, and the focus image generated by the focus image generation unit 213 are transmitted to the information processing device 1a via the communication unit 23 together with the instruction ID.

[0112] The display information generation unit 113a of the information processing device 1a acquires, for each target image, (1) the first judgment information, region of interest information, and image of interest generated by the first judgment information generation device 2a, and (2) feature information generated by the feature information generation device 3. The display information generation unit 113a generates display information by associating the first judgment information, region of interest information, and image of interest with the feature information, with reference to an instruction ID or the like associated with each piece of acquired information. The display information generation unit 113a may store the generated display information together with the instruction ID in the display information database 121a of the storage unit 12a.

[0113] [Display Information Database 121a] The display information database 121a will now be described with reference to Fig. 17. Fig. 17 is a diagram showing an example of the data structure of the display information database 121a.

[0114] The display information database 121a stores the following information in association with each other. ·Instruction ID. The first judgment information, the region of interest information, and the image of interest generated by the first judgment information generating device 2 in accordance with the first generation instruction and the second generation instruction to which each instruction ID is assigned. The characteristic information generated by the characteristic information generating device 3 in response to the third generation instruction to which each instruction ID is assigned.

[0115] Returning to FIG. 16, the display control unit 114 of the information processing device 1a causes the display unit 51 of the display device 5 to display information associated with each instruction ID from the display information database 121a.

[0116] According to this configuration, the determination system 100a can clearly indicate to the user which region of the target image is the region of interest, thereby improving the usability of the region of interest information.

[0117] [Embodiment 4] The feature information is information indicating the position of a feature detected from the target image. Therefore, a configuration may be adopted in which a first feature image is generated by superimposing the feature information on the target image. Here, a determination system 100b including a feature information generation device 3a that generates a feature image by superimposing the feature information on the target image will be described with reference to FIG. 18. FIG. 18 is a block diagram showing an example of the configuration of the determination system 100b. For convenience of explanation, components having the same functions as those described in the above embodiment will be denoted by the same reference numerals, and their description will not be repeated.

[0118] The determination system 100b includes an information processing device 1b, a first determination information generating device 2, and a feature information generating device 3a.

[0119] The characteristic information generating device 3a includes a control unit 31a that controls each unit of the characteristic information generating device 3a in an integrated manner, a memory unit 32 that stores various data used by the control unit 31a, and a communication unit 33 that performs various data communications.

[0120] The control unit 31a includes a second generation unit 311 that generates feature information based on the target image, and a feature image generation unit 312.

[0121] The feature image generating unit 312 generates the first feature image in response to the third generation instruction. The feature image generating unit 312 may generate the first feature image by superimposing the feature information generated by the second generating unit 311 on the target image.

[0122] The feature information generated by the second generation unit 311 and the feature image generated by the feature image generation unit 312 are transmitted to the information processing device 1b via the communication unit 33 together with the instruction ID.

[0123] The display information generation unit 113b of the information processing device 1b acquires, for each target image, (1) the first judgment information and region-of-interest information generated by the first judgment information generation device 2, and (2) the feature information and feature image generated by the feature information generation device 3a. The display information generation unit 113b generates display information by associating the first judgment information and region-of-interest information with the feature information and feature image, with reference to an instruction ID or the like associated with each piece of acquired information. The display information generation unit 113b may store the generated display information together with the instruction ID in the display information database 121b of the storage unit 12b.

[0124] The display information database 121b stores the following information in association with each other. ·Instruction ID. The first judgment information and the region-of-interest information generated by the first judgment information generating device 2 in accordance with the first generation instruction and the second generation instruction to which each instruction ID is assigned. The feature information and the feature image generated by the feature information generating device 3 in response to the third generation instruction to which each instruction ID is assigned.

[0125] The display control unit 114 of the information processing device 1b causes the display unit 51 of the display device 5 to display information associated with each instruction ID from the display information database 121b.

[0126] According to this configuration, the determination system 100b can clearly show the user in which area of ​​the target image the feature was detected, thereby improving the usability of the feature information.

[0127] [Embodiment 5] A configuration in which an image of interest and a feature image are generated may also be used. Here, a determination system 100c including an image of interest generation unit 213 that generates an image of interest by superimposing information about a region of interest on a target image, and a feature information generation device 3a that generates a first feature image by superimposing feature information on the target image will be described with reference to FIG. 19. FIG. 19 is a block diagram showing an example of the configuration of the determination system 100c. For convenience of explanation, components having the same functions as those described in the above embodiment will be denoted by the same reference numerals, and their description will not be repeated.

[0128] The determination system 100c includes an information processing device 1c, a first determination information generating device 2a, and a feature information generating device 3a.

[0129] The display information generation unit 113c of the information processing device 1c acquires, for each target image, (1) the first judgment information, region of interest information, and image of interest generated by the first judgment information generation device 2a, and (2) the feature information and feature image generated by the feature information generation device 3a. The display information generation unit 113c generates display information by associating the first judgment information, region of interest information, and image of interest with the feature information and feature image, with reference to an instruction ID or the like associated with each piece of acquired information. The display information generation unit 113c may store the generated display information together with the instruction ID in the display information database 121c of the storage unit 12c.

[0130] The display control unit 114 of the information processing device 1c causes the display unit 51 of the display device 5 to display information using the information associated with each instruction ID from the display information database 121c.

[0131] [Display Information Database 121c] The display information database 121c will now be described with reference to Fig. 20. Fig. 20 is a diagram showing an example of the data structure of the display information database 121c.

[0132] The display information database 121c stores the following information in association with each other. ·Instruction ID. The first judgment information, the region of interest information, and the image of interest generated by the first judgment information generating device 2a in accordance with the first generation instruction and the second generation instruction to which each instruction ID is assigned. The feature information and the feature image generated by the feature information generating device 3a in response to the third generation instruction to which each instruction ID is assigned.

[0133] Returning to FIG. 19, the display control unit 114 of the information processing device 1c causes the display unit 51 of the display device 5 to display information associated with each instruction ID from the display information database 121c.

[0134] With this configuration, the determination system 100c can clearly show the user which region of the target image is the region of interest and where the feature is detected, thereby improving the usability of the region of interest information and feature information.

[0135] The information processing device 1c may have the functionality of the feature image generation unit 312. In this case, the information processing device 1c can generate a second feature image from the region-of-interest information or image of interest generated by the first judgment information generation device 2a and the feature information acquired from the feature information generation device 3a. The second feature image is an image in which the feature information and region-of-interest information are superimposed on the target image. The second feature image may also be an image in which the feature information is superimposed on the image of interest (see FIGS. 21 and 22).

[0136] (display screen) Next, examples of display screens displayed on the display unit 51 of the display device 5 will be described with reference to Fig. 21 and Fig. 22. Fig. 21 and Fig. 22 are diagrams showing examples of display screens. In Fig. 21, an X-ray image of the subject's lungs is used as the target image, a target image is displayed instead of the region-of-interest information, and a second feature image is displayed instead of the feature information. In Fig. 22, an X-ray image of the subject's knee joint is used as the target image.

[0137] The display screen shown in Figures 21 and 22 has an area R1 that displays the subject ID, an area R2 that displays the first judgment information, and an area R5 that displays the subject image and image ID used to generate the first judgment information.

[0138] 21 and 22 have an area R9 for displaying the image of interest and an area R10 for displaying the second feature image. The first feature image may be displayed in area R10 instead of the second feature image.

[0139] 21 and 22, the image of interest may be, for example, an image in which the region of interest is expressed as a heat map on the target image. On the other hand, as shown in region R10, the feature image may be, for example, an image in which the positions of detected features and the names of each feature are displayed on the target image or the image of interest.

[0140] In this manner, the determination system 100c may display the target image and the second feature image side by side on a display device. By comparing the target image and the second feature image, the user can comprehensively understand the first determination information, the target area information, and the feature information generated from the same target image. This allows the determination system 100c to help the user understand the first determination information and effectively encourage them to utilize the first determination information.

[0141] [Embodiment 6] Other embodiments of the present disclosure will be described below. For convenience of explanation, the same reference numerals will be used to designate components having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0142] Second determination information indicating a method of medical intervention for the subject and the effect of the medical intervention may be generated from the first determination information. The configuration of a determination system 100d capable of generating the second determination information will be described with reference to FIG. 23. FIG. 23 is a functional block diagram showing an example of the configuration of the determination system 100d. For convenience of explanation, components having the same functions as those described in the above embodiment are denoted by the same reference numerals, and their description will not be repeated.

[0143] The determination system 100d includes an information processing device 1d, a first determination information generating device 2d, and a feature information generating device 3.

[0144] The first judgment information generating device 2d includes a control unit 21d that controls each part of the first judgment information generating device 2d in an integrated manner, a memory unit 22 that stores various data used by the control unit 21d, and a communication unit 23 that performs various data communications.

[0145] [Fourth generation unit 214] The fourth generating unit 214 generates second determination information indicating a method of medical intervention for the subject and an effect of the medical intervention, from the first determination information generated by the first generating unit 211. In one example, the fourth generating unit 214 may generate the second determination information in response to the first generation instruction.

[0146] The fourth generating unit 214 generates, from the first determination information generated by the first generating unit 211, second determination information regarding the effect of applying a medical intervention to the subject.

[0147] The fourth generation unit 214 may have an intervention effect assessment model that receives first assessment information of a subject as input and outputs second assessment information indicating a method of intervention for the subject and the effect of the intervention.

[0148] The intervention effect assessment model may be a third neural network trained using effect information for each of a plurality of patients who have received an intervention for a disease in a target area as training data. The effect information may be information in which the intervention applied to the target area of ​​each patient is associated with intervention effect information indicating the effect of the intervention for each patient. This trained third neural network may be used as an intervention effect assessment model capable of outputting second assessment information from first assessment information.

[0149] Here, the intervention may include at least one of diet therapy, exercise therapy, drug therapy, stimulation therapy, manual therapy, ultrasound therapy, use of walking aids, use of braces, orthotic surgery, osteotomy, intra-articular injection, joint preservation surgery, total joint replacement surgery, and spinal instrumentation surgery.

[0150] The first judgment information generated by the first generation unit 211, the area of ​​interest information generated by the third generation unit 212, and the second judgment information generated by the fourth generation unit 214 are transmitted to the information processing device 1d via the communication unit 23 together with the instruction ID.

[0151] The display information generation unit 113d of the information processing device 1d acquires, for each target image, (1) the first judgment information, region of interest information, and second judgment information generated by the first judgment information generation device 2d, and (2) the feature information generated by the feature information generation device 3. The display information generation unit 113d generates display information by associating the first judgment information, region of interest information, and second judgment information with the feature information, with reference to an instruction ID or the like associated with each piece of acquired information. The display information generation unit 113d may store the generated display information together with the instruction ID in the display information database 121d of the storage unit 12d.

[0152] The display control unit 114 of the information processing device 1d causes the display unit 51 of the display device 5 to display information associated with each instruction ID from the display information database 121d.

[0153] According to this configuration, the determination system 100d can provide the user with not only first determination information on the condition of the target part of the subject from the target image of the subject, but also second determination information on the effect of applying medical intervention to the subject, thereby allowing the user of the determination system 100d to appropriately determine whether or not medical intervention is necessary for the subject.

[0154] For example, the fourth generator 214 for lung X-ray images is generated by learning applying the backpropagation method to an LSTM using the following training data and the following teacher data: Such a fourth generator 214 can accurately predict the effect of an intervention on the lungs for an X-ray image of the lungs of any subject. Training data: lung x-ray images of multiple patients combined with the interventions performed on those patients. Training data: Information on the prognostic effect of lungs shown in each of the lung X-ray images of multiple patients (presence or absence of effect, change in hospital stay, etc.).

[0155] For example, the fourth generator 214 for knee X-ray images is generated by learning applying the backpropagation method to an LSTM using the following training data and the following teacher data: Such a fourth generator 214 can accurately predict the effect of intervention on the knee for an X-ray image of the knee of any subject. Training data: knee x-rays from multiple patients combined with the interventions performed on those patients. Training data: Information on the prognostic effect of the knee (e.g., change in pain, improvement in walking ability) shown in each of the knee X-ray images of multiple patients.

[0156] [Embodiment 7] Other embodiments of the present disclosure will be described below. For convenience of explanation, the same reference numerals will be used to designate components having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0157] (Configuration of Determination System 100) 1 may be communicably connected to a LAN disposed in each of a plurality of medical facilities 6 via a communication network 7, rather than being a computer installed in a predetermined medical facility 6. Fig. 24 is a block diagram showing an example configuration of a determination system 100 according to another embodiment of the present disclosure.

[0158] In the determination system 100 shown in FIG. 24, an information processing device 1 is communicably connected to each device in medical facilities 6a and 6b via a communication network 7.

[0159] The medical facility 6a includes a display device 5a, a target image management device 4a, and an electronic medical record management device 9a, all of which are communicatively connected to one another. On the other hand, the medical facility 6b includes a display device 5b, a target image management device 4b, and an electronic medical record management device 9b, all of which are communicatively connected to one another.

[0160] 24 shows an example in which LANs of a medical facility 6a and a medical facility 6b are connected to a communication network 7. The information processing device 1 is not limited to the configuration shown in FIG. 24 as long as it is communicably connected to devices in each of the medical facilities 6a and 6b via the communication network 7.

[0161] In the determination system 100 employing such a configuration, the information processing device 1 can acquire a target image of the subject Pa from the medical facility 6a, and can acquire a target image of the subject Pb from the medical facility 6b.

[0162] In this case, the target image of each subject may include identification information unique to each medical facility 6a, 6b, which is assigned to each medical facility 6 examining each subject, and identification information unique to each subject, which is assigned to each subject. The identification information unique to each medical facility 6a, 6b may be, for example, a facility ID. Furthermore, the identification information unique to each subject may be, for example, a patient ID. The information processing device 1 can transmit various pieces of information acquired from the first determination information generating device 2 and the feature information generating device 3 to the display devices 5a, 5b installed in the medical facilities 6a, 6b that are the source of the target image.

[0163] Based on this identification information, the information processing device 1 can correctly transmit the first judgment information, the area of ​​interest information, and the feature information indicating the condition of the target area of ​​the subject to the display devices 5a, 5b of each medical facility 6a, 6b where the subject received treatment.

[0164] The information processing device 1, the first determination information generation device 2, and the characteristic information generation device 3 may each be connected to a communication network 7. Alternatively, at least one of the first determination information generation device 2 and the characteristic information generation device 3 may be located in either the medical facility 6a or 6b.

[0165] In the determination systems 100a to 100d, the information processing devices 1a to 1d may be configured to be communicably connected to a LAN arranged in each of the medical facilities 6a and 6b via a communication network 7. Alternatively, the information processing devices 1a to 1d, the first determination information generation devices 2 and 2a, and the characteristic information generation devices 3 and 3a may each be connected to the communication network 7. Alternatively, at least one of the first determination information generation devices 2 and 2a and the characteristic information generation devices 3 and 3a may be located in either of the medical facilities 6a and 6b.

[0166] [Appendix 1] In the above-described embodiments, the acquisition unit 111 of the information processing device 1, 1a to 1d acquires a target image from the target image management device 4 and transmits the target image to the first determination information generation device 2, 2a, 2d and the feature information generation device 3, 3a. However, the present invention is not limited to this.

[0167] For example, the first determination information generation devices 2, 2a, 2d and the feature information generation devices 3, 3a may be configured to acquire the target image from the target image management device 4. In this case, various generation instructions transmitted from the information processing devices 1, 1a to 1d to the first determination information generation devices 2, 2a, 2d and the feature information generation devices 3, 3a may include an image ID, which is identification information assigned to the target image.

[0168] The first determination information generating devices 2, 2a, and 2d may acquire from the target image management device 4 the target images to which the image IDs included in the first and second generation instructions received from the information processing devices 1 and 1a to 1d have been assigned.

[0169] [Appendix 2] In the above embodiments, the first determination information generating devices 2, 2a, and 2d generate the first determination information and the region-of-interest information, but the present invention is not limited to this.

[0170] For example, the information processing devices 1, 1a to 1d may have the functions of the first generation unit 211 and the third generation unit 212 (and the image of interest generation unit 213). The information processing devices 1, 1a to 1d may have at least one of the functions of the image of interest generation unit 213 and the function of the fourth generation unit 214.

[0171] [Appendix 3] In the above embodiments, the feature information generating device 3, 3a generates feature information, but the present invention is not limited to this.

[0172] For example, the information processing devices 1, 1a to 1d may have the function of the second generation unit 311. The information processing devices 1, 1a to 1d may have the function of the feature image generation unit 312.

[0173] [Appendix 4] 24, in the above-described embodiments of the information processing device 1 and the determination system 100, the feature information generating device 3, 3a generates feature information as an example. However, the present invention is not limited to this.

[0174] For example, the information processing devices 1, 1a to 1d may have the function of the second generation unit 311. The information processing devices 1, 1a to 1d may have the function of the feature image generation unit 312.

[0175] [Software implementation example] The control blocks of the determination systems 100 to 100d (particularly the control units 11, 11a to 11d, 21, 21a, 21d, 31, 31a) may be realized by a logic circuit (hardware) formed on an integrated circuit (IC chip) or the like, or may be realized by software.

[0176] In the latter case, the determination systems 100 to 100d include a computer that executes instructions of a program, which is software that realizes each function. This computer includes, for example, one or more processors and a computer-readable recording medium storing the program. The object of the present disclosure is achieved when the processor in the computer reads and executes the program from the recording medium. The processor may be, for example, a central processing unit (CPU). The recording medium may be a "non-transitory tangible medium," such as a read-only memory (ROM), tape, disk, card, semiconductor memory, or programmable logic circuit. The system may also include a random access memory (RAM) for loading the program. The program may be supplied to the computer via any transmission medium capable of transmitting the program (such as a communication network or broadcast waves). One aspect of the present disclosure may also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

[0177] The invention according to the present disclosure has been described above based on the drawings and examples. However, the invention according to the present disclosure is not limited to the above-described embodiments. In other words, the invention according to the present disclosure can be modified in various ways within the scope of the present disclosure, and embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the invention according to the present disclosure. In other words, it should be noted that a person skilled in the art can easily make various modifications or corrections based on the present disclosure. It should also be noted that these modifications or corrections are included in the scope of the present disclosure.

[0178] [Other Aspect 1] The present invention also includes the following aspects.

[0179] A judgment system according to one embodiment of the present disclosure includes an acquisition unit that acquires a target image showing a target part of a subject's body, a first generation unit that generates first judgment information indicating the state of the target part from the target image, a second generation unit that generates feature information indicating features related to the state of the subject's body from the target image used to generate the first judgment information, and a display control unit that acquires the first judgment information and the feature information and displays the first judgment information and the feature information on a display device.

[0180] A control method of a judgment system according to one embodiment of the present disclosure includes a target image acquisition step of acquiring a target image showing a target part of a subject's body; a first generation step of generating first judgment information that judges the state of the target part from the target image; a second generation step of generating feature information that indicates features related to the subject's physical state from the target image used to generate the first judgment information; and a display control step of acquiring the first judgment information and the feature information and displaying the first judgment information and the feature information on a display device.

[0181] The determination system according to each aspect of the present disclosure may be realized by a computer. In this case, the control program of the determination system that causes the computer to operate as each part (software element) of the device included in the determination system, and the computer-readable recording medium on which the control program is recorded, also fall within the scope of the present disclosure.

[0182] [Other Aspect 2] The present invention also includes the following aspects.

[0183] A judgment system according to aspect 1 of the present disclosure includes a target image acquisition unit that acquires a target image showing a target part of a subject's body, a first generation unit that generates first judgment information indicating the state of the target part from the target image, a second generation unit that generates feature information indicating features related to the state of the subject's body from the target image, and a third generation unit that generates focus area information indicating the position of a focus area that was focused on during the generation of the first judgment information, wherein the feature information includes identification information including the name of each feature detected from the target image and the name of the position of each feature in the target image.

[0184] A determination system according to a second aspect of the present disclosure is based on the first aspect, and the third generator may generate the region-of-interest information based on a feature map.

[0185] In the determination system according to aspect 3 of the present disclosure, in aspect 1 or 2, the third generation unit may evaluate the degree of influence on the first determination information for each pixel of the target image.

[0186] The judgment system according to aspect 4 of the present disclosure, in any of aspects 1 to 3, may further include a display control unit that acquires the first judgment information, the feature information, and the area of ​​interest information, and displays the first judgment information, the feature information, and the area of ​​interest information on a display device.

[0187] In the determination system according to aspect 5 of the present disclosure, in any one of aspects 1 to 4, the feature information may further include position information indicating a position of each of the features in the target image.

[0188] The judgment system of aspect 6 of the present disclosure, in aspect 4, further includes a focus image generation unit that generates a focus image by superimposing a focus area indicated by focus area information indicating a focus area related to the first judgment information, which is a partial area of ​​the target image, on the target image, and the display control unit may display the focus image on the display device.

[0189] The determination system of aspect 7 of the present disclosure, in aspect 6, further includes a feature image generation unit that generates at least one of a first feature image in which the feature information is superimposed on the target image, and a second feature image in which the feature information and the area of ​​interest information are superimposed on the target image, and the display control unit may display at least one of the first feature image and the second feature image on the display device.

[0190] In the determination system according to aspect 8 of the present disclosure, in addition to aspect 7, the display control unit may cause the display device to display the image of interest and the second feature image side by side.

[0191] In the judgment system of aspect 9 of the present disclosure, in any of aspects 4, 6, 7, and 8, the display control unit may simultaneously display (1) the first judgment information, (2) area of ​​interest information indicating an area of ​​interest related to the first judgment information, which is a partial area of ​​the target image, and (3) the feature information on the display device.

[0192] In the determination system according to aspect 10 of the present disclosure, in any one of aspects 1 to 9, the first determination information may be diagnostic information including a diagnostic result as to whether or not the subject has a disease.

[0193] In the judgment system of aspect 11 of the present disclosure, in any of aspects 1 to 10, the first generation unit may have a judgment information generation model that is capable of generating the first judgment information indicating the state of a target part of the subject using the target image of the subject.

[0194] In the judgment system of aspect 12 of the present disclosure, in aspect 11, the judgment information generation model is a first neural network trained using patient information for each of a plurality of patients having a disease in the target area as training data, and the patient information may include diagnostic information and medical images indicating diagnostic results for the condition of the target area for each patient.

[0195] In the determination system of aspect 13 of the present disclosure, in any of aspects 1 to 12, the second generation unit may have a feature generation model capable of generating the feature information related to the physical condition of the subject from the target image of the subject.

[0196] A determination system according to aspect 14 of the present disclosure is such that, in aspect 13, the feature generation model is a second neural network trained using patient images of a target area of ​​each of a plurality of patients having a disease of the target area as training data, and the patient images may be associated with identification information including the name and annotation of each feature detected from the patient image, and location information indicating the location of each feature in the patient image.

[0197] In the judgment system of aspect 15 of the present disclosure, in any of aspects 1 to 14, the first judgment information may be information that judges the state of the target area at a second point in time after a predetermined period of time has elapsed since the first point in time when the target image was captured.

[0198] The judgment system of aspect 16 of the present disclosure, in any of aspects 4, 6, 7, 8, and 9, further includes a fourth generation unit that generates second judgment information from the first judgment information, the second judgment information indicating a method of medical intervention for the subject and the effects of the medical intervention, and the display control unit may display the first judgment information, the feature information generated from the target image used to generate the first judgment information, and the second judgment information on the display device.

[0199] In the judgment system of aspect 17 of the present disclosure, in aspect 16, the fourth generation unit may have an intervention effect judgment model that takes the first judgment information of the subject as input and outputs second judgment information indicating a method of intervention for the subject and the effect of the intervention.

[0200] In the judgment system of aspect 18 of the present disclosure, in aspect 17, the intervention effect judgment model is a third neural network trained using effect information for each of multiple patients who have undergone intervention for a disease in the target area as training data, and the effect information may be information in which the intervention applied to the target area of ​​each patient and intervention effect information indicating the effect of the intervention are associated for each patient.

[0201] A determination system according to a nineteenth aspect of the present disclosure is in any one of the first to eighteenth aspects, wherein the region of interest information may include a name of the position of the region of interest.

[0202] A judgment system according to aspect 20 of the present disclosure may, in any of aspects 1 to 19, include an information processing device having the target image acquisition unit and a display control unit that acquires the first judgment information, the feature information, and the area of ​​interest information and displays the first judgment information, the feature information, and the area of ​​interest information on a display device; a first judgment information generation device having the first generation unit and the third generation unit; and a feature information generation device having the second generation unit.

[0203] A control method according to aspect 21 of the present disclosure is a control method for a judgment system comprising an information processing device, a first judgment information generation device, and a feature information generation device, and includes a target image acquisition step in which the information processing device acquires a target image showing a target part of the subject's body; a first generation step in which the first judgment information generation device generates first judgment information that judges the state of the target part from the target image; a second generation step in which the feature information generation device generates feature information indicating features related to the subject's physical state from the target image used to generate the first judgment information; and a display control step in which the information processing device acquires the first judgment information and the feature information and displays the first judgment information and the feature information on a display device, and the feature information may include identification information including the names of each feature detected from the target image, and the names of the positions of each feature in the target image.

[0204] The control program of aspect 22 of the present disclosure is a control program for causing a computer to function as a judgment system described in any of aspects 1 to 20, and is a control program for causing a computer to function as the target image acquisition unit, the first generation unit, the second generation unit, and a display control unit that acquires the first judgment information, the feature information, and the area of ​​interest information, and displays the first judgment information, the feature information, and the area of ​​interest information on a display device.

[0205] The control program of aspect 23 of the present disclosure is a control program for causing a computer to function as the information processing device described in aspect 20, and causes the computer to execute a transmission step of transmitting the target image or identification information assigned to the target image to the information processing device and the feature information generating device, and a display control step of acquiring the first judgment information and the feature information generated from the transmitted target image and displaying them on a display device. [Explanation of symbols]

[0206] 1, 1a to 1d information processing device 2, 2a, 2d First judgment information generation device 3, 3a Feature information generation device 4, 4a, 4b Target image management device 5, 5a, 5b display device 6, 6a, 6b Medical Facilities 7. Communication Networks 9, 9a, 9b Electronic medical record management device 11, 11a to 11d, 21, 21a, 21d, 31, 31a Control unit 12, 12a~12d, 22, 32 storage section 13, 23, 33 Communications Department 51 Display section 100, 100a-100d Judgment System 111 Acquisition Department 112 Instruction generation section 113, 113a~113d Display information generation section 114 Display control unit 211 1st generation part 212 Third generation part 213 Image of interest generation unit 214 4th generation part 311 Second generation part 312 Feature Image Generation Unit

Claims

1. a target image acquisition unit that acquires a target image showing a target part of the body of the target person; a first generation unit that generates first determination information related to a disease in the target region from the target image; a second generation unit that generates feature information indicating features related to a physical state of the subject from the target image; a third generation unit that generates region-of-interest information indicating a position of a region of interest that is focused on during the process of generating the first determination information; Equipped with the feature information includes identification information for each feature detected from the target image and position information indicating a position of each feature in the target image; the second generation unit has a second neural network trained using patient images of a plurality of patients having a disease in a target region as training data, the patient images including the target region, as a feature generation model for generating the feature information; The patient image is associated with identification information for each feature detected from the patient image and position information indicating the position of each feature in the patient image. Judging system.

2. the third generation unit generates the region-of-interest information based on a feature map. The determination system according to claim 1 .

3. the third generation unit evaluates the degree of influence on the first determination information for each pixel of the target image. The determination system according to claim 1 or 2.

4. a display control unit that acquires the first determination information, the feature information, and the region-of-interest information and displays the first determination information, the feature information, and the region-of-interest information on a display device; The determination system according to any one of claims 1 to 3.

5. a target image generating unit configured to generate a target image by superimposing a target area indicated by target area information indicating a target area associated with the first determination information, which is a partial area of ​​the target image, on the target image; the display control unit causes the display device to display the image of interest; The determination system according to claim 4 .

6. a feature image generating unit configured to generate at least one of a first feature image in which the feature information is superimposed on the target image and a second feature image in which the feature information and the region-of-interest information are superimposed on the target image, the display control unit causes the display device to display at least one of the first characteristic image and the second characteristic image. The determination system according to claim 5 .

7. the display control unit causes the display device to display the image of interest and the second feature image side by side. The determination system according to claim 6 .

8. The display control unit simultaneously displays, on the display device, (1) the first determination information, (2) area-of-interest information indicating an area of ​​interest related to the first determination information, which is a partial area of ​​the target image, and (3) the feature information. The determination system according to any one of claims 4 to 7.

9. The first determination information is diagnostic information including a diagnostic result regarding whether or not the subject has a disease. The determination system according to any one of claims 1 to 8.

10. the first generation unit has a determination information generation model capable of generating the first determination information indicating a state of a target part of the subject by using the target image of the subject; The determination system according to any one of claims 1 to 9.

11. the determination information generation model is a first neural network trained using patient information on each of a plurality of patients having a disease in the target region as training data, The patient information includes diagnostic information and medical images showing diagnostic results regarding the condition of the target area of ​​each patient. The determination system according to claim 10.

12. The first determination information is information that determines the state of the target part at a second time point after a predetermined period has elapsed since a first time point when the target image was captured. The determination system according to any one of claims 1 to 11.

13. a fourth generation unit that generates second determination information indicating a method of medical intervention for the subject and an effect of the medical intervention from the first determination information; the display control unit causes the display device to display the first determination information, the feature information generated from the target image used to generate the first determination information, and the second determination information. The determination system according to any one of claims 4 to 8.

14. The fourth generation unit has an intervention effect assessment model that receives the first assessment information of the subject as input and outputs second assessment information indicating a method of intervention for the subject and an effect of the intervention. The determination system according to claim 13.

15. the intervention effect assessment model is a third neural network trained using effect information of each of a plurality of patients who have received an intervention for a disease of a target site as training data, The effect information is information in which an intervention applied to a target site of each patient and intervention effect information indicating the effect of the intervention are associated with each patient. The determination system according to claim 14.

16. an information processing device including the target image acquisition unit, and a display control unit that acquires the first determination information, the feature information, and the region-of-interest information, and displays the first determination information, the feature information, and the region-of-interest information on a display device; a first determination information generating device including the first generating unit and the third generating unit; a feature information generation device including the second generation unit, The determination system according to any one of claims 1 to 15.

17. a target image acquisition step of acquiring a target image showing a target part of the body of the target person; a first generation step of generating first determination information related to a disease in the target region from the target image; a second generation step of generating feature information indicating features related to a physical state of the subject from the target image; a third generation step of generating region-of-interest information indicating the position of a region of interest that was noticed during the process of generating the first determination information; Including, the feature information includes identification information for each feature detected from the target image and position information indicating a position of each feature in the target image; In the second generation step, a second neural network trained using patient images of a plurality of patients having a disease in a target region as training data is used as a feature generation model for generating the feature information; The patient image is associated with identification information for each feature detected from the patient image and position information indicating the position of each feature in the patient image. Judging system.

18. A control program for causing a computer to function as the determination system described in any one of claims 1 to 16, the control program causing a computer to function as the target image acquisition unit, the first generation unit, the second generation unit, and the third generation unit.

19. A control program for causing a computer to function as the information processing device according to claim 16, The computer, a transmitting step of transmitting the target image or the identification information assigned to the target image to the information processing device and the feature information generating device; a display control step of acquiring the first determination information and the feature information generated from the transmitted target image and displaying them on a display device;

Images