Prediction system, control method, and control program

Abstract

To enable generation and outputting of a prediction image showing the condition of a target site of a subject.SOLUTION: A prediction system is provided, comprising a prediction information acquisition unit for acquiring (a) a subject image capturing a target site of a subject at a first point in time and (b) first prediction information regarding the target site at a second point in time after a given period from the first point in time; and a prediction image generation unit configured to generate and output a prediction image showing the condition of the target site at the second point in time from the subject image on the basis of the first prediction information.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to a prediction system, a control method, and a control program for predicting the state of a target part of a human body.

Background Art

[0002] As described in Patent Document 1, it has been devised to support osteoporosis diagnosis using a neural network.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] A prediction system according to an aspect of the present disclosure includes a prediction information acquisition unit that acquires (a) a target person image showing a target part of a target person having a predetermined disease at a first time point, and (b) first prediction information regarding the onset or progression degree of the disease regarding the target part at a second time point after a predetermined period has elapsed from the first time point; a prediction image generation unit having a prediction image generation model that generates a prediction image predicting the state of the target part at the second time point from the first prediction information and a converted image in which the file format of the target person image is converted into a predetermined file format; and a prediction information generation unit that generates the first prediction information. The prediction image generation model is generated by machine learning using first training data including first data of the same type as the target person image and second data of the same type as the first prediction information, and first teacher data including third data related to the second data. The first prediction information is generated based on at least the target person image.

[0005] Furthermore, a control method according to one aspect of the present disclosure is a control method for a prediction system, comprising: a prediction information acquisition step of acquiring (a) a subject image showing a subject body part having a predetermined disease at a first time point, and (b) first prediction information relating to the onset or progression of the disease in the subject body part at a second time point after a predetermined period has elapsed from the first time point; a prediction image generation step of generating and outputting a prediction image that predicts the state of the subject body part at a second time point from the first prediction information and a converted image obtained by converting the file format of the subject image to a predetermined file format; and a prediction information generation step of generating the first prediction information, wherein the prediction system has a prediction image generation model capable of generating the prediction image using the subject image and the first prediction information, and the first prediction information is generated at least based on the subject image.

[0006] Furthermore, each aspect of the prediction system described herein may be implemented by a computer. In this case, a control program for the prediction system that enables the computer to implement the prediction system by operating the computer as each component (software element) of the prediction system, and a computer-readable recording medium on which the program is recorded, also fall within the scope of this disclosure.

[0007] If the prediction system is implemented by multiple computers, the prediction system may be implemented by operating each computer as a component (software element) of the multiple computers that make up the prediction system. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example configuration of a prediction system according to one aspect of this disclosure. [Figure 2] This is a block diagram showing an example configuration of a prediction system relating to another aspect of this disclosure. [Figure 3] This block diagram shows an example of the configuration of a prediction system relating to one aspect of this disclosure. [Figure 4] This figure shows an example of the neural network configuration used in the predictive image generation unit. [Figure 5] This flowchart shows an example of the processing flow performed by the prediction system according to Embodiment 1. [Figure 6] This block diagram shows an example of the configuration of a prediction system relating to another aspect of this disclosure. [Figure 7] This figure shows an example of the neural network configuration used in the predictive information generation unit. [Figure 8] This flowchart shows an example of the processing flow performed by the prediction system according to Embodiment 2. [Figure 9] This block diagram shows an example of the configuration of a prediction system relating to another aspect of this disclosure. [Figure 10] This flowchart shows an example of the learning process flow of the neural network in the predictive information generation unit. [Figure 11] This flowchart shows another example of the processing flow performed by the prediction system according to Embodiment 2. [Figure 12] This block diagram shows an example of the configuration of a prediction system relating to another aspect of this disclosure. [Figure 13] This figure shows an example of the neural network configuration used in the predictive image generation unit. [Figure 14] This flowchart shows an example of the learning process flow of the neural network in the intervention effect prediction unit. [Figure 15] This flowchart shows an example of the processing flow performed by the prediction system according to Embodiment 3. [Modes for carrying out the invention]

[0009] To improve the condition of the affected area in the subject, or to slow the onset and progression of the disease, it is desirable to start interventions such as treatment and lifestyle guidance as early as possible.

[0010] In order to initiate intervention early, it is important to make the subject aware of the predicted results regarding the condition of the target area and to make them understand the necessity of the intervention.

[0011] [Embodiment 1] (Overview of the prediction system) A prediction system according to one aspect of this disclosure is a system that generates and outputs a prediction image that predicts a change in the state of a target part of a subject's body. Here, the target part may be any part of the subject's body, for example, any of the whole body, head, eyes, mouth, neck, arms, hands, torso, waist, buttocks, legs, and feet. The prediction image may also be an image that predicts a change in the state of any of the following in the target part: skin, hair, eyeballs, teeth, gums, muscles, fat, bone, cartilage, joints, and intervertebral discs.

[0012] The predictive image may be an image that predicts the changes that will occur in the target area of ​​a subject affected by the disease. For example, the predictive image may be an image that shows at least one of the following: the shape of the target area of ​​the subject (e.g., waist circumference, chest circumference, height, swelling, atrophy, joint angles and curvature, etc.) and the appearance (e.g., posture, wrinkles, spots, redness, cloudiness, darkening, yellowing, etc.).

[0013] In one example, the subject may have a disease in the subject area. In this case, the predicted image may be an image that predicts the change in symptoms in the subject's subject area due to the effects of the disease. This specification describes a prediction system that generates and outputs a predicted image that predicts the change in symptoms in a subject's subject area as an example. In this case, the predicted image is an image that shows the effect of the subject's disease on the subject's subject area. Here, "an image that shows the effect on the subject area" may be any image that shows changes in the shape of the subject area affected by the disease, or qualitative or quantitative changes that have occurred in the tissue of the subject area due to the disease.

[0014] The disease may include at least one of obesity, alopecia, cataract, periodontal disease, rheumatoid arthritis, Heberden's nodes, hallux valgus, osteoarthritis, degenerative spondylosis, compression fracture, and sarcopenia. The disease may include (i) syndromes such as metabolic syndrome and locomotive syndrome that exhibit a collective pathological condition formed by a group of diverse symptoms, and (ii) physical changes such as aging and dentition.

[0015] The prediction system generates a prediction image that predicts the symptoms at the target site at the second time point after a predetermined period has elapsed from the first time point, based on the subject image in which the target site of the subject at the first time point is shown. In this specification, the description "subject image" may sometimes intend the image data showing the subject image.

[0016] Here, the first time point may be, for example, the time when the subject image capturing the target site of the subject is acquired. The first time point may typically be the time when the subject image capturing the current state of the target site of the subject is acquired. That is, the first time point may substantially intend the current time. Also, the predetermined period may be any period elapsed from the first time point, and may be six months, one year, five years, ten years, or fifty years. That is, the second time point may substantially intend any future time point. The predetermined period is not limited to one period and may include a plurality of periods. That is, the prediction image may include images generated by predicting the symptoms of the target site of the subject at a plurality of time points such as six months, one year, five years, ten years, and fifty years after the first time point.

[0017] A subject image is an image showing the subject's target area at the first point in time. In one example, the subject image may be an external image of the subject's whole body, head, upper body, lower body, upper limbs, or lower limbs. In another example, the subject image may be a medical image of the subject's target area taken for the purpose of examining the subject. The medical image may include at least one of the following: X-ray image, CT (Computed Tomography) image, MRI (Magnetic Resonance Imaging) image, PET (Positron Emission Tomography) image, and ultrasound image of the subject. In yet another example, the subject image may be an image showing at least one of the following: shape of the subject's target area (e.g., waist circumference, chest circumference, height, swelling, atrophy, joint angles and curvature, etc.) and appearance (e.g., posture, wrinkles, blemishes, redness, turbidity, darkening, yellowing, etc.).

[0018] Furthermore, the prediction system uses first prediction information regarding the target body part at a second time point, in addition to the subject image, to generate a predicted image.

[0019] The first predictive information may also be information regarding the symptoms of the subject's target area at a second point in time. In this case, the first predictive information may include information showing the symptoms of the subject's target area at multiple points in time, such as six months, one year, five years, ten years, and fifty years after the first point in time. The first predictive information may include, for example, information that includes predictions regarding symptoms that are likely to occur in the subject's target area, the timing of the occurrence of such symptoms, and the degree of progression of the symptoms in the subject's target area.

[0020] The first predictive information may be information relating to at least one of the shape and appearance of the subject's body part at a second point in time. In this case, the information may also be information indicating at least one of the shape (e.g., waist circumference, chest circumference, and height) and appearance (e.g., posture, wrinkles, blemishes) of the subject's body part at multiple points in time, such as six months, one year, five years, ten years, and fifty years after the first point in time.

[0021] The first predictive information may be information related to at least one of the shape and appearance of the target area, relating to a disease of the target area. The first predictive information may include the following information as information that is likely to cause changes in the target area of ​​the subject: For example, the first predictive information may include (i) information related to obesity, such as weight, body mass index (BMI), waist circumference, visceral fat mass, blood pressure, blood glucose level, lipids, uric acid level, liver function values, etc.; (ii) information related to alopecia, such as hair count, sex hormone levels, Norwood classification, Ludwig classification, etc.; (iii) information related to cataracts, such as visual acuity, field of vision, degree of opacity, Emery-Little classification, etc.; (iv) information related to periodontal disease, such as degree of pain and swelling, number of remaining teeth, gingivitis index, periodontal pocket depth, etc.; (v) information related to rheumatoid arthritis, such as degree of pain, degree of swelling, joint angle, range of motion, Larsen classification, Stein (vi) Information related to Heberden's nodes may include information on the degree of pain, degree of swelling, range of motion of the joint, etc. (vii) Information related to hallux valgus may include information on the degree of pain, degree of swelling, range of motion of the joint, HV angle, M1-M2 angle, etc. (viii) Information related to osteoarthritis may include information on the degree of pain, degree of swelling, joint angle, range of motion of the joint, degree of stiffness, thickness of the joint cartilage, Kellgren-Laurence (KL) classification, presence or absence of limping, etc. (ix) In the case of degenerative spondylosis, information may include information on the degree of pain, degree of spinal curvature, range of motion of the spine, KL classification, etc. (x) Information related to compression fractures may include information on the degree of pain, range of motion of the spine, etc. (xi) Information related to sarcopenia may include information on muscle mass, walking speed, grip strength, etc.

[0022] In this way, the prediction system generates and outputs a predicted image that predicts the state of the target body part at the second time point, based on a subject image showing the target body part of the subject at the first time point and first predicted information about the target body part at the second time point, after a predetermined period has elapsed from the first time point.

[0023] The prediction system can output a visually easy-to-understand predicted image of the target area at a second time point. Furthermore, since the predicted image is generated from the subject's own image, it is a realistic and persuasive image for the subject. Therefore, for example, if presented to the subject by a doctor in charge of the subject, it can help the subject recognize the condition of the target area at the second time point and easily understand the need for intervention.

[0024] The predictive image may be an image that mimics an external image of the subject's whole body, head, upper body, lower body, upper limbs, or lower limbs. In another example, the predictive image may be an image that mimics a medical image of the subject's area obtained during the subject's examination. In yet another example, the predictive image may be an image that shows at least one of the following characteristics of the subject's area: shape (e.g., waist circumference, chest circumference, height, swelling, atrophy, joint angles and curvature, etc.) and appearance (e.g., posture, wrinkles, blemishes, redness, cloudiness, darkening, yellowing, etc.). For example, if the subject image is an appearance image showing the subject's current skin appearance (e.g., wrinkles, blemishes, redness, cloudiness, darkening, yellowing, etc.), and the first prediction information is information about the subject's future degree of wrinkles, blemishes, redness, cloudiness, darkening, or yellowing, then based on the subject image and the first prediction information, the prediction system can output an image showing the subject's future skin appearance as a prediction image. For example, if the subject image is a medical image showing the subject's current joint angles, and the first prediction information is information about the subject's future joint angles, the prediction system can output the following two images as prediction images: (1) a medical image showing the subject's current joint angles and, based on the first prediction information, a medical image showing the subject's future joint angles can be output as a prediction image; (2) an appearance image showing the subject's current joint appearance and, based on the first prediction information, an image showing the subject's future joint appearance can be output as a prediction image.

[0025] (Configuration of prediction system 100) Next, the configuration of a prediction system 100, which includes a prediction device 1 that acquires subject images and first prediction information, and generates and outputs a predicted image from the subject images based on the first prediction information, will be described with reference to Figure 1. In one embodiment of this disclosure, the prediction device 1 of the prediction system 100 can function as the prediction system described above on its own. Figure 1 is a block diagram showing an example configuration of the prediction system 100 in a medical facility 5 in which the prediction device 1 has been introduced.

[0026] As shown in Figure 1, the prediction system 100 comprises a prediction device 1 and one or more terminal devices 2 that are communicatively connected to the prediction device 1. Thus, the prediction system 100 may also comprise a prediction device 1 and a device (for example, terminal device 2) capable of displaying the prediction image output from the prediction device 1.

[0027] Prediction device 1 is a computer that acquires subject images and first prediction information, generates and outputs a predicted image from the subject image based on the first prediction information, and transmits the predicted image to terminal device 2. Prediction device 1 may be connected to the LAN of medical facility 5, as shown in Figure 1. The configuration of prediction device 1 will be described later.

[0028] Terminal device 2 receives a predicted image from prediction device 1 and displays the predicted image. Terminal device 2 may be a computer or the like used by medical personnel such as doctors belonging to medical facility 5. Terminal device 2 may be connected to the LAN of medical facility 5, as shown in Figure 1. Terminal device 2 may be, for example, a personal computer, a tablet terminal, a smartphone, etc. Terminal device 2 has a communication unit for sending and receiving data with other devices, an input unit such as a keyboard and microphone, a display unit capable of displaying the predicted image, etc.

[0029] In the prediction system 100, the prediction device 1 and the terminal device 2 are provided as separate components, but the prediction device 1 and the terminal device 2 may be integrated. For example, the prediction device 1 may have a display unit capable of displaying a prediction image, thereby providing the functions of the terminal device 2.

[0030] The prediction system 100 may further include a first prediction information management device 3, a subject image management device 4, and an electronic medical record management device 9.

[0031] The first prediction information management device 3 is a computer that functions as a server for managing the first prediction information. The first prediction information management device 3 may be connected to the LAN of the medical facility 5, as shown in Figure 1. In this case, the prediction device 1 may obtain the first prediction information of the subject from the first prediction information management device 3.

[0032] The subject image management device 4 is a computer that functions as a server for managing subject images. The subject image management device 4 may contain images of a subject who has received an examination regarding the condition of a target area at the medical facility 5. In one example, the subject image may be a medical image taken within the medical facility 5. The subject image management device 4 may be connected to an imaging device, such as an X-ray imaging device, within the medical facility 5 in a communicative manner. In this case, the subject image management device 4 may record images taken by the imaging device, for example, via a LAN. The subject image management device 4 may be connected to the LAN of the medical facility 5, as shown in Figure 1. In this case, the prediction device 1 may acquire subject images from the subject image management device 4.

[0033] The electronic medical record management device 9 is a computer that functions as a server for managing the electronic medical record information of patients who have received medical treatment at the medical facility 5. The electronic medical record management device 9 may be connected to the LAN of the medical facility 5, as shown in Figure 1. In this case, the prediction device 1 may obtain basic information related to the patient from the electronic medical record management device 9. The basic information is information contained in the electronic medical record information and may include at least one of the patient's gender, age, height, weight, and information indicating the condition of the target body part at the first point in time.

[0034] Figure 1 shows an example in which a local area network (LAN) is installed within a medical facility 5, and the prediction device 1, terminal device 2, first prediction information management device 3, patient image management device 4, and electronic medical record management device 9 are connected to the LAN, but is not limited to this example. For example, the network within the medical facility 5 may be the internet, a telephone communication line network, a fiber optic communication network, a cable communication network, or a satellite communication network. The LAN within the medical facility 5 may be connected to an external communication network in a communicative manner. In this case, for example, terminal device 2 may be a computer used by a patient.

[0035] In the prediction system 100, the prediction device 1 and at least one of the terminal device 2, the first prediction information management device 3, the subject image management device 4, and the electronic medical record management device 9 may be directly connected without using a LAN. Furthermore, the number of terminal devices 2, the first prediction information management device 3, the subject image management device 4, and the electronic medical record management device 9 that can communicate with the prediction device 1 may be multiple. Additionally, multiple prediction devices 1 may be installed in the prediction system 100.

[0036] (Configuration of prediction system 100a) The prediction device 1 may not be a computer installed in a predetermined medical facility 5, but may be connected to a LAN installed in each of the multiple medical facilities 5 via a communication network 6. Figure 2 is a block diagram showing an example configuration of a prediction system 100a according to another aspect of the present disclosure.

[0037] The prediction system 100a shown in Figure 2 comprises a medical facility 5a, a medical facility 5b, a prediction device 1 that is communicatively connected to each device in medical facilities 5a and 5b via a communication network 6, and a first prediction information management device 3 that is communicatively connected to the prediction device 1.

[0038] Medical facility 5a is equipped with a terminal device 2a, a patient image management device 4a, and an electronic medical record management device 9a, all of which are connected via communication. Medical facility 5b, on the other hand, is equipped with a terminal device 2b, a patient image management device 4b, and an electronic medical record management device 9b, all of which are connected via communication. In the following, unless otherwise specified, terminal devices 2a and 2b, and medical facilities 5a and 5b will be referred to as "terminal device 2" and "medical facility 5," respectively.

[0039] Figure 2 shows an example where the LANs of medical facilities 5a and 5b are connected to the communication network 6. The prediction device 1 only needs to be connected to the devices within each medical facility via the communication network 6 and is not limited to the configuration shown in Figure 2. For example, the prediction device 1 and the first prediction information management device 3 may be installed within medical facility 5a or medical facility 5b.

[0040] A prediction system 100a employing such a configuration may include a first prediction information management device 3a installed in medical facility 5a and a first prediction information management device 3b installed in medical facility 5b. In this case, the prediction device 1 can acquire first prediction information and subject images of subject Pa from the first prediction information management device 3a and subject image management device 4a in medical facility 5a, respectively. The prediction device 1 can then transmit a predicted image predicting the state of the subject body part of subject Pa to a terminal device 2a installed in medical facility 5a. The prediction device 1 can acquire first prediction information and subject images of subject Pb from the first prediction information management device 3b and subject image management device 4b in medical facility 5b, respectively. The prediction device 1 can then transmit a predicted image predicting the state of the subject body part of subject Pb to a terminal device 2b installed in medical facility 5b.

[0041] In this case, the first prediction information and subject image for each subject should include identification information unique to each medical facility 5 that has been assigned to each subject, and identification information unique to each subject that has been assigned to each subject. The identification information unique to each medical facility 5 may be, for example, a facility ID. The identification information unique to each subject may be, for example, a patient ID. Based on this identification information, the prediction device 1 can correctly transmit the prediction image predicting the state of the subject's target area to the terminal device 2 of each medical facility 5 where the subject was examined.

[0042] (Configuration of prediction systems 100 and 100a) Next, the configuration of the prediction systems 100 and 100a will be explained using Figure 3. Figure 3 is a block diagram showing an example of the configuration of the prediction systems 100 and 100a according to one aspect of this disclosure. For the sake of convenience of explanation, components having the same function as those already described will be denoted by the same reference numerals, and their descriptions will not be repeated.

[0043] The prediction systems 100 and 100a shown in Figure 3 include a prediction device 1, one or more terminal devices 2 connected to the prediction device 1 in a communicative manner, a first prediction information management device 3, and a subject image management device 4.

[0044] (Configuration of prediction device 1) The prediction device 1 comprises a control unit 7 that comprehensively controls each part of the prediction device 1, and a storage unit 8 that stores various data used by the control unit 7. The control unit 7 includes a prediction information acquisition unit 71, a prediction image generation unit 72, and an output control unit 73. The storage unit 8 stores a control program 81, which is a program for performing various controls on the prediction device 1.

[0045] <Predictive Information Acquisition Unit 71> The prediction information acquisition unit 71 acquires subject images from the subject image management device 4 and first prediction information from the first prediction information management device 3. The subject images and first prediction information are input data that is input to the prediction image generation unit 72.

[0046] The subject image and first predictive information will be explained using several diseases as examples. For example, if the disease is obesity, the subject image may be an image of the subject's whole body or abdomen at the time of study, and the first predictive information may be information on the subject's weight, BMI, waist circumference, visceral fat mass, blood pressure, blood glucose level, lipids, uric acid level, or liver function values. For example, if the disease is alopecia, the subject image may be an image of the subject's whole body or head at the time of study, and the first predictive information may be information on the subject's hair count, sex hormone levels, Norwood classification, or Ludwig classification. For example, if the disease is cataract, the subject image may be an image of the subject's head (face) or eyes at the time of study, and the first predictive information may be information on the subject's visual acuity, visual field, degree of clouding of the lens of the eye, or Emery-Little classification. For example, if the disease is periodontal disease, the subject image may be an image of the subject's head (face) or oral cavity at present, and the first predictive information may be information regarding the degree of pain in the subject's teeth or gums, the degree of swelling of the teeth or gums, the number of remaining teeth, the gingivitis index, or the depth of the periodontal pocket. If the disease is periodontal disease, the subject image may be an image of an open mouth or an image of a closed mouth. For example, if the disease is rheumatoid arthritis, the subject image may be an image of the subject's whole body, upper limb, or lower limb at present, and the first predictive information may be information regarding the degree of pain, the degree of swelling, joint angles, range of motion, Larsen classification, or Stein-Blocker classification of the subject's whole body, upper limb, or lower limb. For example, if the disease is Heberden's nodes, the subject image may be an image of the subject's hand at present, and the first predictive information may be information regarding the degree of pain, the degree of swelling, or the range of motion of the subject's hand. For example, if the disease is hallux valgus, the subject image is an image of the subject's foot at present, and the first predictive information may be information regarding the degree of pain, degree of swelling, range of motion of the joint, HV angle, or M1-M2 angle of the subject's foot.For example, if the disease is osteoarthritis, the subject image may be an image of the subject's whole body, upper limbs, or lower limbs in their current state, and the first predictive information may be information regarding the degree of pain, degree of swelling, joint angles, range of motion, or KL classification of the subject's whole body, upper limbs, or lower limbs. For example, if the disease is degenerative spondylosis, the subject image may be an image of the subject's whole body, neck, chest, or lumbar region in their current state, and the first predictive information may be information regarding the degree of spinal curvature, range of motion, or KL classification of the subject. For example, if the disease is a compression fracture, the subject image may be an image of the subject's whole body or lumbar region in their current state, and the first predictive information may be information regarding the degree of spinal curvature, range of motion, or KL classification of the subject. For example, if the disease is sarcopenia, the subject image may be an image of the subject's whole body, upper limbs, or lower limbs in their current state, and the first predictive information may be information regarding the subject's muscle mass. The subject image may be a medical image taken during the treatment of each disease. For example, if the disease is osteoarthritis of the knee, the subject image may be an X-ray image showing the subject's knee joint at present, and the first predictive information may be information about the angle between the subject's tibia and femur two years later.

[0047] <Predictive image generation unit 72> The predictive image generation unit 72 generates and outputs a predictive image that predicts the state of the target area at a second time point, based on the first predictive information, from the subject image. The predictive image generation unit 72 may generate an image that mimics at least a part of the subject image used to generate the predictive image. The predictive image generated by the predictive image generation unit 72 may also be an image that shows the effect of a disease occurring in the target area on that area. The generated predictive image may include images related to the subject's body part that has not changed from the first time point to the second time point. That is, the predictive image may include images related to body parts that have changed from the first time point to the second time point, and images related to body parts that have not changed from the first time point to the second time point.

[0048] The predictive image generation unit 72 may have any known image editing and video editing functions. In this case, the predictive image generation unit 72 generates a predictive image by converting the subject image into an editable file format and then making changes to the subject image based on the first predictive information. For example, if the subject image is an image of the subject's lower limbs at present, and the first predictive information is information about the angle between the subject's tibia and femur two years later, the predictive image generation unit 72 first converts the subject image into a predetermined file format. The predictive image generation unit 72 then modifies the angle between the tibia and femur in the subject image after the file format conversion based on the first predictive information and generates a predictive image.

[0049] The predictive image generation unit 72 may have a predictive image generation model capable of generating a predicted image using the subject image and the first prediction information. Here, the predictive image generation model may be a neural network trained using multiple image data showing the target area as training data. For example, a convolutional neural network (CNN), a generative adversarial network (GAN), or an autoencoder may be applied as the predictive image generation model.

[0050] The predictive image generation unit 72 inputs the subject image and first predictive information into the predictive image generation model and outputs a predictive image. The predictive image generation unit 72 outputs the predictive image output from the predictive image generation model (i.e., the image generated by the predictive image generation unit 72).

[0051] The predictive image generation model is a computational model used by the predictive image generation unit 72 when performing calculations based on the input data. The predictive image generation model is generated by performing machine learning, as described later, on the neural network of the predictive image generation unit 72.

[0052] <Output control unit 73> The output control unit 73 transmits the predicted image output from the predicted image generation unit 72 to the terminal device 2. The output control unit 73 may also transmit to the terminal device 2, along with the predicted image, at least one of the subject image and the first prediction information used to generate the predicted image.

[0053] The prediction device 1 may be configured to include a display unit (not shown). In this case, the output control unit 73 may cause the display unit to display the prediction image. In this case, the output control unit 73 may also cause the display unit to display at least one of the subject image used to generate the prediction image and the first prediction information along with the prediction image.

[0054] By including a predictive image generation unit 72 with a predictive image generation model, the prediction systems 100 and 100a can generate and output realistic predictive images that reflect the state of the target area at a second time point in time onto the subject's image. This allows the prediction systems 100 and 100a to clearly recognize the state of their target area at a second time point in time.

[0055] In one example, a pre-trained predictive image generation model may be pre-installed in the prediction device 1. Alternatively, the prediction device 1 may further include a first learning unit 74 that performs training processing for the predictive image generation unit 72.

[0056] <Learning Section 1, 74> The first learning unit 74 controls the learning process for the neural network of the predictive image generation unit 72.

[0057] (Training process for predictive image generation model) The learning process for generating a predictive image generation model using a Generative Adversarial Network (GAN) will be explained below with reference to Figure 4. Figure 4 shows an example of the neural network configuration of the predictive image generation unit 72.

[0058] As shown in Figure 4, the predictive image generation model applying a generative adversarial network has two networks: a generator network (hereinafter referred to as generator 721) and a discriminator network (hereinafter referred to as discriminator 722). The generator 721 can generate an image that looks like a real image as a predicted image from the first prediction information and the subject image. On the other hand, the discriminator 722 can distinguish between image data from the generator 721 (fake image) and a real image from the first training dataset 82, which will be described later.

[0059] First, the first learning unit 74 acquires the subject image and first prediction information from the memory unit 8 and inputs them to the generator 721.

[0060] The generator 721 generates candidate predicted images (fake images) from the subject images and the first prediction information. The generator 721 may also generate candidate predicted images by referring to real images included in the first training dataset 82.

[0061] Here, the first training dataset 82 is data used for machine learning to generate a predictive image generation model. The first training dataset 82 may include any real images that the generator 721 aims to reproduce as faithfully as possible. For example, the first training dataset 82 may include real medical images taken in the past. Here, the medical images may include, for example, at least one of X-ray image data, CT image data, MRI image data, PET image data, and ultrasound image data taken of each target area of ​​multiple patients.

[0062] The first training dataset 82 may include first training data and first training data. The first training data is, for example, data of the same type as the subject images and data of the same type as the first prediction information. "Data of the same type as the subject images" means image data of the same subject area as the subject image, taken from the same angle, and means image data of the same type, such as medical images or appearance images. "Data of the same type as the first prediction information" means information related to the shape and appearance of the same subject area related to the same disease, if the first prediction information is information related to the shape and appearance of the subject area related to the disease. The first training data is data of the same type as the prediction images, and is data of the same person taken at a later time than the first training data. The first training data is data related to the first training data, "data of the same type as the first prediction information". "Data of the same type as the prediction images" means image data of the same subject area as the prediction image, taken from the same angle, and means image data of the same type, such as medical images or appearance images.

[0063] Next, the first learning unit 74 inputs the predicted image candidates generated by the generator 721 and the real images included in the first training dataset 82 to the classifier 722.

[0064] The classifier 722 takes real images from the first training dataset 82 and predicted image candidates generated by the generator 721 as input, and outputs the probability that each image is a real image.

[0065] The first learning unit 74 calculates a classification error that represents how correct the probability output by the classifier 722 is. The first learning unit 74 repeatedly improves the classifier 722 and the generator 721 using backpropagation. At this time, the weights and biases of the classifier 722 are updated to minimize the classification error (i.e., to maximize the classification performance). On the other hand, the weights and biases of the generator 721 are updated to maximize the classification error (i.e., to maximize the probability that the classifier 722 mistakes a predicted image candidate for a real image).

[0066] The first learning unit 74 updates the weights and biases of the classifier 722 and the generator 721 until the probability output by the classifier 722 meets a predetermined standard. As a result, the predictive image generation unit 72 can generate predictive images that are indistinguishable from real images.

[0067] (Processing performed by prediction systems 100 and 100a) The following describes the processing flow performed by the prediction systems 100 and 100a using Figure 5. Figure 5 is a flowchart showing an example of the processing flow performed by the prediction systems 100 and 100a according to this embodiment.

[0068] First, in step S1, the prediction information acquisition unit 71 acquires the subject image and the first prediction information (input data) (prediction information acquisition step).

[0069] Next, in response to the input of the subject image and the first prediction information, the prediction image generation unit 72 generates a prediction image in step S2 and outputs the prediction image (prediction image generation step).

[0070] [Embodiment 2] Other embodiments of this disclosure are described below. For convenience of explanation, components having the same function as those described in the above embodiments are denoted by the same reference numerals, and their descriptions are not repeated.

[0071] The prediction systems 100 and 100a according to the above-described embodiment include a prediction device 1 that acquires first prediction information from a first prediction information management device 3, but are not limited to this. For example, the prediction system may be configured such that the first prediction information is generated by the prediction device 1A. The configuration of a prediction system 100 and 100a that includes such a prediction device 1A will be explained with reference to Figure 6. Figure 6 is a block diagram showing an example of the configuration of a prediction system 100 and 100a according to another aspect of this disclosure.

[0072] (Configuration of prediction device 1A) The prediction device 1A includes a control unit 7A that comprehensively controls each part of the prediction device 1A, and a storage unit 8 that stores various data used by the control unit 7A. In addition to a prediction information acquisition unit 71, a prediction image generation unit 72, and an output control unit 73, the control unit 7A further includes a prediction information generation unit 75.

[0073] Figure 6 shows an example in which the prediction device 1A includes a first learning unit 74, but is not limited to this. In one example, a trained prediction image generation model may be pre-installed in the prediction device 1A.

[0074] <Predictive Information Generation Unit 75> The prediction information generation unit 75 generates first prediction information regarding the target body part at a second time point, after a predetermined period has elapsed from the first time point, from a subject image of the subject showing the target body part at the first time point, and outputs the first prediction information to the prediction information acquisition unit 71.

[0075] The prediction information generation unit 75 may have a prediction information generation model capable of estimating first prediction information from the subject's image. The prediction information generation model is a model capable of estimating first prediction information from the subject's image and the subject's basic information. Here, the prediction information generation model may be a neural network trained using patient information about patients with diseases of the target area as training data. For example, a convolutional neural network (CNN), a recurrent neural network (RNN), and LSTM (Long Short-Term Memory) may be applied as the prediction information generation model.

[0076] Patient information includes, for example, status information indicating the condition of a target area for each patient, acquired at multiple points in the past, and the status information for each patient is associated with information indicating the time when the status information was acquired.

[0077] The prediction information generation unit 75 inputs data related to the subject image into the prediction information generation model and outputs first prediction information. The prediction information generation unit 75 outputs the first prediction information output from the prediction information generation model (i.e., generated by the prediction information generation unit 75).

[0078] The predictive information generation model is a computational model used by the predictive information generation unit 75 when performing calculations based on input data. The predictive information generation model is generated by performing machine learning, as described later, on the neural network of the predictive information generation unit 75.

[0079] Below, we will further explain the configuration of the prediction information generation unit 75 using Figure 7, taking the case where a neural network is applied as the prediction information generation model as an example. Figure 7 shows an example of the configuration of the neural network in the prediction information generation unit.

[0080] As shown in Figure 7, the prediction information generation unit 75 comprises an input layer 751 and an output layer 752. The prediction information generation unit 75 performs calculations based on a prediction information generation model on the input data input to the input layer 751 and outputs prediction information from the output layer 752.

[0081] The prediction information generation unit 75 in Figure 7 includes a neural network with an input layer 751 and an output layer 752. The neural network can be any neural network suitable for handling time-series information. For example, an LSTM would be suitable. The neural network may also be a neural network suitable for handling time-series information and location information in combination. For example, a ConvLSTM network combining a CNN and an LSTM would be suitable. The input layer 751 can extract time-varying features of the input data. The output layer 752 can calculate new features based on the features extracted by the input layer 751, the time-varying of the input data, and initial values. The input layer 751 and the output layer 752 each have multiple LSTM layers. Each of the input layer 751 and the output layer 752 may have three or more LSTM layers.

[0082] The input data input to the input layer 751 may, for example, be parameters representing feature quantities extracted from a subject image showing the subject's body part at a first time point. In this case, the prediction information generation unit 75 can output first prediction information regarding the subject's body part at a second time point, after a predetermined period has elapsed from the first time point.

[0083] The prediction information generation unit 75 outputs, as first prediction information, for example, a prediction result of the onset or progression of a disease in the target area of ​​the subject at a second time point, after a predetermined period has elapsed from the first time point. Specifically, as first prediction information, the prediction information generation unit 75 outputs, for example, the degree of symptoms of each disease in the subject at the second time point, the classification of each disease, and information indicating when invasive treatment will be necessary for the target area. The first prediction information shown here is just an example and is not limited to these.

[0084] Furthermore, the prediction information generation unit 75 may output information indicating the subject's QOL as third prediction information based on the first prediction information described above. Specifically, the prediction information generation unit 75 outputs at least one of the following as third prediction information: information regarding pain occurring in the subject's target area, information regarding the subject's catastrophic thinking, information regarding the subject's motor skills, information indicating the subject's satisfaction with life, and information regarding the degree of stiffness in the subject's target area.

[0085] Information indicating the quality of life of the subjects must include at least one of the following: • Information regarding pain occurring in the affected area of ​​the subject. • Information regarding the subject's catastrophic thinking. • Information regarding the athletic abilities of the subjects • Information indicating the life satisfaction level of the aforementioned subjects.

[0086] Information indicating the quality of life of the subject may include information regarding the subject's (1) physical function, (2) daily role functioning (physical), (3) physical pain, (4) overall health perception, (5) vitality, (6) social functioning, (7) daily role functioning (mental), and (8) mental health.

[0087] Information related to QOL may include, for example, data from SF-36 (36-Item Short-Form Health Survey), VAS (Visual Analog Scale), NEI VFQ-25 (The 25-item National Eye Institute Visual Function Questionnaire), GOHAI (General Oral Health Assessment Index), WOMAC (Western Ontario and McMaster Universities Osteoarthritis Index), and RDQ (Roland-Morris Disability Questionnaire).

[0088] The prediction information generation unit 75 only needs to generate at least a portion of the first prediction information that the prediction image generation unit 72 uses to generate a prediction image. In that case, the remaining first prediction information may be acquired by the prediction information acquisition unit 71 from the first prediction information management device 3.

[0089] (Processing performed by prediction systems 100 and 100a) The following describes the processing flow performed by the prediction systems 100 and 100a of the second embodiment, using Figure 8 as a reference. Figure 8 is a flowchart showing an example of the processing flow performed by the prediction systems 100 and 100a of the second embodiment.

[0090] First, in step S11, the predictive information generation unit 75 acquires an image of the subject (input data) (image acquisition step).

[0091] Next, in response to the input of the subject image, the prediction information generation unit 75 generates first prediction information in step S12 and outputs the first prediction information to the prediction information acquisition unit 71 (first prediction step).

[0092] Subsequently, the prediction information acquisition unit 71 inputs (a) the first prediction information acquired from the prediction information generation unit 75 and (b) the subject image management device 4, either acquired before the acquisition of the first prediction information or acquired simultaneously with or after the acquisition of the first prediction information (input data), to the prediction image generation unit 72 (not shown).

[0093] Next, in response to the input of the subject image and the first prediction information, the prediction image generation unit 72 generates a prediction image in step S13 and outputs the prediction image (prediction image generation step).

[0094] [Variation] In prediction device 1A, the prediction information generation unit 75 generated first prediction information based on the subject image. In contrast, in prediction device 1B according to this modified embodiment 2, the prediction information generation unit 75B generates first prediction information based on basic information in addition to the subject image. The configuration of prediction systems 100 and 100a equipped with such prediction device 1B will be explained with reference to Figure 9. Figure 9 is a block diagram showing a modified configuration of prediction systems 100 and 100a according to another aspect of the present disclosure. Prediction device 1B further includes a prediction information generation unit 75B in the control unit 7B.

[0095] The prediction device 1B includes a control unit 7B that comprehensively controls each part of the prediction device 1B, and a storage unit 8B that stores various data used by the control unit 7B. In addition to the prediction information acquisition unit 71, the prediction image generation unit 72, and the output control unit 73, the control unit 7B further includes a prediction information generation unit 75B and a basic information acquisition unit 76.

[0096] With the configuration described above, the prediction device 1B can generate information about symptoms that are closer to the symptoms that may occur or have occurred in the target area at the second time point, i.e., more accurate first prediction information, based on the subject image captured at the first time point. As a result, the prediction device 1B can generate a prediction image that shows symptoms that are closer to the symptoms that may occur or have occurred in the target area at the second time point, i.e., a prediction image that shows more accurate prediction information.

[0097] <Basic information acquisition section 76> The basic information acquisition unit 76 acquires basic information, which is information related to the subject, from the electronic medical record management device 9. The electronic medical record management device 9 is a computer that functions as a server for managing the electronic medical record information of subjects who have received medical treatment at medical facility 5 or a medical facility other than medical facility 5. The electronic medical record information may include the subject's basic information and interview information. The basic information is input data that is input to the predictive information generation unit 75B in addition to the subject's image.

[0098] Basic information includes at least one of the following: the subject's sex, age, height, weight, and information indicating the condition of the subject's body part at the first point in time. Basic information may further include at least one of the following: the subject's BMI, race, occupational history, exercise history, medical history of diseases related to the body part, information regarding the shape and appearance of the body part, biomarker information, and genetic information. Basic information may also include, for example, information such as the severity of symptoms of diseases related to the subject's body part. Basic information may also include, for example, information contained in the subject's electronic medical record. Basic information may also be interview information obtained from the subject through an interview conducted at a medical facility 5, and may include, for example, information related to the subject's quality of life at the first point in time.

[0099] With the above configuration, the prediction device 1B can acquire basic information about the subject from the electronic medical record management device 9 in addition to the subject's image, and transmit a predicted image predicting the condition of the subject's target area Pa to the terminal device 2a installed in the medical facility 5a. The prediction device 1B can acquire basic information about the subject from the electronic medical record management device 9 in addition to the subject's image, and transmit a predicted image predicting the condition of the subject's target area Pb to the terminal device 2b installed in the medical facility 5b.

[0100] In one example, a pre-trained predictive information generation model may be pre-installed in the predictive device 1B. Alternatively, the predictive device 1B may further include a second learning unit 77 that performs the learning process for the predictive information generation unit 75B.

[0101] <Second Learning Department 77> The second learning unit 77 controls the learning process for the neural network of the predictive information generation unit 75B. The second learning dataset 83, which will be described later, is used for this learning. A specific example of the learning performed by the second learning unit 77 will be described later.

[0102] (Training process for predictive information generation models) The learning process for generating a predictive information generation model using a neural network will be explained below with reference to Figure 10. Figure 10 is a flowchart showing an example of the learning process flow of the neural network in the predictive information generation unit 75B.

[0103] First, the second learning unit 77 retrieves the second training data contained in the second training dataset 83 from the storage unit 8B (step S21). The second training data contains patient images of multiple patients.

[0104] Next, the second learning unit 77 determines a patient (step S22).

[0105] Next, the second learning unit 77 inputs a patient image of a certain patient at time A, which is included in the second learning data, into the input layer 751 (step S23). The input layer 751 may extract parameters representing features from the input patient image.

[0106] Next, the second learning unit 77 obtains output data from the output layer 752 regarding the symptoms of a target area of ​​a patient (step S24). This output data contains the same content as the second training data.

[0107] Next, the second learning unit 77 acquires the second training data included in the second training dataset 83. Then, the second learning unit 77 compares the acquired output data with state information included in the second training data, which indicates the state of a target area of ​​a patient at time point B, and calculates the error (step S25).

[0108] Subsequently, the second learning unit 77 adjusts the predictive information generation model so that the error is reduced (step S26).

[0109] Any known method can be applied to adjust the predictive information generation model. For example, backpropagation may be used as a method for adjusting the predictive information generation model. The adjusted predictive information generation model becomes the new predictive information generation model, and in subsequent calculations, the predictive information generation unit 75B uses the new predictive information generation model. During the adjustment stage of the predictive information generation model, the parameters used by the predictive information generation unit 75B may be adjusted.

[0110] The parameters include, for example, the parameters used in the input layer 751 and the output layer 752. Specifically, the parameters include the weighting coefficients used in the LSTM layers of the input layer 751 and the output layer 752. The parameters may also include filter coefficients.

[0111] If the error is not within a predetermined range, or if patient images of all patients included in the second training dataset 83 have not been input (NO in step S27), the second learning unit 77 changes the patient (step S28), returns to step S23, and repeats the learning process. If the error is within a predetermined range, or if patient images of all patients included in the second training dataset 83 have been input (YES in step S27), the second learning unit 77 terminates the learning process.

[0112] <Second training dataset 83> The second training dataset 83 is data used for machine learning to generate a predictive information generation model. The second training dataset 83 may include patient information relating to patients with diseases of the target area. Here, patient information may include state information indicating the state of the target area for each patient, acquired at multiple points in the past, and the state information for each patient may be associated with information indicating the time when the state information was acquired. The second training dataset 83 includes second training data for calculating the error between the second training data used as input data and the first predictive information output by the predictive information generation unit 75B.

[0113] The second training data may include, for example, image data of the target area for each of multiple patients. The image data used as the second training data may be image data of the whole body, upper body, lower body, upper limbs, and lower limbs for each of multiple patients. The image data used as the second training data may be medical image data of the target area for each of multiple patients. The medical image data may include, for example, at least one of X-ray image data, CT image data, MRI image data, PET image data, and ultrasound image data of the target area for each of multiple subjects. The second training data is of the same type as the subject images. The second training data is of the same type as the first prediction information.

[0114] The second training data may include state information indicating the condition of each patient's target area at the time the patient image was taken, and symptom information relating to the target area. Here, the state information may include information regarding the progression of symptoms in the target area. Also, here, the symptom information may include information regarding the onset time of the disease in the target area.

[0115] The second training dataset 83 may be a dataset in which the second training data and the second training data are combined. In other words, the second training dataset 83 may be a time-series dataset in which patient images acquired from each of multiple patients at multiple points in the past are associated with state information indicating the state of the target area at the time the patient image was taken. For example, the second training dataset 83 may include parameters that show features extracted from the following information at a certain point in time and one year after that point in time.

[0116] For example, if the disease is obesity, the second training dataset 83 may include weight, BMI, waist circumference, visceral fat mass, blood pressure, blood glucose level, lipid levels, uric acid level, or liver function values. For example, if the disease is alopecia, the second training dataset 83 may include hair count, sex hormone levels, Norwood classification, or Ludwig classification. For example, if the disease is cataract, the second training dataset 83 may include visual acuity, visual field, recommended degree of ocular opacity, or Emery-Little classification. For example, if the disease is periodontal disease, the second training dataset 83 may include the degree of tooth or gum pain, the degree of tooth or gum swelling, number of remaining teeth, gingivitis index, or periodontal pocket depth. For example, if the disease is rheumatoid arthritis, the second training dataset 83 may include the degree of pain, degree of swelling, joint angles, range of motion, Larsen classification or Stein-Blocker classification, etc., for the subject's whole body, upper limbs, or lower limbs. For example, if the disease is Heberden's nodes, the second training dataset 83 may include the degree of pain, degree of swelling, or range of motion, etc., for the subject's hands. For example, if the disease is hallux valgus, the second training dataset 83 may include the degree of pain, degree of swelling, range of motion, HV angle or M1-M2 angle, etc., for the subject's feet. For example, if the disease is osteoarthritis, the second training dataset 83 may include the degree of pain, degree of swelling, joint angles, range of motion, degree of stiffness, thickness of joint cartilage, KL classification or presence or absence of limping, etc., for the subject's whole body, upper limbs, or lower limbs. For example, if the disease is degenerative spondylosis, the second training dataset 83 may include the degree of pain, the degree of spinal curvature, the range of motion of the spine, or the KL classification. For example, if the disease is a compression fracture, the second training dataset 83 may include the degree of pain or the range of motion of the spine. For example, if the disease is sarcopenia, it may include muscle mass, walking speed, or grip strength.

[0117] Furthermore, the second training dataset 83 may include parameters indicating the attributes of the subject. These attributes may include, for example, the gender, age, height, and weight of each subject. If the second training dataset 83 is time-series data, the second learning unit 77 may use a subject image at a certain point in time as the second training data, and use a subject image taken a predetermined period later from that point in time, along with information about the symptoms of the subject area and information about the subject at the time the subject image was taken, as the second training data.

[0118] The second training dataset 83 may include time-series data containing information about the quality of life (QOL) of each of the multiple subjects. For example, it may include information such as SF-36 and VAS. The predictive information generation unit 75B, which has a predictive information generation model generated by machine learning using such a second training dataset 83, can also output information related to the subject's QOL at a second time point from the subject's image.

[0119] Specifically, the input data used during training of the predictive information generation unit 75B is (a) an image of the subject showing the subject's body part at a certain point in time A, which is included in the second training data. Based on the above input data, the predictive information generation unit 75B outputs first predictive information regarding the subject's body part at point B, which is a predetermined period (e.g., 3 years) after point A, as output data. Specifically, the predictive information generation unit 75B outputs information such as the angle around the subject's body part at point B, the degree of enlargement and reduction of the subject's body part, the degree of wrinkles and blemishes on the subject's body part, the timing and degree of pain on the subject's body part, and information indicating when invasive treatment will be necessary for the subject's body part. The output data shown here is just an example and is not limited to these.

[0120] When the predictive information generation unit 75B uses basic information as input data, the second learning unit 77 may, when learning the predictive information generation unit 75B, input to the predictive information generation unit 75B, in addition to the subject image showing the subject's target area at a certain point in time A, general subject symptom information and attribute information as second learning data.

[0121] (Prediction system 100, the process performed by 100) The following describes the processing flow performed by the prediction systems 100 and 100a, which are equipped with a prediction device 1B, using Figure 11 as an example. Figure 11 is a flowchart showing an example of the processing flow performed by the prediction systems 100 and 100a according to this embodiment.

[0122] First, in step S31, the predictive information generation unit 75B acquires a subject image (input data) and basic information (input data) (image and information acquisition step).

[0123] Next, in response to the input of the subject image and basic information, the prediction information generation unit 75B generates first prediction information in step S32 and outputs the first prediction information to the prediction information acquisition unit 71 (first prediction step).

[0124] Subsequently, the prediction information acquisition unit 71 inputs (a) the first prediction information acquired from the prediction information generation unit 75B and (b) the subject image management device 4's subject image (input data), which was acquired before the acquisition of the first prediction information, or which is acquired simultaneously with or after the acquisition of the first prediction information, to the prediction image generation unit 72 (not shown).

[0125] Next, in response to the input of the subject image and the first prediction information, the prediction image generation unit 72 generates a prediction image in step S33 and outputs the prediction image (prediction image generation step).

[0126] [Third Embodiment] Other embodiments of this disclosure are described below. For convenience of explanation, components having the same function as those described in the above embodiments are denoted by the same reference numerals, and their descriptions are not repeated.

[0127] The prediction systems 100 and 100a may include a function to output a predicted image that predicts the state of the target area at a second time point in the event of an intervention on the target area, as well as a method of intervention on the subject and the effect of the intervention. A prediction device 1C having such a function will be described with reference to Figure 12. Figure 12 is a block diagram showing an example of the configuration of the prediction systems 100 and 100a according to another aspect of this disclosure.

[0128] Here, we will explain specific examples of intervention methods. For example, if the disease is obesity, intervention methods may include lifestyle guidance, dietary therapy, drug therapy, exercise therapy, and surgical therapy (liposuction, gastrectomy, gastric banding, etc.). For example, if the disease is alopecia, intervention methods may include lifestyle guidance, dietary therapy, drug therapy, surgical therapy (hair transplantation), and wearing a wig. For example, if the disease is cataracts, intervention methods may include drug therapy, exercise therapy, and surgical therapy (cataract removal, intraocular lens implantation, etc.). For example, if the disease is periodontal disease, intervention methods may include oral care guidance, drug therapy, orthodontic therapy, surgical therapy (periodontal plastic surgery, implant treatment, etc.), and the use of dentures. For example, if the disease is rheumatoid arthritis, intervention methods may include drug therapy and surgical therapy (osteotomy, joint replacement). For example, if the disease is Heberden's nodes, intervention methods may include drug therapy, etc. For example, if the condition is hallux valgus, intervention methods may include shoe guidance, exercise therapy, orthotic therapy, drug therapy, and surgical therapy (osteotomy, fusion, joint replacement, etc.). For example, if the condition is osteoarthritis, intervention methods may include exercise therapy, orthotic therapy, drug therapy, rehabilitation, and surgical therapy (intra-articular injection, arthroscopic surgery, osteotomy, fusion, joint replacement, etc.). For example, if the condition is degenerative spondylosis, intervention methods may include exercise therapy, orthotic therapy, drug therapy, and surgical therapy (spinal instrumentation surgery, etc.). For example, if the condition is compression fracture, intervention methods may include orthotic therapy, drug therapy, and surgical therapy (spinal instrumentation surgery, etc.). For example, if the condition is sarcopenia, intervention methods may include lifestyle guidance, dietary therapy, drug therapy, and exercise therapy.

[0129] (Configuration of prediction device 1C) The prediction device 1C includes a control unit 7C that comprehensively controls each part of the prediction device 1, and a storage unit 8C that stores various data used by the control unit 7C. The control unit 7C includes a prediction information acquisition unit 71, a prediction image generation unit 72C, an output control unit 73C, a first learning unit 74, a prediction information generation unit 75B, a basic information acquisition unit 76, and a second learning unit 77, as well as an intervention effect prediction unit 78 and a third learning unit 79.

[0130] Figure 12 shows a prediction device 1C comprising a first learning unit 74, a second learning unit 77, and a third learning unit 79, but is not limited to this. The prediction device 1C may include any (or all) of the first learning unit 74, the second learning unit 77, and the third learning unit 79, or it may not include any (or all) of them.

[0131] For example, the prediction device 1C does not need to include the first learning unit 74. In this case, it is sufficient that a trained prediction image generation model is pre-installed in the prediction device 1C. Alternatively, the prediction device 1C does not need to include the second learning unit 77. In this case, it is sufficient that a trained prediction information generation model is pre-installed in the prediction device 1C. Alternatively, the prediction device 1C does not need to include the third learning unit 79. In this case, it is sufficient that a trained intervention effect prediction model (described later) is pre-installed in the prediction device 1C.

[0132] The memory unit 8C may store a control program 81, a first training dataset 82, and a second training dataset 83, which are programs for performing various controls on the prediction device 1C, as well as a third training data 84 and intervention information 85, which will be described later.

[0133] Figure 12 shows a prediction device 1C in which a control program 81, a first learning dataset 82, a second learning dataset 83, a third training data 84, and intervention information 85 are stored in the memory unit 8C, but is not limited to this. The memory unit 8C of the prediction device 1C may store any (or all) of the control program 81, the first learning dataset 82, the second learning dataset 83, the third training data 84, and the intervention information 85, or it may not store any (or all) of them.

[0134] <Predictive image generation unit 72C> The predictive image generation unit 72C inputs the subject image, first predictive information, and second predictive information into the predictive image generation model and outputs a predictive image. The predictive image generation unit 72C outputs the predictive image output from the predictive image generation model (i.e., the image generated by the predictive image generation unit 72C).

[0135] <Output Control Unit 73C> The output control unit 73C transmits the predicted image output from the predicted image generation unit 72C to the terminal device 2. As shown in Figure 12, the output control unit 73C may also transmit to the terminal device 2, along with the predicted image, at least one of the subject image used to generate the predicted image, the first prediction information, and the second prediction information.

[0136] <Intervention effect prediction unit 78> The intervention effect prediction unit 78 outputs second prediction information indicating the method of intervention for the subject and the effect of the intervention, based on first prediction information regarding the target area at a second time point after a predetermined period has elapsed from the first time point. The intervention effect prediction unit 78 may have an intervention effect prediction model that can be estimated from the first prediction information.

[0137] The intervention effect prediction model is a calculation model used by the intervention effect prediction unit 78 when performing calculations based on the input data. The intervention effect prediction model is not particularly limited in its configuration, as long as it is a calculation model capable of estimating the second prediction information from the first prediction information.

[0138] The intervention effect prediction model may be a neural network, for example, a trained neural network having an input layer and an output layer. More specifically, the intervention effect prediction model may be a neural network trained using effect information as training data.

[0139] For example, the effect information includes status information indicating the condition of the target area for each patient, acquired at multiple points in the past, and the status information for each patient is associated with intervention information 85 indicating the intervention applied to each patient. The effect information may also include time-series data regarding the status information of each patient, acquired at multiple points in the past from each of the multiple patients to whom the intervention was applied in the past.

[0140] If the intervention effect prediction model is a trained neural network, the intervention effect prediction unit 78 performs calculations based on the intervention effect prediction model in response to the input of the first prediction information as input data to the input layer, and outputs the second prediction information as output data from the output layer.

[0141] The second predictive information includes, for example, information indicating the type of intervention and information indicating the effect of the intervention. The effect of the intervention refers to information describing the symptoms of the target area of ​​the subject at a second time point if the intervention is applied. Alternatively, the effect of the intervention may refer to information describing the extent to which the symptoms of the disease in the target area of ​​the subject at a second time point improve or the extent to which the progression of symptoms is suppressed compared to if the intervention is not applied. The second predictive information may also include information indicating when the intervention should be applied (intervention timing).

[0142] The intervention effect prediction unit 78 may, for example, be configured to extract features from the first prediction information and use them as input data. The following known algorithms can be applied to extract these features. • Convolutional neural network (CNN) • Autoencoder • Recurrent neural network (RNN) ·LSTM (Long Short-Term Memory).

[0143] The configuration of the intervention effect prediction unit 78 will be further explained below using Figure 7, with the example of the case where the intervention effect prediction unit 78 uses a neural network as the intervention effect prediction model. The configuration shown in Figure 7 is just one example, and the configuration of the intervention effect prediction unit 78 is not limited to this.

[0144] As shown in Figure 7, the intervention effect prediction unit 78 comprises an input layer 781 and an output layer 782. The intervention effect prediction unit 78 acquires first prediction information from the prediction information generation unit 75B and uses it as input data to be input to the input layer 781. The intervention effect prediction unit 78 may further acquire subject images and use them as input data. The intervention effect prediction unit 78 may acquire basic information from the basic information acquisition unit 76 and use this basic information as input data to be input to the input layer 781. The intervention effect prediction unit 78 performs calculations based on the intervention effect prediction model on the input data input to the input layer 781 and outputs a predicted image from the output layer 782.

[0145] As shown in Figure 7, the intervention effect prediction unit 78 includes a neural network from the input layer 781 to the output layer 782. The neural network can be any neural network suitable for handling time-series information. For example, an LSTM would be suitable. The neural network may also be a neural network suitable for handling time-series information and location information in combination. For example, a ConvLSTM network combining a CNN and an LSTM would be suitable. The input layer 781 can extract time-varying features of the input data. The output layer 782 can calculate new features based on the features extracted by the input layer 781, the time-varying of the input data, and the initial values. The input layer 781 and the output layer 782 each have multiple LSTM layers. Each of the input layer 781 and the output layer 782 may have three or more LSTM layers.

[0146] An intervention effect prediction model is generated by performing machine learning, as described later, on the neural network of the intervention effect prediction unit 78.

[0147] The input data input to the input layer 781 may, for example, be parameters representing feature quantities extracted from first prediction information regarding the target area at a second time point after a predetermined period has elapsed from the first time point. Alternatively, the input data may be information indicating the intervention method included in the intervention information 85 described later. When the intervention effect prediction unit 78 uses the intervention information 85 as input data, the intervention effect prediction unit 78 can select at least one intervention method from among those included in the intervention information 85 and output second prediction information predicting the effect of the intervention.

[0148] In response to the input data described above being input to the input layer 781, the output layer 782 outputs second predictive information indicating the method of intervention for the subject and the effect of said intervention.

[0149] The second prediction information may be, for example, information representing the degree to which the symptoms of the disease in the target area of ​​the subject will improve or the degree to which the progression of the symptoms will be suppressed if the intervention is applied at a second time point. More specifically, the intervention effect prediction unit 78 may output the following information as the second prediction information. • How close the angle around the target area approaches the normal angle. • How close do the enlargement and reduction of the target area come to normal size? • By what percentage will wrinkles and blemishes in the affected area be reduced? • How well will the condition of the target area be maintained? How much will walking ability (including the ability to climb stairs) improve?

[0150] The second predictive information is the same type of data as the first predictive information. For example, if the disease is obesity, the second predictive information may be information about the subject's weight, BMI, waist circumference, visceral fat mass, blood pressure, blood glucose level, lipid levels, uric acid level, or liver function values. For example, if the disease is alopecia, the second predictive information may be information about the subject's hair count, sex hormone levels, Norwood classification, or Ludwig classification. For example, if the disease is cataract, the second predictive information may be information about the subject's visual acuity, visual field, degree of clouding of the lens of the eye, or Emery-Little classification. For example, if the disease is periodontal disease, the second predictive information may be information about the subject's degree of tooth or gum pain, degree of tooth or gum swelling, number of remaining teeth, gingivitis index, or periodontal pocket depth. For example, if the disease is rheumatoid arthritis, the second predictive information may be information about the subject's whole body, upper or lower limb pain level, degree of swelling, joint angle, range of motion, Larsen classification, or Stein-Blocker classification. For example, if the disease is Heberden's nodes, the second predictive information may be information about the degree of pain, swelling, or range of motion of the subject's hand. For example, if the disease is hallux valgus, the second predictive information may be information about the degree of pain, swelling, range of motion, HV angle, or M1-M2 angle of the subject's foot. For example, if the disease is osteoarthritis, the second predictive information may be information about the degree of pain, swelling, joint angles, range of motion, or KL classification of the subject's whole body, upper limb, or lower limb. For example, if the disease is degenerative spondylosis, the second predictive information may be information about the degree of spinal curvature, range of motion, or KL classification of the subject. For example, if the disease is compression fracture, the second predictive information may be information about the degree of spinal curvature, range of motion, or KL classification of the subject. For example, if the disease is sarcopenia, the second predictive information may be information about the subject's muscle mass.

[0151] <Intervention information 85> Intervention information 85 is information about an intervention whose effect the intervention effect prediction unit 78 estimates. Examples of intervention information 85 whose effect is estimated include non-invasive treatments such as weight restriction, thermotherapy, ultrasound therapy, wearing of orthotic devices, or taking supplements. Intervention information 85 may also include the estimated effect of invasive treatments such as surgical therapy.

[0152] (Neural network learning process of the predictive image generation unit 72C) The learning process for generating a predictive image generation model using a Generative Adversarial Network (GAN) will be explained below with reference to Figure 13. Figure 13 shows an example of the neural network configuration of the predictive image generation unit 72C. As shown in Figure 13, the predictive image generation model using a Generative Adversarial Network has two networks: a generator 721C and a discriminator 722C.

[0153] First, the first learning unit 74 acquires subject images and first prediction information from the memory unit 8C and inputs them to the generator 721C. The first learning unit 74 also inputs the second prediction information generated by the intervention effect prediction unit 78 to the generator 721C.

[0154] The generator 721C generates candidate predicted images (fake images) from the subject image, first prediction information, and second prediction information. The generator 721C may also generate candidate predicted images by referring to real images included in the first training dataset 82.

[0155] Next, the first learning unit 74 inputs the predicted image candidates generated by the generator 721C and the real images included in the first training dataset 82 to the classifier 722C.

[0156] The classifier 722C takes real images from the first training dataset 82 and predicted image candidates generated by the generator 721C as input, and outputs the probability that each image is a real image.

[0157] The first learning unit 74 calculates a classification error that represents how correct the probability output by the classifier 722C is. The first learning unit 74 repeatedly improves the classifier 722C and the generator 721C using backpropagation.

[0158] The first learning unit 74 updates the weights and biases of the classifier 722C and the generator 721C until the probability output by the classifier 722C meets a predetermined standard. As a result, the predictive image generation unit 72C can generate predictive images that are indistinguishable from real images.

[0159] <3rd Learning Department 79> The third learning unit 79 controls the learning process for the neural network of the intervention effect prediction unit 78. The third training data 84 is used for this learning.

[0160] Here, the third training data 84 is data used for machine learning to generate an intervention effect prediction model. The third training data 84 includes third training data used as input data and third training data used to calculate the error between the third learning input data used as input data and the first prediction information output by the intervention effect prediction unit 78.

[0161] The third learning input data may include, for example, information indicating when the intervention was applied to each of several patients to whom the intervention was applied, patient images showing the target area of ​​each patient, and symptom information regarding the onset or progression of symptoms in the target area at the time the patient image of each patient was taken.

[0162] The third training data may include patient images showing the target area of ​​the patient at a later time (e.g., one year later) than when the patient images used in the third training input data were taken, and symptom information regarding the onset or progression of symptoms in the target area of ​​the patient. The third training data may also include symptom information regarding the target area of ​​each patient at the time the patient images were taken. Here, the symptom information may include information regarding the onset of the patient's disease or the progression of symptoms.

[0163] The third training data 84 may include status information indicating the state of the target area of ​​each patient, acquired at multiple points in the past, and may also be information that associates the status information for each patient with intervention information indicating the intervention applied to each patient, i.e., effect information. The third training data 84 may also be time-series data in which patient images acquired at multiple points in time from each of multiple patients to whom an intervention was applied in the past are associated with information regarding the symptoms of the target area at the time the patient image was taken.

[0164] <Training process for the intervention effect prediction model> The learning process for generating an intervention effect prediction model using a neural network will be explained below, referring to Figure 7 and using Figure 14. Figure 14 is a flowchart showing an example of the learning process flow of the neural network in the intervention effect prediction unit 78.

[0165] First, the third learning unit 79 obtains the third learning input data contained in the third training data 84 from the storage unit 8C (step S41). The third learning input data includes, for example, (a) information indicating the time when the intervention was applied to each of a plurality of patients to which the intervention was applied, (b) pixel data of patient images showing the target area of ​​each patient, and (c) symptom information regarding the onset or progression of symptoms in the target area at the time the patient image of each patient was taken.

[0166] Next, the third learning unit 79 determines a patient (step S42).

[0167] Next, the third learning unit 79 inputs the following into the input layer 781, which are included in the third learning input data: (a) information indicating the timing of the intervention for a patient to whom the intervention was applied, (b) pixel data of a patient image showing the target area of ​​a patient, and (c) symptom information regarding the onset or progression of symptoms in the target area at the time the patient image of a patient was taken (step S43).

[0168] Next, the third learning unit 79 obtains output data from the output layer 782, which is information indicating at least one of the method of intervention for a patient and the effect of said intervention (step S44). This output data contains the same content as the third training data.

[0169] Next, the third learning unit 79 acquires the third training data included in the third training data 84. Then, the third learning unit 79 compares the acquired output data with information included in the third training data that shows the method of intervention for a certain patient and the effect of said intervention, and calculates the error (step S45).

[0170] Subsequently, the third learning unit 79 adjusts the intervention effect prediction model so that the error is reduced (step S46).

[0171] Any known method can be applied to adjust the intervention effect prediction model. For example, backpropagation may be used as a method for adjusting the intervention effect prediction model. The adjusted intervention effect prediction model becomes the new intervention effect prediction model, and in subsequent calculations, the intervention effect prediction unit 78 uses the new intervention effect prediction model. During the adjustment stage of the intervention effect prediction model, the parameters used by the intervention effect prediction unit 78 may be adjusted.

[0172] The parameters include, for example, the parameters used in the input layer 781 and the output layer 782. Specifically, the parameters include the weighting coefficients used in the LSTM layers of the input layer 781 and the output layer 782. The parameters may also include filter coefficients.

[0173] If the error is not within a predetermined range, or if patient images of all patients included in the third training data 84 have not been input (NO in step S47), the third learning unit 79 changes the patient (step S48), returns to step S43, and repeats the learning process. If the error is within a predetermined range, or if patient images of all patients included in the third training data 84 have been input (YES in step S47), the third learning unit 79 terminates the learning process.

[0174] <Third Teacher Data 84> The third training data 84 is data used for machine learning to generate an intervention effect prediction model. The third training data 84 includes state information indicating the state of the target area of ​​each patient, acquired at multiple points in the past, and may also include information relating the state information for each patient to intervention information indicating the intervention applied to each patient, i.e., effect information. The third training data 84 includes third training data for calculating the error between the third learning input data used as input data and the first prediction information output by the intervention effect prediction unit 78.

[0175] The third learning input data may include, for example, (a) information indicating when an intervention was applied to a patient to whom the intervention was applied, (b) pixel data of a patient image showing the target area of ​​a patient, and (c) symptom information regarding the onset or progression of symptoms in the target area at the time the patient image of a patient was taken. The image data used as the third learning input data may be medical image data showing the target area of ​​each of multiple patients. The medical image data may include, for example, at least one of X-ray image data, CT image data, MRI image data, PET image data, and ultrasound image data taken of the target area of ​​each of multiple subjects.

[0176] The third training data may include patient images showing the target area of ​​the patient at a later point in time (e.g., one year later) than when the patient images used in the third training input data were taken, as well as state information indicating the condition of the target area of ​​the patient, and symptom information relating to the target area. The third training data may also be time-series data obtained at multiple point in time from each of multiple patients to whom the intervention was previously applied, where patient images are associated with information regarding joint symptoms at the time the patient images were taken.

[0177] Specifically, the input data used during the training of the intervention effect prediction unit 78 is (a) a subject image showing the subject's target area at a certain point in time A, which is included in the third training input data. Based on the above input data, the intervention effect prediction unit 78 outputs first prediction information regarding the target area at point B, which is a predetermined period (e.g., 3 years) after point A, as output data. Specifically, the intervention effect prediction unit 78 outputs information such as the angle around the target area of ​​the subject at point B, the degree of enlargement and reduction of the target area, the degree of wrinkles and blemishes on the target area, the timing and degree of pain on the target area, and information indicating when invasive treatment will be necessary for the target area. The output data shown here is an example and is not limited to these.

[0178] Specifically, the input data used during training of the intervention effect prediction unit 78 is information included in the third training data 84 indicating the onset or progression of a disease in the target area of ​​the patient at a certain time point B, and information included in the intervention information indicating the method of intervention. Based on the above input data, the intervention effect prediction unit 78 outputs information as output data indicating the method of intervention for the subject and the effect of the intervention. Based on the above input data, the intervention effect prediction unit 78 outputs information as output data indicating, for example, the degree to which the symptoms of the disease in the target area of ​​the patient are improved or the degree to which the progression of the symptoms is suppressed if the intervention is applied at time point B. More specifically, the intervention effect prediction unit 78 may output the above-mentioned second prediction information as output data.

[0179] (Processing performed by prediction systems 100 and 100a) The following describes the processing flow performed by the prediction systems 100 and 100a, using Figure 15. Figure 15 is a flowchart showing an example of the processing flow performed by the prediction systems 100 and 100a according to this embodiment.

[0180] First, the predictive information acquisition unit 71 acquires the subject's image. Meanwhile, the basic information acquisition unit 76 acquires basic information (step S51: acquisition step).

[0181] Next, the prediction information generation unit 75B generates first prediction information in response to the input of subject images and basic information, and outputs the first prediction information to the prediction information acquisition unit 71 and the intervention effect prediction unit 78 (Step S52: First information prediction step).

[0182] Next, the intervention effect prediction unit 78 refers to the intervention information 85 and selects at least one of the intervention methods included in the intervention information 85 (step S53: intervention method selection step).

[0183] Furthermore, the intervention effect prediction unit 78 generates second prediction information about the selected intervention method in response to the input of first prediction information, and outputs the second prediction information to the prediction image generation unit 72C and the output control unit 73C (step S54: intervention effect prediction step).

[0184] Next, the predictive image generation unit 72C generates a predictive image in response to the input of the subject image, first predictive information, and second predictive information, and outputs the predictive image to the terminal device 2 (step S55: predictive image generation step).

[0185] The prediction systems 100 and 100a can output visually easy-to-understand prediction images showing how the state of the target area differs depending on whether or not the intervention is effective at a second time point. Furthermore, since the prediction images are generated from images of the subject themselves, they are realistic and persuasive to the subject. Therefore, for example, if presented to the subject by a doctor in charge of the subject, it can effectively help the subject understand the necessity of the intervention and increase the subject's motivation to participate in the intervention.

[0186] [Examples of implementation using software] The control blocks of the prediction devices 1, 1A, 1B, and 1C (especially the control units 7, 7A, 7B, and 7C) may be implemented by logic circuits (hardware) formed on an integrated circuit (IC chip) or by software.

[0187] In the latter case, the prediction devices 1, 1A, 1B, and 1C are equipped with a computer that executes instructions for a program, which is software that implements each function. This computer is equipped with, for example, one or more processors and a computer-readable recording medium that stores the program. The object of this disclosure is achieved when the processor reads the program from the recording medium and executes it in the computer. For example, a CPU (Central Processing Unit) can be used as the processor. As the recording medium, a "tangible medium that is not temporary," such as ROM (Read Only Memory), can be used, as well as tape, disk, card, semiconductor memory, programmable logic circuit, etc. It may also be further equipped with RAM (Random Access Memory) for deploying the program. Furthermore, the program may be supplied to the computer via any transmission medium capable of transmitting the program (such as a communication network or broadcast wave). One aspect of this disclosure can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

[0188] The inventions described in this disclosure have been explained above based on the drawings and embodiments. However, the inventions described in this disclosure are not limited to the embodiments described above. That is, the inventions described in this disclosure can be modified in various ways within the scope shown in this disclosure, and embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the inventions described in this disclosure. In other words, it should be noted that it is easy for those skilled in the art to make various modifications or alterations based on this disclosure. Furthermore, it should be noted that these modifications or alterations are included in the scope of this disclosure.

[0189] [Other aspects] A prediction system according to one aspect of the present disclosure includes: a prediction information acquisition unit that acquires (a) a subject image showing the subject's body part at a first time point in time, and (b) first prediction information regarding the subject's body part at a second time point in time after a predetermined period has elapsed from the first time point; and a prediction image generation unit that generates and outputs a prediction image predicting the state of the subject's body part at the second time point from the first prediction information and the subject image.

[0190] Furthermore, a control method according to one aspect of the present disclosure is a control method for a prediction system, comprising: a prediction information acquisition step of acquiring (a) a subject image showing a subject part of the subject at a first time point, and (b) first prediction information relating to the subject part at a second time point after a predetermined period has elapsed from the first time point; and a prediction image generation step of generating and outputting a prediction image that predicts the state of the subject part at the second time point from the first prediction information and the subject image, wherein the prediction system has a prediction image generation model capable of generating the prediction image using the subject image and the first prediction information. [Explanation of Symbols]

[0191] 1, 1A, 1B, 1C Prediction device 2, 2a, 2b Terminal devices 3, 3a, 3b 1st prediction information management device 4, 4a, 4b Subject image management device 5, 5a, 5b Medical facilities 6. Communication Network 7, 7A, 7B, 7C control section 8, 8B, 8C storage section 9, 9a, 9b Electronic medical record management system 71 Predictive Information Acquisition Unit 72, 72C Predictive Image Generation Unit 73, 73C Output control unit 74. First Learning Department 75, 75B Predictive Information Generation Unit 76 Basic information acquisition department 77 Second Learning Department 78 Intervention Effect Prediction Unit 79 Third Learning Department 81 Control Program 82. First training dataset 83 Second training dataset 84 Third Teacher Data 85 Intervention information 100, 100a prediction system 721, 721C generator 722, 722C discriminator 751, 781 Input Layer 752, 782 Output Layer

Claims

1. A predictive information acquisition unit that acquires (a) an image of the subject at a first time point showing the subject's body part affected by a predetermined disease, and (b) first predictive information regarding the onset or progression of the disease in the subject body at a second time point, after a predetermined period has elapsed from the first time point. A predictive image generation unit having a predictive image generation model that generates a predictive image predicting the state of the target area at a second time point, from the first predictive information and the converted image obtained by converting the file format of the subject image to a predetermined file format, The system comprises a prediction information generation unit that generates the first prediction information, The predictive image generation model is generated by machine learning using first training data, which includes first data of the same type as the subject image and second data of the same type as the first predictive information, and first training data, which includes third data related to the second data. The first prediction information is generated at least based on the subject image. Prediction system.

2. The predicted image is an image that mimics at least a part of the subject image. The prediction system according to claim 1.

3. The aforementioned image of the subject is an external image showing the subject body part. The prediction system according to claim 1 or 2.

4. The aforementioned image of the subject is a medical image showing the aforementioned body part. A prediction system according to any one of claims 1 to 3.

5. The medical image is at least one of the following: an X-ray image, a CT image, an MRI image, a PET image, and an ultrasound image of the subject. The prediction system according to claim 4.

6. The aforementioned predictive image is an image that predicts the effect that a disease occurring in the target area will have on that area. A prediction system according to any one of claims 1 to 5.

7. The aforementioned predictive image generation model is a neural network trained using multiple image data showing the target area as the first training data. The prediction system according to claim 1.

8. The first predictive information is information relating to the shape and appearance of the target area in relation to the disease of the target area. The prediction system according to claim 6.

9. The prediction information generation unit has a prediction information generation model that generates the first prediction information from the subject image, The predictive information generation model is generated by machine learning using second training data including the first data and second training data including fourth data related to the first data. A prediction system according to any one of claims 1 to 8.

10. The fourth data includes information regarding the onset time of the disease in the target area. The prediction system according to claim 9.

11. The system further includes a basic information acquisition unit that acquires basic information including at least one of the following: the gender, age, height, weight of the subject, and information indicating the state of the subject's body part at the first time point in time. The predictive information generation model can estimate the first predictive information from the subject's image and the subject's basic information. The prediction system according to claim 9 or 10.

12. The aforementioned predictive information generation model is a neural network trained using patient information relating to patients with diseases of the target area as the second training data. The aforementioned patient information includes status information indicating the condition of the target area of ​​each patient, acquired at multiple points in the past, and the status information for each patient is associated with information indicating the time when the status information was acquired. The prediction system according to any one of claims 9 to 11.

13. The system further includes an intervention effect prediction unit that takes the first prediction information as input and outputs a second prediction information indicating a method of intervention for the subject and the effect of said intervention. A prediction system according to any one of claims 1 to 12.

14. The intervention effect prediction unit has a neural network trained using effect information as training data as an intervention effect prediction model. The aforementioned effect information includes status information indicating the condition of the target area for each patient, acquired at multiple points in the past, and the aforementioned status information for each patient is associated with intervention information indicating the intervention applied to each patient. The prediction system according to claim 13.

15. The intervention method includes at least one of the following: dietary therapy, exercise therapy, drug therapy, orthotic therapy, rehabilitation, and surgical therapy. The prediction system according to claim 13 or 14.

16. The first predictive information includes a prediction regarding the timing of the occurrence of symptoms that are likely to occur in the target area. A prediction system according to any one of claims 1 to 15.

17. A method for controlling a prediction system, A prediction information acquisition step that acquires (a) a subject image showing the subject body part having a predetermined disease at a first time point, and (b) first prediction information regarding the onset or progression of the disease in the subject body part at a second time point after a predetermined period has elapsed from the first time point. A prediction image generation step involves generating and outputting a prediction image that predicts the state of the target area at a second time point, based on the first prediction information and the converted image obtained by converting the file format of the subject image to a predetermined file format. The step includes a prediction information generation step that generates the first prediction information, The prediction system has a prediction image generation model that can generate the prediction image using the subject image and the first prediction information. A control method wherein the first predictive information is generated at least based on the subject image.

18. A control program for causing a computer to function as a prediction system according to any one of claims 1 to 16, wherein the computer functions as the prediction information acquisition unit, the prediction image generation unit, and the prediction information generation unit.

Images